- gerrit-branch-lock
+.. _lf-global-jjb-jenkins-cfg-merge:
+
Jenkins Configuration Merge
---------------------------
Jenkins job to manage Global Jenkins configuration.
-Global Environment Variables are managed via the
-``jenkins-config/global-vars-SILO.sh`` file in ci-management. Replace SILO with
-the name of the Jenkins silo the variable configuration is for.
-
-The format for this file is ``KEY=value`` for example::
-
- GERRIT_URL=https://git.opendaylight.org/gerrit
- GIT_BASE=git://devvexx.opendaylight.org/mirror/$PROJECT
- GIT_URL=git://devvexx.opendaylight.org/mirror
- JENKINS_HOSTNAME=vex-yul-odl-jenkins-2
- LOGS_SERVER=https://logs.opendaylight.org
- NEXUS_URL=https://nexus.opendaylight.org
- ODLNEXUSPROXY=https://nexus.opendaylight.org
- SILO=sandbox
- SONAR_URL=https://sonar.opendaylight.org
-
.. note::
Requires the jjbini file in Jenkins CFP to contain JJB 2.0 style
.. literalinclude:: ../../.jjb-test/lf-ci-jobs/jenkins-cfg-merge-full.yaml
:language: yaml
+Global Environment Variables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Global Environment Variables are managed via the
+``jenkins-config/global-vars-SILO.sh`` file in ci-management. Replace SILO with
+the name of the Jenkins silo the variable configuration is for.
+
+The format for this file is ``KEY=value`` for example::
+
+ GERRIT_URL=https://git.opendaylight.org/gerrit
+ GIT_BASE=git://devvexx.opendaylight.org/mirror/$PROJECT
+ GIT_URL=git://devvexx.opendaylight.org/mirror
+ JENKINS_HOSTNAME=vex-yul-odl-jenkins-2
+ LOGS_SERVER=https://logs.opendaylight.org
+ NEXUS_URL=https://nexus.opendaylight.org
+ ODLNEXUSPROXY=https://nexus.opendaylight.org
+ SILO=sandbox
+ SONAR_URL=https://sonar.opendaylight.org
+
+Cloud Configuration
+^^^^^^^^^^^^^^^^^^^
+
+This configuration requires the OpenStack Cloud plugin in Jenkins and is
+currently the only cloud plugin supported.
+
+OpenStack Cloud plugin version supported:
+
+* 2.30
+* 2.31
+
+Cloud configuration are managed via a directory structure in ci-management as
+follows:
+
+- jenkins-config/clouds/openstack/
+- jenkins-config/clouds/openstack/cattle/cloud.cfg
+- jenkins-config/clouds/openstack/cattle/centos7-builder-2c-2g.cfg
+- jenkins-config/clouds/openstack/cattle/centos7-builder-4c-4g.cfg
+- jenkins-config/clouds/openstack/cattle/centos7-docker-4c-4g.cfg
+
+The directory name inside of the "openstack" directory is used as the name of
+the cloud configuration. In this case "cattle" is being used as the cloud name.
+
+The ``cloud.cfg`` file is a special file used to configure the main cloud
+configuration in the format ``KEY=value``.
+
+:Cloud Parameters:
+
+ :CLOUD_URL: API endpoint URL for Keystone.
+ (default: "")
+ :CLOUD_IGNORE_SSL: Ignore unverified SSL certificates. (default: false)
+ :CLOUD_ZONE: OpenStack region to use. (default: "")
+ :CLOUD_CREDENTIAL_ID: Credential to use for authentication to OpenStack.
+ (default: "os-cloud")
+ :INSTANCE_CAP: Total number of instances the cloud will allow spin up.
+ (default: null)
+ :SANDBOX_CAP: Total number of instances the clodu will allow to
+ spin up. This applies to "sandbox" systems and overrides the
+ INSTANCE_CAP setting. (default: null)
+
+:Template Parameters:
+
+ .. note::
+
+ In the case of template definitions of a parameter below is not passed
+ the one defined in default clouds will be inherited.
+
+ :IMAGE_NAME: The image name to use for this template.
+ (default: "")
+ :LABELS: Labels to assign to the vm. (default: FILE_NAME)
+ :HARDWARE_ID: OpenStack flavor to use. (default: "")
+ :NETWORK_ID: OpenStack network to use. (default: "")
+ :USER_DATA_ID: User Data to pass into the instance.
+ (default: jenkins-init-script)
+ :INSTANCE_CAP: Total number of instances of this type that can be launched
+ at one time. When defined in clouds.cfg it defines the total for the
+ entire cloud. (default: null)
+ :SANDBOX_CAP: Total number of instances of this type that can be launched
+ at one time. When defined in clouds.cfg it defines the total for the
+ entire cloud. This applies to "sandbox" systems and overrides the
+ INSTANCE_CAP setting. (default: null)
+ :FLOATING_IP_POOL: Floating ip pool to use. (default: "")
+ :SECURITY_GROUPS: Security group to use. (default: "default")
+ :AVAILABILITY_ZONE: OpenStack availability zone to use. (default: "")
+ :START_TIMEOUT: Number of milliseconds to wait for the agent to be
+ provisioned and connected. (default: 600000)
+ :KEY_PAIR_NAME: SSH Public Key Pair to use for authentication.
+ (default: jenkins)
+ :NUM_EXECUTORS: Number of executors to enable for the instance.
+ (default: 1)
+ :JVM_OPTIONS: JVM Options to pass to Java. (default: "")
+ :FS_ROOT: File system root for the workspace. (default: "/w")
+ :RETENTION_TIME: Number of minutes to wait for an idle slave to be used
+ again before it's removed. If set to -1, the slave will be kept
+ forever. (default: 0)
+
+For a live example see the OpenDaylight project jenkins-config directory.
+https://github.com/opendaylight/releng-builder/tree/master/jenkins-config
+
+Troubleshooting
+^^^^^^^^^^^^^^^
+
+:Cloud Configuration:
+
+ The directory ``groovy-inserts`` contains the groovy script output that is
+ used to push to Jenkins. In the event of a job failure this file can be
+ inspected.
+
JJB Deploy Job
--------------
--- /dev/null
+#!/bin/bash
+# SPDX-License-Identifier: EPL-1.0
+##############################################################################
+# Copyright (c) 2018 The Linux Foundation and others.
+#
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Eclipse Public License v1.0
+# which accompanies this distribution, and is available at
+# http://www.eclipse.org/legal/epl-v10.html
+##############################################################################
+# Pulls global variable definitions out of a file.
+#
+# Configuration is read from $WORKSPACE/jenkins-config/clouds/openstack/$cloud/cloud.cfg
+#
+# Requirements: lftools must be installed to /tmp/v/lftools
+# Parameters:
+# jenkins_silos: Space separated list of Jenkins silos to push
+# configuration to. (default: jenkins)
+echo "---> jenkins-configure-clouds.sh"
+
+if [ ! -d "$WORKSPACE/jenkins-config/clouds" ]; then
+ echo "WARN: jenkins-config/clouds does not exist. Skipping cloud management..."
+ exit 0
+fi
+
+#shellcheck source=/tmp/v/lftools/bin/activate disable=SC1091
+source "/tmp/v/lftools/bin/activate"
+
+GROOVY_SCRIPT_FILE="jjb/global-jjb/jenkins-admin/manage_clouds.groovy"
+OS_CLOUD_DIR="$WORKSPACE/jenkins-config/clouds/openstack"
+SCRIPT_DIR="$WORKSPACE/archives/groovy-inserts"
+mkdir -p "$SCRIPT_DIR"
+
+silos="${jenkins_silos:-jenkins}"
+
+set -eu -o pipefail
+
+get_cfg() {
+ if [ -z ${3+x} ]; then
+ >&2 echo "Usage: get_cfg CFG_FILE SETTING DEFAULT"
+ exit 1
+ fi
+
+ local cfg_file="$1"
+ local setting="$2"
+ local default="$3"
+
+ if [ ! -f "$cfg_file" ]; then
+ >&2 echo "ERROR: Configuration file $cfg_file not found."
+ exit 1
+ fi
+
+ cfg=$(grep "${setting^^}" "$cfg_file" | tail -1 | awk -F'=' '{print $2}')
+ cfg=${cfg:-"$default"}
+ echo "$cfg"
+}
+export get_cfg
+
+get_cloud_cfg() {
+ if [ -z $1 ]; then
+ >&2 echo "Usage: get_cloud_cfg CFG_DIR"
+ exit 1
+ fi
+
+ local cfg_dir="$1"
+ local silo="$2"
+ local cfg_file="$cfg_dir/cloud.cfg"
+
+ cloud_name=$(basename "$cfg_dir")
+ cloud_url=$(get_cfg "$cfg_file" CLOUD_URL "")
+ cloud_ignore_ssl=$(get_cfg "$cfg_file" CLOUD_IGNORE_SSL "false")
+ cloud_zone=$(get_cfg "$cfg_file" CLOUD_ZONE "")
+ cloud_credential_id=$(get_cfg "$cfg_file" CLOUD_CREDENTIAL_ID "os-cloud")
+
+ echo "default_options = new SlaveOptions("
+ get_minion_options "$cfg_file" "$silo"
+ echo ")"
+
+ echo "cloud = new JCloudsCloud("
+ echo " \"$cloud_name\","
+ echo " \"$cloud_url\","
+ echo " $cloud_ignore_ssl,"
+ echo " \"$cloud_zone\","
+ echo " default_options,"
+ echo " templates,"
+ echo " \"$cloud_credential_id\""
+ echo ")"
+}
+
+get_minion_options() {
+ if [ -z $1 ]; then
+ >&2 echo "Usage: get_minion_options CFG_FILE"
+ exit 1
+ fi
+
+ local cfg_file="$1"
+ local silo="${2:-}"
+
+ # Create a flavor mapping to manage hardware_id until OpenStack Cloud
+ # plugin supports using names
+ declare -A flavors
+ flavors["v1-standard-1"]="bbcb7eb5-5c8d-498f-9d7e-307c575d3566"
+ flavors["v1-standard-2"]="ca2a6e9c-2236-4107-8905-7ae9427132ff"
+ flavors["v1-standard-4"]="5cf64088-893b-46b5-9bb1-ee020277635d"
+ flavors["v1-standard-8"]="6eec77b4-2286-4e3b-b3f0-cac67aa2c727"
+ flavors["v1-standard-16"]="2f8730dd-7688-4b72-a512-99fb9a482414"
+ flavors["v1-standard-32"]="0da688af-bb0c-4116-a158-cbf37240a8b1"
+ flavors["v1-standard-48"]="69471d69-61fb-40dd-bdf3-e6b7f4e6daa3"
+ flavors["v1-standard-64"]="0c1d9008-f546-4608-9e8f-f8bdaec8dddd"
+ flavors["v1-standard-96"]="5741c775-92a4-4488-bd77-dd7b08e2be81"
+ flavors["v1-standard-128"]="e82d0a5b-8031-4526-9a5d-a15f7b4d48ff"
+ flavors["v2-standard-1"]="52a01f6b-e660-48b5-8c06-5fb2a0fab0ec"
+ flavors["v2-standard-2"]="ac2c4d17-8d6f-4e3c-a9eb-57c155f0a949"
+ flavors["v2-standard-4"]="d9115351-defe-4fac-986b-1a1187e2c31c"
+ flavors["v2-standard-8"]="e6fe2e37-0e38-438c-8fa5-fc2d79d0a7bb"
+ flavors["v2-standard-16"]="9e4b01cd-6744-4120-aafe-1b5e17584919"
+ flavors["v2-standard-360"]="f0d27f44-a410-4f0f-9781-d722f5b5489e"
+ flavors["v2-highcpu-1"]="c04abb7a-2b61-4ed3-8ce8-6c40ad9df750"
+ flavors["v2-highcpu-2"]="03bdf34e-8905-46bc-a4b9-8dbf94b6e06d"
+ flavors["v2-highcpu-4"]="3b72e578-7875-4e0e-91b7-71ed292f3ca2"
+ flavors["v2-highcpu-8"]="221de281-95ec-414f-8e42-c86c9e0b318d"
+ flavors["v2-highcpu-16"]="ddd6863a-ef4f-475c-9aee-61d46898651d"
+ flavors["v2-highcpu-32"]="21dfb8a3-c472-4a2c-a8e1-4da8de415ff8"
+
+ image_name=$(get_cfg "$cfg_file" IMAGE_NAME "")
+ hardware_id=$(get_cfg "$cfg_file" HARDWARE_ID "")
+ network_id=$(get_cfg "$cfg_file" NETWORK_ID "")
+ user_data_id=$(get_cfg "$cfg_file" USER_DATA_ID "jenkins-init-script")
+
+ # Handle Sandbox systems that might have a different cap.
+ instance_cap=$(get_cfg "$cfg_file" INSTANCE_CAP "null")
+ if [ "$silo" == "sandbox" ]; then
+ instance_cap=$(get_cfg "$cfg_file" SANDBOX_CAP "null")
+ fi
+
+ floating_ip_pool=$(get_cfg "$cfg_file" FLOATING_IP_POOL "")
+ security_groups=$(get_cfg "$cfg_file" SECURITY_GROUPS "default")
+ availability_zone=$(get_cfg "$cfg_file" AVAILABILITY_ZONE "")
+ start_timeout=$(get_cfg "$cfg_file" START_TIMEOUT "600000")
+ key_pair_name=$(get_cfg "$cfg_file" KEY_PAIR_NAME "jenkins")
+ num_executors=$(get_cfg "$cfg_file" NUM_EXECUTORS "1")
+ jvm_options=$(get_cfg "$cfg_file" JVM_OPTIONS "")
+ fs_root=$(get_cfg "$cfg_file" FS_ROOT "/w")
+ retention_time=$(get_cfg "$cfg_file" RETENTION_TIME "0")
+
+ echo " new BootSource.Image(\"$image_name\"),"
+ echo " \"${flavors[${hardware_id}]}\","
+ echo " \"$network_id\","
+ echo " \"$user_data_id\","
+ echo " $instance_cap,"
+ echo " \"$floating_ip_pool\","
+ echo " \"$security_groups\","
+ echo " \"$availability_zone\","
+ echo " $start_timeout,"
+ echo " \"$key_pair_name\","
+ echo " $num_executors,"
+ echo " \"$jvm_options\","
+ echo " \"$fs_root\","
+ echo " new LauncherFactory.SSH(\"jenkins\", \"\"),"
+ echo " $retention_time"
+}
+
+get_template_cfg() {
+ if [ -z $1 ]; then
+ >&2 echo "Usage: get_template_cfg CFG_FILE"
+ exit 1
+ fi
+
+ local cfg_file="$1"
+ local minion_prefix="${2:-}"
+
+ template_name=$(basename $cfg_file .cfg)
+ labels=$(get_cfg "$cfg_file" LABELS "")
+
+ echo "minion_options = new SlaveOptions("
+ get_minion_options "$cfg_file"
+ echo ")"
+
+ echo "template = new JCloudsSlaveTemplate("
+ # TODO: Figure out how to insert the "prd / snd" prefix into template name.
+ echo " \"${minion_prefix}${template_name}\","
+ echo " \"$template_name $labels\","
+ echo " minion_options,"
+ echo ")"
+}
+
+mapfile -t clouds < <(ls -d1 $OS_CLOUD_DIR/*/)
+
+for silo in $silos; do
+
+ script_file="$SCRIPT_DIR/${silo}-cloud-cfg.groovy"
+ cp "$GROOVY_SCRIPT_FILE" "$script_file"
+
+ # Linux Foundation Jenkins systems use "prd-" and "snd-" to mark
+ # production and sandbox servers.
+ if [ "$silo" == "releng" ] || [ "$silo" == "production" ]; then
+ node_prefix="prd-"
+ elif [ "$silo" == "sandbox" ]; then
+ node_prefix="snd-"
+ else
+ node_prefix="${silo}-"
+ fi
+
+ echo "-----> Groovy script $script_file"
+ for cloud in "${clouds[@]}"; do
+ cfg_dir="${cloud}"
+ echo "Processing $cfg_dir"
+ insert_file="$SCRIPT_DIR/$silo/$(basename $cloud)/cloud-cfg.txt"
+ mkdir -p "$(dirname $insert_file)"
+ rm -f "$insert_file"
+
+ echo "" >> "$insert_file"
+ echo "//////////////////////////////////////////////////" >> "$insert_file"
+ echo "// Cloud config for $(basename $cloud)" >> "$insert_file"
+ echo "//////////////////////////////////////////////////" >> "$insert_file"
+ echo "" >> "$insert_file"
+
+ echo "templates = []" >> $insert_file
+ mapfile -t templates < <(find $cfg_dir -maxdepth 1 -not -type d -not -name "cloud.cfg")
+ for template in "${templates[@]}"; do
+ get_template_cfg "$template" "$node_prefix" >> "$insert_file"
+ echo "templates.add(template)" >> "$insert_file"
+ done
+
+ get_cloud_cfg "$cfg_dir" "$silo" >> "$insert_file"
+ echo "clouds.add(cloud)" >> "$insert_file"
+
+ cat "$insert_file" >> "$script_file"
+ done
+
+ set +x # Disable `set -x` to prevent printing passwords
+ echo "Configuring $silo"
+ JENKINS_URL=$(crudini --get "$HOME"/.config/jenkins_jobs/jenkins_jobs.ini "$silo" url)
+ JENKINS_USER=$(crudini --get "$HOME"/.config/jenkins_jobs/jenkins_jobs.ini "$silo" user)
+ JENKINS_PASSWORD=$(crudini --get "$HOME"/.config/jenkins_jobs/jenkins_jobs.ini "$silo" password)
+ export JENKINS_URL
+ export JENKINS_USER
+ export JENKINS_PASSWORD
+ lftools jenkins groovy "$script_file"
+done
Code Review / releng / global-jjb.git / commitdiff