From: Thanh Ha Date: Thu, 18 Jan 2018 16:09:36 +0000 (-0500) Subject: Add script to manage Jenkins cloud cfg X-Git-Tag: v0.17.0~5 X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=commitdiff_plain;h=c26afec952891def4ae050c4a9d9d3a13dedd149;p=releng%2Fglobal-jjb.git Add script to manage Jenkins cloud cfg A groovy script to manage the Jenkins global cloud cfg programmatically. This script reads configuration from the directory jenkins-config/clouds to configure cloud configuration in Jenkins. Refer to the documentation for details. No upgrade consideration is necessary as this script automatically detects the absence of the jenkins-config/clouds directory and disables the Cloud configuration. Issue: RELENG-804 Change-Id: Idad3078942618b7d8fd08a40bf9e4e5ed5a0dc48 Signed-off-by: Thanh Ha --- diff --git a/docs/configuration.rst b/docs/configuration.rst index af98c6e2..27918df9 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -46,3 +46,14 @@ Create a "Custom file" with contents: [global] timeout = 60 index-url = https://nexus3.opendaylight.org/repository/PyPi/simple + + +Jenkins CI Jobs +=============== + +jenkins-cfg-merge +----------------- + +This job manages Jenkins Global configuration. Refer to +the :ref:`CI Documentation ` for job +configuration details. diff --git a/docs/jjb/lf-ci-jobs.rst b/docs/jjb/lf-ci-jobs.rst index 19de22d1..2cb767a0 100644 --- a/docs/jjb/lf-ci-jobs.rst +++ b/docs/jjb/lf-ci-jobs.rst @@ -109,27 +109,13 @@ Job submits a patch to lock or unlock a project's branch. - gerrit-branch-lock +.. _lf-global-jjb-jenkins-cfg-merge: + Jenkins Configuration Merge --------------------------- Jenkins job to manage Global Jenkins configuration. -Global Environment Variables are managed via the -``jenkins-config/global-vars-SILO.sh`` file in ci-management. Replace SILO with -the name of the Jenkins silo the variable configuration is for. - -The format for this file is ``KEY=value`` for example:: - - GERRIT_URL=https://git.opendaylight.org/gerrit - GIT_BASE=git://devvexx.opendaylight.org/mirror/$PROJECT - GIT_URL=git://devvexx.opendaylight.org/mirror - JENKINS_HOSTNAME=vex-yul-odl-jenkins-2 - LOGS_SERVER=https://logs.opendaylight.org - NEXUS_URL=https://nexus.opendaylight.org - ODLNEXUSPROXY=https://nexus.opendaylight.org - SILO=sandbox - SONAR_URL=https://sonar.opendaylight.org - .. note:: Requires the jjbini file in Jenkins CFP to contain JJB 2.0 style @@ -164,6 +150,113 @@ Full Example: .. literalinclude:: ../../.jjb-test/lf-ci-jobs/jenkins-cfg-merge-full.yaml :language: yaml +Global Environment Variables +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Global Environment Variables are managed via the +``jenkins-config/global-vars-SILO.sh`` file in ci-management. Replace SILO with +the name of the Jenkins silo the variable configuration is for. + +The format for this file is ``KEY=value`` for example:: + + GERRIT_URL=https://git.opendaylight.org/gerrit + GIT_BASE=git://devvexx.opendaylight.org/mirror/$PROJECT + GIT_URL=git://devvexx.opendaylight.org/mirror + JENKINS_HOSTNAME=vex-yul-odl-jenkins-2 + LOGS_SERVER=https://logs.opendaylight.org + NEXUS_URL=https://nexus.opendaylight.org + ODLNEXUSPROXY=https://nexus.opendaylight.org + SILO=sandbox + SONAR_URL=https://sonar.opendaylight.org + +Cloud Configuration +^^^^^^^^^^^^^^^^^^^ + +This configuration requires the OpenStack Cloud plugin in Jenkins and is +currently the only cloud plugin supported. + +OpenStack Cloud plugin version supported: + +* 2.30 +* 2.31 + +Cloud configuration are managed via a directory structure in ci-management as +follows: + +- jenkins-config/clouds/openstack/ +- jenkins-config/clouds/openstack/cattle/cloud.cfg +- jenkins-config/clouds/openstack/cattle/centos7-builder-2c-2g.cfg +- jenkins-config/clouds/openstack/cattle/centos7-builder-4c-4g.cfg +- jenkins-config/clouds/openstack/cattle/centos7-docker-4c-4g.cfg + +The directory name inside of the "openstack" directory is used as the name of +the cloud configuration. In this case "cattle" is being used as the cloud name. + +The ``cloud.cfg`` file is a special file used to configure the main cloud +configuration in the format ``KEY=value``. + +:Cloud Parameters: + + :CLOUD_URL: API endpoint URL for Keystone. + (default: "") + :CLOUD_IGNORE_SSL: Ignore unverified SSL certificates. (default: false) + :CLOUD_ZONE: OpenStack region to use. (default: "") + :CLOUD_CREDENTIAL_ID: Credential to use for authentication to OpenStack. + (default: "os-cloud") + :INSTANCE_CAP: Total number of instances the cloud will allow spin up. + (default: null) + :SANDBOX_CAP: Total number of instances the clodu will allow to + spin up. This applies to "sandbox" systems and overrides the + INSTANCE_CAP setting. (default: null) + +:Template Parameters: + + .. note:: + + In the case of template definitions of a parameter below is not passed + the one defined in default clouds will be inherited. + + :IMAGE_NAME: The image name to use for this template. + (default: "") + :LABELS: Labels to assign to the vm. (default: FILE_NAME) + :HARDWARE_ID: OpenStack flavor to use. (default: "") + :NETWORK_ID: OpenStack network to use. (default: "") + :USER_DATA_ID: User Data to pass into the instance. + (default: jenkins-init-script) + :INSTANCE_CAP: Total number of instances of this type that can be launched + at one time. When defined in clouds.cfg it defines the total for the + entire cloud. (default: null) + :SANDBOX_CAP: Total number of instances of this type that can be launched + at one time. When defined in clouds.cfg it defines the total for the + entire cloud. This applies to "sandbox" systems and overrides the + INSTANCE_CAP setting. (default: null) + :FLOATING_IP_POOL: Floating ip pool to use. (default: "") + :SECURITY_GROUPS: Security group to use. (default: "default") + :AVAILABILITY_ZONE: OpenStack availability zone to use. (default: "") + :START_TIMEOUT: Number of milliseconds to wait for the agent to be + provisioned and connected. (default: 600000) + :KEY_PAIR_NAME: SSH Public Key Pair to use for authentication. + (default: jenkins) + :NUM_EXECUTORS: Number of executors to enable for the instance. + (default: 1) + :JVM_OPTIONS: JVM Options to pass to Java. (default: "") + :FS_ROOT: File system root for the workspace. (default: "/w") + :RETENTION_TIME: Number of minutes to wait for an idle slave to be used + again before it's removed. If set to -1, the slave will be kept + forever. (default: 0) + +For a live example see the OpenDaylight project jenkins-config directory. +https://github.com/opendaylight/releng-builder/tree/master/jenkins-config + +Troubleshooting +^^^^^^^^^^^^^^^ + +:Cloud Configuration: + + The directory ``groovy-inserts`` contains the groovy script output that is + used to push to Jenkins. In the event of a job failure this file can be + inspected. + JJB Deploy Job -------------- diff --git a/jenkins-admin/manage_clouds.groovy b/jenkins-admin/manage_clouds.groovy new file mode 100644 index 00000000..472109bc --- /dev/null +++ b/jenkins-admin/manage_clouds.groovy @@ -0,0 +1,27 @@ +/* + * SPDX-License-Identifier: EPL-1.0 + * Copyright (c) 2018 The Linux Foundation and others. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + */ + + /** + * Manage Jenkins OpenStack cloud configuration + * + * This file is used for auto-generation so is quite minimal. The generator + * code is produced by shell/jenkins-configure-clouds.sh. + */ + +import jenkins.plugins.openstack.compute.JCloudsCloud +import jenkins.plugins.openstack.compute.JCloudsSlaveTemplate +import jenkins.plugins.openstack.compute.SlaveOptions +import jenkins.plugins.openstack.compute.slaveopts.BootSource +import jenkins.plugins.openstack.compute.slaveopts.LauncherFactory + +def clouds = Jenkins.instance.clouds +clouds.removeAll { it instanceof JCloudsCloud } + +// Code below is auto-generated by jenkins-configure-clouds.sh diff --git a/jjb/lf-ci-jobs.yaml b/jjb/lf-ci-jobs.yaml index 558d3858..fc49f524 100644 --- a/jjb/lf-ci-jobs.yaml +++ b/jjb/lf-ci-jobs.yaml @@ -256,6 +256,15 @@ # Jenkins Configuration Management Merge # ########################################## +- builder: + name: lf-jenkins-cfg-clouds + builders: + - lf-infra-jjbini + - inject: + properties-content: 'jenkins_silos={jenkins-silos}' + - shell: !include-raw-escape: ../shell/jenkins-configure-clouds.sh + - shell: rm "$HOME/.config/jenkins_jobs/jenkins_jobs.ini" + - builder: name: lf-jenkins-cfg-global-vars builders: @@ -316,6 +325,8 @@ - ../shell/lftools-install.sh - lf-jenkins-cfg-global-vars: jenkins-silos: '{jenkins-silos}' + - lf-jenkins-cfg-clouds: + jenkins-silos: '{jenkins-silos}' publishers: - lf-infra-publish diff --git a/shell/jenkins-configure-clouds.sh b/shell/jenkins-configure-clouds.sh new file mode 100644 index 00000000..ab7f9098 --- /dev/null +++ b/shell/jenkins-configure-clouds.sh @@ -0,0 +1,240 @@ +#!/bin/bash +# SPDX-License-Identifier: EPL-1.0 +############################################################################## +# Copyright (c) 2018 The Linux Foundation and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Eclipse Public License v1.0 +# which accompanies this distribution, and is available at +# http://www.eclipse.org/legal/epl-v10.html +############################################################################## +# Pulls global variable definitions out of a file. +# +# Configuration is read from $WORKSPACE/jenkins-config/clouds/openstack/$cloud/cloud.cfg +# +# Requirements: lftools must be installed to /tmp/v/lftools +# Parameters: +# jenkins_silos: Space separated list of Jenkins silos to push +# configuration to. (default: jenkins) +echo "---> jenkins-configure-clouds.sh" + +if [ ! -d "$WORKSPACE/jenkins-config/clouds" ]; then + echo "WARN: jenkins-config/clouds does not exist. Skipping cloud management..." + exit 0 +fi + +#shellcheck source=/tmp/v/lftools/bin/activate disable=SC1091 +source "/tmp/v/lftools/bin/activate" + +GROOVY_SCRIPT_FILE="jjb/global-jjb/jenkins-admin/manage_clouds.groovy" +OS_CLOUD_DIR="$WORKSPACE/jenkins-config/clouds/openstack" +SCRIPT_DIR="$WORKSPACE/archives/groovy-inserts" +mkdir -p "$SCRIPT_DIR" + +silos="${jenkins_silos:-jenkins}" + +set -eu -o pipefail + +get_cfg() { + if [ -z ${3+x} ]; then + >&2 echo "Usage: get_cfg CFG_FILE SETTING DEFAULT" + exit 1 + fi + + local cfg_file="$1" + local setting="$2" + local default="$3" + + if [ ! -f "$cfg_file" ]; then + >&2 echo "ERROR: Configuration file $cfg_file not found." + exit 1 + fi + + cfg=$(grep "${setting^^}" "$cfg_file" | tail -1 | awk -F'=' '{print $2}') + cfg=${cfg:-"$default"} + echo "$cfg" +} +export get_cfg + +get_cloud_cfg() { + if [ -z $1 ]; then + >&2 echo "Usage: get_cloud_cfg CFG_DIR" + exit 1 + fi + + local cfg_dir="$1" + local silo="$2" + local cfg_file="$cfg_dir/cloud.cfg" + + cloud_name=$(basename "$cfg_dir") + cloud_url=$(get_cfg "$cfg_file" CLOUD_URL "") + cloud_ignore_ssl=$(get_cfg "$cfg_file" CLOUD_IGNORE_SSL "false") + cloud_zone=$(get_cfg "$cfg_file" CLOUD_ZONE "") + cloud_credential_id=$(get_cfg "$cfg_file" CLOUD_CREDENTIAL_ID "os-cloud") + + echo "default_options = new SlaveOptions(" + get_minion_options "$cfg_file" "$silo" + echo ")" + + echo "cloud = new JCloudsCloud(" + echo " \"$cloud_name\"," + echo " \"$cloud_url\"," + echo " $cloud_ignore_ssl," + echo " \"$cloud_zone\"," + echo " default_options," + echo " templates," + echo " \"$cloud_credential_id\"" + echo ")" +} + +get_minion_options() { + if [ -z $1 ]; then + >&2 echo "Usage: get_minion_options CFG_FILE" + exit 1 + fi + + local cfg_file="$1" + local silo="${2:-}" + + # Create a flavor mapping to manage hardware_id until OpenStack Cloud + # plugin supports using names + declare -A flavors + flavors["v1-standard-1"]="bbcb7eb5-5c8d-498f-9d7e-307c575d3566" + flavors["v1-standard-2"]="ca2a6e9c-2236-4107-8905-7ae9427132ff" + flavors["v1-standard-4"]="5cf64088-893b-46b5-9bb1-ee020277635d" + flavors["v1-standard-8"]="6eec77b4-2286-4e3b-b3f0-cac67aa2c727" + flavors["v1-standard-16"]="2f8730dd-7688-4b72-a512-99fb9a482414" + flavors["v1-standard-32"]="0da688af-bb0c-4116-a158-cbf37240a8b1" + flavors["v1-standard-48"]="69471d69-61fb-40dd-bdf3-e6b7f4e6daa3" + flavors["v1-standard-64"]="0c1d9008-f546-4608-9e8f-f8bdaec8dddd" + flavors["v1-standard-96"]="5741c775-92a4-4488-bd77-dd7b08e2be81" + flavors["v1-standard-128"]="e82d0a5b-8031-4526-9a5d-a15f7b4d48ff" + flavors["v2-standard-1"]="52a01f6b-e660-48b5-8c06-5fb2a0fab0ec" + flavors["v2-standard-2"]="ac2c4d17-8d6f-4e3c-a9eb-57c155f0a949" + flavors["v2-standard-4"]="d9115351-defe-4fac-986b-1a1187e2c31c" + flavors["v2-standard-8"]="e6fe2e37-0e38-438c-8fa5-fc2d79d0a7bb" + flavors["v2-standard-16"]="9e4b01cd-6744-4120-aafe-1b5e17584919" + flavors["v2-standard-360"]="f0d27f44-a410-4f0f-9781-d722f5b5489e" + flavors["v2-highcpu-1"]="c04abb7a-2b61-4ed3-8ce8-6c40ad9df750" + flavors["v2-highcpu-2"]="03bdf34e-8905-46bc-a4b9-8dbf94b6e06d" + flavors["v2-highcpu-4"]="3b72e578-7875-4e0e-91b7-71ed292f3ca2" + flavors["v2-highcpu-8"]="221de281-95ec-414f-8e42-c86c9e0b318d" + flavors["v2-highcpu-16"]="ddd6863a-ef4f-475c-9aee-61d46898651d" + flavors["v2-highcpu-32"]="21dfb8a3-c472-4a2c-a8e1-4da8de415ff8" + + image_name=$(get_cfg "$cfg_file" IMAGE_NAME "") + hardware_id=$(get_cfg "$cfg_file" HARDWARE_ID "") + network_id=$(get_cfg "$cfg_file" NETWORK_ID "") + user_data_id=$(get_cfg "$cfg_file" USER_DATA_ID "jenkins-init-script") + + # Handle Sandbox systems that might have a different cap. + instance_cap=$(get_cfg "$cfg_file" INSTANCE_CAP "null") + if [ "$silo" == "sandbox" ]; then + instance_cap=$(get_cfg "$cfg_file" SANDBOX_CAP "null") + fi + + floating_ip_pool=$(get_cfg "$cfg_file" FLOATING_IP_POOL "") + security_groups=$(get_cfg "$cfg_file" SECURITY_GROUPS "default") + availability_zone=$(get_cfg "$cfg_file" AVAILABILITY_ZONE "") + start_timeout=$(get_cfg "$cfg_file" START_TIMEOUT "600000") + key_pair_name=$(get_cfg "$cfg_file" KEY_PAIR_NAME "jenkins") + num_executors=$(get_cfg "$cfg_file" NUM_EXECUTORS "1") + jvm_options=$(get_cfg "$cfg_file" JVM_OPTIONS "") + fs_root=$(get_cfg "$cfg_file" FS_ROOT "/w") + retention_time=$(get_cfg "$cfg_file" RETENTION_TIME "0") + + echo " new BootSource.Image(\"$image_name\")," + echo " \"${flavors[${hardware_id}]}\"," + echo " \"$network_id\"," + echo " \"$user_data_id\"," + echo " $instance_cap," + echo " \"$floating_ip_pool\"," + echo " \"$security_groups\"," + echo " \"$availability_zone\"," + echo " $start_timeout," + echo " \"$key_pair_name\"," + echo " $num_executors," + echo " \"$jvm_options\"," + echo " \"$fs_root\"," + echo " new LauncherFactory.SSH(\"jenkins\", \"\")," + echo " $retention_time" +} + +get_template_cfg() { + if [ -z $1 ]; then + >&2 echo "Usage: get_template_cfg CFG_FILE" + exit 1 + fi + + local cfg_file="$1" + local minion_prefix="${2:-}" + + template_name=$(basename $cfg_file .cfg) + labels=$(get_cfg "$cfg_file" LABELS "") + + echo "minion_options = new SlaveOptions(" + get_minion_options "$cfg_file" + echo ")" + + echo "template = new JCloudsSlaveTemplate(" + # TODO: Figure out how to insert the "prd / snd" prefix into template name. + echo " \"${minion_prefix}${template_name}\"," + echo " \"$template_name $labels\"," + echo " minion_options," + echo ")" +} + +mapfile -t clouds < <(ls -d1 $OS_CLOUD_DIR/*/) + +for silo in $silos; do + + script_file="$SCRIPT_DIR/${silo}-cloud-cfg.groovy" + cp "$GROOVY_SCRIPT_FILE" "$script_file" + + # Linux Foundation Jenkins systems use "prd-" and "snd-" to mark + # production and sandbox servers. + if [ "$silo" == "releng" ] || [ "$silo" == "production" ]; then + node_prefix="prd-" + elif [ "$silo" == "sandbox" ]; then + node_prefix="snd-" + else + node_prefix="${silo}-" + fi + + echo "-----> Groovy script $script_file" + for cloud in "${clouds[@]}"; do + cfg_dir="${cloud}" + echo "Processing $cfg_dir" + insert_file="$SCRIPT_DIR/$silo/$(basename $cloud)/cloud-cfg.txt" + mkdir -p "$(dirname $insert_file)" + rm -f "$insert_file" + + echo "" >> "$insert_file" + echo "//////////////////////////////////////////////////" >> "$insert_file" + echo "// Cloud config for $(basename $cloud)" >> "$insert_file" + echo "//////////////////////////////////////////////////" >> "$insert_file" + echo "" >> "$insert_file" + + echo "templates = []" >> $insert_file + mapfile -t templates < <(find $cfg_dir -maxdepth 1 -not -type d -not -name "cloud.cfg") + for template in "${templates[@]}"; do + get_template_cfg "$template" "$node_prefix" >> "$insert_file" + echo "templates.add(template)" >> "$insert_file" + done + + get_cloud_cfg "$cfg_dir" "$silo" >> "$insert_file" + echo "clouds.add(cloud)" >> "$insert_file" + + cat "$insert_file" >> "$script_file" + done + + set +x # Disable `set -x` to prevent printing passwords + echo "Configuring $silo" + JENKINS_URL=$(crudini --get "$HOME"/.config/jenkins_jobs/jenkins_jobs.ini "$silo" url) + JENKINS_USER=$(crudini --get "$HOME"/.config/jenkins_jobs/jenkins_jobs.ini "$silo" user) + JENKINS_PASSWORD=$(crudini --get "$HOME"/.config/jenkins_jobs/jenkins_jobs.ini "$silo" password) + export JENKINS_URL + export JENKINS_USER + export JENKINS_PASSWORD + lftools jenkins groovy "$script_file" +done