Move openstack-cron job to global-jjb

Move OpenDaylight's builder-openstack-cron job to global-jjb. This is
an initial patch that only migrates the "image-protection" script.

image-protection is used to parse jenkins-config/ and jjb/ directories
for any mention of images starting with "ZZCI" and flagging them as
protected. The assumption here is that ZZCI images are CI managed
and if they are referred to anywher by anything then they are
effectively considered in-use and should not be deletable so flag
them as "protected: true" in OpenStack.

Change-Id: Ib706663f0c08bd7a9160b0a6ce69dcd3b802f8f5
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

diff --git a/.jjb-test/lf-ci-jobs.yaml b/.jjb-test/lf-ci-jobs.yaml
index ecf365d..c9dfdfc 100644 (file)
--- a/.jjb-test/lf-ci-jobs.yaml
+++ b/.jjb-test/lf-ci-jobs.yaml
@@ -15,6 +15,21 @@
     project-name: github-ciman
 
 
+- project:
+    name: gerrit-openstack-jobs
+    jobs:
+      - gerrit-openstack-cron
+
+    project-name: gerrit-ciman
+
+- project:
+    name: github-openstack-jobs
+    jobs:
+      - github-openstack-cron
+
+    project-name: github-ciman
+
+
 - project:
     name: gerrit-packer-jobs
     jobs:

diff --git a/docs/jjb/lf-ci-jobs.rst b/docs/jjb/lf-ci-jobs.rst
index dc9690e..25d60c5 100644 (file)
--- a/docs/jjb/lf-ci-jobs.rst
+++ b/docs/jjb/lf-ci-jobs.rst
@@ -510,6 +510,41 @@ Job to scan projects for files missing license headers.
     :project-pattern: The ANT based pattern for Gerrit Trigger to choose which
         projects to trigger job against. (default: '**')
 
+.. _gjjb-openstack-cron:
+
+OpenStack Cron
+--------------
+
+Cron job that runs regularly to perform periodic tasks against OpenStack.
+
+This job requires a Config File Provider file named ``clouds-yaml`` available
+containing the credentials for the cloud.
+
+:Template Names:
+    - {project-name}-openstack-cron
+    - gerrit-openstack-cron
+    - github-openstack-cron
+
+:Required parameters:
+
+    :build-node: The node to run build on.
+    :jenkins-ssh-credential: Credential to use for SSH. (Generally should
+        be configured in defaults.yaml)
+
+:Optional parameters:
+
+    :branch: Git branch to fetch for the build. (default: master)
+    :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+    :build-timeout: Timeout in minutes before aborting build. (default: 90)
+    :cron: Time when the packer image should be rebuilt (default: @daily)
+    :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+    :openstack-cloud: OS_CLOUD setting to pass to openstack client.
+        (default: vex)
+    :stream: Keyword that can be used to represent a release code-name.
+        Often the same as the branch. (default: master)
+    :submodule-recursive: Whether to checkout submodules recursively.
+        (default: true)
+
 .. _gjjb-packer-merge:
 
 Packer Merge

diff --git a/jjb/lf-ci-jobs.yaml b/jjb/lf-ci-jobs.yaml
index 5532a3e..57e4054 100644 (file)
--- a/jjb/lf-ci-jobs.yaml
+++ b/jjb/lf-ci-jobs.yaml
@@ -1025,6 +1025,99 @@
           white-list-target-branches:
             - '{branch}'
 
+##################
+# OPENSTACK CRON #
+##################
+
+- lf_openstack_cron: &lf_openstack_cron
+    name: lf-openstack-cron
+
+    ######################
+    # Default parameters #
+    ######################
+
+    branch: master
+    build-days-to-keep: 7
+    build-timeout: 10
+    cron: '@daily'
+    git-url: '$GIT_URL/$PROJECT'
+    github-url: 'https://github.com'
+    openstack-cloud: vex
+    stream: master
+
+    #####################
+    # Job Configuration #
+    #####################
+
+    project-type: freestyle
+    node: '{build-node}'
+    concurrent: false
+
+    properties:
+      - lf-infra-properties:
+          build-days-to-keep: '{build-days-to-keep}'
+
+    parameters:
+      - lf-infra-parameters:
+          project: '{project}'
+          stream: '{stream}'
+          branch: '{branch}'
+          lftools-version: '{lftools-version}'
+
+    wrappers:
+      - lf-infra-wrappers:
+          build-timeout: '{build-timeout}'
+          jenkins-ssh-credential: '{jenkins-ssh-credential}'
+      # Listed after to override openstack-infra-wrappers clouds.yaml definition
+      - config-file-provider:
+          files:
+            - file-id: clouds-yaml
+              target: '$HOME/.config/openstack/clouds.yaml'
+            - file-id: npmrc
+              target: '$HOME/.npmrc'
+            - file-id: pipconf
+              target: '$HOME/.config/pip/pip.conf'
+
+    triggers:
+      - timed: '{obj:cron}'
+
+    builders:
+      - inject:
+          properties-content: OS_CLOUD={openstack-cloud}
+      - shell: !include-raw-escape: ../shell/openstack-install.sh
+      - shell: !include-raw-escape: ../shell/openstack-protect-in-use-images.sh
+
+    publishers:
+      - lf-infra-publish
+
+- job-template:
+    name: '{project-name}-openstack-cron'
+    id: gerrit-openstack-cron
+    <<: *lf_openstack_cron
+
+    scm:
+      - lf-infra-gerrit-scm:
+          git-url: '{git-url}'
+          refspec: 'refs/heads/{branch}'
+          branch: '{branch}'
+          submodule-recursive: true
+          choosing-strategy: default
+          jenkins-ssh-credential: '{jenkins-ssh-credential}'
+
+- job-template:
+    name: '{project-name}-openstack-cron'
+    id: github-openstack-cron
+    <<: *lf_openstack_cron
+
+    scm:
+      - lf-infra-github-scm:
+          url: '{git-clone-url}{github-org}/{project}'
+          refspec: ''
+          branch: 'refs/heads/{branch}'
+          submodule-recursive: true
+          choosing-strategy: default
+          jenkins-ssh-credential: '{jenkins-ssh-credential}'
+
 ################
 # Packer Merge #
 ################

diff --git a/shell/openstack-install.sh b/shell/openstack-install.sh
new file mode 100644 (file)
index 0000000..a7e5099
--- /dev/null
+++ b/shell/openstack-install.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+# SPDX-License-Identifier: EPL-1.0
+##############################################################################
+# Copyright (c) 2018 The Linux Foundation and others.
+#
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Eclipse Public License v1.0
+# which accompanies this distribution, and is available at
+# http://www.eclipse.org/legal/epl-v10.html
+##############################################################################
+# Install the openstack cli
+echo "---> Install openstack cli"
+
+set -eux -o pipefail
+
+pip install --user --quiet --upgrade "pip<10.0.0" setuptools
+pip install --user --quiet --upgrade python-openstackclient python-heatclient
+pip freeze

diff --git a/shell/openstack-protect-in-use-images.sh b/shell/openstack-protect-in-use-images.sh
new file mode 100644 (file)
index 0000000..a23a4ed
--- /dev/null
+++ b/shell/openstack-protect-in-use-images.sh
@@ -0,0 +1,43 @@
+#!/bin/bash -l
+# SPDX-License-Identifier: EPL-1.0
+##############################################################################
+# Copyright (c) 2017, 2018 The Linux Foundation and others.
+#
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Eclipse Public License v1.0
+# which accompanies this distribution, and is available at
+# http://www.eclipse.org/legal/epl-v10.html
+##############################################################################
+# Checks the image "protected" value and set "True" marker
+#
+# The script searches the ciman repo for the images presently used and ensures
+# the protection setting is set for those images to prevent the image from
+# getting purged by the image cleanup script.
+# This script assumes images prefixed with the string "ZZCI - " are ci-managed
+# images.
+echo "---> Protect in-use images"
+
+os_cloud="${OS_CLOUD:-vex}"
+
+set -eu -o pipefail
+
+declare -a images
+declare -a cfg_images
+declare -a yaml_images
+readarray -t cfg_images <<< "$(grep -r IMAGE_NAME --include \*.cfg \
+    | awk -F'=' '{print $2}' | sort -u)"
+readarray -t yaml_images <<< "$(grep -r 'ZZCI - ' --include \*.yaml \
+    | awk -F": " '{print $3}' | sed "s:'::;s:'$::;/^$/d" | sort -u)"
+mapfile -t images < <(for R in "${cfg_images[@]}" "${yaml_images[@]}" ; do echo "$R" ; done | sort -u)
+
+
+for image in "${images[@]}"; do
+    os_image_protected=$(openstack --os-cloud "$os_cloud" \
+        image show "$image" -f value -c protected)
+    echo "Protected setting for $image: $os_image_protected"
+
+    if [[ $os_image_protected != True ]]; then
+        echo "    Image NOT set as protected, changing the protected value."
+        openstack --os-cloud "$os_cloud" image set --protected "$image"
+    fi
+done