From: Jessica Wagantall Date: Wed, 13 Feb 2019 18:13:19 +0000 (-0800) Subject: Add Docker Verify and Merge jobs X-Git-Tag: v0.38.0^2 X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=commitdiff_plain;h=8f00e059ea64956604ea2d23ebe46e4fda788cdf;p=releng%2Fglobal-jjb.git Add Docker Verify and Merge jobs Add {project-name}-docker-verify-{stream} and {project-name}-docker-merge-{stream} jobs. Issue: RELENG-908 Change-Id: I009fea61fd4c13daebac0d14f5dc00e1a90011e7 Signed-off-by: Jessica Wagantall --- diff --git a/.coafile b/.coafile index 7e9db1c8..44f62150 100644 --- a/.coafile +++ b/.coafile @@ -34,6 +34,7 @@ use_spaces = true ignore = .git/**, .tox/**, docs/jjb/lf-ci-jobs.rst, + docs/jjb/lf-docker-jobs.rst, docs/jjb/lf-macros.rst, docs/jjb/lf-maven-jobs.rst diff --git a/.jjb-test/lf-docker-jobs/docker-jobs.yaml b/.jjb-test/lf-docker-jobs/docker-jobs.yaml new file mode 100644 index 00000000..bb05ea4b --- /dev/null +++ b/.jjb-test/lf-docker-jobs/docker-jobs.yaml @@ -0,0 +1,13 @@ +--- +- project: + name: '{project-name}-docker' + jobs: + - '{project-name}-gerrit-docker-jobs' + + project: docker/project + project-name: docker-project + container-public-registry: pub-registry + container-snapshot-registry: snap-registry + docker-name: docker-image-name + branch: master + mvn-settings: docker-project-settings diff --git a/docs/jjb/lf-docker-jobs.rst b/docs/jjb/lf-docker-jobs.rst new file mode 100644 index 00000000..05a0e3cd --- /dev/null +++ b/docs/jjb/lf-docker-jobs.rst @@ -0,0 +1,147 @@ +########### +Docker Jobs +########### + +Job Groups +========== + +.. include:: ../job-groups.rst + +Below is a list of Docker job groups: + +.. literalinclude:: ../../jjb/lf-docker-job-groups.yaml + :language: yaml + + +Macros +====== + +lf-docker-get-container-tag +--------------------------- + +Calls docker-get-git-describe.sh or docker-get-yaml-tag.sh (depending on the +'docker-use-params-from' condition) to obtain the tag to build. + +lf-docker-build +--------------- + +Calls docker build to build the container. + +lf-docker-push +-------------- + +Calls docker-push.sh script to push docker images. + +Job Templates +============= + +Docker Verify +------------- + +Executes a docker build task. + +:Template Names: + + - {project-name}-docker-verify-{stream} + - gerrit-docker-verify + - github-docker-verify + +:Comment Trigger: recheck|reverify + +:Required parameters: + + :build-node: The node to run build on. + :container-public-registry: Docker registry source with base images. + :docker-name: Name of the Docker image. + :docker-use-params-from: Used to select the source of the tag information. + Options are "git-describe-params" or "yaml-file-params". (yaml-file-params + expects the tag to be defined in a local file "container-tag.yaml"). + :jenkins-ssh-credential: Credential to use for SSH. (Generally should + be configured in defaults.yaml) + :mvn-settings: Maven settings.xml file containing credentials to use. + +:Optional parameters: + + :branch: Git branch to fetch for the build. (default: master) + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 60) + :docker-build-args: Additional arguments for the docker build command. + :docker-root: Path of the Dockerfile within the repo. + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :pre_docker_build_script: Optional build script to execute before the main verify + builder steps. + :post_docker_build_script: Optional build script to execute after the main verify + builder steps. + :stream: Keyword that can be used to represent a release code-name. + Often the same as the branch. (default: master) + :submodule-recursive: Whether to checkout submodules recursively. + (default: true) + :submodule-timeout: Timeout (in minutes) for checkout operation. + (default: 10) + + :gerrit_verify_triggers: Override Gerrit Triggers. + :gerrit_trigger_file_paths: Override file paths which can be used to + filter which file modifications will trigger a build. + +container-tag.yaml example: + +.. code-block:: yaml + + --- + tag: 1.0.0 + +Docker Merge +------------ + +Executes a docker build task and publishes the resulting images to a specified Docker registry. + +:Template Names: + + - {project-name}-docker-merge-{stream} + - gerrit-docker-merge + - github-docker-merge + +:Comment Trigger: remerge + +:Required parameters: + + :build-node: The node to run build on. + :container-public-registry: Docker registry source with base images. + :container-push-registry: Docker registry target for the deploy action. + :docker-name: Name of the Docker image. + :docker-use-params-from: Used to select the source of the tag information. + Options are "git-describe-params" or "yaml-file-params". (yaml-file-params + expects the tag to be defined in a local file "container-tag.yaml"). + :jenkins-ssh-credential: Credential to use for SSH. (Generally should + be configured in defaults.yaml) + :mvn-settings: Maven settings.xml file containing credentials to use. + +:Optional parameters: + + :branch: Git branch to fetch for the build. (default: master) + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 60) + :docker-build-args: Additional arguments for the docker build command. + :docker-root: Path of the Dockerfile within the repo. + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :pre_docker_build_script: Optional build script to execute before the main merge + builder steps. + :post_docker_build_script: Optional build script to execute after the main merge + builder steps. + :stream: Keyword that can be used to represent a release code-name. + Often the same as the branch. (default: master) + :submodule-recursive: Whether to checkout submodules recursively. + (default: true) + :submodule-timeout: Timeout (in minutes) for checkout operation. + (default: 10) + + :gerrit_verify_triggers: Override Gerrit Triggers. + :gerrit_trigger_file_paths: Override file paths which can be used to + filter which file modifications will trigger a build. + +container-tag.yaml example: + +.. code-block:: yaml + + --- + tag: 1.0.0 diff --git a/jjb/lf-docker-job-groups.yaml b/jjb/lf-docker-job-groups.yaml new file mode 100644 index 00000000..4635b0b3 --- /dev/null +++ b/jjb/lf-docker-job-groups.yaml @@ -0,0 +1,20 @@ +--- +- job-group: + name: '{project-name}-gerrit-docker-jobs' + + # This job group contains all the recommended jobs that should be deployed + # for any docker project ci. + + jobs: + - gerrit-docker-verify + - gerrit-docker-merge + +- job-group: + name: '{project-name}-github-docker-jobs' + + # This job group contains all the recommended jobs that should be deployed + # for any docker project ci. + + jobs: + - github-docker-verify + - github-docker-merge diff --git a/jjb/lf-docker-jobs.yaml b/jjb/lf-docker-jobs.yaml new file mode 100644 index 00000000..0de21fb2 --- /dev/null +++ b/jjb/lf-docker-jobs.yaml @@ -0,0 +1,310 @@ +--- +# This file contains job templates for Docker projects. + +########## +# Macros # +########## + +- builder: + name: lf-docker-get-container-tag + builders: + - conditional-step: + condition-kind: regex-match + regex: git-describe-params + label: '{docker-use-params-from}' + steps: + - shell: !include-raw-escape: + - ../shell/docker-get-git-describe.sh + - conditional-step: + condition-kind: regex-match + regex: yaml-file-params + label: '{docker-use-params-from}' + steps: + - shell: !include-raw-escape: + - ../shell/docker-get-yaml-tag.sh + - inject: + # Import the docker image information from the previous build step + properties-file: 'env_docker_inject.txt' + + +- builder: + name: lf-docker-build + builders: + - inject: + properties-content: | + DOCKER_ARGS={docker-build-args} + DOCKER_NAME={docker-name} + DOCKER_ROOT={docker-root} + CONTAINER_PULL_REGISTRY={container-public-registry} + - shell: !include-raw-escape: + - ../shell/docker-build.sh + +- builder: + name: lf-docker-push + builders: + - inject: + properties-content: | + CONTAINER_PUSH_REGISTRY={container-push-registry} + - shell: !include-raw-escape: + - ../shell/docker-push.sh + +#################### +# COMMON FUNCTIONS # +#################### + +- lf_docker_common: &lf_docker_common + name: lf-docker-common + + project-type: freestyle + node: '{build-node}' + + ###################### + # Default parameters # + ###################### + + branch: master + build-days-to-keep: 7 + build-timeout: 60 + submodule-disable: false + submodule-recursive: true + submodule-timeout: 10 + pre_docker_build_script: '' + post_docker_build_script: '' + disable-job: 'false' + docker-root: '$WORKSPACE' + docker-build-args: '' + git-url: '$GIT_URL/$PROJECT' + + ##################### + # Job Configuration # + ##################### + + disabled: '{disable-job}' + + properties: + - lf-infra-properties: + build-days-to-keep: '{build-days-to-keep}' + + parameters: + - lf-infra-parameters: + branch: '{branch}' + project: '{project}' + refspec: 'refs/heads/{branch}' + stream: '{stream}' + + wrappers: + - lf-infra-wrappers: + build-timeout: '{build-timeout}' + jenkins-ssh-credential: '{jenkins-ssh-credential}' + + publishers: + - lf-infra-publish + +- docker_verify_common: &docker_verify_common + name: docker-verify-common + + concurrent: true + + scm: + - lf-infra-gerrit-scm: + jenkins-ssh-credential: '{jenkins-ssh-credential}' + git-url: '{git-url}' + refspec: '$GERRIT_REFSPEC' + branch: '$GERRIT_BRANCH' + submodule-disable: '{submodule-disable}' + submodule-recursive: '{submodule-recursive}' + submodule-timeout: '{submodule-timeout}' + choosing-strategy: gerrit + + gerrit_verify_triggers: + - patchset-created-event: + exclude-drafts: true + exclude-trivial-rebase: false + exclude-no-code-change: false + - draft-published-event + - comment-added-contains-event: + comment-contains-value: '^Patch Set[ ]+[0-9]+:([ ]+|[\n]+)(recheck|reverify)$' + + builders: + - lf-infra-pre-build + - lf-infra-docker-login: + global-settings-file: 'global-settings' + settings-file: '{mvn-settings}' + - shell: '{pre_docker_build_script}' + - lf-docker-get-container-tag + - lf-docker-build + - shell: '{post_docker_build_script}' + - lf-provide-maven-settings-cleanup + +- docker_merge_common: &docker_merge_common + name: docker-merge-common + + scm: + - lf-infra-gerrit-scm: + jenkins-ssh-credential: '{jenkins-ssh-credential}' + git-url: '{git-url}' + refspec: '$GERRIT_REFSPEC' + branch: '$GERRIT_BRANCH' + submodule-disable: '{submodule-disable}' + submodule-recursive: '{submodule-recursive}' + submodule-timeout: '{submodule-timeout}' + choosing-strategy: gerrit + + gerrit_merge_triggers: + - change-merged-event + - comment-added-contains-event: + comment-contains-value: remerge$ + + builders: + - lf-infra-pre-build + - lf-infra-docker-login: + global-settings-file: 'global-settings' + settings-file: '{mvn-settings}' + - shell: '{pre_docker_build_script}' + - lf-docker-get-container-tag + - lf-docker-build + - shell: '{post_docker_build_script}' + # Provided all steps have already passed, push the docker image + - lf-docker-push + - lf-provide-maven-settings-cleanup + +################# +# Docker Verify # +################# + +- job-template: + name: '{project-name}-docker-verify-{stream}' + id: gerrit-docker-verify + # Job template for Docker verify jobs + # + # The purpose of this job template is to run a docker build, and potentially + # test validation of the docker image + + <<: *lf_docker_common + # yamllint disable-line rule:key-duplicates + <<: *docker_verify_common + + triggers: + - gerrit: + server-name: '{gerrit-server-name}' + trigger-on: '{obj:gerrit_verify_triggers}' + projects: + - project-compare-type: ANT + project-pattern: '{project}' + branches: + - branch-compare-type: ANT + branch-pattern: '**/{branch}' + +- job-template: + name: '{project-name}-docker-verify-{stream}' + id: github-docker-verify + # Job template for Docker verify jobs + # + # The purpose of this job template is to run a docker build, and potentially + # test validation of the docker image + + <<: *lf_docker_common + # yamllint disable-line rule:key-duplicates + <<: *docker_verify_common + + properties: + - lf-infra-properties: + build-days-to-keep: '{build-days-to-keep}' + - github: + url: '{github-url}/{github-org}/{project}' + + scm: + - lf-infra-github-scm: + url: '{git-clone-url}{github-org}/{project}' + refspec: '+refs/pull/*:refs/remotes/origin/pr/*' + branch: '$sha1' + submodule-recursive: '{submodule-recursive}' + submodule-timeout: '{submodule-timeout}' + submodule-disable: '{submodule-disable}' + choosing-strategy: default + jenkins-ssh-credential: '{jenkins-ssh-credential}' + + triggers: + - github-pull-request: + trigger-phrase: '^(recheck|reverify)$' + only-trigger-phrase: false + status-context: 'Docker Verify' + permit-all: true + github-hooks: true + white-list-target-branches: + - '{branch}' + included-regions: '{obj:github_included_regions}' + +################ +# Docker Merge # +################ + +- job-template: + name: '{project-name}-docker-merge-{stream}' + id: gerrit-docker-merge + # Job template for Docker merge jobs + # + # The purpose of this job template is to run a docker build, and potentially + # test validation of the docker image + + <<: *lf_docker_common + # yamllint disable-line rule:key-duplicates + <<: *docker_merge_common + + triggers: + - gerrit: + server-name: '{gerrit-server-name}' + trigger-on: '{obj:gerrit_merge_triggers}' + projects: + - project-compare-type: ANT + project-pattern: '{project}' + branches: + - branch-compare-type: ANT + branch-pattern: '**/{branch}' + +- job-template: + name: '{project-name}-docker-merge-{stream}' + id: github-docker-merge + # Job template for Docker merge jobs + # + # The purpose of this job template is to run a docker build, and potentially + # test validation of the docker image + + <<: *lf_docker_common + # yamllint disable-line rule:key-duplicates + <<: *docker_merge_common + + properties: + - lf-infra-properties: + build-days-to-keep: '{build-days-to-keep}' + - github: + url: '{github-url}/{github-org}/{project}' + + scm: + - lf-infra-github-scm: + url: '{git-clone-url}{github-org}/{project}' + refspec: '' + branch: 'refs/heads/{branch}' + submodule-recursive: '{submodule-recursive}' + submodule-timeout: '{submodule-timeout}' + submodule-disable: '{submodule-disable}' + choosing-strategy: default + jenkins-ssh-credential: '{jenkins-ssh-credential}' + + triggers: + - github + - pollscm: + cron: '' + - github-pull-request: + trigger-phrase: '^remerge$' + only-trigger-phrase: true + status-context: 'Docker Merge' + permit-all: true + github-hooks: true + org-list: + - '{github-org}' + white-list: '{obj:github_pr_whitelist}' + admin-list: '{obj:github_pr_admin_list}' + white-list-target-branches: + - '{branch}' + included-regions: '{obj:github_included_regions}' diff --git a/releasenotes/notes/lf-docker-jobs-a6927a60ad849d2d.yaml b/releasenotes/notes/lf-docker-jobs-a6927a60ad849d2d.yaml new file mode 100644 index 00000000..43ade8f3 --- /dev/null +++ b/releasenotes/notes/lf-docker-jobs-a6927a60ad849d2d.yaml @@ -0,0 +1,8 @@ +--- +features: + - | + gerrit-docker-verify runs for new commits and runs a build of the affected + Docker images. + - | + gerrit-docker-merge runs for merged commits, runs a build of the affected + Docker images and pushes the images to a specified Docker registry. diff --git a/requirements.txt b/requirements.txt index d2a80712..eb6884eb 100644 --- a/requirements.txt +++ b/requirements.txt @@ -2,3 +2,4 @@ reno~=2.11.2 sphinx~=1.7.9 sphinxcontrib-programoutput sphinx_bootstrap_theme>=0.6.0 +yq diff --git a/shell/docker-build.sh b/shell/docker-build.sh new file mode 100644 index 00000000..982a61f5 --- /dev/null +++ b/shell/docker-build.sh @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: EPL-1.0 +############################################################################## +# Copyright (c) 2019 The Linux Foundation and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Eclipse Public License v1.0 +# which accompanies this distribution, and is available at +# http://www.eclipse.org/legal/epl-v10.html +############################################################################## +echo "---> docker-build.sh" +# Docker image build script + +set -eu -o pipefail + +cd "$DOCKER_ROOT" +# DOCKER_IMAGE variable gets constructed after lf-docker-get-container-tag builder step +# is executed. It constructs the image name and the appropriate tag in the same varaiable. +docker build "$DOCKER_ARGS" . -t "$DOCKER_IMAGE" | tee "$WORKSPACE/docker_build_log.txt" diff --git a/shell/docker-get-git-describe.sh b/shell/docker-get-git-describe.sh new file mode 100644 index 00000000..326f41a3 --- /dev/null +++ b/shell/docker-get-git-describe.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# SPDX-License-Identifier: EPL-1.0 +############################################################################## +# Copyright (c) 2019 The Linux Foundation and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Eclipse Public License v1.0 +# which accompanies this distribution, and is available at +# http://www.eclipse.org/legal/epl-v10.html +############################################################################## +echo "---> docker-get-git-describe.sh" +# Gets the container tag using git describe. + +set -eu -o pipefail + +cd "$DOCKER_ROOT" + +image_build_tag=$(git describe) + +if [ -z "$image_build_tag" ] +then + echo "git describe returned an empty value, make sure a version tag is applied" + exit 1 +else + image_name="$CONTAINER_PUSH_REGISTRY/$DOCKER_NAME:$image_build_tag" +fi + +# Write DOCKER_IMAGE information to a file so it can be injected into the +# environment for following steps +echo "DOCKER_IMAGE=$image_name" >> "$WORKSPACE/env_docker_inject.txt" diff --git a/shell/docker-get-yaml-tag.sh b/shell/docker-get-yaml-tag.sh new file mode 100644 index 00000000..2723eee4 --- /dev/null +++ b/shell/docker-get-yaml-tag.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# SPDX-License-Identifier: EPL-1.0 +############################################################################## +# Copyright (c) 2019 The Linux Foundation and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Eclipse Public License v1.0 +# which accompanies this distribution, and is available at +# http://www.eclipse.org/legal/epl-v10.html +############################################################################## +echo "---> docker-get-yaml-tag.sh" +# Gets the container tag from a yaml file. + +set -eu -o pipefail + +cd "$DOCKER_ROOT" +container_tag_file=container-tag.yaml + +if [ -f "$container_tag_file" ] +then + image_name=$(yq -r .tag "$container_tag_file") +else + echo "$container_tag_file file not found. Make sure this file exists." + exit 1 +fi + +# Write DOCKER_IMAGE information to a file so it can be injected into the +# environment for following steps +echo "DOCKER_IMAGE=$image_name" >> "$WORKSPACE/env_docker_inject.txt" diff --git a/shell/docker-push.sh b/shell/docker-push.sh new file mode 100644 index 00000000..83996fab --- /dev/null +++ b/shell/docker-push.sh @@ -0,0 +1,17 @@ +#!/bin/bash +# SPDX-License-Identifier: EPL-1.0 +############################################################################## +# Copyright (c) 2018 The Linux Foundation and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Eclipse Public License v1.0 +# which accompanies this distribution, and is available at +# http://www.eclipse.org/legal/epl-v10.html +############################################################################## +echo "---> docker-push.sh" +# Docker image push script + +# Ensure we fail the job if any steps fail +set -ue -o pipefail + +docker push "$CONTAINER_PUSH_REGISTRY/$DOCKER_IMAGE"