From: Lott, Christopher (cl778h) Date: Mon, 10 Jun 2019 14:56:30 +0000 (-0400) Subject: Refactor templates and scripts for container tag X-Git-Tag: v0.38.4~5 X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=commitdiff_plain;h=f88c7c24b2b1a6f04f85afa8c0a91d8321aef8ea;p=releng%2Fglobal-jjb.git Refactor templates and scripts for container tag Change parameter names used to specify container tag method, with a new default to use the fixed string 'latest' as a tag. Merge two shell scripts into one instead of using Jenkins conditional steps. Extend to accept a custom directory for the container-tag.yaml if the yaml-file method is used to set the docker tag information; this is an optional variable which is set to empty by default, and falls back to DOCKER_ROOT. Signed-off-by: Lott, Christopher (cl778h) Change-Id: I19a8809650fa29706acaf0f06f4b666bef0067d7 --- diff --git a/.jjb-test/lf-docker-jobs/docker-jobs.yaml b/.jjb-test/lf-docker-jobs/docker-jobs.yaml index a414ab99..fed0e092 100644 --- a/.jjb-test/lf-docker-jobs/docker-jobs.yaml +++ b/.jjb-test/lf-docker-jobs/docker-jobs.yaml @@ -8,9 +8,10 @@ project-name: docker-project container-public-registry: pub-registry container-push-registry: push-registry + container-tag-method: yaml-file + container-tag-yaml-dir: '' docker-build-args: '' docker-name: docker-image-name docker-root: docker - docker-use-params-from: yaml-file-params branch: master mvn-settings: docker-project-settings diff --git a/docs/jjb/lf-docker-jobs.rst b/docs/jjb/lf-docker-jobs.rst index b661f527..4a91dc7f 100644 --- a/docs/jjb/lf-docker-jobs.rst +++ b/docs/jjb/lf-docker-jobs.rst @@ -19,8 +19,16 @@ Macros lf-docker-get-container-tag --------------------------- -Calls docker-get-git-describe.sh or docker-get-yaml-tag.sh (depending on the -'docker-use-params-from' condition) to obtain the tag to build. +Chooses a container tag to label the image based on the 'container-tag-method' +parameter. If container-tag-method: latest, the tag 'latest' is used. +If container-tag-method: git-describe, the tag is obtained using +the git describe command, which requires that the repository has a git tag. +If container-tag-method: yaml-file, the tag is obtained using +the yq command, which requires that the repository has a YAML file named +'container-tag.yaml'. The script checks the docker-root directory by +default or the directory specified by parameter container-tag-yaml-dir. +An example file appears below. + lf-docker-build --------------- @@ -53,9 +61,6 @@ Executes a docker build task. :build-node: The node to run build on. :container-public-registry: Docker registry source with base images. :docker-name: Name of the Docker image. - :docker-use-params-from: Used to select the source of the tag information. - Options are "git-describe-params" or "yaml-file-params". (yaml-file-params - expects the tag to be defined in a local file "container-tag.yaml"). :jenkins-ssh-credential: Credential to use for SSH. (Generally should be configured in defaults.yaml) :mvn-settings: Maven settings.xml file containing credentials to use. @@ -65,13 +70,19 @@ Executes a docker build task. :branch: Git branch to fetch for the build. (default: master) :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) :build-timeout: Timeout in minutes before aborting build. (default: 60) + :container-tag-method: Specifies the docker tag-choosing method. + Options are "latest", "git-describe" or "yaml-file". + Option git-describe requires a git tag to exist in the repository. + Option yaml-file requires a file "container-tag.yaml" to exist in the repository. + (default: latest) + :container-tag-yaml-dir: Directory with container-tag.yaml. (default: $DOCKER_ROOT) :docker-build-args: Additional arguments for the docker build command. - :docker-root: Path of the Dockerfile within the repo. + :docker-root: Build directory within the repo. (default: $WORKSPACE, the repo root) :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) - :pre_docker_build_script: Optional build script to execute before the main verify - builder steps. - :post_docker_build_script: Optional build script to execute after the main verify - builder steps. + :pre_docker_build_script: Build script to execute before the main verify + builder steps. (default: a string with only a comment) + :post_docker_build_script: Build script to execute after the main verify + builder steps. (default: a string with only a comment) :stream: Keyword that can be used to represent a release code-name. Often the same as the branch. (default: master) :submodule-recursive: Whether to checkout submodules recursively. @@ -113,9 +124,6 @@ Executes a docker build task and publishes the resulting images to a specified D :container-public-registry: Docker registry source with base images. :container-push-registry: Docker registry target for the deploy action. :docker-name: Name of the Docker image. - :docker-use-params-from: Used to select the source of the tag information. - Options are "git-describe-params" or "yaml-file-params". (yaml-file-params - expects the tag to be defined in a local file "container-tag.yaml"). :jenkins-ssh-credential: Credential to use for SSH. (Generally should be configured in defaults.yaml) :mvn-settings: Maven settings.xml file containing credentials to use. @@ -125,17 +133,23 @@ Executes a docker build task and publishes the resulting images to a specified D :branch: Git branch to fetch for the build. (default: master) :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) :build-timeout: Timeout in minutes before aborting build. (default: 60) + :container-tag-method: Specifies the docker tag-choosing method. + Options are "latest", "git-describe" or "yaml-file". + Option git-describe requires a git tag to exist in the repository. + Option yaml-file requires a file "container-tag.yaml" to exist in the repository. + (default: latest) + :container-tag-yaml-dir: Directory with container-tag.yaml. (default: $DOCKER_ROOT) :cron: Cron schedule when to trigger the job. This parameter also supports multiline input via YAML pipe | character in cases where one may want to provide more than 1 cron timer. No default. Use '@daily' to run daily or 'H H * * 0' to run weekly. :docker-build-args: Additional arguments for the docker build command. - :docker-root: Path of the Dockerfile within the repo. + :docker-root: Build directory within the repo. (default: $WORKSPACE, the repo root) :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) - :pre_docker_build_script: Optional build script to execute before the main merge - builder steps. - :post_docker_build_script: Optional build script to execute after the main merge - builder steps. + :pre_docker_build_script: Build script to execute before the main merge + builder steps. (default: a string with only a comment) + :post_docker_build_script: Build script to execute after the main merge + builder steps. (default: a string with only a comment) :stream: Keyword that can be used to represent a release code-name. Often the same as the branch. (default: master) :submodule-recursive: Whether to checkout submodules recursively. @@ -150,7 +164,8 @@ Executes a docker build task and publishes the resulting images to a specified D used to filter which file modifications will trigger a build; must match parameter gerrit_trigger_file_paths -container-tag.yaml example: +Sample container-tag.yaml File +------------------------------ .. code-block:: yaml diff --git a/jjb/lf-docker-jobs.yaml b/jjb/lf-docker-jobs.yaml index d4d265a4..6fbf09bf 100644 --- a/jjb/lf-docker-jobs.yaml +++ b/jjb/lf-docker-jobs.yaml @@ -10,26 +10,15 @@ builders: - inject: properties-content: | + CONTAINER_TAG_METHOD={container-tag-method} + CONTAINER_TAG_YAML_DIR={container-tag-yaml-dir} DOCKER_ROOT={docker-root} - - conditional-step: - condition-kind: regex-match - regex: git-describe-params - label: '{docker-use-params-from}' - steps: - - shell: !include-raw-escape: - - ../shell/docker-get-git-describe.sh - - conditional-step: - condition-kind: regex-match - regex: yaml-file-params - label: '{docker-use-params-from}' - steps: - - shell: !include-raw-escape: - - ../shell/docker-get-yaml-tag.sh + - shell: !include-raw-escape: + - ../shell/docker-get-container-tag.sh - inject: - # Import the docker image information from the previous build step + # Import the container tag set by this build step properties-file: 'env_docker_inject.txt' - - builder: name: lf-docker-build builders: @@ -74,11 +63,13 @@ submodule-timeout: 10 pre_docker_build_script: '# pre docker build script goes here' post_docker_build_script: '# post docker build script goes here' - docker-use-params-from: '' disable-job: 'false' docker-root: '$WORKSPACE' docker-build-args: '' git-url: '$GIT_URL/$PROJECT' + container-tag-method: 'latest' + # TODO: how to interpolate value of {docker-root} parameter? + container-tag-yaml-dir: '' ##################### # Job Configuration # @@ -145,7 +136,8 @@ settings-file: '{mvn-settings}' - shell: '{pre_docker_build_script}' - lf-docker-get-container-tag: - docker-use-params-from: '{docker-use-params-from}' + container-tag-method: '{container-tag-method}' + container-tag-yaml-dir: '{container-tag-yaml-dir}' docker-root: '{docker-root}' - lf-docker-build: docker-build-args: '{docker-build-args}' @@ -191,7 +183,8 @@ settings-file: '{mvn-settings}' - shell: '{pre_docker_build_script}' - lf-docker-get-container-tag: - docker-use-params-from: '{docker-use-params-from}' + container-tag-method: '{container-tag-method}' + container-tag-yaml-dir: '{container-tag-yaml-dir}' docker-root: '{docker-root}' - lf-docker-build: docker-build-args: '{docker-build-args}' diff --git a/releasenotes/notes/add-container-tag-file-path-854b7a89a533a11f.yaml b/releasenotes/notes/add-container-tag-file-path-854b7a89a533a11f.yaml new file mode 100644 index 00000000..a862ae80 --- /dev/null +++ b/releasenotes/notes/add-container-tag-file-path-854b7a89a533a11f.yaml @@ -0,0 +1,11 @@ +--- +fixes: + - | + Change parameter names used to specify container tag method, + with a new default to use the fixed string 'latest' as a tag. + Merge two shell scripts into one instead of using Jenkins + conditional steps. + Extend to accept a custom directory for the container-tag.yaml + if the yaml-file method is used to set the docker tag information; + this is an optional variable which is set to empty by default, + and falls back to DOCKER_ROOT. diff --git a/shell/docker-get-container-tag.sh b/shell/docker-get-container-tag.sh new file mode 100644 index 00000000..e522e220 --- /dev/null +++ b/shell/docker-get-container-tag.sh @@ -0,0 +1,42 @@ +#!/bin/bash +# SPDX-License-Identifier: EPL-1.0 +############################################################################## +# Copyright (c) 2019 The Linux Foundation and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Eclipse Public License v1.0 +# which accompanies this distribution, and is available at +# http://www.eclipse.org/legal/epl-v10.html +############################################################################## +echo "---> docker-get-container-tag.sh" +# Gets the container tag per $CONTAINER_TAG_METHOD +# For YAML file use directory $CONTAINER_TAG_YAML_DIR +# if value is provided, else fall back to $DOCKER_ROOT + +set -feu -o pipefail + +tag="" +if [[ $CONTAINER_TAG_METHOD == "latest" ]]; then + tag="latest" +elif [[ $CONTAINER_TAG_METHOD == "git-describe" ]]; then + tag=$(git describe) +elif [[ $CONTAINER_TAG_METHOD == "yaml-file" ]]; then + # TODO: set default value in JJB? + dir=${CONTAINER_TAG_YAML_DIR:-$DOCKER_ROOT} + tag_file=$dir/container-tag.yaml + if [[ -f $tag_file ]]; then + tag=$(yq -r .tag tag_file) + else + echo "File $tag_file not found." + fi +else + echo "Method $CONTAINER_TAG_METHOD not implemented (yet)" +fi +if [[ -z $tag ]]; then + echo "Failed to get a container tag using method $CONTAINER_TAG_METHOD" + exit 1 +fi +echo "---> Docker image tag found: $tag" +# Write DOCKER_IMAGE_TAG information to a file so it can be +# injected into the environment for following steps +echo "DOCKER_IMAGE_TAG=$tag" >> "$WORKSPACE/env_docker_inject.txt" diff --git a/shell/docker-get-git-describe.sh b/shell/docker-get-git-describe.sh deleted file mode 100644 index c1a3d21e..00000000 --- a/shell/docker-get-git-describe.sh +++ /dev/null @@ -1,28 +0,0 @@ -#!/bin/bash -# SPDX-License-Identifier: EPL-1.0 -############################################################################## -# Copyright (c) 2019 The Linux Foundation and others. -# -# All rights reserved. This program and the accompanying materials -# are made available under the terms of the Eclipse Public License v1.0 -# which accompanies this distribution, and is available at -# http://www.eclipse.org/legal/epl-v10.html -############################################################################## -echo "---> docker-get-git-describe.sh" -# Gets the container tag using git describe. - -set -eu -o pipefail - -cd "$DOCKER_ROOT" - -image_build_tag=$(git describe) - -if [ -z "$image_build_tag" ] -then - echo "git describe returned an empty value, make sure a version tag is applied" - exit 1 -fi -echo "---> Tag found: $image_build_tag" -# Write DOCKER_IMAGE_TAG information to a file so it can be injected into the -# environment for following steps -echo "DOCKER_IMAGE_TAG=$image_build_tag" >> "$WORKSPACE/env_docker_inject.txt" diff --git a/shell/docker-get-yaml-tag.sh b/shell/docker-get-yaml-tag.sh deleted file mode 100644 index 8b0e16ee..00000000 --- a/shell/docker-get-yaml-tag.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/bin/bash -l -# SPDX-License-Identifier: EPL-1.0 -############################################################################## -# Copyright (c) 2019 The Linux Foundation and others. -# -# All rights reserved. This program and the accompanying materials -# are made available under the terms of the Eclipse Public License v1.0 -# which accompanies this distribution, and is available at -# http://www.eclipse.org/legal/epl-v10.html -############################################################################## -echo "---> docker-get-yaml-tag.sh" -# Gets the container tag from a yaml file. - -set -eu -o pipefail - -cd "$DOCKER_ROOT" -container_tag_file=container-tag.yaml - -if [ -f "$container_tag_file" ] -then - image_build_tag=$(yq -r .tag "$container_tag_file") -else - echo "$container_tag_file file not found. Make sure this file exists." - exit 1 -fi -echo "---> Tag found: $image_build_tag" -# Write DOCKER_IMAGE_TAG information to a file so it can be injected into the -# environment for following steps -echo "DOCKER_IMAGE_TAG=$image_build_tag" >> "$WORKSPACE/env_docker_inject.txt"