From: Aric Gardner Date: Wed, 7 Aug 2019 18:05:51 +0000 (-0400) Subject: Add support for container releases X-Git-Tag: v0.42.0~4 X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=commitdiff_plain;h=d528f52b9ebd3c9982045e4cb754eb8bd0f1e007;p=releng%2Fglobal-jjb.git Add support for container releases - Add support for distribution_type container - Add conditional build step. If {build-node} is ^.*-docker-.* docker login will occur for verify and merge. - Trigger updated to support hidden releases directory. (releases\/.*\.yaml|\.releases\/.*\.yaml) - Container support: container_release_file example: containers: - name: example version: 1.5.1-20190806T184921Z - Verify pulls container - Grab the image_id - Merge - Apply tag - Push the container ISSUE: RELENG-2292 Signed-off-by: Aric Gardner Change-Id: Ibc6cf1a22bd6c0136a94d2b434c6cd332c12b43f --- diff --git a/.jjb-test/lf-release-jobs.yaml b/.jjb-test/lf-release-jobs.yaml index 5f928519..6dc72bca 100644 --- a/.jjb-test/lf-release-jobs.yaml +++ b/.jjb-test/lf-release-jobs.yaml @@ -7,3 +7,4 @@ project: self-release/project project-name: self-release-project branch: master + mvn-settings: gerrit-maven-project-settings diff --git a/docs/jjb/lf-release-jobs.rst b/docs/jjb/lf-release-jobs.rst index f5572473..f07c4e55 100644 --- a/docs/jjb/lf-release-jobs.rst +++ b/docs/jjb/lf-release-jobs.rst @@ -4,14 +4,31 @@ Self Serve Release Jobs ####################### -Self serve release jobs allow a project to create a releases directory and then place a release file in it. -Jenkins will pick this up and then promote the artifact from the staging log directory (log_dir) and tag the release -with the defined version. maven_central_url is optional +Self serve release jobs allow a project to create a releases/ or .releases/ directory and then place a release yaml file in it. +Jenkins will pick this up and sign the ref extrapolated by log_dir and promote the artifact, whether maven or container. + +Maven release jobs can also trigger via "Build with parameters" negating the need for a release file. +The parameters will need to be filled out in the same was as a release file's would, excepting the speacial +RELEASE_FILE parameter which will need to be set to False to inform the job that it should not expect a release file. +The Special Parameters are as follows: + +GERRIT_BRANCH = master +VERSION = 1.0.0 +LOG_DIR = example-project-maven-stage-master/17/ +DISTRIBUTION_TYPE = maven +RELEASE_FILE = False .. note:: Example of a maven release file: +.. note:: + + Release files regex: (releases\/.*\.yaml|\.releases\/.*\.yaml) + directory can be .releases/ or releases/ + file can be ANYTHING.yaml + + .. code-block:: bash $ cat releases/maven-1.0.0.yaml @@ -30,8 +47,12 @@ with the defined version. maven_central_url is optional --- distribution_type: 'container' version: '1.0.0' - project: 'example-project' - log_dir: 'example-project-maven-docker-stage-master/17/' + project: 'test' + containers: + - name: test-backend + version: 1.0.0-20190806T184921Z + - name: test-frontend + version: 1.0.0-20190806T184921Z .. note:: @@ -109,7 +130,7 @@ Jenkins ======= Add a global credential to Jenkins called ``jenkins-release`` and set the ID: ``'jenkins-release'`` -as its value insert the private portion of the ``ssh-key`` that you created for your Gerrit user. +as its value insert the private half of the ``ssh-key`` that you created for your Gerrit user. Add Global vars in Jenkins: Jenkins configure -> Global properties -> Environment variables @@ -161,21 +182,8 @@ Job Templates Release Merge ------------- -Runs: - -- sigul-install -- sigul-configuration -- checkout ref from taglist.log -- applies the $PROJECT.bundle -- signs, tags and pushes - -.. code-block:: bash - - lftools nexus release --server $NEXUS_URL $STAGING_REPO - - :Template Name: - - {project-name}-release-merge-{stream} + - {project-name}-release-merge :Comment Trigger: remerge @@ -198,25 +206,15 @@ Runs: file modifications will trigger a build. **default**:: - - compare-type: ANT - pattern: 'releases/*.yaml' + - compare-type: REG_EXP + pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' Release Verify ------------------ -Release verify job checks the schema and ensures that the staging-repo.txt.gz -is available on the job. - -- sigul-install -- sigul-configuration -- checkout ref from taglist.log -- applies the $PROJECT.bundle -- signs and shows signature - - :Template Names: - - {project-name}-release-verify-{stream} + - {project-name}-release-verify :Comment Trigger: recheck|reverify @@ -244,5 +242,5 @@ is available on the job. file modifications will trigger a build. **default**:: - - compare-type: ANT - pattern: 'releases/*.yaml' + - compare-type: REG_EXP + pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' diff --git a/jjb/lf-release-jobs.yaml b/jjb/lf-release-jobs.yaml index 29deacaf..200b14b5 100644 --- a/jjb/lf-release-jobs.yaml +++ b/jjb/lf-release-jobs.yaml @@ -61,7 +61,7 @@ exclude-no-code-change: false - draft-published-event - comment-added-contains-event: - comment-contains-value: '^Patch Set[ ]+[0-9]+:([ ]+|[\n]+)(recheck|reverify)$' + comment-contains-value: "^Patch Set[ ]+[0-9]+:([ ]+|[\n]+)(recheck|reverify)$" ##################### # Job Configuration # @@ -71,6 +71,16 @@ builders: - lf-infra-pre-build + - conditional-step: + condition-kind: regex-match + regex: "^.*-docker-.*" + label: "{build-node}" + steps: + - lf-provide-maven-settings: + global-settings-file: "global-settings" + settings-file: "{mvn-settings}" + - shell: !include-raw-escape: ../shell/docker-login.sh + - lf-provide-maven-settings-cleanup - config-file-provider: files: - file-id: sigul-config @@ -119,8 +129,8 @@ - branch-compare-type: "ANT" branch-pattern: "**" file-paths: - - compare-type: ANT - pattern: "releases/*.yaml" + - compare-type: REG_EXP + pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' skip-vote: successful: "{gerrit-skip-vote}" failed: "{gerrit-skip-vote}" @@ -157,6 +167,16 @@ builders: - lf-infra-pre-build + - conditional-step: + condition-kind: regex-match + regex: "^.*-docker-.*" + label: "{build-node}" + steps: + - lf-provide-maven-settings: + global-settings-file: "global-settings" + settings-file: "{mvn-settings}" + - shell: !include-raw-escape: ../shell/docker-login.sh + - lf-provide-maven-settings-cleanup - config-file-provider: files: - file-id: sigul-config @@ -205,5 +225,5 @@ - branch-compare-type: "ANT" branch-pattern: "**" file-paths: - - compare-type: ANT - pattern: "releases/*.yaml" + - compare-type: REG_EXP + pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' diff --git a/releasenotes/notes/lf-release-jobs-21bd4681a7454451.yaml b/releasenotes/notes/lf-release-jobs-21bd4681a7454451.yaml new file mode 100644 index 00000000..798f43f7 --- /dev/null +++ b/releasenotes/notes/lf-release-jobs-21bd4681a7454451.yaml @@ -0,0 +1,21 @@ +--- +features: + - | + Add support for distribution_type "container" + - | + Add function maven_release_file and container_release_file + and the logic to choose the correct one. + No functional change to maven_release_file. + - | + Add docker login step when docker releases are being processed. + - | + container_release_file downloads log_dir/console.log.gz + and parses it to get a list of container name and version. + Verifies pulls container and grabs the image_id then + performs the merge then tags and pushes the container. +upgrade: + - | + release-verify and merge will need to run on a docker build-node + for example centos7-docker-8c-8g + Lftools will need to be updated to 0.26.0 so that -v is supported for + lftools nexus release diff --git a/schema/release-container-schema.yaml b/schema/release-container-schema.yaml new file mode 100644 index 00000000..f12c8ee0 --- /dev/null +++ b/schema/release-container-schema.yaml @@ -0,0 +1,37 @@ +# SPDX-License-Identifier: EPL-1.0 +############################################################################## +# Copyright (c) 2019 The Linux Foundation and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Eclipse Public License v1.0 +# which accompanies this distribution, and is available at +# http://www.eclipse.org/legal/epl-v10.html +############################################################################## +--- +$schema: "http://json-schema.org/schema#" +$id: "https://github.com/lfit/releng-global-jjb/blob/master/release-container-schema.yaml" + +required: + - "containers" + - "distribution_type" + - "project" + - "version" + - "ref" + +properties: + containers: + type: "array" + properties: + name: + type: "string" + version: + type: "string" + additionalProperties: false + distribution_type: + type: "string" + project: + type: "string" + version: + type: "string" + ref: + type: "string" diff --git a/shell/release-job.sh b/shell/release-job.sh index 31721d11..0ada4617 100644 --- a/shell/release-job.sh +++ b/shell/release-job.sh @@ -82,9 +82,47 @@ nexus_release(){ done } + container_release_file(){ echo "---> Processing container release" - # Container-release code is addressed in a new feature patchset. + local lfn_umbrella + lfn_umbrella="$(echo "$GERRIT_HOST" | awk -F"." '{print $2}')" + + + for namequoted in $(cat $release_file | yq '.containers[].name'); do + versionquoted=$(cat $release_file | yq ".containers[] |select(.name=="$namequoted") |.version") + + #Remove extra yaml quotes + name="${namequoted#\"}" + name="${name%\"}" + version="${versionquoted#\"}" + version="${version%\"}" + + echo "$name" + echo "$version" + echo "---> INFO: Merge will release $name $version as $VERSION" + #Pull from public, to see if we have already tagged this. + if docker pull "$DOCKER_REGISTRY":10002/"$lfn_umbrella"/"$name":"$VERSION"; then + echo "---> OK: $VERSION is already released for image $name, Continuing..." + else + echo "---> OK: $VERSION not found in releases, release will be prepared. Continuing..." + docker pull "$DOCKER_REGISTRY":10001/"$lfn_umbrella"/"$name":"$version" + container_image_id="$(docker images | grep $name | grep $version | awk '{print $3}')" + echo "---> INFO: Merge will run the following commands:" + echo "docker tag $container_image_id $DOCKER_REGISTRY:10002/$lfn_umbrella/$name:$VERSION" + echo "docker push $DOCKER_REGISTRY:10002/$lfn_umbrella/$name:$VERSION" + if [[ "$JOB_NAME" =~ "merge" ]]; then + docker tag "$container_image_id" "$DOCKER_REGISTRY":10002/"$lfn_umbrella"/"$name":"$VERSION" + docker push "$DOCKER_REGISTRY":10002/"$lfn_umbrella"/"$name":"$VERSION" + fi + echo "#########################" + fi + done + + ref="$(niet ".ref" "$release_file")" + echo "---> INFO: Merge will tag ref: $ref" + git checkout "$ref" + tag } maven_release_file(){ @@ -178,7 +216,9 @@ if [[ "$DISTRIBUTION_TYPE" == "maven" ]]; then fi maven_release_file elif [[ "$DISTRIBUTION_TYPE" == "container" ]]; then - # Container-release code is addressed in a new feature patchset. + wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/release-container-schema.yaml + RELEASE_SCHEMA="release-container-schema.yaml" + verify_schema container_release_file else echo "---> ERROR: distribution_type: $DISTRIBUTION_TYPE not supported" @@ -188,4 +228,3 @@ fi ########################################## echo "########### End Script release-job.sh ###################################" -