From: Anil Belur Date: Tue, 21 Apr 2020 01:44:31 +0000 (+0000) Subject: Merge "Add packer-verify-build jobs" X-Git-Tag: v0.53.0 X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=commitdiff_plain;h=refs%2Ftags%2Fv0.53.0;hp=7e25ce08cc805721308a72aa6f91d29ea31658cc;p=releng%2Fglobal-jjb.git Merge "Add packer-verify-build jobs" --- diff --git a/docs/jjb/lf-ci-jobs.rst b/docs/jjb/lf-ci-jobs.rst index 0323bd2e..1ec54851 100644 --- a/docs/jjb/lf-ci-jobs.rst +++ b/docs/jjb/lf-ci-jobs.rst @@ -744,9 +744,9 @@ Packer Merge job runs `packer build` to build system images in the cloud. :mvn-settings: The name of settings file containing credentials for the project. :platforms: Platform or distribution to build. Typically json file - found in the packer/vars directory. (Example: centos) - :template: System template to build. Typically shell script found in - the packer/provision directory. (Example: java-builder) + found in the packer/vars directory. (Example: centos-7) + :templates: System template to build. Typically a yaml file or shell script + found in the packer/provision directory. (Example: docker) :Optional parameters: @@ -754,6 +754,7 @@ Packer Merge job runs `packer build` to build system images in the cloud. :branch: Git branch to fetch for the build. (default: master) :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) :build-timeout: Timeout in minutes before aborting build. (default: 90) + :gerrit_verify_triggers: Override Gerrit Triggers. :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) :openstack: Packer template uses an OpenStack builder (default: true). :openstack-cloud: Sets OS_CLOUD variable to the value of this parameter. @@ -769,18 +770,16 @@ Packer Merge job runs `packer build` to build system images in the cloud. (default: 10) :submodule-disable: Disable submodule checkout operation. (default: false) - - :gerrit_verify_triggers: Override Gerrit Triggers. - :update-cloud-image: Submit a change request to update new built cloud + :update-cloud-image: Submit a change request to update newly built cloud image to Jenkins. (default: false) Test an in-progress patch ^^^^^^^^^^^^^^^^^^^^^^^^^ -To test an in-progress patch from a GitHub Pull Request. Upload this +To test an in-progress patch from a GitHub Pull Request, upload this job to the :doc:`Jenkins Sandbox `. Then when manually -building the job replace the GERRIT_REFSPEC parameter with the GitHub Pull +building the job, replace the GERRIT_REFSPEC parameter with the GitHub Pull Request number of the patch you would like to test. Example GitHub: @@ -789,12 +788,15 @@ Example GitHub: GERRIT_REFSPEC: origin/pr/49/merge + .. _gjjb-packer-verify: Packer Verify ------------- -Packer Verify job runs `packer validate` to verify packer configuration. +Packer Verify job runs `packer validate` to verify packer configuration. The +verify job checks superficial syntax of the template and other files. It does +not attempt to build an image, and cannot detect all possible build issues. :Template Names: - {project-name}-packer-verify @@ -816,6 +818,9 @@ Packer Verify job runs `packer validate` to verify packer configuration. :branch: Git branch to fetch for the build. (default: master) :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) :build-timeout: Timeout in minutes before aborting build. (default: 10) + :gerrit_trigger_file_paths: Override file paths which can be used to + filter which file modifications will trigger a build. + :gerrit_verify_triggers: Override Gerrit Triggers. :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) :openstack: Packer template uses an OpenStack builder (default: true). :openstack-cloud: Sets OS_CLOUD variable to the value of this parameter. @@ -832,9 +837,61 @@ Packer Verify job runs `packer validate` to verify packer configuration. :submodule-disable: Disable submodule checkout operation. (default: false) - :gerrit_verify_triggers: Override Gerrit Triggers. - :gerrit_trigger_file_paths: Override file paths which can be used to - filter which file modifications will trigger a build. + +.. _gjjb-packer-verify-build: + +Packer Verify Build +------------------- + +Packer Verify Build job is essentially the same as the +:ref:`Packer Merge job`. It is triggered only by its keyword, +and will build a useable image. If the last patch set before a merge has a +successful verify build, the merge job will not build the same image. + +:Template Names: + - {project-name}-packer-verify-build-{platforms}-{templates} + - gerrit-packer-verify-build + - github-packer-verify-build + +:Comment Trigger: verify-build|packer-build + +:Required parameters: + + :build-node: The node to run build on. + :jenkins-ssh-credential: Credential to use for SSH. (Generally should + be configured in defaults.yaml) + :mvn-settings: The name of settings file containing credentials for + the project. + :platforms: Platform or distribution to build. Typically json file + found in the packer/vars directory. (Example: centos-7) + :templates: System template to build. Typically a yaml file or shell script + found in the packer/provision directory. (Example: docker) + +:Optional parameters: + + :branch: Git branch to fetch for the build. (default: master) + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 10) + :gerrit_trigger_file_paths: Override file paths which can be used to + filter which file modifications will trigger a build. + :gerrit_verify_triggers: Override Gerrit Triggers. + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :openstack: Packer template uses an OpenStack builder (default: true). + :openstack-cloud: Sets OS_CLOUD variable to the value of this parameter. + (default: vex). + :packer-cloud-settings: Name of settings file containing credentials + for the cloud that packer will build on. (default: packer-cloud-env) + :packer-version: Version of packer to install / use in build. (default: 1.0.2) + :stream: Keyword that can be used to represent a release code-name. + Often the same as the branch. (default: master) + :submodule-recursive: Whether to checkout submodules recursively. + (default: true) + :submodule-timeout: Timeout (in minutes) for checkout operation. + (default: 10) + :submodule-disable: Disable submodule checkout operation. + (default: false) + :update-cloud-image: Submit a change request to update new built cloud + image to Jenkins. (default: false) Puppet Verify diff --git a/jjb/lf-ci-job-groups.yaml b/jjb/lf-ci-job-groups.yaml index 945079cb..9da9578f 100644 --- a/jjb/lf-ci-job-groups.yaml +++ b/jjb/lf-ci-job-groups.yaml @@ -40,6 +40,7 @@ jobs: - gerrit-packer-merge - gerrit-packer-verify + - gerrit-packer-verify-build - job-group: name: "{project-name}-github-packer-jobs" @@ -47,6 +48,7 @@ jobs: - github-packer-merge - github-packer-verify + - github-packer-verify-build - job-group: name: "{project-name}-openstack-jobs" diff --git a/jjb/lf-ci-jobs.yaml b/jjb/lf-ci-jobs.yaml index 635bc67e..cc0b2d5e 100644 --- a/jjb/lf-ci-jobs.yaml +++ b/jjb/lf-ci-jobs.yaml @@ -1581,6 +1581,112 @@ white-list-target-branches: - "{branch}" +- job-template: + name: "{project-name}-packer-verify-build-{platforms}-{templates}" + id: gerrit-packer-verify-build + <<: *lf_packer_common + # yamllint disable-line rule:key-duplicates + <<: *lf_packer_merge + + ###################### + # Default parameters # + ###################### + + git-url: "$GIT_URL/$GERRIT_PROJECT" + + gerrit_verify_triggers: + - comment-added-contains-event: + comment-contains-value: '^Patch Set\s+\d+:\s+(verify-build|packer-build)\s*$' + + ##################### + # Job Configuration # + ##################### + + scm: + - lf-infra-gerrit-scm: + git-url: "{git-url}" + refspec: "$GERRIT_REFSPEC" + branch: "$GERRIT_BRANCH" + submodule-recursive: "{submodule-recursive}" + submodule-timeout: "{submodule-timeout}" + submodule-disable: "{submodule-disable}" + choosing-strategy: gerrit + jenkins-ssh-credential: "{jenkins-ssh-credential}" + + triggers: + - gerrit: + server-name: "{gerrit-server-name}" + trigger-on: "{obj:gerrit_verify_triggers}" + projects: + - project-compare-type: ANT + project-pattern: "{project}" + branches: + - branch-compare-type: ANT + branch-pattern: "**/{branch}" + file-paths: + - compare-type: REG_EXP + pattern: 'packer\/provision\/.*\.sh' + - compare-type: REG_EXP + pattern: 'packer\/provision\/.*\.bat' + - compare-type: REG_EXP + pattern: 'packer\/provision\/.*\.ps1' + - compare-type: REG_EXP + pattern: 'packer\/provision\/{templates}\.yaml' + - compare-type: REG_EXP + pattern: 'packer\/provision\/local-{templates}\.yaml' + - compare-type: REG_EXP + pattern: 'packer\/templates\/{templates}\.json' + - compare-type: REG_EXP + pattern: 'packer\/vars\/{platforms}\.json' + +- job-template: + name: "{project-name}-packer-verify-build-{platforms}-{templates}" + id: github-packer-verify-build + <<: *lf_packer_common + # yamllint disable-line rule:key-duplicates + <<: *lf_packer_merge + + ##################### + # Job Configuration # + ##################### + + properties: + - lf-infra-properties: + project: "{project}" + build-days-to-keep: 7 + - github: + url: "{github-url}/{github-org}/{project}" + + scm: + - lf-infra-github-scm: + url: "{git-clone-url}{github-org}/{project}" + refspec: "+refs/pull/*:refs/remotes/origin/pr/*" + branch: "$sha1" + submodule-recursive: "{submodule-recursive}" + submodule-timeout: "{submodule-timeout}" + submodule-disable: "{submodule-disable}" + choosing-strategy: default + jenkins-ssh-credential: "{jenkins-ssh-credential}" + + triggers: + - github-pull-request: + trigger-phrase: "^(verify-build|packer-build)$" + only-trigger-phrase: true + status-context: "Packer {platforms}-{templates} Verify Build" + permit-all: true + github-hooks: true + white-list-target-branches: + - "{branch}" + # included-regions MUST match Gerrit Trigger file-paths + included-regions: + - 'packer\/provision\/.*\.sh' + - 'packer\/provision\/.*\.bat' + - 'packer\/provision\/.*\.ps1' + - 'packer\/provision\/{templates}\.yaml' + - 'packer\/provision\/local-{templates}\.yaml' + - 'packer\/templates\/{templates}\.json' + - 'packer\/vars\/{platforms}\.json' + ################# # Packer Verify # ################# diff --git a/releasenotes/notes/packer-verify-build-job-2e70b4020d426621.yaml b/releasenotes/notes/packer-verify-build-job-2e70b4020d426621.yaml new file mode 100644 index 00000000..c2246cb7 --- /dev/null +++ b/releasenotes/notes/packer-verify-build-job-2e70b4020d426621.yaml @@ -0,0 +1,8 @@ +--- +features: + - | + Add a packer-verify-build job. This job is made to be manually triggered in + order to build the packer image, so that the full build process can be + validated before merging. If this is done on the final patch that is merged, + the merge job will not run another build (thereby avoiding building two + identical images). diff --git a/shell/packer-build.sh b/shell/packer-build.sh index 0d9d68d5..3349a723 100644 --- a/shell/packer-build.sh +++ b/shell/packer-build.sh @@ -36,6 +36,32 @@ packer.io validate \ -var-file="$CLOUDENV" \ -var-file="$platform_file" \ "templates/$PACKER_TEMPLATE.json" + +set -x +# If this is a Gerrit system, check patch comments for successful verify build. +if [[ -n ${GERRIT_URL:-} ]] && \ + [[ -n ${GERRIT_CHANGE_NUMBER:-} ]] && \ + [[ -n ${GERRIT_PATCHSET_NUMBER:-} ]] && \ + curl -s "${GERRIT_URL}/changes/${GERRIT_CHANGE_NUMBER}/detail" \ + | tail -n +2 | jq .messages[].message? \ + | grep "Patch Set ${GERRIT_PATCHSET_NUMBER}:.*Build Successful.*verify-build-${PACKER_PLATFORM}-${PACKER_TEMPLATE}" +then + echo "Build already successful for this patch set. Skipping merge build..." + exit +# If this is Github, check the last non-merge commit for a successful Packer +# Verify Build status. +elif [[ "${GIT_BASE:-}" =~ https://github.com ]]; then + LAST_CHANGE_SHA=$(git log --no-merges -1 --format=%H) + API_BASE=$(echo "$GIT_BASE" | sed -E 's#(www.)?github.com#api.github.com/repos#') + STATUS=$(curl "${API_BASE}/statuses/${LAST_CHANGE_SHA}" \ + | jq ".[] | select(.state == \"success\" and .context == \"Packer ${PACKER_PLATFORM}-${PACKER_TEMPLATE} Verify Build\")") + if [[ -n ${STATUS} ]]; then + echo "Build already successful for this patch set. Skipping merge build..." + exit + fi +fi +set +x + packer.io build -color=false \ -var-file="$CLOUDENV" \ -var-file="$platform_file" \