From: Anil Belur Date: Tue, 21 Jul 2020 23:22:25 +0000 (+0000) Subject: Merge "Fix port clean up race condition" X-Git-Tag: v0.55.3^0 X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=commitdiff_plain;h=6b4eba38370203ded897068371a4e50255ef588b;hp=b8cc2d59ce5bd319a4c3d15e6fdab3ef6875c8f5;p=releng%2Fglobal-jjb.git Merge "Fix port clean up race condition" --- diff --git a/docs/jjb/lf-ci-jobs.rst b/docs/jjb/lf-ci-jobs.rst index 116de87a..694bf2de 100644 --- a/docs/jjb/lf-ci-jobs.rst +++ b/docs/jjb/lf-ci-jobs.rst @@ -566,6 +566,22 @@ other file changes. Verifies INFO.yaml files follow the schema defined in :gerrit_verify_triggers: Override Gerrit Triggers. +.. _lf_pipelines_verify: + +LF Pipelines Verify +------------------- + +Verify job for the LF RelEng pipeline library. This can be implemented on any +Jenkins machine that has the appropriate Pipelines plugins installed. It will +look for a Gerrit system named "lf-releng" (which should be mapped to +https://gerrit.linuxfoundation.org/infra/), and pull in the Jenkinsfile in the +root directory of the repo. + +:Template Names: + - lf-pipelines-verify + +:Comment Trigger: recheck|reverify + .. _license-checker: License Checker diff --git a/docs/jjb/lf-rtdv3-jobs.rst b/docs/jjb/lf-rtdv3-jobs.rst index 15dc81e4..0385f806 100644 --- a/docs/jjb/lf-rtdv3-jobs.rst +++ b/docs/jjb/lf-rtdv3-jobs.rst @@ -4,9 +4,12 @@ ReadTheDocs Version:3 Jobs ########################## -ReadTheDocs V3 jobs support documentation that is structured as a -master documentation project plus a sub-project for each software -component. The master project files are usually maintained in a +ReadTheDocs Projects can be configured in a nested manner, by configuring a +project as a subproject of another project. This allows for documentation +projects to share a search index and a namespace or custom domain, but still be +maintained independently. + +The master Read The Docs project files are usually maintained in a "docs" git repository and should contain an index with links to all the sub-projects. Each sub-project must maintain its documentation files in a "docs" subdirectory within that software component's git @@ -14,11 +17,11 @@ repository. The RTDv3 Jenkins jobs publish documentation by triggering builds at ReadTheDocs.io. That build process clones the appropriate repository -and transforms Real Simple Text (RST) and other files into HTML. -Master project builds are performed separately from sub-project +and transforms reStructuredText (RST) and other files into HTML. +All project's Read the Docs builds are performed separately from sub-project builds. -The ReadTheDocs site supports multiple versions of documentation for +The Read The Docs site supports multiple versions of documentation for the master project and every sub-project. Every project should have a development branch that's published at ReadTheDocs under the title "latest"; in git this is usually the "master" branch. Most projects @@ -33,7 +36,7 @@ discussed below. User setup ---------- -To transform your rst documentation into a read the docs page, this job must be +To transform your rst documentation into a Read The Docs page, this job must be configured and created as described in Admin setup below. Once this is complete the following files must be added to your repository: @@ -69,8 +72,8 @@ of ``doc-dir: "docs/_build/html"`` to ``doc-dir: "_build/html"``, as the relativ path in the tox run has changed. -Once these files are correctly configured in your repository you can test -locally: +Once these files are correctly configured in your repository you can build +the rst files locally to test: .. code-block:: bash @@ -81,23 +84,15 @@ Stable Branch Instructions -------------------------- If your project does not create branches, you can skip this step. -Once you branch your project modify your conf.yaml and add the following line: - -.. code-block:: bash - - version: 'ReleaseBranchName' - -This will update the docs and change "master" on the top bar to your branch -name. This change should be done against your release branch, this change will -trigger a Read The Docs build which will create a new landing point for your -documentation. -This landing point is called /stable/ and is selectable as a version in the -bottom right corner of all Read The Docs pages. Once all projects have -branched and modified their conf.py they will have available their /stable/ -documentation. The process to release the documentation (that is to change the -default landing point of your docs from /latest/ to /stable/) is to change the -default-version in the jenkins job config as follows: +For Read The Docs to see your new branch, a build needs to be triggered. +A trivial change to any file in your project's /docs/ directory +on your newly minted branch is sufficient to build and activate your project's +new branch on Read The Docs. This will create a new selectable version +in the bottom right corner of your project's Read The Docs page. +Once all projects have branched the process to release the documentation +(that is to change the default landing point of your docs from /latest/ to /branchname/) +is to change the default-version in the jenkins job config as follows: From: @@ -113,10 +108,12 @@ To: Admin setup: +------------ -This is a global job that only needs to be added once to your project's ci-mangement git -repository. It leverages the read the docs v3 api to create projects on the fly, as well -as setting up subproject associations with the master doc. +This part of the documentation explains how to enable this job so that It will trigger +on docs/* changes for all projects in a Gerrit instance. It leverages the +Read The Docs v3 api to create projects on the fly, as well as setting up +sub-project associations with the master doc. Jobs will run but skip any actual verification until a .readthedocs.yaml is placed in the root of the repository @@ -212,7 +209,7 @@ a tox.ini that lived inside your docs/ dir - master: branch: '*' -Github jobs must be per project, and will be covered by a diffrent set of jobs once these are proven. +Github jobs must be per project, and will be covered by a different set of jobs once these are proven. Job requires an lftools config section, this is to provide api access to read the docs. @@ -222,7 +219,7 @@ Job requires an lftools config section, this is to provide api access to read th endpoint = https://readthedocs.org/api/v3/ token = [hidden] -Merge Job will create a project on read the docs if none exist. +Merge Job will create a project on Read The Docs if none exist. Merge Job will assign a project as a subproject of the master project. Merge job will trigger a build to update docs. Merge job will change the default version if needed. @@ -243,7 +240,7 @@ Job Templates ReadTheDocs v3 Merge -------------------- -Merge job which triggers a build of the docs to readthedocs. +Merge job which triggers a build of the docs to Read The Docs. :Template Names: - rtdv3-global-merge-{stream} @@ -273,7 +270,7 @@ Merge job which triggers a build of the docs to readthedocs. (default: 10) :submodule-disable: Disable submodule checkout operation. (default: false) - :tox-dir: Directory containing the project's read the docs tox.ini + :tox-dir: Directory containing the project's Read The Docs tox.ini :doc-dir: Relative directory project's docs generated by tox :gerrit_merge_triggers: Override Gerrit Triggers. :gerrit_trigger_file_paths: Override file paths filter which checks which @@ -318,7 +315,7 @@ Also outputs some info on the build. (default: 10) :submodule-disable: Disable submodule checkout operation. (default: false) - :tox-dir: Directory containing the project's read the docs tox.ini + :tox-dir: Directory containing the project's Read The Docs tox.ini :doc-dir: Relative directory project's docs generated by tox :gerrit_verify_triggers: Override Gerrit Triggers. :gerrit_trigger_file_paths: Override file paths filter which checks which diff --git a/jjb/lf-ci-jobs.yaml b/jjb/lf-ci-jobs.yaml index 9c62d971..a48f3251 100644 --- a/jjb/lf-ci-jobs.yaml +++ b/jjb/lf-ci-jobs.yaml @@ -623,6 +623,9 @@ project: "{project}" stream: "{stream}" branch: "{branch}" + - lf-infra-jjb-parameters: + jjb-cache: "{jjb-cache}" + jjb-version: "{jjb-version}" wrappers: - lf-infra-wrappers: @@ -639,6 +642,8 @@ - job-template: name: "{project-name}-jjb-deploy-job" id: gerrit-jjb-deploy-job + <<: *lf_jjb_common + # yamllint disable-line rule:key-duplicates <<: *lf_jjb_deploy_job scm: @@ -671,6 +676,8 @@ - job-template: name: "{project-name}-jjb-deploy-job" id: github-jjb-deploy-job + <<: *lf_jjb_common + # yamllint disable-line rule:key-duplicates <<: *lf_jjb_deploy_job properties: @@ -2338,3 +2345,65 @@ <<: *lf_sonar_builders_prescan_script # yamllint disable-line rule:key-duplicates <<: *lf_sonar_github_common + +############# +# Pipelines # +############# + +- lf_pipelines_common: &lf_pipelines_common + name: lf-pipelines-common + + ###################### + # Default parameters # + ###################### + + branch: master + build-timeout: 90 + disable-job: false + stream: master + submodule-recursive: true + submodule-timeout: 10 + submodule-disable: false + +- lf_global_pipelines_common: &lf_global_pipelines_common + name: lf-global-pipelines-common + + # All jobs are triggering from the same source + git-url: https://gerrit.linuxfoundation.org/infra/releng/pipelines + triggers: + - gerrit: + server-name: "lf-releng" + trigger-on: + - patchset-created-event: + exclude-drafts: true + exclude-trivial-rebase: false + exclude-no-code-change: false + - draft-published-event + - comment-added-contains-event: + comment-contains-value: '^Patch Set\s+\d+:\s+(recheck|reverify)\s*$' + projects: + - project-compare-type: ANT + project-pattern: "releng/pipelines" + branches: + - branch-compare-type: ANT + branch-pattern: "**/master" + +- job-template: + name: "lf-pipelines-verify" + id: lf-pipelines-verify + <<: *lf_pipelines_common + <<: *lf_global_pipelines_common + + project-type: pipeline + pipeline-scm: + scm: + - lf-infra-gerrit-scm: + git-url: "{git-url}" + refspec: "$GERRIT_REFSPEC" + branch: "$GERRIT_BRANCH" + submodule-recursive: "{submodule-recursive}" + submodule-timeout: "{submodule-timeout}" + submodule-disable: "{submodule-disable}" + choosing-strategy: gerrit + jenkins-ssh-credential: "{jenkins-ssh-credential}" + sandbox: true diff --git a/releasenotes/notes/include-jjb-version-param-deploy-jjb-b2c2f915274405f2.yaml b/releasenotes/notes/include-jjb-version-param-deploy-jjb-b2c2f915274405f2.yaml new file mode 100644 index 00000000..de143651 --- /dev/null +++ b/releasenotes/notes/include-jjb-version-param-deploy-jjb-b2c2f915274405f2.yaml @@ -0,0 +1,17 @@ +--- +fixes: + - | + Fix the jjb-deploy job to accept a JJB_VERSION parameter in the parameters + field so that projects can select which version of JJB they want for the job. + + The change that ended up breaking jjb-deploy was caused by this Gerrit + patch here: + + https://gerrit.linuxfoundation.org/infra/c/releng/global-jjb/+/62788 + + The bug was introduced in the lf-env.sh when the lf-pip-install macro was + removed. Prior to the removal the jjb-deploy job was installing the latest + and greatest JJB version. However now jjb-deploy is now installing the + fallback version of JJB_VERSION from the lf-env.sh script which is 2.8.0. + + Reference: `RELENG-3073 `_ diff --git a/releasenotes/notes/lf-pipelines-verify-86d7003f660e7974.yaml b/releasenotes/notes/lf-pipelines-verify-86d7003f660e7974.yaml new file mode 100644 index 00000000..6b2b273a --- /dev/null +++ b/releasenotes/notes/lf-pipelines-verify-86d7003f660e7974.yaml @@ -0,0 +1,4 @@ +--- +features: + - | + Introduce lf-pipelines-verify job to test the LF's global pipeline library. diff --git a/releasenotes/notes/maven-deploy-87ad380c53d8e2c2.yaml b/releasenotes/notes/maven-deploy-87ad380c53d8e2c2.yaml new file mode 100644 index 00000000..e6167196 --- /dev/null +++ b/releasenotes/notes/maven-deploy-87ad380c53d8e2c2.yaml @@ -0,0 +1,19 @@ +--- +fixes: + - | + Resolves the lftools_activate failure below. + + .. code-block:: bash + + ---> maven-deploy.sh + ImportError: cannot import name 'enquote_executable' + + Fix is to replace lftools_activate with lf-activate-venv + by sourcing lf-env.sh + lf-activate-venv is a more error resistant way to make sure a pip + package is installed + +deprecates: + - | + lftools_activate should no longer be used and will be removed + in a future release. diff --git a/shell/common-variables.sh b/shell/common-variables.sh index 346eb727..5092a8ea 100644 --- a/shell/common-variables.sh +++ b/shell/common-variables.sh @@ -23,6 +23,7 @@ echo "$MAVEN_OPTIONS" # Activates the lftools virtualenv lftools_activate() { + echo "WARNING: lftools_activate should no longer be used and will be removed in a future release." virtualenv --quiet "/tmp/v/lftools" set +u # Ignore unbound variables in activate # shellcheck source=/tmp/v/lftools/bin/activate disable=SC1091 diff --git a/shell/maven-deploy.sh b/shell/maven-deploy.sh index 38c2f7dd..f9e28893 100644 --- a/shell/maven-deploy.sh +++ b/shell/maven-deploy.sh @@ -21,8 +21,6 @@ set -eu -o pipefail m2repo_dir="$WORKSPACE/m2repo" nexus_repo_url="$NEXUS_URL/content/repositories/$NEXUS_REPO" -lftools_activate - echo "-----> Remove metadata files that were not updated" set +e # Temporarily disable to run diff command. mapfile -t metadata_files <<< "$(diff -s -r "$m2repo_dir" "$WORKSPACE/m2repo-backup" \ @@ -41,5 +39,10 @@ set -u # Re-enable. find "$m2repo_dir" -type d -empty -delete +echo "-----> Install lftools" +# shellcheck disable=SC1090 +source ~/lf-env.sh +lf-activate-venv lftools + echo "-----> Upload files to Nexus" lftools deploy nexus -s "$nexus_repo_url" "$m2repo_dir"