From: Lott, Christopher (cl778h) Date: Wed, 23 Oct 2019 01:43:27 +0000 (-0400) Subject: Revise PyPI release jobs to use a staging index X-Git-Tag: v0.47.0^0 X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=commitdiff_plain;h=refs%2Fchanges%2F67%2F62067%2F23;p=releng%2Fglobal-jjb.git Revise PyPI release jobs to use a staging index Rewrite PyPI release-verify and release-merge templates to download distribution files from a PyPI staging index and upload the files to a PyPI release index. Remove the tox etc. builders from the release jobs. Install and upgrade python packages setuptools, twine etc. Drop the branch/stream parameters for the pypi release templates; only need one release-verify and one release-merge job per project. Split the PyPI job groups because the verify & merge templates do not accept the same arguments as release-verify & release-merge templates, especially no stream/branch expected by release templates. Extend release-job.sh with PyPI release features, drop the old file pypi-tag-release.sh Move the PyPI release features to the lf-release-jobs.(rst,yaml) files from the lf-python-jobs.(rst,yaml) files. Refactor the PyPI verify and merge macros and templates due to moving out the release-verify and release-merge features. Restore lf-python-jobs.rst to Coala processing by WriteGoodLintBear, but turn off processing for sections with the word 'validate'. Change-Id: Ie27071076f9c018ccf8765c9dd948a4f5c876ec9 Issue-ID: RELENG-2455, RELENG-2497 Signed-off-by: Lott, Christopher (cl778h) --- diff --git a/.coafile b/.coafile old mode 100644 new mode 100755 index 998c7a33..f2e8f4a5 --- a/.coafile +++ b/.coafile @@ -37,7 +37,6 @@ ignore = .git/**, docs/jjb/lf-docker-jobs.rst, docs/jjb/lf-macros.rst, docs/jjb/lf-maven-jobs.rst, - docs/jjb/lf-python-jobs.rst, docs/jjb/lf-rtdv3-jobs.rst [all.ShellCheck] diff --git a/.jjb-test/lf-python-jobs.yaml b/.jjb-test/lf-python-jobs.yaml index dd94c646..bb7aadce 100644 --- a/.jjb-test/lf-python-jobs.yaml +++ b/.jjb-test/lf-python-jobs.yaml @@ -3,10 +3,11 @@ name: gerrit-python-jobs jobs: - "{project-name}-python-jobs" - - gerrit-tox-sonar - gerrit-pypi-merge - - gerrit-pypi-release-verify - gerrit-pypi-release-merge + - gerrit-pypi-release-verify + - gerrit-pypi-verify + - gerrit-tox-sonar project-name: gerrit-python @@ -14,10 +15,11 @@ name: github-python-jobs jobs: - "{project-name}-github-python-jobs" - - github-tox-sonar - github-pypi-merge - - github-pypi-release-verify - github-pypi-release-merge + - github-pypi-release-verify + - github-tox-sonar + - github-pypi-verify project-name: github-python diff --git a/docs/jjb/lf-python-jobs.rst b/docs/jjb/lf-python-jobs.rst index c87e7807..217a7b35 100644 --- a/docs/jjb/lf-python-jobs.rst +++ b/docs/jjb/lf-python-jobs.rst @@ -25,27 +25,6 @@ Runs CLM scanning against a Python project. :clm-project-name: Project name in Nexus IQ to send results to. -lf-infra-pypi-tag-release -------------------------- - -Checks the format of the release version string and checks the git -repository for that tag. In a merge job, if the tag does not exist, -adds the tag to the repository, signs the tag, and pushes the tag -to the git server. Signing requires sigul, which is only available -on a CentOS build node. - -lf-infra-pypi-upload --------------------- - -Uploads distribution files from subdirectory "dist" to a PyPI repository -using a Python virtual enviroment to install required packages. The -Jenkins server must have a configuration file ".pypirc". - -:Required Parameters: - - :pypi-repo: PyPI repository key in .pypirc configuration file; - e.g., "pypi-test" or "pypi". - lf-infra-tox-install -------------------- @@ -137,11 +116,15 @@ The coverage commands define the code that gets executed by the test suites. Checking coverage does not guarantee that the tests execute properly, but it identifies code that is not executed by any test. +.. comment Start ignoring WriteGoodLintBear + This job reuses the Sonar builders used for Java/Maven projects which run maven twice. The first invocation does nothing for Python projects, so the job uses the goal 'validate' by default. The second invocation publishes results using the goal 'sonar:sonar' by default. +.. comment Stop ignoring + For example: .. code-block:: bash @@ -173,6 +156,8 @@ https://docs.sonarqube.org/display/PLUG/Python+Coverage+Results+Import get configured in defaults.yaml) :mvn-settings: The name of the settings file with credentials for the project. +.. comment Start ignoring WriteGoodLintBear + :Optional parameters: :branch: Git branch, should be master (default: master) @@ -186,17 +171,17 @@ https://docs.sonarqube.org/display/PLUG/Python+Coverage+Results+Import :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) :github-url: URL for Github. (default: https://github.com) :java-version: Version of Java to use for the build. (default: openjdk8) - :mvn-global-settings: The name of the Maven global settings to use for + :mvn-global-settings: The name of the Maven global settings to use :mvn-goals: The Maven goal to run first. (default: validate) :mvn-version: Version of maven to use. (default: mvn35) :parallel: Boolean indicator for tox to run tests in parallel or series. - (default: false, in series) + (default: false, in series) :pre-build-script: Shell script to execute before the Sonar builder. For example, install prerequisites or move files to the repo root. (default: a string with a shell comment) :python-version: Python version to invoke pip install of tox-pyenv (default: python2) - :sonarcloud: Whether or not to use SonarCloud ``true|false``. + :sonarcloud: Boolean indicator to use SonarCloud ``true|false``. (default: false) :sonarcloud-project-key: SonarCloud project key. (default: '') :sonarcloud-project-organization: SonarCloud project organization. @@ -221,6 +206,8 @@ https://docs.sonarqube.org/display/PLUG/Python+Coverage+Results+Import modifications trigger a build. Refer to JJB documentation for "file-path" details. https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit +.. comment Stop ignoring + Tox Verify ---------- @@ -317,7 +304,7 @@ variables before running. :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) :pre-build-script: Shell script to execute before the CLM builder. For example, install prerequisites or move files to the repo root. - (default: a string with only a comment) + (default: a string with a shell comment) :python-version: Python version to invoke pip install of tox-pyenv (default: python2) :stream: Keyword representing a release code-name. @@ -369,6 +356,8 @@ pyenv variables before running. :build-node: The node to run the build on. :jenkins-ssh-credential: Credential to use for SSH. (Generally set in defaults.yaml) + :project: Git repository name + :project-name: Jenkins job name prefix :Optional Parameters: @@ -408,10 +397,11 @@ PyPI Merge Creates and uploads distribution files on merge of a patch set. Runs tox, builds a source distribution and (optionally) a binary distribution, and uploads the distribution(s) to a PyPI repository. -This job should be configured to use a test PyPI repository like -testpypi.python.org, not a public release area like the global PyPI -repository. Like the verify job, this requires a setup.py file for -packaging the component. +This job should use a staging repository like testpypi.python.org, +which sets up use of release jobs to promote the distributions later. +This job can also use a public release area like the global PyPI +repository. The project git repository must have a setup.py file +with configuration for packaging the component. The tox runner is pyenv aware so if the image contains an installation of pyenv at /opt/pyenv it will pick it up and run Python tests with @@ -425,8 +415,8 @@ pyenv variables before running. Requires a .pypirc configuration file in the Jenkins builder home directory, -an example appears next that uses API tokens. No repository is needed in the -PyPI section. +an example appears next that uses API tokens. Note that in the [pypi] entry +the repository key-value pair is optional, it defaults to pypi.org. .. code-block:: bash @@ -456,6 +446,8 @@ PyPI section. :build-node: The node to run the build on. :jenkins-ssh-credential: Credential to use for SSH. (Generally set in defaults.yaml) + :project: Git repository name + :project-name: Jenkins job name prefix :Optional Parameters: @@ -472,8 +464,8 @@ PyPI section. (default: false, in series) :pre-build-script: Shell script to execute before the tox builder. For example, install system prerequisites. (default: a shell comment) - :pypi-repo: Key for PyPI repository parameters in the .pypirc file. - Merge jobs should use a server like testpypi.python.org. (default: pypi-test) + :pypi-repo: Key for the PyPI target repository in the .pypirc file, + ideally a server like test.pypy.org. (default: pypi-test) :python-version: Python version to invoke pip install of tox-pyenv (default: python3) :stream: Keyword representing a release code-name. @@ -492,196 +484,3 @@ PyPI section. :gerrit_trigger_file_paths: Override file paths used to filter which file modifications trigger a build. Refer to JJB documentation for "file-path" details. https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit - - -PyPI Release Verify -------------------- - -Verifies a Python library project on creation of a patch set with a -release yaml file. Runs tox, builds source and (optionally) binary -distributions, checks the format of the version string, checks that -the distribution file names contain the release version string, and -checks if the tag exists in the code repository for the release -version. - -To initiate the release process, create a releases/ or .releases/ -directory at the root of the project repository, add one release yaml -file to it, and submit a change set with that release yaml file. A -schema and and an example for the release yaml file appear below. The -version in the release yaml file must be a valid Semantic Versioning -(SemVer) string, matching either the pattern "v#.#.#" or "#.#.#" where -"#" is one or more digits. - -This job is similar to the PyPI verify job, but is only triggered by a -patch set with a release yaml file. - -The build node for PyPI release verify jobs must be CentOS, which -supports the sigul client for accessing a signing server. - -.. note:: - - The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml). - In words, the directory name can be ".releases" or "releases"; the file - name can be anything with suffix ".yaml". - -The JSON schema for a pypi release file appears below. - -.. code-block:: none - - --- - $schema: "http://json-schema.org/schema#" - $id: "https://github.com/lfit/releng-global-jjb/blob/master/release-pypi-schema.yaml" - - required: - - "distribution_type" - - "project" - - "version" - - properties: - distribution_type: - type: "string" - project: - type: "string" - version: - type: "string" - - -An example of a pypi release file appears below. - -.. code-block:: none - - $ cat releases/1.0.0-pypi.yaml - --- - distribution_type: pypi - version: 1.0.0 - project: 'example-project' - - -:Template Names: - - - {project-name}-pypi-release-verify-{stream} - - gerrit-pypi-release-verify - - github-pypi-release-verify - -:Comment Trigger: recheck - -:Required Parameters: - - :build-node: The node to run build on, which must be Centos. - :jenkins-ssh-credential: Credential to use for SSH. (Generally set - in defaults.yaml) - -:Optional Parameters: - - :branch: The branch to build against. (default: master) - :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) - :build-timeout: Timeout in minutes before aborting build. (default: 15) - :disable-job: Whether to disable the job (default: false) - :dist-binary: Whether to build a binary wheel distribution. (default: true) - :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) - :parallel: Boolean indicator for tox to run tests in parallel or series. - (default: false, in series) - :pre-build-script: Shell script to execute before the tox builder. - For example, install prerequisites or move files to the repo root. - (default: a string with a shell comment) - :pypi-repo: Key for PyPI repository parameters in the .pypirc file. - Release jobs should use a server like pypy.org. (default: pypi) - :python-version: Python version to invoke pip install of tox-pyenv - (default: python3) - :stream: Keyword representing a release code-name. - Often the same as the branch. (default: master) - :submodule-recursive: Whether to checkout submodules recursively. - (default: true) - :submodule-timeout: Timeout (in minutes) for checkout operation. - (default: 10) - :submodule-disable: Disable submodule checkout operation. - (default: false) - :tox-dir: Directory containing the project's tox.ini relative to - the workspace. The default uses tox.ini at the project root. - (default: '.') - :tox-envs: Tox environments to run. If blank run everything described - in tox.ini. (default: '') - :use-release-file: Whether to use the release file. (default: true) - -PyPI Release Merge ------------------- - -Publishes a Python library on merge of a patch set with a release yaml -file. Runs tox, builds source and (optionally) binary distributions, -checks the format of the version string, checks that the distribution -file names contain the release version string, checks if the tag -exists in the code repository for the release version, then if the tag -does not exist, tags the code repository with the release version, -signs the tag and pushes the tag to the git server. Finally this -uploads the distributions to a PyPI repository. - -This job is similar to the PyPI merge job, but is only triggered by -merge of a release yaml file, also this checks the version and tag -before uploading to a public repository such as PyPI. - -See the PyPI Release Verify job above for documentation of the release -yaml file format. - -The build node for PyPI release merge jobs must be CentOS, which -supports the sigul client for accessing a signing server. - -A Jenkins user can also trigger this release job via the "Build with -parameters" action, removing the need to merge a release yaml file. -The user must enter parameters in the same way as a release yaml file, -except for the special USE_RELEASE_FILE and DRY_RUN check boxes. The -user must uncheck the USE_RELEASE_FILE check box if the job should run -with a release file, while passing the required information as build -parameters. Similarly, the user must uncheck the DRY_RUN check box to -test the job while skipping upload of files to a repository. - -The special parameters are as follows:: - - VERSION = 1.0.0 - USE_RELEASE_FILE = false - DRY_RUN = false - -:Template Names: - - - {project-name}-pypi-release-merge-{stream} - - gerrit-pypi-release-merge - - github-pypi-release-merge - -:Comment Trigger: remerge - -:Required Parameters: - - :build-node: The node to run build on, which must be Centos. - :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set - in defaults.yaml) - -:Optional Parameters: - - :branch: The branch to build against. (default: master) - :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) - :build-timeout: Timeout in minutes before aborting build. (default: 15) - :disable-job: Whether to disable the job (default: false) - :dist-binary: Whether to build a binary wheel distribution. (default: true) - :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) - :parallel: Boolean indicator for tox to run tests in parallel or series. - (default: false, in series) - :pre-build-script: Shell script to execute before the tox builder. - For example, install prerequisites or move files to the repo root. - (default: a string with a shell comment) - :pypi-repo: Key for PyPI repository parameters in the .pypirc file. - Release jobs should use a server like pypy.org. (default: pypi) - :python-version: Python version to invoke pip install of tox-pyenv - (default: python3) - :stream: Keyword representing a release code-name. - Often the same as the branch. (default: master) - :submodule-recursive: Whether to checkout submodules recursively. - (default: true) - :submodule-timeout: Timeout (in minutes) for checkout operation. - (default: 10) - :submodule-disable: Disable submodule checkout operation. - (default: false) - :tox-dir: Directory containing the project's tox.ini relative to - the workspace. The default uses tox.ini at the project root. - (default: '.') - :tox-envs: Tox environments to run. If blank run everything described - in tox.ini. (default: '') - :use-release-file: Whether to use the release file. (default: true) diff --git a/docs/jjb/lf-release-jobs.rst b/docs/jjb/lf-release-jobs.rst index d0f5340f..60ec72c1 100644 --- a/docs/jjb/lf-release-jobs.rst +++ b/docs/jjb/lf-release-jobs.rst @@ -1,32 +1,43 @@ .. _lf-global-jjb-release: -####################### Self-Serve Release Jobs -####################### - -Self-serve release jobs allow a project team to direct Jenkins to -promote a jar file or container image from a staging area to a release -area. To trigger the action, create a releases/ or .releases/ -directory, add a release yaml file to it, and submit a change set with -one release yaml file to Gerrit. Upon merge of the change, Jenkins will -sign the reference extrapolated by log_dir and promote the artifact. The -expected format of the release yaml file appears in schemas and examples -below. - -The build node for maven and container release jobs must be CentOS, -which supports the sigul client for accessing a signing server. The -build node for container release jobs must have Docker installed. - -A Jenkins user can also trigger a release job via the "Build with -parameters" action, removing the need for a release yaml file. The -user must enter parameters in the same way as a release yaml file, -except for the special USE_RELEASE_FILE and DRY_RUN check boxes. The -user must uncheck the USE_RELEASE_FILE check box if the job should -run with a release file, while passing the required information as -build parameters. Similarly, the user must uncheck the DRY_RUN check -box to test the job while skipping repository promotion to Nexus. - -The special parameters are as follows:: +======================= + +Self-serve release jobs allow project committers to direct Jenkins to +promote a jar file, container image or Python package from a staging +area to a release area. + +To use the self-release process, create a releases/ or .releases/ +directory at the root of the project repository, add one release yaml +file to it, and submit a change set with that release yaml file. The +required contents of the release yaml file are different for each type +of release, see the schemas and examples shown below. The version in +the release yaml file must be a valid Semantic Versioning (SemVer) +string, matching either the pattern "v#.#.#" or "#.#.#" where "#" is +one or more digits. Upon merge of the change, Jenkins will sign the +reference extrapolated by log_dir and promote the artifact. + +.. note:: + + The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml). + In words, the directory name can be ".releases" or "releases"; the file + name can be anything with suffix ".yaml". + +The build node for all release jobs must be CentOS, which supports the +sigul client for accessing a signing server. The build node for +container release jobs must have Docker installed. + +A Jenkins admin user can also trigger a release job via the "Build +with parameters" action, removing the need to create and merge a +release yaml file. The user must enter parameters in the same way as +a release yaml file, except for the special USE_RELEASE_FILE and +DRY_RUN check boxes. The user must uncheck the USE_RELEASE_FILE check +box if the job should run without a release file, instead passing the +required information as build parameters. The user can check the +DRY_RUN check box to test the job while skipping upload of files to +the release repository. + +For example, the parameters for a Maven release are as follows:: GERRIT_BRANCH = master VERSION = 1.0.0 @@ -35,13 +46,35 @@ The special parameters are as follows:: USE_RELEASE_FILE = false DRY_RUN = false -.. note:: +Maven Release Files +------------------- - The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml). - In words, the directory name can be ".releases" or "releases"; the file - name can be anything with suffix ".yaml". +An example of a maven release file appears below. + +.. code-block:: none -The JSON schema for a maven release job appears below. + $ cat releases/maven-release.yaml + --- + distribution_type: maven + log_dir: example-project-maven-stage-master/17/ + project: example-project + version: 1.0.0 + + +The following parameters must appear in a maven release yaml file. + +:Required Parameters: + + :distribution_type: Must be "maven". + :log_dir: The suffix of the logs URL reported on completion by the + Jenkins stage job that created and pushed the artifact + to the staging repository. For example, use value + "example-project-maven-stage-master/17" for the logs URL + https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-maven-stage-master/17 + :project: The name of the project. + :version: The semantic version string used for the artifact. + +The JSON schema for a maven release file appears below. .. code-block:: none @@ -66,19 +99,46 @@ The JSON schema for a maven release job appears below. type: "string" -Example of a maven release file: +Container Release Files +----------------------- -.. code-block:: bash +An example of a container release file appears below. - $ cat releases/1.0.0-maven.yaml +.. code-block:: none + + $ cat releases/container-release.yaml --- - distribution_type: 'maven' - version: '1.0.0' - project: 'example-project' - log_dir: 'example-project-maven-stage-master/17/' + distribution_type: container + container_release_tag: 1.0.0 + container_pull_registry: nexus.onap.org:10003 + container_push_registry: nexus.onap.org:10002 + project: test + ref: d1b9cd2dd345fbeec0d3e2162e008358b8b663b2 + containers: + - name: test-backend + version: 1.0.0-20190806T184921Z + - name: test-frontend + version: 1.0.0-20190806T184921Z + +The following parameters must appear in a container release yaml file. + +:Required Parameters: -The JSON schema for a container release job appears below. + :distribution_type: Must be "container". + :container_release_tag: The string to use as a Docker tag on all + released containers. + :container_pull_registry: The Nexus registry that supplies the staged + image(s). + :container_push_registry: The Nexus registry that receives the released + image(s). + :project: The name of the project. + :ref: The git commit reference (SHA-1 code) to tag with the version string. + :containers: A list of name and version (tag) pairs that specify the + Docker images in the container-pull registry to promote to the + container-push registry. + +The JSON schema for a container release file appears below. .. code-block:: none @@ -116,30 +176,74 @@ The JSON schema for a container release job appears below. type: "string" -An example of a container release file appears below. The job tags the -git repository at the specified commit reference. The job applies the -container_release_tag string to all released containers. The job uses the -per-container version strings to pull images from the container registry. +PyPI Release Files +------------------ -.. code-block:: bash +An example of a PyPI release file appears below. + +.. code-block:: none - $ cat releases/1.0.0-container.yaml + $ cat releases/pypi-release.yaml --- - distribution_type: 'container' - container_release_tag: '1.0.0' - container_pull_registry: 'nexus.onap.org:10003" - container_push_registry: 'nexus.onap.org:10002" - project: 'test' - ref: d1b9cd2dd345fbeec0d3e2162e008358b8b663b2 - containers: - - name: test-backend - version: 1.0.0-20190806T184921Z - - name: test-frontend - version: 1.0.0-20190806T184921Z + distribution_type: pypi + pypi_project: mypackage + python_version: 3.4 + version: 1.0.0 + +The following parameters must appear in the PyPI release yaml file. +These are not part of the Jenkins job definition to allow independent +self-release of a package maintained in a git repository with other +packages. -Example of a Jenkins job configuration that uses the global-jjb -templates for Gerrit: +:Required Parameters: + + :distribution_type: Must be "pypi". + :log_dir: The suffix of the logs URL reported on completion by the + Jenkins merge job that created and pushed the distribution files + to the staging repository. For example, use value + "example-project-pypi-merge-master/17" for the logs URL + https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-pypi-merge-master/17 + :pypi_project: The PyPI project name at the staging and + release repositories, for example "mypackage". + :python_version: The Python interpreter version to use for pip + "Requires-Python" compatibility checks, for example "3" or "3.7.0". + :version: The semantic version string used for the package in the + setup.py file. + +The JSON schema for a PyPI release file appears below. + +.. code-block:: none + + --- + $schema: "http://json-schema.org/schema#" + $id: "https://github.com/lfit/releng-global-jjb/blob/master/release-pypi-schema.yaml" + + required: + - "distribution_type" + - "log_dir" + - "pypi_project" + - "python_version" + - "version" + + properties: + distribution_type: + type: "string" + log_dir: + type: "string" + pypi_project: + type: "string" + python_version: + type: "string" + version: + type: "string" + + +Jenkins Jobs +------------ + +An example of a Jenkins job configuration that uses the global-jjb +templates for maven and container release jobs appears next. .. code-block:: none @@ -158,11 +262,167 @@ templates for Gerrit: Release Engineers: please follow the setup guide below before adding the job definition. +JJB Macros +---------- + +lf-release +~~~~~~~~~~ + +Release verify and merge jobs are the same except for their scm, +trigger, and builders definition. This anchor is the common template. + +JJB Templates +------------- + +Release Merge +~~~~~~~~~~~~~ + +This template supports Maven and Container release jobs. + +:Template Name: {project-name}-release-merge + +:Comment Trigger: remerge + +:Required parameters: + + :build-node: The node to run build on. + :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + :project: Git repository name + :project-name: Jenkins job name prefix + +:Optional parameters: + + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 15) + + :gerrit_merge_triggers: Override Gerrit Triggers. + :gerrit_trigger_file_paths: Override file paths filter which checks which + file modifications will trigger a build. + **default**:: + + - compare-type: REG_EXP + pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' + + +Release Verify +~~~~~~~~~~~~~~ + +This template supports Maven and Container release jobs. + +:Template Name: {project-name}-release-verify + +:Comment Trigger: recheck|reverify + +:Required Parameters: + + :build-node: The node to run build on. + :jenkins-ssh-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + :project: Git repository name + :project-name: Jenkins job name prefix + +:Optional Parameters: + + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-node: The node to run build on. + :build-timeout: Timeout in minutes before aborting build. (default: 15) + :gerrit-skip-vote: Skip voting for this job. (default: false) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + + :gerrit_verify_triggers: Override Gerrit Triggers. + :gerrit_trigger_file_paths: Override file paths filter which checks which + file modifications will trigger a build. + **default**:: + + - compare-type: REG_EXP + pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' + + +PyPI Release Merge +~~~~~~~~~~~~~~~~~~ + +Publishes a Python package on merge of a patch set with a release yaml +file. Checks the format of the version string, downloads the package +artifacts from the PyPI staging repository, uploads the package +artifacts to the PyPI release repository, tags the git repository, +signs the tag and pushes the tag to the git server. The release verify +template accepts neither a branch nor a stream parameter. + +:Template Names: + + - {project-name}-pypi-release-merge + - gerrit-pypi-release-merge + - github-pypi-release-merge + +:Comment Trigger: remerge + +:Required Parameters: + + :build-node: The node to run build on, which must be Centos. + :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + :project: Git repository name + :project-name: Jenkins job name prefix + +:Optional Parameters: + + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 15) + :disable-job: Whether to disable the job (default: false) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :pypi-stage-index: Base URL of the PyPI staging repository. + (default https://test.pypi.org/simple) + :pypi-repo: Key for the PyPI release repository in the .pypirc file, + should be the repository pypy.org. (default: pypi) + :use-release-file: Whether to use the release file. (default: true) + + +PyPI Release Verify +~~~~~~~~~~~~~~~~~~~ + +Verifies a Python package project on creation of a patch set with a +release yaml file. Checks the contents of the release yaml file, +checks the format of the version string, and downloads the release +artifacts from the specified PyPI staging repository. The release +verify template accepts neither a branch nor a stream parameter. + +:Template Names: + + - {project-name}-pypi-release-verify + - gerrit-pypi-release-verify + - github-pypi-release-verify + +:Comment Trigger: recheck + +:Required Parameters: + + :build-node: The node to run build on, which must be Centos. + :jenkins-ssh-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + :project: Git repository name + :project-name: Jenkins job name prefix + +:Optional Parameters: + + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 15) + :disable-job: Whether to disable the job (default: false) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :pypi-stage-index: Base URL of the PyPI staging repository. + (default https://test.pypi.org/simple) + :pypi-repo: Key for the PyPI release repository in the .pypirc file, + should be the repository pypy.org (default: pypi) + :use-release-file: Whether to use the release file. (default: true) + + Setup for LFID, Nexus, Jenkins and Gerrit -========================================= +----------------------------------------- + +This section is for the Linux Foundation release engineering team. LFID -==== +~~~~ Create an ``lfid`` and an ``ssh-key`` @@ -181,14 +441,14 @@ ssh-key example: Nexus -===== +~~~~~ Create a Nexus account called ``'jenkins-release'`` with promote privileges. .. image:: ../_static/nexus-promote-privs.png Gerrit -====== +~~~~~~ Log into your Gerrit with ``YOUR_RELEASE_USERNAME``, upload the public part of the ``ssh-key`` you created earlier. Log out of Gerrit and log @@ -214,17 +474,17 @@ In All project, grant group self-serve-release the following: Jenkins -======= +~~~~~~~ Add a global credential to Jenkins called ``jenkins-release`` and set the ID: ``'jenkins-release'`` as its value insert the private half of the ``ssh-key`` that you created for your Gerrit user. -Add Global vars in Jenkins: -Jenkins configure -> Global properties -> Environment variables +Add Global variables in Jenkins: +Jenkins configure -> Global properties -> Environment variables:: -``RELEASE_USERNAME = YOUR_RELEASE_USERNAME`` -``RELEASE_EMAIL = YOUR_RELEASE_EMAIL`` + RELEASE_USERNAME = YOUR_RELEASE_USERNAME + RELEASE_EMAIL = YOUR_RELEASE_EMAIL .. note:: @@ -234,12 +494,14 @@ Jenkins configure -> Global properties -> Environment variables Jenkins configure -> Managed Files -> Add a New Config -> Custom File -id: signing-pubkey -Name: SIGNING_PUBKEY (optional) -Comment: SIGNING_PUBKEY (optional) +.. code-block:: none + + id: signing-pubkey + Name: SIGNING_PUBKEY (optional) + Comment: SIGNING_PUBKEY (optional) -Content: (Ask Andy for the public signing key) ------BEGIN PGP PUBLIC KEY BLOCK----- + Content: (Ask Andy for the public signing key) + -----BEGIN PGP PUBLIC KEY BLOCK----- Add or edit the managed file in Jenkins called ``lftoolsini``, @@ -253,87 +515,11 @@ appending a nexus section: Jenkins Settings -> Managed files -> Add password= Ci-management -============= +~~~~~~~~~~~~~ Upgrade your project's global-jjb if needed, then add the following to your global defaults file (e.g., jjb/defaults.yaml). .. code-block:: none - jenkins-ssh-release-credential: 'jenkins-release' - -Macros -====== - -lf-release ----------- - -Release verify and merge jobs are the same except for their scm, -trigger, and builders definition. This anchor is the common template. - -Job Templates -============= - -Release Merge -------------- - -:Template Name: {project-name}-release-merge - -:Comment Trigger: remerge - -:Required parameters: - - :build-node: The node to run build on. - :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set - in defaults.yaml) - :stream: run this job against: ** - -:Optional parameters: - - :branch: Git branch to fetch for the build. (default: all) - :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) - :build-timeout: Timeout in minutes before aborting build. (default: 15) - :project-pattern: Project to trigger build against. (default: \*\*) - - :gerrit_merge_triggers: Override Gerrit Triggers. - :gerrit_trigger_file_paths: Override file paths filter which checks which - file modifications will trigger a build. - **default**:: - - - compare-type: REG_EXP - pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' - - -Release Verify ------------------- - -:Template Name: {project-name}-release-verify - -:Comment Trigger: recheck|reverify - -:Required Parameters: - - :build-node: The node to run build on. - :jenkins-ssh-credential: Credential to use for SSH. (Generally set - in defaults.yaml) - :stream: run this job against: ** - -:Optional Parameters: - - :branch: Git branch to fetch for the build. (default: all) - :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) - :build-node: The node to run build on. - :build-timeout: Timeout in minutes before aborting build. (default: 15) - :doc-dir: Directory where tox will place built docs. - as defined in the tox.ini (default: docs/_build/html) - :gerrit-skip-vote: Skip voting for this job. (default: false) - :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) - :project-pattern: Project to trigger build against. (default: \*\*) - - :gerrit_verify_triggers: Override Gerrit Triggers. - :gerrit_trigger_file_paths: Override file paths filter which checks which - file modifications will trigger a build. - **default**:: - - - compare-type: REG_EXP - pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' + jenkins-ssh-release-credential: jenkins-release diff --git a/jjb/lf-python-job-groups.yaml b/jjb/lf-python-job-groups.yaml index 60ecc660..ad022355 100644 --- a/jjb/lf-python-job-groups.yaml +++ b/jjb/lf-python-job-groups.yaml @@ -24,23 +24,39 @@ - job-group: name: "{project-name}-gerrit-pypi-jobs" - # This job group contains all the recommended jobs that should be deployed for - # a Gerrit-based Python project to test, build and deploy a library to PyPI. + # This job group contains the recommended jobs that should be deployed for + # a Gerrit-based Python project to test, build and deploy a package. jobs: - - gerrit-pypi-verify - gerrit-pypi-merge - - gerrit-pypi-release-verify - - gerrit-pypi-release-merge + - gerrit-pypi-verify - job-group: name: "{project-name}-github-pypi-jobs" - # This job group contains all the recommended jobs that should be deployed for - # a Github-based Python project to test, build and deploy a library to PyPI. + # This job group contains the recommended jobs that should be deployed for + # a Github-based Python project to test, build and deploy a package. jobs: - - github-pypi-verify - github-pypi-merge - - github-pypi-release-verify + - github-pypi-verify + +- job-group: + name: "{project-name}-gerrit-pypi-release-jobs" + + # This job group contains the recommended jobs that should be deployed for + # a Gerrit-based Python project to promote a package from staging to pypi. + + jobs: + - gerrit-pypi-release-merge + - gerrit-pypi-release-verify + +- job-group: + name: "{project-name}-github-pypi-release-jobs" + + # This job group contains the recommended jobs that should be deployed for + # a Github-based Python project to promote a package from staging to pypi. + + jobs: - github-pypi-release-merge + - github-pypi-release-verify diff --git a/jjb/lf-python-jobs.yaml b/jjb/lf-python-jobs.yaml index a0f818e9..606d5ec1 100644 --- a/jjb/lf-python-jobs.yaml +++ b/jjb/lf-python-jobs.yaml @@ -10,34 +10,6 @@ properties-content: "CLM_PROJECT_NAME={clm-project-name}" - shell: !include-raw-escape: ../shell/nexus-iq-cli.sh -- builder: - name: lf-infra-pypi-tag-release - builders: - - config-file-provider: - files: - - file-id: sigul-config - variable: SIGUL_CONFIG - - file-id: sigul-password - variable: SIGUL_PASSWORD - - file-id: sigul-pki - variable: SIGUL_PKI - - file-id: signing-pubkey - variable: SIGNING_PUBKEY - - shell: !include-raw: ../shell/sigul-configuration.sh - - shell: !include-raw: ../shell/sigul-install.sh - - shell: !include-raw: ../shell/pypi-tag-release.sh - -- builder: - name: lf-infra-pypi-upload - builders: - - config-file-provider: - files: - - file-id: pypirc - target: "$HOME/.pypirc" - - inject: - properties-content: "REPOSITORY={pypi-repo}" - - shell: !include-raw-escape: ../shell/pypi-upload.sh - - builder: name: lf-infra-tox-install builders: @@ -740,31 +712,19 @@ - bool: name: BUILD_BDIST_WHEEL default: "{dist-binary}" - description: "Set to True to build a wheel" + description: "Set to True (checked) to build a binary distribution" - bool: name: DRY_RUN default: false - description: | - If DRY_RUN is enabled artifacts are not published. - - publishers: - - lf-infra-publish - -- lf_pypi_common_wrappers: &lf_pypi_common_wrappers - name: lf-pypi-common-wrappers + description: "Set to True (checked) to skip uploading artifacts" wrappers: - lf-infra-wrappers: build-timeout: "{build-timeout}" jenkins-ssh-credential: "{jenkins-ssh-credential}" -- lf_pypi_release_wrappers: &lf_pypi_release_wrappers - name: lf-pypi-release-wrappers - - wrappers: - - lf-infra-wrappers: - build-timeout: "{build-timeout}" - jenkins-ssh-credential: "{jenkins-ssh-release-credential}" + publishers: + - lf-infra-publish - lf_pypi_verify_builders: &lf_pypi_verify_builders name: lf-pypi-verify-builders @@ -789,42 +749,18 @@ - lf-infra-tox-run: parallel: "{parallel}" - shell: !include-raw-escape: ../shell/pypi-dist-build.sh - - lf-infra-pypi-upload: - pypi-repo: "{pypi-repo}" - -- lf_pypi_release_verify_builders: &lf_pypi_release_verify_builders - name: lf-pypi-release-verify-builders - - builders: - - lf-infra-pre-build - - lf-infra-tox-install: - python-version: "{python-version}" - - shell: "{pre-build-script}" - - lf-infra-tox-run: - parallel: "{parallel}" - - shell: !include-raw-escape: ../shell/pypi-dist-build.sh - - lf-infra-pypi-tag-release - -- lf_pypi_release_merge_builders: &lf_pypi_release_merge_builders - name: lf-pypi-release-merge-builders - - builders: - - lf-infra-pre-build - - lf-infra-tox-install: - python-version: "{python-version}" - - shell: "{pre-build-script}" - - lf-infra-tox-run: - parallel: "{parallel}" - - shell: !include-raw-escape: ../shell/pypi-dist-build.sh - - lf-infra-pypi-tag-release - - lf-infra-pypi-upload: - pypi-repo: "{pypi-repo}" + - config-file-provider: + files: + - file-id: pypirc + target: "$HOME/.pypirc" + - inject: + properties-content: "REPOSITORY={pypi-repo}" + - shell: !include-raw-escape: ../shell/pypi-upload.sh - job-template: name: "{project-name}-pypi-verify-{stream}" id: gerrit-pypi-verify <<: *lf_pypi_common - <<: *lf_pypi_common_wrappers <<: *lf_pypi_verify_builders scm: @@ -862,7 +798,6 @@ name: "{project-name}-pypi-verify-{stream}" id: github-pypi-verify <<: *lf_pypi_common - <<: *lf_pypi_common_wrappers <<: *lf_pypi_verify_builders properties: @@ -895,7 +830,6 @@ name: "{project-name}-pypi-merge-{stream}" id: gerrit-pypi-merge <<: *lf_pypi_common - <<: *lf_pypi_common_wrappers <<: *lf_pypi_merge_builders cron: "" @@ -932,7 +866,6 @@ name: "{project-name}-pypi-merge-{stream}" id: github-pypi-merge <<: *lf_pypi_common - <<: *lf_pypi_common_wrappers <<: *lf_pypi_merge_builders cron: "" @@ -968,188 +901,3 @@ white-list-target-branches: - "{branch}" included-regions: "{obj:github_included_regions}" - -- lf_pypi_release_common: &lf_pypi_release_common - name: lf-pypi-release-common - - dist-binary: true - pypi-repo: pypi - use-release-file: true - - # define once and use twice; jobs MUST NOT override - gerrit_release_trigger_file_paths: - - compare-type: REG_EXP - pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' - - # yamllint disable-line rule:line-length - # github_release_included_regions MUST match gerrit_release_trigger_file_paths - github_release_included_regions: - - 'releases\/.*\.yaml' - - '.releases\/.*\.yaml' - - parameters: - - lf-infra-parameters: - project: "{project}" - branch: "{branch}" - stream: "{stream}" - - lf-infra-tox-parameters: - tox-dir: "{tox-dir}" - tox-envs: "{tox-envs}" - - bool: - name: BUILD_BDIST_WHEEL - default: "{dist-binary}" - description: "Set to True to build a wheel" - - string: - name: VERSION - default: "" - description: "This is the version, example: 1.0.0" - - bool: - name: USE_RELEASE_FILE - default: "{use-release-file}" - description: "Set to False for job built with parameters" - - bool: - name: DRY_RUN - default: false - description: | - If DRY_RUN is enabled artifacts are not published. - -- job-template: - name: "{project-name}-pypi-release-verify-{stream}" - id: gerrit-pypi-release-verify - <<: *lf_pypi_common - <<: *lf_pypi_common_wrappers - <<: *lf_pypi_release_common - <<: *lf_pypi_release_verify_builders - - scm: - - lf-infra-gerrit-scm: - jenkins-ssh-credential: "{jenkins-ssh-credential}" - git-url: "{git-url}" - refspec: "$GERRIT_REFSPEC" - branch: "$GERRIT_BRANCH" - submodule-recursive: "{submodule-recursive}" - submodule-timeout: "{submodule-timeout}" - submodule-disable: "{submodule-disable}" - choosing-strategy: gerrit - - triggers: - - gerrit: - server-name: "{gerrit-server-name}" - trigger-on: - - patchset-created-event: - exclude-drafts: true - exclude-trivial-rebase: false - exclude-no-code-change: false - - draft-published-event - - comment-added-contains-event: - # yamllint disable-line rule:line-length - comment-contains-value: '^Patch Set\s+\d+:\s+(recheck|reverify)\s*$' - projects: - - project-compare-type: "ANT" - project-pattern: "{project}" - branches: - - branch-compare-type: ANT - branch-pattern: "**/{branch}" - file-paths: "{obj:gerrit_release_trigger_file_paths}" - -- job-template: - name: "{project-name}-pypi-release-verify-{stream}" - id: github-pypi-release-verify - <<: *lf_pypi_common - <<: *lf_pypi_common_wrappers - <<: *lf_pypi_release_common - <<: *lf_pypi_release_verify_builders - - properties: - - github: - url: "{github-url}/{github-org}/{project}" - - scm: - - lf-infra-github-scm: - url: "{git-clone-url}{github-org}/{project}" - refspec: "" - branch: "refs/heads/{branch}" - submodule-recursive: "{submodule-recursive}" - submodule-timeout: "{submodule-timeout}" - submodule-disable: "{submodule-disable}" - choosing-strategy: default - jenkins-ssh-credential: "{jenkins-ssh-credential}" - - triggers: - - github-pull-request: - trigger-phrase: "^(recheck|reverify)$" - only-trigger-phrase: false - status-context: "PyPI Release Verify" - permit-all: true - github-hooks: true - white-list-target-branches: - - "{branch}" - included-regions: "{obj:github_release_included_regions}" - -- job-template: - name: "{project-name}-pypi-release-merge-{stream}" - id: gerrit-pypi-release-merge - <<: *lf_pypi_common - <<: *lf_pypi_release_wrappers - <<: *lf_pypi_release_common - <<: *lf_pypi_release_merge_builders - - scm: - - lf-infra-gerrit-scm: - jenkins-ssh-credential: "{jenkins-ssh-credential}" - git-url: "{git-url}" - refspec: "$GERRIT_REFSPEC" - branch: "$GERRIT_BRANCH" - submodule-recursive: "{submodule-recursive}" - submodule-timeout: "{submodule-timeout}" - submodule-disable: "{submodule-disable}" - choosing-strategy: gerrit - - triggers: - - gerrit: - server-name: "{gerrit-server-name}" - trigger-on: - - change-merged-event - - comment-added-contains-event: - comment-contains-value: '^Patch Set\s+\d+:\s+remerge\s*$' - projects: - - project-compare-type: "ANT" - project-pattern: "{project}" - branches: - - branch-compare-type: ANT - branch-pattern: "**/{branch}" - file-paths: "{obj:gerrit_release_trigger_file_paths}" - -- job-template: - name: "{project-name}-pypi-release-merge-{stream}" - id: github-pypi-release-merge - <<: *lf_pypi_common - <<: *lf_pypi_release_wrappers - <<: *lf_pypi_release_common - <<: *lf_pypi_release_merge_builders - - properties: - - github: - url: "{github-url}/{github-org}/{project}" - - scm: - - lf-infra-github-scm: - url: "{git-clone-url}{github-org}/{project}" - refspec: "" - branch: "refs/heads/{branch}" - submodule-recursive: "{submodule-recursive}" - submodule-timeout: "{submodule-timeout}" - submodule-disable: "{submodule-disable}" - choosing-strategy: default - jenkins-ssh-credential: "{jenkins-ssh-credential}" - - triggers: - - github-pull-request: - trigger-phrase: "^(remerge)$" - only-trigger-phrase: false - status-context: "PyPI Release Merge" - permit-all: true - github-hooks: true - white-list-target-branches: - - "{branch}" - included-regions: "{obj:github_release_included_regions}" diff --git a/jjb/lf-release-jobs.yaml b/jjb/lf-release-jobs.yaml index fbef3fae..c2bb7492 100644 --- a/jjb/lf-release-jobs.yaml +++ b/jjb/lf-release-jobs.yaml @@ -53,7 +53,8 @@ exclude-no-code-change: false - draft-published-event - comment-added-contains-event: - comment-contains-value: "^Patch Set[ ]+[0-9]+:([ ]+|[\n]+)(recheck|reverify)$" + comment-contains-value: | + ^Patch Set[ ]+[0-9]+:([ ]+|[\n]+)(recheck|reverify)$ ##################### # Job Configuration # @@ -240,3 +241,248 @@ file-paths: - compare-type: REG_EXP pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' + +################ +# PyPI RELEASE # +################ + +- lf_pypi_verify_wrappers: &lf_pypi_verify_wrappers + name: lf-pypi-verify-wrappers + + wrappers: + - lf-infra-wrappers: + build-timeout: "{build-timeout}" + jenkins-ssh-credential: "{jenkins-ssh-credential}" + +- lf_pypi_release_wrappers: &lf_pypi_release_wrappers + name: lf-pypi-release-wrappers + + wrappers: + - lf-infra-wrappers: + build-timeout: "{build-timeout}" + jenkins-ssh-credential: "{jenkins-ssh-release-credential}" + +- lf_pypi_release: &lf_pypi_release + name: lf-pypi-release + + ###################### + # Default parameters # + ###################### + + branch: master # for github + build-days-to-keep: 7 + build-timeout: 15 + disable-job: false + gerrit-skip-vote: false + git-url: "$GIT_URL/$PROJECT" + github-url: "https://github.com" + pypi-repo: pypi + pypi-stage-index: https://test.pypi.org/simple + submodule-disable: true + submodule-recursive: false + submodule-timeout: 10 + use-release-file: true + + # define once and use twice; jobs MUST NOT override + gerrit_release_trigger_file_paths: + - compare-type: REG_EXP + pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' + + # yamllint disable-line rule:line-length + # github_release_included_regions MUST match gerrit_release_trigger_file_paths + github_release_included_regions: + - 'releases\/.*\.yaml' + - '.releases\/.*\.yaml' + + parameters: + - lf-infra-parameters: + project: "{project}" + branch: "$GERRIT_BRANCH" + stream: "$GERRIT_BRANCH" + # accept all entries defined in the release-yaml file + - string: + name: DISTRIBUTION_TYPE + default: "pypi" + description: "The Jenkins release job distribution type." + - string: + name: LOG_DIR + default: "" + description: "The partial path of logs from the PyPI merge job." + - string: + name: PYPI_PROJECT + default: "" + description: "The PyPI project name." + - string: + name: PYTHON_VERSION + default: "" + description: "The Python compatibility version, example: 3.6" + - string: + name: VERSION + default: "" + description: "The module version, example: 1.0.0" + # special parameters for manual use of the Jenkins job + - bool: + name: USE_RELEASE_FILE + default: true + description: "Set to False (unchecked) to build with parameters" + - bool: + name: DRY_RUN + default: false + description: "Set to True (checked) to skip uploading artifacts" + + builders: + - lf-infra-pre-build + - config-file-provider: + files: + - file-id: sigul-config + variable: SIGUL_CONFIG + - file-id: sigul-password + variable: SIGUL_PASSWORD + - file-id: sigul-pki + variable: SIGUL_PKI + - file-id: signing-pubkey + variable: SIGNING_PUBKEY + - shell: !include-raw-escape: ../shell/sigul-configuration.sh + - shell: !include-raw-escape: ../shell/sigul-install.sh + - inject: + properties-content: | + PYPI_INDEX={pypi-stage-index} + REPOSITORY={pypi-repo} + - shell: !include-raw-escape: ../shell/release-job.sh + +- job-template: + name: "{project-name}-pypi-release-merge" + id: gerrit-pypi-release-merge + <<: *lf_release_common + <<: *lf_pypi_release_wrappers + <<: *lf_pypi_release + + scm: + - lf-infra-gerrit-scm: + jenkins-ssh-credential: "{jenkins-ssh-credential}" + git-url: "{git-url}" + refspec: "$GERRIT_REFSPEC" + branch: "$GERRIT_BRANCH" + submodule-recursive: "{submodule-recursive}" + submodule-timeout: "{submodule-timeout}" + submodule-disable: "{submodule-disable}" + choosing-strategy: gerrit + + triggers: + - gerrit: + server-name: "{gerrit-server-name}" + trigger-on: + - change-merged-event + - comment-added-contains-event: + comment-contains-value: '^Patch Set\s+\d+:\s+remerge\s*$' + projects: + - project-compare-type: "ANT" + project-pattern: "{project}" + branches: + - branch-compare-type: ANT + branch-pattern: "**" + file-paths: "{obj:gerrit_release_trigger_file_paths}" + +- job-template: + name: "{project-name}-pypi-release-merge" + id: github-pypi-release-merge + <<: *lf_release_common + <<: *lf_pypi_release_wrappers + <<: *lf_pypi_release + + properties: + - github: + url: "{github-url}/{github-org}/{project}" + + scm: + - lf-infra-github-scm: + url: "{git-clone-url}{github-org}/{project}" + refspec: "" + branch: "refs/heads/{branch}" + submodule-recursive: "{submodule-recursive}" + submodule-timeout: "{submodule-timeout}" + submodule-disable: "{submodule-disable}" + choosing-strategy: default + jenkins-ssh-credential: "{jenkins-ssh-credential}" + + triggers: + - github-pull-request: + trigger-phrase: "^(remerge)$" + only-trigger-phrase: false + status-context: "PyPI Release Merge" + permit-all: true + github-hooks: true + white-list-target-branches: + - "{branch}" + included-regions: "{obj:github_release_included_regions}" + +- job-template: + name: "{project-name}-pypi-release-verify" + id: gerrit-pypi-release-verify + <<: *lf_release_common + <<: *lf_pypi_verify_wrappers + <<: *lf_pypi_release + + scm: + - lf-infra-gerrit-scm: + jenkins-ssh-credential: "{jenkins-ssh-credential}" + git-url: "{git-url}" + refspec: "$GERRIT_REFSPEC" + branch: "$GERRIT_BRANCH" + submodule-recursive: "{submodule-recursive}" + submodule-timeout: "{submodule-timeout}" + submodule-disable: "{submodule-disable}" + choosing-strategy: gerrit + + triggers: + - gerrit: + server-name: "{gerrit-server-name}" + trigger-on: + - patchset-created-event: + exclude-drafts: true + exclude-trivial-rebase: false + exclude-no-code-change: false + - draft-published-event + - comment-added-contains-event: + # yamllint disable-line rule:line-length + comment-contains-value: '^Patch Set\s+\d+:\s+(recheck|reverify)\s*$' + projects: + - project-compare-type: "ANT" + project-pattern: "{project}" + branches: + - branch-compare-type: ANT + branch-pattern: "**" + file-paths: "{obj:gerrit_release_trigger_file_paths}" + +- job-template: + name: "{project-name}-pypi-release-verify" + id: github-pypi-release-verify + <<: *lf_release_common + <<: *lf_pypi_verify_wrappers + <<: *lf_pypi_release + + properties: + - github: + url: "{github-url}/{github-org}/{project}" + + scm: + - lf-infra-github-scm: + url: "{git-clone-url}{github-org}/{project}" + refspec: "" + branch: "refs/heads/{branch}" + submodule-recursive: "{submodule-recursive}" + submodule-timeout: "{submodule-timeout}" + submodule-disable: "{submodule-disable}" + choosing-strategy: default + jenkins-ssh-credential: "{jenkins-ssh-credential}" + + triggers: + - github-pull-request: + trigger-phrase: "^(recheck|reverify)$" + only-trigger-phrase: false + status-context: "PyPI Release Verify" + permit-all: true + github-hooks: true + white-list-target-branches: + - "{branch}" + included-regions: "{obj:github_release_included_regions}" diff --git a/releasenotes/notes/pypi-refactor-release-833955d759789d57.yaml b/releasenotes/notes/pypi-refactor-release-833955d759789d57.yaml new file mode 100644 index 00000000..0d99b941 --- /dev/null +++ b/releasenotes/notes/pypi-refactor-release-833955d759789d57.yaml @@ -0,0 +1,16 @@ +--- +features: + - | + Refactor PyPI release-verify and release-merge templates to download + distribution files from a PyPI staging index and upload the files to a + PyPI release index. Remove all the builders that were previously used. + Call pip upgrade to install latest version of setuptools and twine. + Split the PyPI job groups because the verify & merge templates do not + accept the same arguments as release-verify & release-merge templates. + Remove stream, branch usage; just one PyPI release job per project now. + Extend the PyPI release yaml file schema for log_dir and other values. + All this makes the PyPI release ver/mrg templates highly similar to the + release-job ver/mrg templates. Revise documentation appropriately. + Move the PyPI release features to the lf-release-jobs.(rst,yaml) files. + Extend the release-job.sh script to have pypi release functions, and + drop the pypi-tag-release.sh script. diff --git a/schema/release-pypi-schema.yaml b/schema/release-pypi-schema.yaml index e56eed69..7edc4942 100644 --- a/schema/release-pypi-schema.yaml +++ b/schema/release-pypi-schema.yaml @@ -13,13 +13,19 @@ $id: "https://github.com/lfit/releng-global-jjb/blob/master/release-pypi-schema. required: - "distribution_type" - - "project" + - "log_dir" + - "pypi_project" + - "python_version" - "version" properties: distribution_type: type: "string" - project: + log_dir: + type: "string" + pypi_project: + type: "string" + python_version: type: "string" version: type: "string" diff --git a/shell/pypi-dist-build.sh b/shell/pypi-dist-build.sh index 27b4b3df..9341a1ff 100644 --- a/shell/pypi-dist-build.sh +++ b/shell/pypi-dist-build.sh @@ -16,16 +16,16 @@ echo "---> pypi-dist-build.sh" # Ensure we fail the job if any steps fail. set -eu -o pipefail +echo "INFO: creating virtual environment" virtualenv -p python3 /tmp/pypi PATH=/tmp/pypi/bin:$PATH - -echo "INFO: installing twine to check distributions" -pip install -q twine +pipup="python -m pip install -q --upgrade setuptools twine wheel" +echo "INFO: $pipup" +$pipup bdist="" if $BUILD_BDIST_WHEEL; then - echo "INFO: installing wheel to build binary distribution" - pip install -q wheel + echo "INFO: adding wheel to build binary distribution" bdist="bdist_wheel" fi diff --git a/shell/pypi-tag-release.sh b/shell/pypi-tag-release.sh deleted file mode 100644 index 2d098ee5..00000000 --- a/shell/pypi-tag-release.sh +++ /dev/null @@ -1,149 +0,0 @@ -#!/bin/bash -# SPDX-License-Identifier: EPL-1.0 -############################################################################## -# Copyright (c) 2019 The Linux Foundation and others. -# -# All rights reserved. This program and the accompanying materials -# are made available under the terms of the Eclipse Public License v1.0 -# which accompanies this distribution, and is available at -# http://www.eclipse.org/legal/epl-v10.html -############################################################################## -echo "---> pypi-tag-release.sh" - -# Ensure we fail the job if any steps fail. -set -eu -o pipefail - -# Functions. - -set_variables(){ - echo "INFO: Setting variables" - # Verify if using release file or parameters - if $USE_RELEASE_FILE; then - echo "INFO: Checking number of release yaml files" - release_files=$(git diff-tree --no-commit-id -r "$GIT_COMMIT" --name-only -- "releases/" ".releases/") - if (( $(echo "$release_files" | wc -w) != 1 )); then - echo "ERROR: RELEASE FILES: $release_files" - echo "ERROR: Committing multiple release files in the same commit OR rename/amend of existing files is not supported." - exit 1 - else - release_file="$release_files" - echo "INFO: RELEASE FILE: $release_file" - fi - else - echo "INFO: This job is built with parameters, no release file" - release_file="None" - fi - - if [[ -z ${DISTRIBUTION_TYPE:-} ]]; then - echo "INFO: reading DISTRIBUTION_TYPE from file $release_file" - DISTRIBUTION_TYPE="$(niet ".distribution_type" "$release_file")" - fi - if [[ -z ${VERSION:-} ]]; then - echo "INFO: reading VERSION from file $release_file" - VERSION="$(niet ".version" "$release_file")" - fi - - # Display Release Information - printf "\t%-30s\n" RELEASE_ENVIRONMENT_INFO: - printf "\t%-30s %s\n" RELEASE_FILE: $release_file - printf "\t%-30s %s\n" JENKINS_HOSTNAME: $JENKINS_HOSTNAME - printf "\t%-30s %s\n" SILO: $SILO - printf "\t%-30s %s\n" PROJECT: $PROJECT - printf "\t%-30s %s\n" PROJECT-DASHED: ${PROJECT//\//-} - printf "\t%-30s %s\n" DISTRIBUTION_TYPE: $DISTRIBUTION_TYPE - printf "\t%-30s %s\n" VERSION: $VERSION -} - -# needs to run in the repository root -verify_schema(){ - echo "INFO: Fetching schema" - pypi_schema="release-pypi-schema.yaml" - wget https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/${pypi_schema} - echo "INFO: Verifying $release_file against schema $pypi_schema" - lftools schema verify "$release_file" "$pypi_schema" - echo "INFO: $release_file passed schema verification" -} - -verify_version(){ - # Verify allowed patterns "v#.#.#" or "#.#.#" aka SemVer - echo "INFO: Verifying version string $VERSION" - allowed_version_regex="^((v?)([0-9]+)\.([0-9]+)\.([0-9]+))$" - if [[ $VERSION =~ $allowed_version_regex ]]; then - echo "INFO: The version $VERSION is a valid semantic version" - else - echo "ERROR: The version $VERSION is not a valid semantic version" - echo "ERROR: Allowed versions are \"v#.#.#\" or \"#.#.#\" aka SemVer" - echo "ERROR: See https://semver.org/ for more details on SemVer" - exit 1 - fi -} - -verify_dist(){ - # Verify all file names in dist folder have the expected version string - dir="$WORKSPACE/$TOX_DIR/dist" - echo "INFO: Listing files in $dir" - ls $dir - echo "INFO: Checking files in $dir for $VERSION" - if unex_files=$(find $dir | grep -v $VERSION | egrep -v "^$dir$"); then - echo "ERROR: found unexpected files: $unex_files" - exit 1 - else - echo "INFO: All file names have expected string ${VERSION}" - fi -} - -# sigul is only available on Centos -tag_gerrit(){ - echo "INFO: Verifying tag $VERSION in repo" - # Import public signing key - gpg --import "$SIGNING_PUBKEY" - if git tag -v "$VERSION"; then - echo "INFO: Repo already tagged" - return 0 - fi - echo "INFO: Tagging repo" - git tag -am "${PROJECT//\//-} $VERSION" "$VERSION" - echo "INFO: Signing tag" - sigul --batch -c "$SIGUL_CONFIG" sign-git-tag "$SIGUL_KEY" "$VERSION" < "$SIGUL_PASSWORD" - echo "INFO: Verifying tag" - # may fail due to missing public key - if ! git tag -v "$VERSION"; then - echo "WARN: failed to verify tag, continuing anyhow" - fi - # The verify job also calls this script - if [[ ! $JOB_NAME =~ "merge" ]] ; then - echo "INFO: job is not a merge, skipping push" - else - echo "INFO: configuring Gerrit remote" - gerrit_ssh=$(echo "$GERRIT_URL" | awk -F"/" '{print $3}') - git remote set-url origin "ssh://$RELEASE_USERNAME@$gerrit_ssh:29418/$PROJECT" - git config user.name "$RELEASE_USERNAME" - git config user.email "$RELEASE_EMAIL" - echo -e "Host $gerrit_ssh\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config - chmod 600 ~/.ssh/config - if $DRY_RUN; then - echo "INFO: dry run, skipping push" - else - echo "INFO: pushing tag" - git push origin "$VERSION" - fi - fi -} - -# Main -virtualenv -p python3 /tmp/pypi -PATH=/tmp/pypi/bin:$PATH -pip install lftools jsonschema niet -set_variables -if [[ $DISTRIBUTION_TYPE != "pypi" ]]; then - echo "ERROR: unexpected distribution type $DISTRIBUTION_TYPE" - exit 1 -fi -if $USE_RELEASE_FILE; then - verify_schema -fi -verify_version -verify_dist -# TODO: write tag_github function -tag_gerrit -echo "---> pypi-tag-release.sh ends" diff --git a/shell/pypi-upload.sh b/shell/pypi-upload.sh index 86fff164..3219cadf 100644 --- a/shell/pypi-upload.sh +++ b/shell/pypi-upload.sh @@ -17,11 +17,12 @@ echo "---> pypi-upload.sh" # Ensure we fail the job if any steps fail. set -eu -o pipefail +echo "INFO: creating virtual environment" virtualenv -p python3 /tmp/pypi PATH=/tmp/pypi/bin:$PATH - -echo "INFO: installing twine to upload distributions" -pip install -q twine +pipup="python -m pip install -q --upgrade twine" +echo "INFO: $pipup" +$pipup echo "INFO: cd to tox-dir $TOX_DIR" cd "$WORKSPACE/$TOX_DIR" @@ -29,10 +30,13 @@ cd "$WORKSPACE/$TOX_DIR" cmd="twine upload -r $REPOSITORY dist/*" if $DRY_RUN; then echo "INFO: dry-run is set, echoing command only" - echo $cmd + echo "$cmd" else echo "INFO: uploading distributions to repo $REPOSITORY" $cmd + # emit message and files on single line for release-job + # shellcheck disable=SC2046 + echo "INFO: successfully uploaded distributions: " $(ls dist) fi echo "---> pypi-upload.sh ends" diff --git a/shell/release-job.sh b/shell/release-job.sh index 7b22b280..5ee06f55 100644 --- a/shell/release-job.sh +++ b/shell/release-job.sh @@ -11,12 +11,12 @@ echo "---> release-job.sh" set -eu -o pipefail -set +u -python3 -m venv /tmp/v/venv/ -# shellcheck disable=SC1091 -source /tmp/v/venv/bin/activate -set -u -python -m pip install lftools[nexus] jsonschema niet yq +echo "INFO: creating virtual environment" +virtualenv -p python3 /tmp/venv +PATH=/tmp/venv/bin:$PATH +pipup="python -m pip install -q --upgrade pip lftools[nexus] jsonschema niet twine yq" +echo "INFO: $pipup" +$pipup #Functions. @@ -30,7 +30,7 @@ set_variables_common(){ NEXUS_PATH="${SILO}/${JENKINS_HOSTNAME}/" # Verify if using release file or parameters if $USE_RELEASE_FILE ; then - release_files=$(git diff-tree --no-commit-id -r "$GERRIT_PATCHSET_REVISION" --name-only -- "releases/" ".releases/") + release_files=$(git diff-tree --no-commit-id -r "$GIT_COMMIT" --name-only -- "releases/" ".releases/") if (( $(grep -c . <<<"$release_files") > 1 )); then echo "INFO: RELEASE FILES ARE AS FOLLOWS: $release_files" echo "ERROR: Committing multiple release files in the same commit OR rename/amend of existing files is not supported." @@ -108,16 +108,50 @@ set_variables_container(){ printf "\t%-30s %s\n" GERRIT_REF_TO_TAG: $ref } +set_variables_pypi(){ + # use Jenkins parameter if set; else get value from release file + echo "INFO: Setting pypi variables" + LOG_DIR="${LOG_DIR:-None}" + if [[ $LOG_DIR == "None" ]]; then + LOG_DIR="$(yq -er .log_dir "$release_file")" + fi + LOGS_URL="${LOGS_SERVER}/${NEXUS_PATH}${LOG_DIR}" + LOGS_URL=${LOGS_URL%/} # strip any trailing '/' + PYPI_PROJECT="${PYPI_PROJECT:-None}" + if [[ $PYPI_PROJECT == "None" ]]; then + PYPI_PROJECT="$(yq -er .pypi_project "$release_file")" + fi + PYTHON_VERSION="${PYTHON_VERSION:-None}" + if [[ $PYTHON_VERSION == "None" ]]; then + PYTHON_VERSION="$(yq -er .python_version "$release_file")" + fi + VERSION="${VERSION:-None}" + if [[ $VERSION == "None" ]]; then + VERSION="$(yq -er .version "$release_file")" + fi + + # Continuing displaying Release Information (pypi) + printf "\t%-30s\n" RELEASE_PYPI_INFO: + printf "\t%-30s %s\n" LOG_DIR: "$LOG_DIR" + printf "\t%-30s %s\n" LOGS_URL: "$LOGS_URL" + printf "\t%-30s %s\n" PYPI_INDEX: "$PYPI_INDEX" # from job configuration + printf "\t%-30s %s\n" PYPI_PROJECT: "$PYPI_PROJECT" + printf "\t%-30s %s\n" PYTHON_VERSION: "$PYTHON_VERSION" + printf "\t%-30s %s\n" VERSION: "$VERSION" +} + verify_schema(){ echo "INFO: Verifying $release_file schema." lftools schema verify "$release_file" "$RELEASE_SCHEMA" } verify_version(){ - # Verify allowed versions - # Allowed versions are "v#.#.#" or "#.#.#" aka SemVer + # Verify allowed patterns "v#.#.#" or "#.#.#" aka SemVer + echo "INFO: Verifying version string $VERSION" allowed_version_regex="^((v?)([0-9]+)\.([0-9]+)\.([0-9]+))$" - if [[ ! $VERSION =~ $allowed_version_regex ]]; then + if [[ $VERSION =~ $allowed_version_regex ]]; then + echo "INFO: The version $VERSION is a valid semantic version" + else echo "INFO: The version $VERSION is not a semantic valid version" echo "INFO: Allowed versions are \"v#.#.#\" or \"#.#.#\" aka SemVer" echo "INFO: See https://semver.org/ for more details on SemVer" @@ -137,6 +171,21 @@ verify_version_match_release(){ fi } +# check prerequisites to detect mistakes in the release YAML file +verify_pypi_match_release(){ + wget -q -P /tmp "${LOGS_URL}/"console.log.gz + echo "INFO: Searching for strings >$PYPI_PROJECT< and >$VERSION< in job log" + # pypi-upload.sh generates success message with file list + if zgrep -i "uploaded" /tmp/console.log.gz | grep "$PYPI_PROJECT" | grep "$VERSION" ; then + echo "INFO: found expected strings in job log" + else + echo "ERROR: failed to find expected strings in job log" + exit 1 + fi +} + +# sigul is only available on Centos +# TODO: write tag_github function tag(){ # Import public signing key gpg --import "$SIGNING_PUBKEY" @@ -244,19 +293,62 @@ maven_release_file(){ tag } -echo "########### Start Script release-job.sh ###################################" +# calls pip to download binary and source distributions from the specified index, +# which requires a recent-in-2019 version. Uploads the files it received. +pypi_release_file(){ + tgtdir=dist + mkdir $tgtdir + pip_pfx="pip download -d $tgtdir --no-deps --python-version $PYTHON_VERSION -i $PYPI_INDEX" + module="$PYPI_PROJECT==$VERSION" + pip_bin="$pip_pfx $module" + echo "INFO: downloading binary: $pip_bin" + if ! $pip_bin ; then + echo "WARN: failed to download binary distribution" + fi + pip_src="$pip_pfx --no-binary=:all: $module" + echo "INFO: downloading source: $pip_src" + if ! $pip_src ; then + echo "WARN: failed to download source distribution" + fi + echo "INFO: Checking files in $tgtdir" + filecount=$(ls $tgtdir | wc -l) + if [[ $filecount = 0 ]] ; then + echo "ERROR: no files downloaded" + exit 1 + else + # shellcheck disable=SC2046 + echo "INFO: downloaded $filecount distributions: " $(ls $tgtdir) + fi -# Check if this is a container or maven release: release-container-schema.yaml vs release-schema.yaml -# Logic to determine what we are releasing. -########################################## + if [[ ! "$JOB_NAME" =~ "merge" ]] ; then + echo "INFO: not a merge job, not uploading files" + return + fi + + cmd="twine upload -r $REPOSITORY $tgtdir/*" + if $DRY_RUN; then + echo "INFO: dry-run is set, echoing command only" + echo "$cmd" + else + echo "INFO: uploading $filecount distributions to repo $REPOSITORY" + $cmd + fi + tag +} # Set common environment variables set_variables_common +# Determine the type of release: +# - container, release-container-schema.yaml +# - maven, release-schema.yaml +# - pypi, release-pypi-schema.yaml + if [[ "$DISTRIBUTION_TYPE" == "maven" ]]; then - wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/release-schema.yaml - RELEASE_SCHEMA="release-schema.yaml" if $USE_RELEASE_FILE ; then + RELEASE_SCHEMA="release-schema.yaml" + echo "INFO: Fetching schema $RELEASE_SCHEMA" + wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/release-schema.yaml verify_schema fi set_variables_maven @@ -264,17 +356,29 @@ if [[ "$DISTRIBUTION_TYPE" == "maven" ]]; then verify_version_match_release maven_release_file elif [[ "$DISTRIBUTION_TYPE" == "container" ]]; then - wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/release-container-schema.yaml - RELEASE_SCHEMA="release-container-schema.yaml" - verify_schema + if $USE_RELEASE_FILE ; then + RELEASE_SCHEMA="release-container-schema.yaml" + echo "INFO: Fetching schema $RELEASE_SCHEMA" + wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/${RELEASE_SCHEMA} + verify_schema + fi set_variables_container verify_version container_release_file +elif [[ "$DISTRIBUTION_TYPE" == "pypi" ]]; then + if $USE_RELEASE_FILE ; then + RELEASE_SCHEMA="release-pypi-schema.yaml" + echo "INFO: Fetching schema $RELEASE_SCHEMA" + wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/${RELEASE_SCHEMA} + verify_schema + fi + set_variables_pypi + verify_version + verify_pypi_match_release + pypi_release_file else echo "ERROR: distribution_type: $DISTRIBUTION_TYPE not supported" - echo "Must be maven or container" exit 1 fi -########################################## -echo "########### End Script release-job.sh ###################################" +echo "---> release-job.sh ends"