docs/jjb/lf-docker-jobs.rst,
docs/jjb/lf-macros.rst,
docs/jjb/lf-maven-jobs.rst,
- docs/jjb/lf-python-jobs.rst,
docs/jjb/lf-rtdv3-jobs.rst
[all.ShellCheck]
name: gerrit-python-jobs
jobs:
- "{project-name}-python-jobs"
- - gerrit-tox-sonar
- gerrit-pypi-merge
- - gerrit-pypi-release-verify
- gerrit-pypi-release-merge
+ - gerrit-pypi-release-verify
+ - gerrit-pypi-verify
+ - gerrit-tox-sonar
project-name: gerrit-python
name: github-python-jobs
jobs:
- "{project-name}-github-python-jobs"
- - github-tox-sonar
- github-pypi-merge
- - github-pypi-release-verify
- github-pypi-release-merge
+ - github-pypi-release-verify
+ - github-tox-sonar
+ - github-pypi-verify
project-name: github-python
:clm-project-name: Project name in Nexus IQ to send results to.
-lf-infra-pypi-tag-release
--------------------------
-
-Checks the format of the release version string and checks the git
-repository for that tag. In a merge job, if the tag does not exist,
-adds the tag to the repository, signs the tag, and pushes the tag
-to the git server. Signing requires sigul, which is only available
-on a CentOS build node.
-
-lf-infra-pypi-upload
---------------------
-
-Uploads distribution files from subdirectory "dist" to a PyPI repository
-using a Python virtual enviroment to install required packages. The
-Jenkins server must have a configuration file ".pypirc".
-
-:Required Parameters:
-
- :pypi-repo: PyPI repository key in .pypirc configuration file;
- e.g., "pypi-test" or "pypi".
-
lf-infra-tox-install
--------------------
suites. Checking coverage does not guarantee that the tests execute
properly, but it identifies code that is not executed by any test.
+.. comment Start ignoring WriteGoodLintBear
+
This job reuses the Sonar builders used for Java/Maven projects which
run maven twice. The first invocation does nothing for Python
projects, so the job uses the goal 'validate' by default. The second
invocation publishes results using the goal 'sonar:sonar' by default.
+.. comment Stop ignoring
+
For example:
.. code-block:: bash
get configured in defaults.yaml)
:mvn-settings: The name of the settings file with credentials for the project.
+.. comment Start ignoring WriteGoodLintBear
+
:Optional parameters:
:branch: Git branch, should be master (default: master)
:git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
:github-url: URL for Github. (default: https://github.com)
:java-version: Version of Java to use for the build. (default: openjdk8)
- :mvn-global-settings: The name of the Maven global settings to use for
+ :mvn-global-settings: The name of the Maven global settings to use
:mvn-goals: The Maven goal to run first. (default: validate)
:mvn-version: Version of maven to use. (default: mvn35)
:parallel: Boolean indicator for tox to run tests in parallel or series.
- (default: false, in series)
+ (default: false, in series)
:pre-build-script: Shell script to execute before the Sonar builder.
For example, install prerequisites or move files to the repo root.
(default: a string with a shell comment)
:python-version: Python version to invoke pip install of tox-pyenv
(default: python2)
- :sonarcloud: Whether or not to use SonarCloud ``true|false``.
+ :sonarcloud: Boolean indicator to use SonarCloud ``true|false``.
(default: false)
:sonarcloud-project-key: SonarCloud project key. (default: '')
:sonarcloud-project-organization: SonarCloud project organization.
modifications trigger a build. Refer to JJB documentation for "file-path" details.
https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+.. comment Stop ignoring
+
Tox Verify
----------
:git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
:pre-build-script: Shell script to execute before the CLM builder.
For example, install prerequisites or move files to the repo root.
- (default: a string with only a comment)
+ (default: a string with a shell comment)
:python-version: Python version to invoke pip install of tox-pyenv
(default: python2)
:stream: Keyword representing a release code-name.
:build-node: The node to run the build on.
:jenkins-ssh-credential: Credential to use for SSH. (Generally set
in defaults.yaml)
+ :project: Git repository name
+ :project-name: Jenkins job name prefix
:Optional Parameters:
Creates and uploads distribution files on merge of a patch set. Runs
tox, builds a source distribution and (optionally) a binary
distribution, and uploads the distribution(s) to a PyPI repository.
-This job should be configured to use a test PyPI repository like
-testpypi.python.org, not a public release area like the global PyPI
-repository. Like the verify job, this requires a setup.py file for
-packaging the component.
+This job should use a staging repository like testpypi.python.org,
+which sets up use of release jobs to promote the distributions later.
+This job can also use a public release area like the global PyPI
+repository. The project git repository must have a setup.py file
+with configuration for packaging the component.
The tox runner is pyenv aware so if the image contains an installation
of pyenv at /opt/pyenv it will pick it up and run Python tests with
Requires a .pypirc configuration file in the Jenkins builder home directory,
-an example appears next that uses API tokens. No repository is needed in the
-PyPI section.
+an example appears next that uses API tokens. Note that in the [pypi] entry
+the repository key-value pair is optional, it defaults to pypi.org.
.. code-block:: bash
:build-node: The node to run the build on.
:jenkins-ssh-credential: Credential to use for SSH. (Generally set
in defaults.yaml)
+ :project: Git repository name
+ :project-name: Jenkins job name prefix
:Optional Parameters:
(default: false, in series)
:pre-build-script: Shell script to execute before the tox builder. For
example, install system prerequisites. (default: a shell comment)
- :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
- Merge jobs should use a server like testpypi.python.org. (default: pypi-test)
+ :pypi-repo: Key for the PyPI target repository in the .pypirc file,
+ ideally a server like test.pypy.org. (default: pypi-test)
:python-version: Python version to invoke pip install of tox-pyenv
(default: python3)
:stream: Keyword representing a release code-name.
:gerrit_trigger_file_paths: Override file paths used to filter which file
modifications trigger a build. Refer to JJB documentation for "file-path" details.
https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
-
-
-PyPI Release Verify
--------------------
-
-Verifies a Python library project on creation of a patch set with a
-release yaml file. Runs tox, builds source and (optionally) binary
-distributions, checks the format of the version string, checks that
-the distribution file names contain the release version string, and
-checks if the tag exists in the code repository for the release
-version.
-
-To initiate the release process, create a releases/ or .releases/
-directory at the root of the project repository, add one release yaml
-file to it, and submit a change set with that release yaml file. A
-schema and and an example for the release yaml file appear below. The
-version in the release yaml file must be a valid Semantic Versioning
-(SemVer) string, matching either the pattern "v#.#.#" or "#.#.#" where
-"#" is one or more digits.
-
-This job is similar to the PyPI verify job, but is only triggered by a
-patch set with a release yaml file.
-
-The build node for PyPI release verify jobs must be CentOS, which
-supports the sigul client for accessing a signing server.
-
-.. note::
-
- The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml).
- In words, the directory name can be ".releases" or "releases"; the file
- name can be anything with suffix ".yaml".
-
-The JSON schema for a pypi release file appears below.
-
-.. code-block:: none
-
- ---
- $schema: "http://json-schema.org/schema#"
- $id: "https://github.com/lfit/releng-global-jjb/blob/master/release-pypi-schema.yaml"
-
- required:
- - "distribution_type"
- - "project"
- - "version"
-
- properties:
- distribution_type:
- type: "string"
- project:
- type: "string"
- version:
- type: "string"
-
-
-An example of a pypi release file appears below.
-
-.. code-block:: none
-
- $ cat releases/1.0.0-pypi.yaml
- ---
- distribution_type: pypi
- version: 1.0.0
- project: 'example-project'
-
-
-:Template Names:
-
- - {project-name}-pypi-release-verify-{stream}
- - gerrit-pypi-release-verify
- - github-pypi-release-verify
-
-:Comment Trigger: recheck
-
-:Required Parameters:
-
- :build-node: The node to run build on, which must be Centos.
- :jenkins-ssh-credential: Credential to use for SSH. (Generally set
- in defaults.yaml)
-
-:Optional Parameters:
-
- :branch: The branch to build against. (default: master)
- :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
- :build-timeout: Timeout in minutes before aborting build. (default: 15)
- :disable-job: Whether to disable the job (default: false)
- :dist-binary: Whether to build a binary wheel distribution. (default: true)
- :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
- :parallel: Boolean indicator for tox to run tests in parallel or series.
- (default: false, in series)
- :pre-build-script: Shell script to execute before the tox builder.
- For example, install prerequisites or move files to the repo root.
- (default: a string with a shell comment)
- :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
- Release jobs should use a server like pypy.org. (default: pypi)
- :python-version: Python version to invoke pip install of tox-pyenv
- (default: python3)
- :stream: Keyword representing a release code-name.
- Often the same as the branch. (default: master)
- :submodule-recursive: Whether to checkout submodules recursively.
- (default: true)
- :submodule-timeout: Timeout (in minutes) for checkout operation.
- (default: 10)
- :submodule-disable: Disable submodule checkout operation.
- (default: false)
- :tox-dir: Directory containing the project's tox.ini relative to
- the workspace. The default uses tox.ini at the project root.
- (default: '.')
- :tox-envs: Tox environments to run. If blank run everything described
- in tox.ini. (default: '')
- :use-release-file: Whether to use the release file. (default: true)
-
-PyPI Release Merge
-------------------
-
-Publishes a Python library on merge of a patch set with a release yaml
-file. Runs tox, builds source and (optionally) binary distributions,
-checks the format of the version string, checks that the distribution
-file names contain the release version string, checks if the tag
-exists in the code repository for the release version, then if the tag
-does not exist, tags the code repository with the release version,
-signs the tag and pushes the tag to the git server. Finally this
-uploads the distributions to a PyPI repository.
-
-This job is similar to the PyPI merge job, but is only triggered by
-merge of a release yaml file, also this checks the version and tag
-before uploading to a public repository such as PyPI.
-
-See the PyPI Release Verify job above for documentation of the release
-yaml file format.
-
-The build node for PyPI release merge jobs must be CentOS, which
-supports the sigul client for accessing a signing server.
-
-A Jenkins user can also trigger this release job via the "Build with
-parameters" action, removing the need to merge a release yaml file.
-The user must enter parameters in the same way as a release yaml file,
-except for the special USE_RELEASE_FILE and DRY_RUN check boxes. The
-user must uncheck the USE_RELEASE_FILE check box if the job should run
-with a release file, while passing the required information as build
-parameters. Similarly, the user must uncheck the DRY_RUN check box to
-test the job while skipping upload of files to a repository.
-
-The special parameters are as follows::
-
- VERSION = 1.0.0
- USE_RELEASE_FILE = false
- DRY_RUN = false
-
-:Template Names:
-
- - {project-name}-pypi-release-merge-{stream}
- - gerrit-pypi-release-merge
- - github-pypi-release-merge
-
-:Comment Trigger: remerge
-
-:Required Parameters:
-
- :build-node: The node to run build on, which must be Centos.
- :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
- in defaults.yaml)
-
-:Optional Parameters:
-
- :branch: The branch to build against. (default: master)
- :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
- :build-timeout: Timeout in minutes before aborting build. (default: 15)
- :disable-job: Whether to disable the job (default: false)
- :dist-binary: Whether to build a binary wheel distribution. (default: true)
- :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
- :parallel: Boolean indicator for tox to run tests in parallel or series.
- (default: false, in series)
- :pre-build-script: Shell script to execute before the tox builder.
- For example, install prerequisites or move files to the repo root.
- (default: a string with a shell comment)
- :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
- Release jobs should use a server like pypy.org. (default: pypi)
- :python-version: Python version to invoke pip install of tox-pyenv
- (default: python3)
- :stream: Keyword representing a release code-name.
- Often the same as the branch. (default: master)
- :submodule-recursive: Whether to checkout submodules recursively.
- (default: true)
- :submodule-timeout: Timeout (in minutes) for checkout operation.
- (default: 10)
- :submodule-disable: Disable submodule checkout operation.
- (default: false)
- :tox-dir: Directory containing the project's tox.ini relative to
- the workspace. The default uses tox.ini at the project root.
- (default: '.')
- :tox-envs: Tox environments to run. If blank run everything described
- in tox.ini. (default: '')
- :use-release-file: Whether to use the release file. (default: true)
.. _lf-global-jjb-release:
-#######################
Self-Serve Release Jobs
-#######################
-
-Self-serve release jobs allow a project team to direct Jenkins to
-promote a jar file or container image from a staging area to a release
-area. To trigger the action, create a releases/ or .releases/
-directory, add a release yaml file to it, and submit a change set with
-one release yaml file to Gerrit. Upon merge of the change, Jenkins will
-sign the reference extrapolated by log_dir and promote the artifact. The
-expected format of the release yaml file appears in schemas and examples
-below.
-
-The build node for maven and container release jobs must be CentOS,
-which supports the sigul client for accessing a signing server. The
-build node for container release jobs must have Docker installed.
-
-A Jenkins user can also trigger a release job via the "Build with
-parameters" action, removing the need for a release yaml file. The
-user must enter parameters in the same way as a release yaml file,
-except for the special USE_RELEASE_FILE and DRY_RUN check boxes. The
-user must uncheck the USE_RELEASE_FILE check box if the job should
-run with a release file, while passing the required information as
-build parameters. Similarly, the user must uncheck the DRY_RUN check
-box to test the job while skipping repository promotion to Nexus.
-
-The special parameters are as follows::
+=======================
+
+Self-serve release jobs allow project committers to direct Jenkins to
+promote a jar file, container image or Python package from a staging
+area to a release area.
+
+To use the self-release process, create a releases/ or .releases/
+directory at the root of the project repository, add one release yaml
+file to it, and submit a change set with that release yaml file. The
+required contents of the release yaml file are different for each type
+of release, see the schemas and examples shown below. The version in
+the release yaml file must be a valid Semantic Versioning (SemVer)
+string, matching either the pattern "v#.#.#" or "#.#.#" where "#" is
+one or more digits. Upon merge of the change, Jenkins will sign the
+reference extrapolated by log_dir and promote the artifact.
+
+.. note::
+
+ The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml).
+ In words, the directory name can be ".releases" or "releases"; the file
+ name can be anything with suffix ".yaml".
+
+The build node for all release jobs must be CentOS, which supports the
+sigul client for accessing a signing server. The build node for
+container release jobs must have Docker installed.
+
+A Jenkins admin user can also trigger a release job via the "Build
+with parameters" action, removing the need to create and merge a
+release yaml file. The user must enter parameters in the same way as
+a release yaml file, except for the special USE_RELEASE_FILE and
+DRY_RUN check boxes. The user must uncheck the USE_RELEASE_FILE check
+box if the job should run without a release file, instead passing the
+required information as build parameters. The user can check the
+DRY_RUN check box to test the job while skipping upload of files to
+the release repository.
+
+For example, the parameters for a Maven release are as follows::
GERRIT_BRANCH = master
VERSION = 1.0.0
USE_RELEASE_FILE = false
DRY_RUN = false
-.. note::
+Maven Release Files
+-------------------
- The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml).
- In words, the directory name can be ".releases" or "releases"; the file
- name can be anything with suffix ".yaml".
+An example of a maven release file appears below.
+
+.. code-block:: none
-The JSON schema for a maven release job appears below.
+ $ cat releases/maven-release.yaml
+ ---
+ distribution_type: maven
+ log_dir: example-project-maven-stage-master/17/
+ project: example-project
+ version: 1.0.0
+
+
+The following parameters must appear in a maven release yaml file.
+
+:Required Parameters:
+
+ :distribution_type: Must be "maven".
+ :log_dir: The suffix of the logs URL reported on completion by the
+ Jenkins stage job that created and pushed the artifact
+ to the staging repository. For example, use value
+ "example-project-maven-stage-master/17" for the logs URL
+ https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-maven-stage-master/17
+ :project: The name of the project.
+ :version: The semantic version string used for the artifact.
+
+The JSON schema for a maven release file appears below.
.. code-block:: none
type: "string"
-Example of a maven release file:
+Container Release Files
+-----------------------
-.. code-block:: bash
+An example of a container release file appears below.
- $ cat releases/1.0.0-maven.yaml
+.. code-block:: none
+
+ $ cat releases/container-release.yaml
---
- distribution_type: 'maven'
- version: '1.0.0'
- project: 'example-project'
- log_dir: 'example-project-maven-stage-master/17/'
+ distribution_type: container
+ container_release_tag: 1.0.0
+ container_pull_registry: nexus.onap.org:10003
+ container_push_registry: nexus.onap.org:10002
+ project: test
+ ref: d1b9cd2dd345fbeec0d3e2162e008358b8b663b2
+ containers:
+ - name: test-backend
+ version: 1.0.0-20190806T184921Z
+ - name: test-frontend
+ version: 1.0.0-20190806T184921Z
+
+The following parameters must appear in a container release yaml file.
+
+:Required Parameters:
-The JSON schema for a container release job appears below.
+ :distribution_type: Must be "container".
+ :container_release_tag: The string to use as a Docker tag on all
+ released containers.
+ :container_pull_registry: The Nexus registry that supplies the staged
+ image(s).
+ :container_push_registry: The Nexus registry that receives the released
+ image(s).
+ :project: The name of the project.
+ :ref: The git commit reference (SHA-1 code) to tag with the version string.
+ :containers: A list of name and version (tag) pairs that specify the
+ Docker images in the container-pull registry to promote to the
+ container-push registry.
+
+The JSON schema for a container release file appears below.
.. code-block:: none
type: "string"
-An example of a container release file appears below. The job tags the
-git repository at the specified commit reference. The job applies the
-container_release_tag string to all released containers. The job uses the
-per-container version strings to pull images from the container registry.
+PyPI Release Files
+------------------
-.. code-block:: bash
+An example of a PyPI release file appears below.
+
+.. code-block:: none
- $ cat releases/1.0.0-container.yaml
+ $ cat releases/pypi-release.yaml
---
- distribution_type: 'container'
- container_release_tag: '1.0.0'
- container_pull_registry: 'nexus.onap.org:10003"
- container_push_registry: 'nexus.onap.org:10002"
- project: 'test'
- ref: d1b9cd2dd345fbeec0d3e2162e008358b8b663b2
- containers:
- - name: test-backend
- version: 1.0.0-20190806T184921Z
- - name: test-frontend
- version: 1.0.0-20190806T184921Z
+ distribution_type: pypi
+ pypi_project: mypackage
+ python_version: 3.4
+ version: 1.0.0
+
+The following parameters must appear in the PyPI release yaml file.
+These are not part of the Jenkins job definition to allow independent
+self-release of a package maintained in a git repository with other
+packages.
-Example of a Jenkins job configuration that uses the global-jjb
-templates for Gerrit:
+:Required Parameters:
+
+ :distribution_type: Must be "pypi".
+ :log_dir: The suffix of the logs URL reported on completion by the
+ Jenkins merge job that created and pushed the distribution files
+ to the staging repository. For example, use value
+ "example-project-pypi-merge-master/17" for the logs URL
+ https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-pypi-merge-master/17
+ :pypi_project: The PyPI project name at the staging and
+ release repositories, for example "mypackage".
+ :python_version: The Python interpreter version to use for pip
+ "Requires-Python" compatibility checks, for example "3" or "3.7.0".
+ :version: The semantic version string used for the package in the
+ setup.py file.
+
+The JSON schema for a PyPI release file appears below.
+
+.. code-block:: none
+
+ ---
+ $schema: "http://json-schema.org/schema#"
+ $id: "https://github.com/lfit/releng-global-jjb/blob/master/release-pypi-schema.yaml"
+
+ required:
+ - "distribution_type"
+ - "log_dir"
+ - "pypi_project"
+ - "python_version"
+ - "version"
+
+ properties:
+ distribution_type:
+ type: "string"
+ log_dir:
+ type: "string"
+ pypi_project:
+ type: "string"
+ python_version:
+ type: "string"
+ version:
+ type: "string"
+
+
+Jenkins Jobs
+------------
+
+An example of a Jenkins job configuration that uses the global-jjb
+templates for maven and container release jobs appears next.
.. code-block:: none
Release Engineers: please follow the setup guide below before adding the job definition.
+JJB Macros
+----------
+
+lf-release
+~~~~~~~~~~
+
+Release verify and merge jobs are the same except for their scm,
+trigger, and builders definition. This anchor is the common template.
+
+JJB Templates
+-------------
+
+Release Merge
+~~~~~~~~~~~~~
+
+This template supports Maven and Container release jobs.
+
+:Template Name: {project-name}-release-merge
+
+:Comment Trigger: remerge
+
+:Required parameters:
+
+ :build-node: The node to run build on.
+ :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
+ in defaults.yaml)
+ :project: Git repository name
+ :project-name: Jenkins job name prefix
+
+:Optional parameters:
+
+ :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+ :build-timeout: Timeout in minutes before aborting build. (default: 15)
+
+ :gerrit_merge_triggers: Override Gerrit Triggers.
+ :gerrit_trigger_file_paths: Override file paths filter which checks which
+ file modifications will trigger a build.
+ **default**::
+
+ - compare-type: REG_EXP
+ pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)'
+
+
+Release Verify
+~~~~~~~~~~~~~~
+
+This template supports Maven and Container release jobs.
+
+:Template Name: {project-name}-release-verify
+
+:Comment Trigger: recheck|reverify
+
+:Required Parameters:
+
+ :build-node: The node to run build on.
+ :jenkins-ssh-credential: Credential to use for SSH. (Generally set
+ in defaults.yaml)
+ :project: Git repository name
+ :project-name: Jenkins job name prefix
+
+:Optional Parameters:
+
+ :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+ :build-node: The node to run build on.
+ :build-timeout: Timeout in minutes before aborting build. (default: 15)
+ :gerrit-skip-vote: Skip voting for this job. (default: false)
+ :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+
+ :gerrit_verify_triggers: Override Gerrit Triggers.
+ :gerrit_trigger_file_paths: Override file paths filter which checks which
+ file modifications will trigger a build.
+ **default**::
+
+ - compare-type: REG_EXP
+ pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)'
+
+
+PyPI Release Merge
+~~~~~~~~~~~~~~~~~~
+
+Publishes a Python package on merge of a patch set with a release yaml
+file. Checks the format of the version string, downloads the package
+artifacts from the PyPI staging repository, uploads the package
+artifacts to the PyPI release repository, tags the git repository,
+signs the tag and pushes the tag to the git server. The release verify
+template accepts neither a branch nor a stream parameter.
+
+:Template Names:
+
+ - {project-name}-pypi-release-merge
+ - gerrit-pypi-release-merge
+ - github-pypi-release-merge
+
+:Comment Trigger: remerge
+
+:Required Parameters:
+
+ :build-node: The node to run build on, which must be Centos.
+ :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
+ in defaults.yaml)
+ :project: Git repository name
+ :project-name: Jenkins job name prefix
+
+:Optional Parameters:
+
+ :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+ :build-timeout: Timeout in minutes before aborting build. (default: 15)
+ :disable-job: Whether to disable the job (default: false)
+ :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+ :pypi-stage-index: Base URL of the PyPI staging repository.
+ (default https://test.pypi.org/simple)
+ :pypi-repo: Key for the PyPI release repository in the .pypirc file,
+ should be the repository pypy.org. (default: pypi)
+ :use-release-file: Whether to use the release file. (default: true)
+
+
+PyPI Release Verify
+~~~~~~~~~~~~~~~~~~~
+
+Verifies a Python package project on creation of a patch set with a
+release yaml file. Checks the contents of the release yaml file,
+checks the format of the version string, and downloads the release
+artifacts from the specified PyPI staging repository. The release
+verify template accepts neither a branch nor a stream parameter.
+
+:Template Names:
+
+ - {project-name}-pypi-release-verify
+ - gerrit-pypi-release-verify
+ - github-pypi-release-verify
+
+:Comment Trigger: recheck
+
+:Required Parameters:
+
+ :build-node: The node to run build on, which must be Centos.
+ :jenkins-ssh-credential: Credential to use for SSH. (Generally set
+ in defaults.yaml)
+ :project: Git repository name
+ :project-name: Jenkins job name prefix
+
+:Optional Parameters:
+
+ :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+ :build-timeout: Timeout in minutes before aborting build. (default: 15)
+ :disable-job: Whether to disable the job (default: false)
+ :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+ :pypi-stage-index: Base URL of the PyPI staging repository.
+ (default https://test.pypi.org/simple)
+ :pypi-repo: Key for the PyPI release repository in the .pypirc file,
+ should be the repository pypy.org (default: pypi)
+ :use-release-file: Whether to use the release file. (default: true)
+
+
Setup for LFID, Nexus, Jenkins and Gerrit
-=========================================
+-----------------------------------------
+
+This section is for the Linux Foundation release engineering team.
LFID
-====
+~~~~
Create an ``lfid`` and an ``ssh-key``
Nexus
-=====
+~~~~~
Create a Nexus account called ``'jenkins-release'`` with promote privileges.
.. image:: ../_static/nexus-promote-privs.png
Gerrit
-======
+~~~~~~
Log into your Gerrit with ``YOUR_RELEASE_USERNAME``, upload the public
part of the ``ssh-key`` you created earlier. Log out of Gerrit and log
Jenkins
-=======
+~~~~~~~
Add a global credential to Jenkins called ``jenkins-release`` and set
the ID: ``'jenkins-release'`` as its value insert the private half of
the ``ssh-key`` that you created for your Gerrit user.
-Add Global vars in Jenkins:
-Jenkins configure -> Global properties -> Environment variables
+Add Global variables in Jenkins:
+Jenkins configure -> Global properties -> Environment variables::
-``RELEASE_USERNAME = YOUR_RELEASE_USERNAME``
-``RELEASE_EMAIL = YOUR_RELEASE_EMAIL``
+ RELEASE_USERNAME = YOUR_RELEASE_USERNAME
+ RELEASE_EMAIL = YOUR_RELEASE_EMAIL
.. note::
Jenkins configure -> Managed Files -> Add a New Config -> Custom File
-id: signing-pubkey
-Name: SIGNING_PUBKEY (optional)
-Comment: SIGNING_PUBKEY (optional)
+.. code-block:: none
+
+ id: signing-pubkey
+ Name: SIGNING_PUBKEY (optional)
+ Comment: SIGNING_PUBKEY (optional)
-Content: (Ask Andy for the public signing key)
------BEGIN PGP PUBLIC KEY BLOCK-----
+ Content: (Ask Andy for the public signing key)
+ -----BEGIN PGP PUBLIC KEY BLOCK-----
Add or edit the managed file in Jenkins called ``lftoolsini``,
password=<plaintext password>
Ci-management
-=============
+~~~~~~~~~~~~~
Upgrade your project's global-jjb if needed, then add the following to
your global defaults file (e.g., jjb/defaults.yaml).
.. code-block:: none
- jenkins-ssh-release-credential: 'jenkins-release'
-
-Macros
-======
-
-lf-release
-----------
-
-Release verify and merge jobs are the same except for their scm,
-trigger, and builders definition. This anchor is the common template.
-
-Job Templates
-=============
-
-Release Merge
--------------
-
-:Template Name: {project-name}-release-merge
-
-:Comment Trigger: remerge
-
-:Required parameters:
-
- :build-node: The node to run build on.
- :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
- in defaults.yaml)
- :stream: run this job against: **
-
-:Optional parameters:
-
- :branch: Git branch to fetch for the build. (default: all)
- :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
- :build-timeout: Timeout in minutes before aborting build. (default: 15)
- :project-pattern: Project to trigger build against. (default: \*\*)
-
- :gerrit_merge_triggers: Override Gerrit Triggers.
- :gerrit_trigger_file_paths: Override file paths filter which checks which
- file modifications will trigger a build.
- **default**::
-
- - compare-type: REG_EXP
- pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)'
-
-
-Release Verify
-------------------
-
-:Template Name: {project-name}-release-verify
-
-:Comment Trigger: recheck|reverify
-
-:Required Parameters:
-
- :build-node: The node to run build on.
- :jenkins-ssh-credential: Credential to use for SSH. (Generally set
- in defaults.yaml)
- :stream: run this job against: **
-
-:Optional Parameters:
-
- :branch: Git branch to fetch for the build. (default: all)
- :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
- :build-node: The node to run build on.
- :build-timeout: Timeout in minutes before aborting build. (default: 15)
- :doc-dir: Directory where tox will place built docs.
- as defined in the tox.ini (default: docs/_build/html)
- :gerrit-skip-vote: Skip voting for this job. (default: false)
- :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
- :project-pattern: Project to trigger build against. (default: \*\*)
-
- :gerrit_verify_triggers: Override Gerrit Triggers.
- :gerrit_trigger_file_paths: Override file paths filter which checks which
- file modifications will trigger a build.
- **default**::
-
- - compare-type: REG_EXP
- pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)'
+ jenkins-ssh-release-credential: jenkins-release
- job-group:
name: "{project-name}-gerrit-pypi-jobs"
- # This job group contains all the recommended jobs that should be deployed for
- # a Gerrit-based Python project to test, build and deploy a library to PyPI.
+ # This job group contains the recommended jobs that should be deployed for
+ # a Gerrit-based Python project to test, build and deploy a package.
jobs:
- - gerrit-pypi-verify
- gerrit-pypi-merge
- - gerrit-pypi-release-verify
- - gerrit-pypi-release-merge
+ - gerrit-pypi-verify
- job-group:
name: "{project-name}-github-pypi-jobs"
- # This job group contains all the recommended jobs that should be deployed for
- # a Github-based Python project to test, build and deploy a library to PyPI.
+ # This job group contains the recommended jobs that should be deployed for
+ # a Github-based Python project to test, build and deploy a package.
jobs:
- - github-pypi-verify
- github-pypi-merge
- - github-pypi-release-verify
+ - github-pypi-verify
+
+- job-group:
+ name: "{project-name}-gerrit-pypi-release-jobs"
+
+ # This job group contains the recommended jobs that should be deployed for
+ # a Gerrit-based Python project to promote a package from staging to pypi.
+
+ jobs:
+ - gerrit-pypi-release-merge
+ - gerrit-pypi-release-verify
+
+- job-group:
+ name: "{project-name}-github-pypi-release-jobs"
+
+ # This job group contains the recommended jobs that should be deployed for
+ # a Github-based Python project to promote a package from staging to pypi.
+
+ jobs:
- github-pypi-release-merge
+ - github-pypi-release-verify
properties-content: "CLM_PROJECT_NAME={clm-project-name}"
- shell: !include-raw-escape: ../shell/nexus-iq-cli.sh
-- builder:
- name: lf-infra-pypi-tag-release
- builders:
- - config-file-provider:
- files:
- - file-id: sigul-config
- variable: SIGUL_CONFIG
- - file-id: sigul-password
- variable: SIGUL_PASSWORD
- - file-id: sigul-pki
- variable: SIGUL_PKI
- - file-id: signing-pubkey
- variable: SIGNING_PUBKEY
- - shell: !include-raw: ../shell/sigul-configuration.sh
- - shell: !include-raw: ../shell/sigul-install.sh
- - shell: !include-raw: ../shell/pypi-tag-release.sh
-
-- builder:
- name: lf-infra-pypi-upload
- builders:
- - config-file-provider:
- files:
- - file-id: pypirc
- target: "$HOME/.pypirc"
- - inject:
- properties-content: "REPOSITORY={pypi-repo}"
- - shell: !include-raw-escape: ../shell/pypi-upload.sh
-
- builder:
name: lf-infra-tox-install
builders:
- bool:
name: BUILD_BDIST_WHEEL
default: "{dist-binary}"
- description: "Set to True to build a wheel"
+ description: "Set to True (checked) to build a binary distribution"
- bool:
name: DRY_RUN
default: false
- description: |
- If DRY_RUN is enabled artifacts are not published.
-
- publishers:
- - lf-infra-publish
-
-- lf_pypi_common_wrappers: &lf_pypi_common_wrappers
- name: lf-pypi-common-wrappers
+ description: "Set to True (checked) to skip uploading artifacts"
wrappers:
- lf-infra-wrappers:
build-timeout: "{build-timeout}"
jenkins-ssh-credential: "{jenkins-ssh-credential}"
-- lf_pypi_release_wrappers: &lf_pypi_release_wrappers
- name: lf-pypi-release-wrappers
-
- wrappers:
- - lf-infra-wrappers:
- build-timeout: "{build-timeout}"
- jenkins-ssh-credential: "{jenkins-ssh-release-credential}"
+ publishers:
+ - lf-infra-publish
- lf_pypi_verify_builders: &lf_pypi_verify_builders
name: lf-pypi-verify-builders
- lf-infra-tox-run:
parallel: "{parallel}"
- shell: !include-raw-escape: ../shell/pypi-dist-build.sh
- - lf-infra-pypi-upload:
- pypi-repo: "{pypi-repo}"
-
-- lf_pypi_release_verify_builders: &lf_pypi_release_verify_builders
- name: lf-pypi-release-verify-builders
-
- builders:
- - lf-infra-pre-build
- - lf-infra-tox-install:
- python-version: "{python-version}"
- - shell: "{pre-build-script}"
- - lf-infra-tox-run:
- parallel: "{parallel}"
- - shell: !include-raw-escape: ../shell/pypi-dist-build.sh
- - lf-infra-pypi-tag-release
-
-- lf_pypi_release_merge_builders: &lf_pypi_release_merge_builders
- name: lf-pypi-release-merge-builders
-
- builders:
- - lf-infra-pre-build
- - lf-infra-tox-install:
- python-version: "{python-version}"
- - shell: "{pre-build-script}"
- - lf-infra-tox-run:
- parallel: "{parallel}"
- - shell: !include-raw-escape: ../shell/pypi-dist-build.sh
- - lf-infra-pypi-tag-release
- - lf-infra-pypi-upload:
- pypi-repo: "{pypi-repo}"
+ - config-file-provider:
+ files:
+ - file-id: pypirc
+ target: "$HOME/.pypirc"
+ - inject:
+ properties-content: "REPOSITORY={pypi-repo}"
+ - shell: !include-raw-escape: ../shell/pypi-upload.sh
- job-template:
name: "{project-name}-pypi-verify-{stream}"
id: gerrit-pypi-verify
<<: *lf_pypi_common
- <<: *lf_pypi_common_wrappers
<<: *lf_pypi_verify_builders
scm:
name: "{project-name}-pypi-verify-{stream}"
id: github-pypi-verify
<<: *lf_pypi_common
- <<: *lf_pypi_common_wrappers
<<: *lf_pypi_verify_builders
properties:
name: "{project-name}-pypi-merge-{stream}"
id: gerrit-pypi-merge
<<: *lf_pypi_common
- <<: *lf_pypi_common_wrappers
<<: *lf_pypi_merge_builders
cron: ""
name: "{project-name}-pypi-merge-{stream}"
id: github-pypi-merge
<<: *lf_pypi_common
- <<: *lf_pypi_common_wrappers
<<: *lf_pypi_merge_builders
cron: ""
white-list-target-branches:
- "{branch}"
included-regions: "{obj:github_included_regions}"
-
-- lf_pypi_release_common: &lf_pypi_release_common
- name: lf-pypi-release-common
-
- dist-binary: true
- pypi-repo: pypi
- use-release-file: true
-
- # define once and use twice; jobs MUST NOT override
- gerrit_release_trigger_file_paths:
- - compare-type: REG_EXP
- pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)'
-
- # yamllint disable-line rule:line-length
- # github_release_included_regions MUST match gerrit_release_trigger_file_paths
- github_release_included_regions:
- - 'releases\/.*\.yaml'
- - '.releases\/.*\.yaml'
-
- parameters:
- - lf-infra-parameters:
- project: "{project}"
- branch: "{branch}"
- stream: "{stream}"
- - lf-infra-tox-parameters:
- tox-dir: "{tox-dir}"
- tox-envs: "{tox-envs}"
- - bool:
- name: BUILD_BDIST_WHEEL
- default: "{dist-binary}"
- description: "Set to True to build a wheel"
- - string:
- name: VERSION
- default: ""
- description: "This is the version, example: 1.0.0"
- - bool:
- name: USE_RELEASE_FILE
- default: "{use-release-file}"
- description: "Set to False for job built with parameters"
- - bool:
- name: DRY_RUN
- default: false
- description: |
- If DRY_RUN is enabled artifacts are not published.
-
-- job-template:
- name: "{project-name}-pypi-release-verify-{stream}"
- id: gerrit-pypi-release-verify
- <<: *lf_pypi_common
- <<: *lf_pypi_common_wrappers
- <<: *lf_pypi_release_common
- <<: *lf_pypi_release_verify_builders
-
- scm:
- - lf-infra-gerrit-scm:
- jenkins-ssh-credential: "{jenkins-ssh-credential}"
- git-url: "{git-url}"
- refspec: "$GERRIT_REFSPEC"
- branch: "$GERRIT_BRANCH"
- submodule-recursive: "{submodule-recursive}"
- submodule-timeout: "{submodule-timeout}"
- submodule-disable: "{submodule-disable}"
- choosing-strategy: gerrit
-
- triggers:
- - gerrit:
- server-name: "{gerrit-server-name}"
- trigger-on:
- - patchset-created-event:
- exclude-drafts: true
- exclude-trivial-rebase: false
- exclude-no-code-change: false
- - draft-published-event
- - comment-added-contains-event:
- # yamllint disable-line rule:line-length
- comment-contains-value: '^Patch Set\s+\d+:\s+(recheck|reverify)\s*$'
- projects:
- - project-compare-type: "ANT"
- project-pattern: "{project}"
- branches:
- - branch-compare-type: ANT
- branch-pattern: "**/{branch}"
- file-paths: "{obj:gerrit_release_trigger_file_paths}"
-
-- job-template:
- name: "{project-name}-pypi-release-verify-{stream}"
- id: github-pypi-release-verify
- <<: *lf_pypi_common
- <<: *lf_pypi_common_wrappers
- <<: *lf_pypi_release_common
- <<: *lf_pypi_release_verify_builders
-
- properties:
- - github:
- url: "{github-url}/{github-org}/{project}"
-
- scm:
- - lf-infra-github-scm:
- url: "{git-clone-url}{github-org}/{project}"
- refspec: ""
- branch: "refs/heads/{branch}"
- submodule-recursive: "{submodule-recursive}"
- submodule-timeout: "{submodule-timeout}"
- submodule-disable: "{submodule-disable}"
- choosing-strategy: default
- jenkins-ssh-credential: "{jenkins-ssh-credential}"
-
- triggers:
- - github-pull-request:
- trigger-phrase: "^(recheck|reverify)$"
- only-trigger-phrase: false
- status-context: "PyPI Release Verify"
- permit-all: true
- github-hooks: true
- white-list-target-branches:
- - "{branch}"
- included-regions: "{obj:github_release_included_regions}"
-
-- job-template:
- name: "{project-name}-pypi-release-merge-{stream}"
- id: gerrit-pypi-release-merge
- <<: *lf_pypi_common
- <<: *lf_pypi_release_wrappers
- <<: *lf_pypi_release_common
- <<: *lf_pypi_release_merge_builders
-
- scm:
- - lf-infra-gerrit-scm:
- jenkins-ssh-credential: "{jenkins-ssh-credential}"
- git-url: "{git-url}"
- refspec: "$GERRIT_REFSPEC"
- branch: "$GERRIT_BRANCH"
- submodule-recursive: "{submodule-recursive}"
- submodule-timeout: "{submodule-timeout}"
- submodule-disable: "{submodule-disable}"
- choosing-strategy: gerrit
-
- triggers:
- - gerrit:
- server-name: "{gerrit-server-name}"
- trigger-on:
- - change-merged-event
- - comment-added-contains-event:
- comment-contains-value: '^Patch Set\s+\d+:\s+remerge\s*$'
- projects:
- - project-compare-type: "ANT"
- project-pattern: "{project}"
- branches:
- - branch-compare-type: ANT
- branch-pattern: "**/{branch}"
- file-paths: "{obj:gerrit_release_trigger_file_paths}"
-
-- job-template:
- name: "{project-name}-pypi-release-merge-{stream}"
- id: github-pypi-release-merge
- <<: *lf_pypi_common
- <<: *lf_pypi_release_wrappers
- <<: *lf_pypi_release_common
- <<: *lf_pypi_release_merge_builders
-
- properties:
- - github:
- url: "{github-url}/{github-org}/{project}"
-
- scm:
- - lf-infra-github-scm:
- url: "{git-clone-url}{github-org}/{project}"
- refspec: ""
- branch: "refs/heads/{branch}"
- submodule-recursive: "{submodule-recursive}"
- submodule-timeout: "{submodule-timeout}"
- submodule-disable: "{submodule-disable}"
- choosing-strategy: default
- jenkins-ssh-credential: "{jenkins-ssh-credential}"
-
- triggers:
- - github-pull-request:
- trigger-phrase: "^(remerge)$"
- only-trigger-phrase: false
- status-context: "PyPI Release Merge"
- permit-all: true
- github-hooks: true
- white-list-target-branches:
- - "{branch}"
- included-regions: "{obj:github_release_included_regions}"
exclude-no-code-change: false
- draft-published-event
- comment-added-contains-event:
- comment-contains-value: "^Patch Set[ ]+[0-9]+:([ ]+|[\n]+)(recheck|reverify)$"
+ comment-contains-value: |
+ ^Patch Set[ ]+[0-9]+:([ ]+|[\n]+)(recheck|reverify)$
#####################
# Job Configuration #
file-paths:
- compare-type: REG_EXP
pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)'
+
+################
+# PyPI RELEASE #
+################
+
+- lf_pypi_verify_wrappers: &lf_pypi_verify_wrappers
+ name: lf-pypi-verify-wrappers
+
+ wrappers:
+ - lf-infra-wrappers:
+ build-timeout: "{build-timeout}"
+ jenkins-ssh-credential: "{jenkins-ssh-credential}"
+
+- lf_pypi_release_wrappers: &lf_pypi_release_wrappers
+ name: lf-pypi-release-wrappers
+
+ wrappers:
+ - lf-infra-wrappers:
+ build-timeout: "{build-timeout}"
+ jenkins-ssh-credential: "{jenkins-ssh-release-credential}"
+
+- lf_pypi_release: &lf_pypi_release
+ name: lf-pypi-release
+
+ ######################
+ # Default parameters #
+ ######################
+
+ branch: master # for github
+ build-days-to-keep: 7
+ build-timeout: 15
+ disable-job: false
+ gerrit-skip-vote: false
+ git-url: "$GIT_URL/$PROJECT"
+ github-url: "https://github.com"
+ pypi-repo: pypi
+ pypi-stage-index: https://test.pypi.org/simple
+ submodule-disable: true
+ submodule-recursive: false
+ submodule-timeout: 10
+ use-release-file: true
+
+ # define once and use twice; jobs MUST NOT override
+ gerrit_release_trigger_file_paths:
+ - compare-type: REG_EXP
+ pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)'
+
+ # yamllint disable-line rule:line-length
+ # github_release_included_regions MUST match gerrit_release_trigger_file_paths
+ github_release_included_regions:
+ - 'releases\/.*\.yaml'
+ - '.releases\/.*\.yaml'
+
+ parameters:
+ - lf-infra-parameters:
+ project: "{project}"
+ branch: "$GERRIT_BRANCH"
+ stream: "$GERRIT_BRANCH"
+ # accept all entries defined in the release-yaml file
+ - string:
+ name: DISTRIBUTION_TYPE
+ default: "pypi"
+ description: "The Jenkins release job distribution type."
+ - string:
+ name: LOG_DIR
+ default: ""
+ description: "The partial path of logs from the PyPI merge job."
+ - string:
+ name: PYPI_PROJECT
+ default: ""
+ description: "The PyPI project name."
+ - string:
+ name: PYTHON_VERSION
+ default: ""
+ description: "The Python compatibility version, example: 3.6"
+ - string:
+ name: VERSION
+ default: ""
+ description: "The module version, example: 1.0.0"
+ # special parameters for manual use of the Jenkins job
+ - bool:
+ name: USE_RELEASE_FILE
+ default: true
+ description: "Set to False (unchecked) to build with parameters"
+ - bool:
+ name: DRY_RUN
+ default: false
+ description: "Set to True (checked) to skip uploading artifacts"
+
+ builders:
+ - lf-infra-pre-build
+ - config-file-provider:
+ files:
+ - file-id: sigul-config
+ variable: SIGUL_CONFIG
+ - file-id: sigul-password
+ variable: SIGUL_PASSWORD
+ - file-id: sigul-pki
+ variable: SIGUL_PKI
+ - file-id: signing-pubkey
+ variable: SIGNING_PUBKEY
+ - shell: !include-raw-escape: ../shell/sigul-configuration.sh
+ - shell: !include-raw-escape: ../shell/sigul-install.sh
+ - inject:
+ properties-content: |
+ PYPI_INDEX={pypi-stage-index}
+ REPOSITORY={pypi-repo}
+ - shell: !include-raw-escape: ../shell/release-job.sh
+
+- job-template:
+ name: "{project-name}-pypi-release-merge"
+ id: gerrit-pypi-release-merge
+ <<: *lf_release_common
+ <<: *lf_pypi_release_wrappers
+ <<: *lf_pypi_release
+
+ scm:
+ - lf-infra-gerrit-scm:
+ jenkins-ssh-credential: "{jenkins-ssh-credential}"
+ git-url: "{git-url}"
+ refspec: "$GERRIT_REFSPEC"
+ branch: "$GERRIT_BRANCH"
+ submodule-recursive: "{submodule-recursive}"
+ submodule-timeout: "{submodule-timeout}"
+ submodule-disable: "{submodule-disable}"
+ choosing-strategy: gerrit
+
+ triggers:
+ - gerrit:
+ server-name: "{gerrit-server-name}"
+ trigger-on:
+ - change-merged-event
+ - comment-added-contains-event:
+ comment-contains-value: '^Patch Set\s+\d+:\s+remerge\s*$'
+ projects:
+ - project-compare-type: "ANT"
+ project-pattern: "{project}"
+ branches:
+ - branch-compare-type: ANT
+ branch-pattern: "**"
+ file-paths: "{obj:gerrit_release_trigger_file_paths}"
+
+- job-template:
+ name: "{project-name}-pypi-release-merge"
+ id: github-pypi-release-merge
+ <<: *lf_release_common
+ <<: *lf_pypi_release_wrappers
+ <<: *lf_pypi_release
+
+ properties:
+ - github:
+ url: "{github-url}/{github-org}/{project}"
+
+ scm:
+ - lf-infra-github-scm:
+ url: "{git-clone-url}{github-org}/{project}"
+ refspec: ""
+ branch: "refs/heads/{branch}"
+ submodule-recursive: "{submodule-recursive}"
+ submodule-timeout: "{submodule-timeout}"
+ submodule-disable: "{submodule-disable}"
+ choosing-strategy: default
+ jenkins-ssh-credential: "{jenkins-ssh-credential}"
+
+ triggers:
+ - github-pull-request:
+ trigger-phrase: "^(remerge)$"
+ only-trigger-phrase: false
+ status-context: "PyPI Release Merge"
+ permit-all: true
+ github-hooks: true
+ white-list-target-branches:
+ - "{branch}"
+ included-regions: "{obj:github_release_included_regions}"
+
+- job-template:
+ name: "{project-name}-pypi-release-verify"
+ id: gerrit-pypi-release-verify
+ <<: *lf_release_common
+ <<: *lf_pypi_verify_wrappers
+ <<: *lf_pypi_release
+
+ scm:
+ - lf-infra-gerrit-scm:
+ jenkins-ssh-credential: "{jenkins-ssh-credential}"
+ git-url: "{git-url}"
+ refspec: "$GERRIT_REFSPEC"
+ branch: "$GERRIT_BRANCH"
+ submodule-recursive: "{submodule-recursive}"
+ submodule-timeout: "{submodule-timeout}"
+ submodule-disable: "{submodule-disable}"
+ choosing-strategy: gerrit
+
+ triggers:
+ - gerrit:
+ server-name: "{gerrit-server-name}"
+ trigger-on:
+ - patchset-created-event:
+ exclude-drafts: true
+ exclude-trivial-rebase: false
+ exclude-no-code-change: false
+ - draft-published-event
+ - comment-added-contains-event:
+ # yamllint disable-line rule:line-length
+ comment-contains-value: '^Patch Set\s+\d+:\s+(recheck|reverify)\s*$'
+ projects:
+ - project-compare-type: "ANT"
+ project-pattern: "{project}"
+ branches:
+ - branch-compare-type: ANT
+ branch-pattern: "**"
+ file-paths: "{obj:gerrit_release_trigger_file_paths}"
+
+- job-template:
+ name: "{project-name}-pypi-release-verify"
+ id: github-pypi-release-verify
+ <<: *lf_release_common
+ <<: *lf_pypi_verify_wrappers
+ <<: *lf_pypi_release
+
+ properties:
+ - github:
+ url: "{github-url}/{github-org}/{project}"
+
+ scm:
+ - lf-infra-github-scm:
+ url: "{git-clone-url}{github-org}/{project}"
+ refspec: ""
+ branch: "refs/heads/{branch}"
+ submodule-recursive: "{submodule-recursive}"
+ submodule-timeout: "{submodule-timeout}"
+ submodule-disable: "{submodule-disable}"
+ choosing-strategy: default
+ jenkins-ssh-credential: "{jenkins-ssh-credential}"
+
+ triggers:
+ - github-pull-request:
+ trigger-phrase: "^(recheck|reverify)$"
+ only-trigger-phrase: false
+ status-context: "PyPI Release Verify"
+ permit-all: true
+ github-hooks: true
+ white-list-target-branches:
+ - "{branch}"
+ included-regions: "{obj:github_release_included_regions}"
--- /dev/null
+---
+features:
+ - |
+ Refactor PyPI release-verify and release-merge templates to download
+ distribution files from a PyPI staging index and upload the files to a
+ PyPI release index. Remove all the builders that were previously used.
+ Call pip upgrade to install latest version of setuptools and twine.
+ Split the PyPI job groups because the verify & merge templates do not
+ accept the same arguments as release-verify & release-merge templates.
+ Remove stream, branch usage; just one PyPI release job per project now.
+ Extend the PyPI release yaml file schema for log_dir and other values.
+ All this makes the PyPI release ver/mrg templates highly similar to the
+ release-job ver/mrg templates. Revise documentation appropriately.
+ Move the PyPI release features to the lf-release-jobs.(rst,yaml) files.
+ Extend the release-job.sh script to have pypi release functions, and
+ drop the pypi-tag-release.sh script.
required:
- "distribution_type"
- - "project"
+ - "log_dir"
+ - "pypi_project"
+ - "python_version"
- "version"
properties:
distribution_type:
type: "string"
- project:
+ log_dir:
+ type: "string"
+ pypi_project:
+ type: "string"
+ python_version:
type: "string"
version:
type: "string"
# Ensure we fail the job if any steps fail.
set -eu -o pipefail
+echo "INFO: creating virtual environment"
virtualenv -p python3 /tmp/pypi
PATH=/tmp/pypi/bin:$PATH
-
-echo "INFO: installing twine to check distributions"
-pip install -q twine
+pipup="python -m pip install -q --upgrade setuptools twine wheel"
+echo "INFO: $pipup"
+$pipup
bdist=""
if $BUILD_BDIST_WHEEL; then
- echo "INFO: installing wheel to build binary distribution"
- pip install -q wheel
+ echo "INFO: adding wheel to build binary distribution"
bdist="bdist_wheel"
fi
+++ /dev/null
-#!/bin/bash
-# SPDX-License-Identifier: EPL-1.0
-##############################################################################
-# Copyright (c) 2019 The Linux Foundation and others.
-#
-# All rights reserved. This program and the accompanying materials
-# are made available under the terms of the Eclipse Public License v1.0
-# which accompanies this distribution, and is available at
-# http://www.eclipse.org/legal/epl-v10.html
-##############################################################################
-echo "---> pypi-tag-release.sh"
-
-# Ensure we fail the job if any steps fail.
-set -eu -o pipefail
-
-# Functions.
-
-set_variables(){
- echo "INFO: Setting variables"
- # Verify if using release file or parameters
- if $USE_RELEASE_FILE; then
- echo "INFO: Checking number of release yaml files"
- release_files=$(git diff-tree --no-commit-id -r "$GIT_COMMIT" --name-only -- "releases/" ".releases/")
- if (( $(echo "$release_files" | wc -w) != 1 )); then
- echo "ERROR: RELEASE FILES: $release_files"
- echo "ERROR: Committing multiple release files in the same commit OR rename/amend of existing files is not supported."
- exit 1
- else
- release_file="$release_files"
- echo "INFO: RELEASE FILE: $release_file"
- fi
- else
- echo "INFO: This job is built with parameters, no release file"
- release_file="None"
- fi
-
- if [[ -z ${DISTRIBUTION_TYPE:-} ]]; then
- echo "INFO: reading DISTRIBUTION_TYPE from file $release_file"
- DISTRIBUTION_TYPE="$(niet ".distribution_type" "$release_file")"
- fi
- if [[ -z ${VERSION:-} ]]; then
- echo "INFO: reading VERSION from file $release_file"
- VERSION="$(niet ".version" "$release_file")"
- fi
-
- # Display Release Information
- printf "\t%-30s\n" RELEASE_ENVIRONMENT_INFO:
- printf "\t%-30s %s\n" RELEASE_FILE: $release_file
- printf "\t%-30s %s\n" JENKINS_HOSTNAME: $JENKINS_HOSTNAME
- printf "\t%-30s %s\n" SILO: $SILO
- printf "\t%-30s %s\n" PROJECT: $PROJECT
- printf "\t%-30s %s\n" PROJECT-DASHED: ${PROJECT//\//-}
- printf "\t%-30s %s\n" DISTRIBUTION_TYPE: $DISTRIBUTION_TYPE
- printf "\t%-30s %s\n" VERSION: $VERSION
-}
-
-# needs to run in the repository root
-verify_schema(){
- echo "INFO: Fetching schema"
- pypi_schema="release-pypi-schema.yaml"
- wget https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/${pypi_schema}
- echo "INFO: Verifying $release_file against schema $pypi_schema"
- lftools schema verify "$release_file" "$pypi_schema"
- echo "INFO: $release_file passed schema verification"
-}
-
-verify_version(){
- # Verify allowed patterns "v#.#.#" or "#.#.#" aka SemVer
- echo "INFO: Verifying version string $VERSION"
- allowed_version_regex="^((v?)([0-9]+)\.([0-9]+)\.([0-9]+))$"
- if [[ $VERSION =~ $allowed_version_regex ]]; then
- echo "INFO: The version $VERSION is a valid semantic version"
- else
- echo "ERROR: The version $VERSION is not a valid semantic version"
- echo "ERROR: Allowed versions are \"v#.#.#\" or \"#.#.#\" aka SemVer"
- echo "ERROR: See https://semver.org/ for more details on SemVer"
- exit 1
- fi
-}
-
-verify_dist(){
- # Verify all file names in dist folder have the expected version string
- dir="$WORKSPACE/$TOX_DIR/dist"
- echo "INFO: Listing files in $dir"
- ls $dir
- echo "INFO: Checking files in $dir for $VERSION"
- if unex_files=$(find $dir | grep -v $VERSION | egrep -v "^$dir$"); then
- echo "ERROR: found unexpected files: $unex_files"
- exit 1
- else
- echo "INFO: All file names have expected string ${VERSION}"
- fi
-}
-
-# sigul is only available on Centos
-tag_gerrit(){
- echo "INFO: Verifying tag $VERSION in repo"
- # Import public signing key
- gpg --import "$SIGNING_PUBKEY"
- if git tag -v "$VERSION"; then
- echo "INFO: Repo already tagged"
- return 0
- fi
- echo "INFO: Tagging repo"
- git tag -am "${PROJECT//\//-} $VERSION" "$VERSION"
- echo "INFO: Signing tag"
- sigul --batch -c "$SIGUL_CONFIG" sign-git-tag "$SIGUL_KEY" "$VERSION" < "$SIGUL_PASSWORD"
- echo "INFO: Verifying tag"
- # may fail due to missing public key
- if ! git tag -v "$VERSION"; then
- echo "WARN: failed to verify tag, continuing anyhow"
- fi
- # The verify job also calls this script
- if [[ ! $JOB_NAME =~ "merge" ]] ; then
- echo "INFO: job is not a merge, skipping push"
- else
- echo "INFO: configuring Gerrit remote"
- gerrit_ssh=$(echo "$GERRIT_URL" | awk -F"/" '{print $3}')
- git remote set-url origin "ssh://$RELEASE_USERNAME@$gerrit_ssh:29418/$PROJECT"
- git config user.name "$RELEASE_USERNAME"
- git config user.email "$RELEASE_EMAIL"
- echo -e "Host $gerrit_ssh\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config
- chmod 600 ~/.ssh/config
- if $DRY_RUN; then
- echo "INFO: dry run, skipping push"
- else
- echo "INFO: pushing tag"
- git push origin "$VERSION"
- fi
- fi
-}
-
-# Main
-virtualenv -p python3 /tmp/pypi
-PATH=/tmp/pypi/bin:$PATH
-pip install lftools jsonschema niet
-set_variables
-if [[ $DISTRIBUTION_TYPE != "pypi" ]]; then
- echo "ERROR: unexpected distribution type $DISTRIBUTION_TYPE"
- exit 1
-fi
-if $USE_RELEASE_FILE; then
- verify_schema
-fi
-verify_version
-verify_dist
-# TODO: write tag_github function
-tag_gerrit
-echo "---> pypi-tag-release.sh ends"
# Ensure we fail the job if any steps fail.
set -eu -o pipefail
+echo "INFO: creating virtual environment"
virtualenv -p python3 /tmp/pypi
PATH=/tmp/pypi/bin:$PATH
-
-echo "INFO: installing twine to upload distributions"
-pip install -q twine
+pipup="python -m pip install -q --upgrade twine"
+echo "INFO: $pipup"
+$pipup
echo "INFO: cd to tox-dir $TOX_DIR"
cd "$WORKSPACE/$TOX_DIR"
cmd="twine upload -r $REPOSITORY dist/*"
if $DRY_RUN; then
echo "INFO: dry-run is set, echoing command only"
- echo $cmd
+ echo "$cmd"
else
echo "INFO: uploading distributions to repo $REPOSITORY"
$cmd
+ # emit message and files on single line for release-job
+ # shellcheck disable=SC2046
+ echo "INFO: successfully uploaded distributions: " $(ls dist)
fi
echo "---> pypi-upload.sh ends"
echo "---> release-job.sh"
set -eu -o pipefail
-set +u
-python3 -m venv /tmp/v/venv/
-# shellcheck disable=SC1091
-source /tmp/v/venv/bin/activate
-set -u
-python -m pip install lftools[nexus] jsonschema niet yq
+echo "INFO: creating virtual environment"
+virtualenv -p python3 /tmp/venv
+PATH=/tmp/venv/bin:$PATH
+pipup="python -m pip install -q --upgrade pip lftools[nexus] jsonschema niet twine yq"
+echo "INFO: $pipup"
+$pipup
#Functions.
NEXUS_PATH="${SILO}/${JENKINS_HOSTNAME}/"
# Verify if using release file or parameters
if $USE_RELEASE_FILE ; then
- release_files=$(git diff-tree --no-commit-id -r "$GERRIT_PATCHSET_REVISION" --name-only -- "releases/" ".releases/")
+ release_files=$(git diff-tree --no-commit-id -r "$GIT_COMMIT" --name-only -- "releases/" ".releases/")
if (( $(grep -c . <<<"$release_files") > 1 )); then
echo "INFO: RELEASE FILES ARE AS FOLLOWS: $release_files"
echo "ERROR: Committing multiple release files in the same commit OR rename/amend of existing files is not supported."
printf "\t%-30s %s\n" GERRIT_REF_TO_TAG: $ref
}
+set_variables_pypi(){
+ # use Jenkins parameter if set; else get value from release file
+ echo "INFO: Setting pypi variables"
+ LOG_DIR="${LOG_DIR:-None}"
+ if [[ $LOG_DIR == "None" ]]; then
+ LOG_DIR="$(yq -er .log_dir "$release_file")"
+ fi
+ LOGS_URL="${LOGS_SERVER}/${NEXUS_PATH}${LOG_DIR}"
+ LOGS_URL=${LOGS_URL%/} # strip any trailing '/'
+ PYPI_PROJECT="${PYPI_PROJECT:-None}"
+ if [[ $PYPI_PROJECT == "None" ]]; then
+ PYPI_PROJECT="$(yq -er .pypi_project "$release_file")"
+ fi
+ PYTHON_VERSION="${PYTHON_VERSION:-None}"
+ if [[ $PYTHON_VERSION == "None" ]]; then
+ PYTHON_VERSION="$(yq -er .python_version "$release_file")"
+ fi
+ VERSION="${VERSION:-None}"
+ if [[ $VERSION == "None" ]]; then
+ VERSION="$(yq -er .version "$release_file")"
+ fi
+
+ # Continuing displaying Release Information (pypi)
+ printf "\t%-30s\n" RELEASE_PYPI_INFO:
+ printf "\t%-30s %s\n" LOG_DIR: "$LOG_DIR"
+ printf "\t%-30s %s\n" LOGS_URL: "$LOGS_URL"
+ printf "\t%-30s %s\n" PYPI_INDEX: "$PYPI_INDEX" # from job configuration
+ printf "\t%-30s %s\n" PYPI_PROJECT: "$PYPI_PROJECT"
+ printf "\t%-30s %s\n" PYTHON_VERSION: "$PYTHON_VERSION"
+ printf "\t%-30s %s\n" VERSION: "$VERSION"
+}
+
verify_schema(){
echo "INFO: Verifying $release_file schema."
lftools schema verify "$release_file" "$RELEASE_SCHEMA"
}
verify_version(){
- # Verify allowed versions
- # Allowed versions are "v#.#.#" or "#.#.#" aka SemVer
+ # Verify allowed patterns "v#.#.#" or "#.#.#" aka SemVer
+ echo "INFO: Verifying version string $VERSION"
allowed_version_regex="^((v?)([0-9]+)\.([0-9]+)\.([0-9]+))$"
- if [[ ! $VERSION =~ $allowed_version_regex ]]; then
+ if [[ $VERSION =~ $allowed_version_regex ]]; then
+ echo "INFO: The version $VERSION is a valid semantic version"
+ else
echo "INFO: The version $VERSION is not a semantic valid version"
echo "INFO: Allowed versions are \"v#.#.#\" or \"#.#.#\" aka SemVer"
echo "INFO: See https://semver.org/ for more details on SemVer"
fi
}
+# check prerequisites to detect mistakes in the release YAML file
+verify_pypi_match_release(){
+ wget -q -P /tmp "${LOGS_URL}/"console.log.gz
+ echo "INFO: Searching for strings >$PYPI_PROJECT< and >$VERSION< in job log"
+ # pypi-upload.sh generates success message with file list
+ if zgrep -i "uploaded" /tmp/console.log.gz | grep "$PYPI_PROJECT" | grep "$VERSION" ; then
+ echo "INFO: found expected strings in job log"
+ else
+ echo "ERROR: failed to find expected strings in job log"
+ exit 1
+ fi
+}
+
+# sigul is only available on Centos
+# TODO: write tag_github function
tag(){
# Import public signing key
gpg --import "$SIGNING_PUBKEY"
tag
}
-echo "########### Start Script release-job.sh ###################################"
+# calls pip to download binary and source distributions from the specified index,
+# which requires a recent-in-2019 version. Uploads the files it received.
+pypi_release_file(){
+ tgtdir=dist
+ mkdir $tgtdir
+ pip_pfx="pip download -d $tgtdir --no-deps --python-version $PYTHON_VERSION -i $PYPI_INDEX"
+ module="$PYPI_PROJECT==$VERSION"
+ pip_bin="$pip_pfx $module"
+ echo "INFO: downloading binary: $pip_bin"
+ if ! $pip_bin ; then
+ echo "WARN: failed to download binary distribution"
+ fi
+ pip_src="$pip_pfx --no-binary=:all: $module"
+ echo "INFO: downloading source: $pip_src"
+ if ! $pip_src ; then
+ echo "WARN: failed to download source distribution"
+ fi
+ echo "INFO: Checking files in $tgtdir"
+ filecount=$(ls $tgtdir | wc -l)
+ if [[ $filecount = 0 ]] ; then
+ echo "ERROR: no files downloaded"
+ exit 1
+ else
+ # shellcheck disable=SC2046
+ echo "INFO: downloaded $filecount distributions: " $(ls $tgtdir)
+ fi
-# Check if this is a container or maven release: release-container-schema.yaml vs release-schema.yaml
-# Logic to determine what we are releasing.
-##########################################
+ if [[ ! "$JOB_NAME" =~ "merge" ]] ; then
+ echo "INFO: not a merge job, not uploading files"
+ return
+ fi
+
+ cmd="twine upload -r $REPOSITORY $tgtdir/*"
+ if $DRY_RUN; then
+ echo "INFO: dry-run is set, echoing command only"
+ echo "$cmd"
+ else
+ echo "INFO: uploading $filecount distributions to repo $REPOSITORY"
+ $cmd
+ fi
+ tag
+}
# Set common environment variables
set_variables_common
+# Determine the type of release:
+# - container, release-container-schema.yaml
+# - maven, release-schema.yaml
+# - pypi, release-pypi-schema.yaml
+
if [[ "$DISTRIBUTION_TYPE" == "maven" ]]; then
- wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/release-schema.yaml
- RELEASE_SCHEMA="release-schema.yaml"
if $USE_RELEASE_FILE ; then
+ RELEASE_SCHEMA="release-schema.yaml"
+ echo "INFO: Fetching schema $RELEASE_SCHEMA"
+ wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/release-schema.yaml
verify_schema
fi
set_variables_maven
verify_version_match_release
maven_release_file
elif [[ "$DISTRIBUTION_TYPE" == "container" ]]; then
- wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/release-container-schema.yaml
- RELEASE_SCHEMA="release-container-schema.yaml"
- verify_schema
+ if $USE_RELEASE_FILE ; then
+ RELEASE_SCHEMA="release-container-schema.yaml"
+ echo "INFO: Fetching schema $RELEASE_SCHEMA"
+ wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/${RELEASE_SCHEMA}
+ verify_schema
+ fi
set_variables_container
verify_version
container_release_file
+elif [[ "$DISTRIBUTION_TYPE" == "pypi" ]]; then
+ if $USE_RELEASE_FILE ; then
+ RELEASE_SCHEMA="release-pypi-schema.yaml"
+ echo "INFO: Fetching schema $RELEASE_SCHEMA"
+ wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/${RELEASE_SCHEMA}
+ verify_schema
+ fi
+ set_variables_pypi
+ verify_version
+ verify_pypi_match_release
+ pypi_release_file
else
echo "ERROR: distribution_type: $DISTRIBUTION_TYPE not supported"
- echo "Must be maven or container"
exit 1
fi
-##########################################
-echo "########### End Script release-job.sh ###################################"
+echo "---> release-job.sh ends"
Code Review / releng / global-jjb.git / commitdiff