Macros
======
-lf-infra-clm-python
--------------------
+lf-infra-nexus-iq-python-cli
+----------------------------
-Runs CLM scanning against a Python project.
+Runs Nexus IQ command-line interface CLM scan on Python package requirements.
:Required Parameters:
- :clm-project-name: Project name in Nexus IQ to send results to.
-
-lf-infra-pypi-tag-release
--------------------------
-
-Checks the format of the release version string and checks the git
-repository for that tag. In a merge job, if the tag does not exist,
-adds the tag to the repository, signs the tag, and pushes the tag
-to the git server. Signing requires sigul, which is only available
-on a CentOS build node.
-
-lf-infra-pypi-upload
---------------------
-
-Uploads distribution files from subdirectory "dist" to a PyPI repository
-using a Python virtual enviroment to install required packages. The
-Jenkins server must have a configuration file ".pypirc".
-
-:Required Parameters:
-
- :pypi-repo: PyPI repository key in .pypirc configuration file;
- e.g., "pypi-test" or "pypi".
+ :nexus-iq-project-name: Project name in Nexus IQ to send results to.
+ :requirements-file: File name with output of pip freeze.
lf-infra-tox-install
--------------------
:Required Parameters:
- :parallel: Boolean. If true use detox (distributed tox);
- else use regular tox.
+ :parallel: If different from false, try pass this parameter to tox option
+ "--parallel" to parallelize jobs in the envlist (and then activate the
+ option "--parallel-live" to display output in logs).
+ Possible values are "auto" (equivalent to "true" for legacy),
+ "all" or any integer. Any other value is equivalent to "false".
Job Templates
=============
-Python XC CLM
--------------
+Tox Nexus IQ CLM
+----------------
+
+The Nexus IQ job invokes tox and the Nexus IQ scanner to analyze packages for
+component lifecycle management (CLM). Runs tox to discover the required packages,
+downloads the command-line interface (CLI) scanner, runs the scanner on the package
+list, then uploads the results to a Nexus IQ server. The project's tox.ini file must
+define a test environment that runs 'pip freeze' and captures the output; that
+environment does not need to execute any tests. For example:
+
+.. code-block:: bash
+
+ [testenv:clm]
+ # use pip to report dependencies with versions
+ whitelist_externals = sh
+ commands = sh -c 'pip freeze > requirements.txt'
-CLM scans for Python based repos. This job will call the Nexus IQ CLI
-directly to run the scans.
-A new credential named "nexus-iq-xc-clm" needs to exist in the Jenkins
-credentials. The credential should contain the username and password
-to access Nexus IQ Server.
+This job runs on the master branch because the basic Nexus IQ configuration
+does not support multi-branch.
:Template Names:
- - {project-name}-python-clm-{stream}
- - gerrit-python-xc-clm
- - github-python-xc-clm
+ - {project-name}-tox-nexus-iq-clm
+ - gerrit-tox-nexus-iq-clm
+ - github-tox-nexus-iq-clm
-:Comment Trigger: run-clm
+:Comment Trigger: ``run-clm``
:Required parameters:
- :build-node: The node to run build on.
- :jenkins-ssh-credential: Credential to use for SSH. (Generally should
- get configured in defaults.yaml)
+ :build-node: The node to run the build on.
+ (Commonly in defaults.yaml)
+ :jenkins-ssh-credential: Credential to use for SSH.
+ (Commonly in defaults.yaml)
+ :project: The git repository name.
+ :project-name: Prefix used to name jobs.
-:Optional parameters:
+:Optional Parameters:
+ :archive-artifacts: Pattern for files to archive to the logs server
+ (default: '\*\*/\*.log')
+ :branch: Git branch, should be master (default: master)
:build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+ :build-timeout: Timeout in minutes before aborting build. (default: 15)
+ :cron: Cron schedule when to trigger the job. This parameter also
+ supports multiline input via the YAML pipe | character to allow
+ more than 1 cron timer. (default: @weekly)
+ :disable-job: Whether to disable the job (default: false)
+ :gerrit_nexusiq_triggers: Override Gerrit Triggers.
+ :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+ :github-url: URL for Github. (default: https://github.com)
+ :java-version: Version of Java to use for the scan. (default: openjdk11)
:nexus-iq-cli-version: Nexus IQ CLI package version to download and use.
- (default: 1.44.0-01)
+ (default is a string like 1.89.0-02, see file lf-python-jobs.yaml)
:nexus-iq-namespace: Insert a namespace to project AppID for projects that
share a Nexus IQ system to avoid project name collision. We recommend
inserting a trailing - dash if using this parameter.
For example 'odl-'. (default: '')
+ :nexus-target-build: Target directory or file for scanning by Nexus IQ CLI
+ (default: "\*\*/\*")
+ :pre-build-script: Shell script to run before tox. Useful for setting up
+ dependencies. (default: a string with a shell comment)
+ :python-version: Python version to invoke pip install of tox-pyenv
+ (default: python3)
+ :requirements-file: Name of file with output of pip freeze.
+ (default: requirements.txt)
+ :submodule-recursive: Whether to checkout submodules recursively.
+ (default: true)
+ :submodule-timeout: Timeout (in minutes) for checkout operation.
+ (default: 10)
+ :submodule-disable: Disable submodule checkout operation.
+ (default: false)
+ :tox-dir: Directory containing the project's tox.ini relative to
+ the workspace. The default uses tox.ini at the project root.
+ (default: '.')
+ :tox-envs: Tox environment with the appropriate pip freeze invocation.
+ (default: 'clm')
+
+Python Snyk CLI
+---------------
+
+Builds the code, downloads and runs a Snyk CLI scan of the code into the Snyk dashboard.
+
+:Template Names:
+
+ - {project-name}-python-snyk-cli-{stream}
+ - gerrit-python-snyk-cli
+ - github-python-snyk-cli
+
+:Comment Trigger: run-snyk
+
+:Required parameters:
+
+ :build-node: The node to run build on.
+ :jenkins-ssh-credential: Credential to use for SSH. (Generally configured in defaults.yaml)
+ :snyk-token-credential-id: Snyk API token to communicate with Jenkins.
+ :snyk-org-credential-id: Snyk organization ID.
+
+:Optional parameters:
+
+ :branch: The branch to build against. (default: master)
+ :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
:build-timeout: Timeout in minutes before aborting build. (default: 60)
:git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
- :java-version: Version of Java to use for the build. (default: openjdk8)
- :pre-build-script: Shell script to execute before the CLM builder.
+ :pre-build-script: Shell script to execute before the Tox builder.
For example, install prerequisites or move files to the repo root.
(default: a string with a shell comment)
+ :parallel: If different from false, try pass this parameter to tox option
+ "--parallel" to parallelize jobs in the envlist (and then activate the
+ option "--parallel-live" to display output in logs).
+ Possible values are "auto" (equivalent to "true" for legacy),
+ "all" or any integer. Any other value is equivalent to "false".
+ (default: false, in series)
+ :python-version: Python version to invoke pip install of tox-pyenv
+ (default: python3)
+ :snyk-cli-options: Additional Snyk CLI options. (default: '')
+ :stream: Keyword representing a release code-name.
+ Often the same as the branch. (default: master)
+ :submodule-recursive: Whether to checkout submodules recursively.
+ (default: true)
+ :submodule-timeout: Timeout (in minutes) for checkout operation.
+ (default: 10)
+ :submodule-disable: Disable submodule checkout operation.
+ (default: false)
+ :tox-dir: Directory containing the project's tox.ini relative to
+ the workspace. The default uses tox.ini at the project root.
+ (default: '.')
+ :tox-envs: Tox environments to run. If blank run everything described
+ in tox.ini. (default: '')
+ :gerrit_trigger_file_paths: Override file paths used to filter which file
+ modifications trigger a build. Refer to JJB documentation for "file-path" details.
+ https://jenkins-job-builder.readthedocs.io/en/latest/triggers.html#triggers.gerrit
+
+Python Sonar with CLI
+---------------------
+
+Sonar scans for non Maven based repos. This job downloads the CLI
+and runs a scan to publish the report to SonarCloud.
+
+As suggested in SonarCoud's UI instructions, the job downloads and unzips
+the Sonar SLI and executes a sonar-scanner command to process the
+report.
+
+For more details refer to sonar documentation:
+
+https://docs.sonarcloud.io/advanced-setup/ci-based-analysis/sonarscanner-cli/
+
+:Template Names:
+
+ - {project-name}-cli-sonar
+ - gerrit-cli-sonar
+ - github-cli-sonar
+
+:Comment Trigger: **run-sonar** post a comment with the trigger to launch
+ this job manually. Do not include any other text or vote in the
+ same comment.
+
+:Required parameters:
+
+ :build-node: The node to run build on.
+ :jenkins-ssh-credential: Credential to use for SSH. (Generally should
+ get configured in defaults.yaml)
+ :mvn-settings: The name of the settings file with credentials for the project.
+
+.. comment Start ignoring WriteGoodLintBear
+
+:Optional parameters:
+
+ :branch: Git branch, should be master (default: master)
+ :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+ :build-timeout: Timeout in minutes before aborting build. (default: 60)
+ :cron: Cron schedule when to trigger the job. This parameter also
+ supports multiline input via YAML pipe | character in cases where
+ one may want to provide more than 1 cron timer. (default: H 11 * * *
+ to run once a day)
+ :disable-job: Whether to disable the job (default: false)
+ :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+ :github-url: URL for Github. (default: https://github.com)
+ :java-version: Version of Java to use for the build. (default: openjdk11)
+ :mvn-global-settings: The name of the Maven global settings to use
+ :mvn-goals: The Maven goal to run first. (default: validate)
+ :mvn-version: Version of maven to use. (default: mvn35)
+ :parallel: If different from false, try pass this parameter to tox option
+ "--parallel" to parallelize jobs in the envlist (and then activate the
+ option "--parallel-live" to display output in logs).
+ Possible values are "auto" (equivalent to "true" for legacy),
+ "all" or any integer. Any other value is equivalent to "false".
+ (default: false, in series)
+ :pre-build-script: Shell script to execute before the Sonar builder.
+ For example, install prerequisites or move files to the repo root.
+ (default: a string with a shell comment)
+ :python-version: Python version to invoke pip install of tox-pyenv
+ (default: python2)
+ :sonarcloud-project-key: SonarCloud project key. (default: '')
+ :sonarcloud-project-organization: SonarCloud project organization.
+ (default: '')
+ :sonarcloud-api-token-cred-id: Jenkins credential ID which has the SonarCloud API Token.
+ This one SHOULDN'T be overwritten as per we are standarizing the credential ID for all
+ projects (default: 'sonarcloud-api-token')
+ :sonar-scanner-home: Sonar scanner home directory.
+ (default: $WORKSPACE/.sonar/sonar-scanner-$SONAR_SCANNER_VERSION-linux)
+ :sonar-scanner-opts: Sonar scanner Java options. (default: '-server')
+ :sonar-scanner-version: Version of sonar scanner to use. (default: 4.7.0.2747)
:stream: Keyword used to represent a release code-name.
Often the same as the branch. (default: master)
:submodule-recursive: Whether to checkout submodules recursively.
(default: 10)
:submodule-disable: Disable submodule checkout operation.
(default: false)
- :gerrit_clm_triggers: Override Gerrit Triggers.
+ :tox-dir: Directory containing the project's tox.ini relative to
+ the workspace. The default uses tox.ini at the project root.
+ (default: '.')
+ :tox-envs: Tox environments to run. If blank run everything described
+ in tox.ini. (default: '')
+ :gerrit_sonar_triggers: Override Gerrit Triggers.
:gerrit_trigger_file_paths: Override file paths used to filter which file
modifications trigger a build. Refer to JJB documentation for "file-path" details.
- https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+ https://jenkins-job-builder.readthedocs.io/en/latest/triggers.html#triggers.gerrit
+.. comment Stop ignoring
Python Sonar with Tox
---------------------
Sonar scans for Python based repos. This job invokes tox to run tests
and gather coverage statistics from the test results, then invokes
-Maven to publish the results to a Sonar server.
+Maven to publish the results to either a Sonar server or SonarCloud.
+
+**Deprecated**, new projects should use Tox Sonarqube.
To get the Sonar coverage results, file tox.ini must exist and contain
coverage commands to run.
suites. Checking coverage does not guarantee that the tests execute
properly, but it identifies code that is not executed by any test.
-This job reuses the Sonar builder used in Java/Maven projects which
-runs maven twice. The first invocation does nothing for Python
-projects, so the job uses the goal 'validate' by default. The second
-invocation publishes results using the goal 'sonar:sonar' by default.
+This job reuses the Sonar builders used for Java/Maven projects which
+run maven twice. The first invocation does nothing for Python
+projects, so the job uses the goal ``validate`` by default. The second
+invocation publishes results using the goal ``sonar:sonar`` by default.
For example:
.. code-block:: bash
- [testenv:py27]
+ [testenv:py3]
commands =
coverage run --module pytest --junitxml xunit-results.xml
- coverage xml --omit=".tox/py27/*","tests/*"
- coverage report --omit=".tox/py27/*","tests/*"
+ coverage xml --omit=".tox/py3/*","tests/*"
+ coverage report --omit=".tox/py3/*","tests/*"
For more details refer to coverage and sonar documentation:
- gerrit-tox-sonar
- github-tox-sonar
-:Comment Trigger: run-sonar
+:Comment Trigger: **run-sonar** post a comment with the trigger to launch
+ this job manually. Do not include any other text or vote in the
+ same comment.
:Required parameters:
get configured in defaults.yaml)
:mvn-settings: The name of the settings file with credentials for the project.
+.. comment Start ignoring WriteGoodLintBear
+
:Optional parameters:
:branch: Git branch, should be master (default: master)
:disable-job: Whether to disable the job (default: false)
:git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
:github-url: URL for Github. (default: https://github.com)
- :java-version: Version of Java to use for the build. (default: openjdk8)
- :mvn-global-settings: The name of the Maven global settings to use for
+ :java-version: Version of Java to use for the build. (default: openjdk11)
+ :mvn-global-settings: The name of the Maven global settings to use
:mvn-goals: The Maven goal to run first. (default: validate)
:mvn-version: Version of maven to use. (default: mvn35)
- :parallel: Boolean indicator for tox to run tests in parallel or series.
- (default: false, in series)
+ :parallel: If different from false, try pass this parameter to tox option
+ "--parallel" to parallelize jobs in the envlist (and then activate the
+ option "--parallel-live" to display output in logs).
+ Possible values are "auto" (equivalent to "true" for legacy),
+ "all" or any integer. Any other value is equivalent to "false".
+ (default: false, in series)
:pre-build-script: Shell script to execute before the Sonar builder.
For example, install prerequisites or move files to the repo root.
(default: a string with a shell comment)
:python-version: Python version to invoke pip install of tox-pyenv
(default: python2)
+ :sonarcloud: Boolean indicator to use SonarCloud ``true|false``.
+ (default: false)
+ :sonarcloud-project-key: SonarCloud project key. (default: '')
+ :sonarcloud-project-organization: SonarCloud project organization.
+ (default: '')
+ :sonarcloud-api-token-cred-id: Jenkins credential ID which has the SonarCloud API Token.
+ This one SHOULDN'T be overwritten as per we are standarizing the credential ID for all
+ projects (default: 'sonarcloud-api-token')
:sonar-mvn-goal: The Maven goal to run the Sonar plugin. (default: sonar:sonar)
:stream: Keyword used to represent a release code-name.
Often the same as the branch. (default: master)
(default: 10)
:submodule-disable: Disable submodule checkout operation.
(default: false)
+ :tox-dir: Directory containing the project's tox.ini relative to
+ the workspace. The default uses tox.ini at the project root.
+ (default: '.')
+ :tox-envs: Tox environments to run. If blank run everything described
+ in tox.ini. (default: '')
:gerrit_sonar_triggers: Override Gerrit Triggers.
:gerrit_trigger_file_paths: Override file paths used to filter which file
modifications trigger a build. Refer to JJB documentation for "file-path" details.
- https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+ https://jenkins-job-builder.readthedocs.io/en/latest/triggers.html#triggers.gerrit
+
+.. comment Stop ignoring
+
+
+Tox SonarQube
+-------------
+
+The SonarQube job invokes tox to run tests and generate code-coverage
+statistics, then runs the SonarQube Scanner Jenkins plug-in to analyze
+code, gather coverage data, and upload the results to a SonarQube server
+such as SonarCloud.io. Optionally runs a shell script before tox.
+
+Requires ``SonarQube Scanner for Jenkins``
+
+This job runs on the master branch because the basic Sonar configuration
+does not support multi-branch.
+
+Plug-in configurations
+ Manage Jenkins --> Configure System --> SonarQube servers
+ - Name: Sonar (fixed)
+ - Server URL: https://sonar.project.org/ or https://sonarcloud.io
+ - Server authentication token: none for local, API token (saved as
+ a "secret text" credential) for Sonarcloud
+
+ Manage Jenkins --> Global Tool Configuration --> SonarQube Scanner
+ - Name: SonarQube Scanner (fixed)
+ - Install automatically
+ - Select latest version
+
+:Template Names:
+
+ - {project-name}-tox-sonarqube
+ - gerrit-tox-sonarqube
+ - github-tox-sonarqube
+
+:Comment Trigger: ``run-sonar``
+
+:Required parameters:
+
+ :build-node: The node to run the build on.
+ (Commonly in defaults.yaml)
+ :jenkins-ssh-credential: Credential to use for SSH.
+ (Commonly in defaults.yaml)
+ :project: The git repository name.
+ :project-name: Prefix used to name jobs.
+
+.. comment Start ignoring WriteGoodLintBear
+
+:Optional Parameters:
+
+ :archive-artifacts: Pattern for files to archive to the logs server
+ (default: '\*\*/\*.log')
+ :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+ :build-timeout: Timeout in minutes before aborting build. (default: 15)
+ :cron: Cron schedule when to trigger the job. This parameter also
+ supports multiline input via YAML pipe | character in cases where
+ one may want to provide more than 1 cron timer. (default: @weekly)
+ :disable-job: Whether to disable the job (default: false)
+ :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+ :github-url: URL for Github. (default: https://github.com)
+ :parallel: If different from false, try pass this parameter to tox option
+ "--parallel" to parallelize jobs in the envlist (and then activate the
+ option "--parallel-live" to display output in logs).
+ Possible values are "auto" (equivalent to "true" for legacy),
+ "all" or any integer. Any other value is equivalent to "false".
+ (default: false, in series)
+ :pre-build-script: Shell script to run before tox. Useful for setting up
+ dependencies. (default: a string with a shell comment)
+ :python-version: Python version to invoke pip install of tox-pyenv
+ (default: python3)
+ :sonar-additional-args: Command line arguments. (default: '')
+ :sonar-java-opts: JVM options. For example, use option -Xmx
+ to increase the memory size limit. (default: '')
+ :sonar-jdk: JDK version to use. (default: openjdk17)
+ :sonar-project-file: The file name with Sonar configuration properties
+ (default: sonar-project.properties)
+ :sonar-properties: Sonar configuration properties. (default: '')
+ :sonar-task: Sonar task to run. (default: '')
+ :tox-dir: Directory containing the project's tox.ini relative to
+ the workspace. The default uses tox.ini at the project root.
+ (default: '.')
+ :tox-envs: Tox environments to run. If blank run everything described
+ in tox.ini. (default: '')
+
+.. comment Stop ignoring
+
+.. note:: A job definition must provide one of the optional parameters
+ ``sonar-project-file`` and ``sonar-properties``; they cannot both be
+ empty. Set Sonar properties directly in the job definition by setting
+ the ``sonar-project-file`` property to ``""`` and adding all properties
+ under ``sonar-properties``.
+
+:Required Sonar Properties:
+
+ - sonar.login: The API token for authentication at SonarCloud.
+ Commonly defined as key "sonarcloud_api_token" in defaults.yaml.
+ - sonar.organization: The umbrella project name; e.g., "opendaylight".
+ Commonly defined as key "sonarcloud_project_organization" in defaults.yaml.
+ - sonar.projectName: The git repository name without slashes; e.g., "infrautils".
+ - sonar.projectKey: The globally unique key for the report in SonarCloud. Most
+ teams use the catenation of sonar.organization, an underscore, and
+ sonar.projectName; e.g., "opendaylight_infrautils".
+
+:Optional Sonar Properties:
+
+ - sonar.cfamily.gcov.reportsPath: directory with GCOV output files
+ - Documentation of SonarQube properties is here:
+ https://docs.sonarqube.org/latest/analysis/overview/
+
+
+Example job definition
+^^^^^^^^^^^^^^^^^^^^^^
+
+The following example defines a job for a basic Python project. This definition
+uses configuration parameters in the umbrella project's defaults.yaml file.
+
+.. code-block:: yaml
+
+ - project:
+ name: my-package-sonar
+ project: my/package
+ project-name: my-package
+ sonar-project-file: ""
+ sonar-properties: |
+ sonar.login={sonarcloud_api_token}
+ sonar.projectKey={sonarcloud_project_organization}_{project-name}
+ sonar.projectName={project-name}
+ sonar.organization={sonarcloud_project_organization}
+ sonar.sourceEncoding=UTF-8
+ sonar.sources=mypackage
+ sonar.exclusions=tests/*,setup.py
+ sonar.python.coverage.reportPaths=coverage.xml
+ jobs:
+ - gerrit-tox-sonarqube
Tox Verify
- gerrit-tox-verify
- github-tox-verify
-:Comment Trigger: recheck|reverify
+:Comment Trigger: **recheck|reverify** post a comment with one of the
+ triggers to launch this job manually. Do not include any other
+ text or vote in the same comment.
:Required Parameters:
:pre-build-script: Shell script to execute before the Tox builder.
For example, install prerequisites or move files to the repo root.
(default: a string with a shell comment)
- :parallel: Boolean indicator for tox to run tests in parallel or series.
- (default: false, in series)
+ :parallel: If different from false, try pass this parameter to tox option
+ "--parallel" to parallelize jobs in the envlist (and then activate the
+ option "--parallel-live" to display output in logs).
+ Possible values are "auto" (equivalent to "true" for legacy),
+ "all" or any integer. Any other value is equivalent to "false".
+ (default: false, in series)
:python-version: Python version to invoke pip install of tox-pyenv
(default: python2)
:stream: Keyword representing a release code-name.
in tox.ini. (default: '')
:gerrit_trigger_file_paths: Override file paths used to filter which file
modifications trigger a build. Refer to JJB documentation for "file-path" details.
- https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+ https://jenkins-job-builder.readthedocs.io/en/latest/triggers.html#triggers.gerrit
Tox Merge
- gerrit-tox-merge
- github-tox-merge
-:Comment Trigger: remerge
+:Comment Trigger: **remerge** post a comment with the trigger to launch
+ this job manually. Do not include any other text or vote in the
+ same comment.
:Required Parameters:
:git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
:pre-build-script: Shell script to execute before the CLM builder.
For example, install prerequisites or move files to the repo root.
- (default: a string with only a comment)
+ (default: a string with a shell comment)
:python-version: Python version to invoke pip install of tox-pyenv
(default: python2)
:stream: Keyword representing a release code-name.
in tox.ini. (default: '')
:gerrit_trigger_file_paths: Override file paths used to filter which file
modifications trigger a build. Refer to JJB documentation for "file-path" details.
- https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+ https://jenkins-job-builder.readthedocs.io/en/latest/triggers.html#triggers.gerrit
-PyPI Verify
------------
+PyPI Merge
+----------
-Verifies a Python library project on creation of a patch set. Runs tox
-then builds a source distribution and (optionally) a binary
-distribution. The project repository must have a setup.py file with
-configuration for packaging the component.
+Creates and uploads package distribution files on merge of a patch set.
+Runs tox, builds a source distribution and (optionally) a binary
+distribution, and uploads the distribution(s) to a PyPI repository.
+The project git repository must have a setup.py file
+with configuration for packaging the component.
+
+Projects can choose **either** this template to publish on merge,
+**or** the Stage template to publish on command.
+
+This job should use a staging repository like testpypi.python.org,
+which sets up use of release jobs to promote the distributions later.
+This job can also use a public release area like the global PyPI
+repository if the release process is not needed. These PyPI
+repositories allow upload of a package at a specific version once,
+they do not allow overwrite of a package. This means that a merge job
+will fail in the upload step if the package version already exists in
+the target repository.
The tox runner is pyenv aware so if the image contains an installation
of pyenv at /opt/pyenv it will pick it up and run Python tests with
export PYENV_ROOT="/opt/pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
+See the recommended directory layout documented in the PyPI Verify job.
+
+Jobs using this PyPI template depend on a .pypirc configuration file
+in the Jenkins builder home directory. An example appears next that uses
+API tokens. Note that in the [pypi] entry the repository key-value pair
+is optional, it defaults to pypi.org.
+
+.. code-block:: bash
+
+ [distutils] # this tells distutils what package indexes you can push to
+ index-servers = pypi-test pypi
+
+ [pypi-test]
+ repository: https://test.pypi.org/legacy/
+ username: __token__
+ password: pypi-test-api-token-goes-here
+
+ [pypi]
+ username: __token__
+ password: pypi-api-token-goes-here
+
+
:Template Names:
- - {project-name}-pypi-verify-{stream}
- - gerrit-pypi-verify
- - github-pypi-verify
+ - {project-name}-pypi-merge-{stream}
+ - gerrit-pypi-merge
+ - github-pypi-merge
-:Comment Trigger: recheck
+:Comment Trigger: **remerge** post a comment with the trigger to launch
+ this job manually. Do not include any other text or vote in the
+ same comment.
:Required Parameters:
:build-node: The node to run the build on.
:jenkins-ssh-credential: Credential to use for SSH. (Generally set
in defaults.yaml)
+ :mvn-settings: The settings file with credentials for the project
+ :project: Git repository name
+ :project-name: Jenkins job name prefix
:Optional Parameters:
:branch: The branch to build against. (default: master)
:build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
:build-timeout: Timeout in minutes before aborting build. (default: 15)
+ :cron: Cron schedule when to trigger the job. Supports regular builds.
+ Not useful when publishing to pypi.org because that rejects a package
+ if the version exists. (default: empty)
:disable-job: Whether to disable the job (default: false)
:dist-binary: Whether to build a binary wheel distribution. (default: true)
:git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
- :parallel: Boolean indicator for tox to run tests in parallel or series.
- (default: false, in series)
+ :mvn-opts: Sets MAVEN_OPTS to start up the JVM running Maven. (default: '')
+ :mvn-params: Parameters to pass to the mvn CLI. (default: '')
+ :mvn-version: Version of maven to use. (default: mvn35)
+ :parallel: If different from false, try pass this parameter to tox option
+ "--parallel" to parallelize jobs in the envlist (and then activate the
+ option "--parallel-live" to display output in logs).
+ Possible values are "auto" (equivalent to "true" for legacy),
+ "all" or any integer. Any other value is equivalent to "false".
+ (default: false, in series)
:pre-build-script: Shell script to execute before the tox builder. For
example, install system prerequisites. (default: a shell comment)
+ :pypi-repo: Key for the PyPI target repository in the .pypirc file,
+ ideally a server like test.pypi.org. (default: pypi-test)
:python-version: Python version to invoke pip install of tox-pyenv
(default: python3)
:stream: Keyword representing a release code-name.
in tox.ini. (default: '')
:gerrit_trigger_file_paths: Override file paths used to filter which file
modifications trigger a build. Refer to JJB documentation for "file-path" details.
- https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+ https://jenkins-job-builder.readthedocs.io/en/latest/triggers.html#triggers.gerrit
-PyPI Merge
+PyPI Stage
----------
-Creates and uploads distribution files on merge of a patch set. Runs
-tox, builds a source distribution and (optionally) a binary
+Creates and uploads package distribution files on receipt of a comment.
+Runs tox, builds a source distribution and (optionally) a binary
distribution, and uploads the distribution(s) to a PyPI repository.
-This job should be configured to use a test PyPI repository like
-testpypi.python.org, not a public release area like the global PyPI
-repository. Like the verify job, this requires a setup.py file for
-packaging the component.
+The project git repository must have a setup.py file with configuration
+for packaging the component.
+
+Projects can choose **either** this template to publish on command,
+**or** the Merge template to publish on merge.
+
+This job should use a staging repository like testpypi.python.org,
+which sets up use of release jobs to promote the distributions later.
+This job can also use a public release area like the global PyPI
+repository if the release process is not needed. These PyPI
+repositories allow upload of a package at a specific version once,
+they do not allow overwrite of a package. This means that a job
+will fail in the upload step if the package version already exists in
+the target repository.
The tox runner is pyenv aware so if the image contains an installation
of pyenv at /opt/pyenv it will pick it up and run Python tests with
export PYENV_ROOT="/opt/pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
+See the recommended directory layout documented in the PyPI Verify job.
-Requires a .pypirc configuration file in the Jenkins builder home directory,
-an example appears next that uses API tokens. No repository is needed in the
-PyPI section.
+Jobs using this PyPI template depend on a .pypirc configuration file
+in the Jenkins builder home directory. An example appears next that uses
+API tokens. Note that in the [pypi] entry the repository key-value pair
+is optional, it defaults to pypi.org.
.. code-block:: bash
:Template Names:
- - {project-name}-pypi-merge-{stream}
- - gerrit-pypi-merge
- - github-pypi-merge
+ - {project-name}-pypi-stage-{stream}
+ - gerrit-pypi-stage
+ - github-pypi-stage
-:Comment Trigger: pypi-remerge
+:Comment Trigger: **stage-release** post a comment with the trigger to launch
+ this job manually. Do not include any other text or vote in the
+ same comment.
:Required Parameters:
:build-node: The node to run the build on.
:jenkins-ssh-credential: Credential to use for SSH. (Generally set
in defaults.yaml)
+ :mvn-settings: The settings file with credentials for the project
+ :project: Git repository name
+ :project-name: Jenkins job name prefix
:Optional Parameters:
:branch: The branch to build against. (default: master)
:build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
:build-timeout: Timeout in minutes before aborting build. (default: 15)
- :cron: Cron schedule when to trigger the job. Supports daily builds.
- This parameter also supports multiline input via YAML pipe | character in
- cases where one may want to provide more than 1 cron timer. (default: empty)
+ :cron: Cron schedule when to trigger the job. Supports regular builds.
+ Not useful when publishing to pypi.org because that rejects a package
+ if the version exists. (default: empty)
:disable-job: Whether to disable the job (default: false)
:dist-binary: Whether to build a binary wheel distribution. (default: true)
:git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
- :parallel: Boolean indicator for tox to run tests in parallel or series.
- (default: false, in series)
+ :mvn-opts: Sets MAVEN_OPTS to start up the JVM running Maven. (default: '')
+ :mvn-params: Parameters to pass to the mvn CLI. (default: '')
+ :mvn-version: Version of maven to use. (default: mvn35)
+ :parallel: If different from false, try pass this parameter to tox option
+ "--parallel" to parallelize jobs in the envlist (and then activate the
+ option "--parallel-live" to display output in logs).
+ Possible values are "auto" (equivalent to "true" for legacy),
+ "all" or any integer. Any other value is equivalent to "false".
+ (default: false, in series)
:pre-build-script: Shell script to execute before the tox builder. For
example, install system prerequisites. (default: a shell comment)
- :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
- Merge jobs should use a server like testpypi.python.org. (default: pypi-test)
+ :pypi-repo: Key for the PyPI target repository in the .pypirc file,
+ ideally a server like test.pypi.org. (default: pypi-test)
:python-version: Python version to invoke pip install of tox-pyenv
(default: python3)
:stream: Keyword representing a release code-name.
in tox.ini. (default: '')
:gerrit_trigger_file_paths: Override file paths used to filter which file
modifications trigger a build. Refer to JJB documentation for "file-path" details.
- https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
-
-
-PyPI Release Verify
--------------------
-
-Verifies a Python library project on creation of a patch set with a
-release yaml file. Runs tox, builds source and (optionally) binary
-distributions, checks the format of the version string, checks that
-the distribution file names contain the release version string, and
-checks if the tag exists in the code repository for the release
-version.
+ https://jenkins-job-builder.readthedocs.io/en/latest/triggers.html#triggers.gerrit
-To initiate the release process, create a releases/ or .releases/
-directory at the root of the project repository, add one release yaml
-file to it, and submit a change set with that release yaml file. A
-schema and and an example for the release yaml file appear below. The
-version in the release yaml file must be a valid Semantic Versioning
-(SemVer) string, matching either the pattern "v#.#.#" or "#.#.#" where
-"#" is one or more digits.
-
-This job is similar to the PyPI verify job, but is only triggered by a
-patch set with a release yaml file.
-
-The build node for PyPI release verify jobs must be CentOS, which
-supports the sigul client for accessing a signing server.
-
-.. note::
-
- The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml).
- In words, the directory name can be ".releases" or "releases"; the file
- name can be anything with suffix ".yaml".
-
-The JSON schema for a pypi release file appears below.
+PyPI Verify
+-----------
-.. code-block:: none
+Verifies a Python library project on creation of a patch set. Runs tox
+then builds a source distribution and (optionally) a binary
+distribution. The project repository must have a setup.py file with
+configuration for packaging the component.
- ---
- $schema: "http://json-schema.org/schema#"
- $id: "https://github.com/lfit/releng-global-jjb/blob/master/release-pypi-schema.yaml"
+Installable package projects should use the directory layout shown
+below. All Python files are in a repo subdirectory separate from
+non-Python files like documentation. This layout allows highly
+specific build-job triggers in Jenkins using the subdirectory
+paths. For example, a PyPI publisher job should not run on a non-Python
+file change such as documentation, because the job cannot upload the
+same package twice.
- required:
- - "distribution_type"
- - "project"
- - "version"
+To make the document files available for building a Python package
+long description in setup.py, add a symbolic link "docs" in the
+package subdirectory pointing to the top-level docs directory.
- properties:
- distribution_type:
- type: "string"
- project:
- type: "string"
- version:
- type: "string"
+.. code-block:: bash
+ git-repo-name/
+ │
+ ├── docs/
+ │ ├── index.rst
+ │ └── release-notes.rst
+ │
+ ├── helloworld-package/
+ │ │
+ │ └── helloworld/
+ │ │ ├── __init__.py
+ │ │ ├── helloworld.py
+ │ │ └── helpers.py
+ │ │
+ │ ├── tests/
+ │ │ ├── helloworld_tests.py
+ │ │ └── helloworld_mocks.py
+ │ │
+ │ ├── requirements.txt
+ │ └── setup.py
+ │ └── tox.ini
+ │
+ ├── releases/
+ │ └── pypi-helloworld.yaml
+ │
+ ├── .gitignore
+ ├── LICENSE
+ └── README.md
-An example of a pypi release file appears below.
-.. code-block:: none
+The tox runner is pyenv aware so if the image contains an installation
+of pyenv at /opt/pyenv it will pick it up and run Python tests with
+the appropriate Python versions. The tox runner sets the following
+pyenv variables before running.
- $ cat releases/1.0.0-pypi.yaml
- ---
- distribution_type: pypi
- version: 1.0.0
- project: 'example-project'
+.. code:: bash
+ export PYENV_ROOT="/opt/pyenv"
+ export PATH="$PYENV_ROOT/bin:$PATH"
:Template Names:
- - {project-name}-pypi-release-verify-{stream}
- - gerrit-pypi-release-verify
- - github-pypi-release-verify
-
-:Required Parameters:
-
- :build-node: The node to run build on, which must be Centos.
- :jenkins-ssh-credential: Credential to use for SSH. (Generally set
- in defaults.yaml)
-
-:Optional Parameters:
-
- :branch: The branch to build against. (default: master)
- :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
- :build-timeout: Timeout in minutes before aborting build. (default: 15)
- :disable-job: Whether to disable the job (default: false)
- :dist-binary: Whether to build a binary wheel distribution. (default: true)
- :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
- :parallel: Boolean indicator for tox to run tests in parallel or series.
- (default: false, in series)
- :pre-build-script: Shell script to execute before the tox builder.
- For example, install prerequisites or move files to the repo root.
- (default: a string with a shell comment)
- :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
- Release jobs should use a server like pypy.org. (default: pypi)
- :python-version: Python version to invoke pip install of tox-pyenv
- (default: python3)
- :stream: Keyword representing a release code-name.
- Often the same as the branch. (default: master)
- :submodule-recursive: Whether to checkout submodules recursively.
- (default: true)
- :submodule-timeout: Timeout (in minutes) for checkout operation.
- (default: 10)
- :submodule-disable: Disable submodule checkout operation.
- (default: false)
- :tox-dir: Directory containing the project's tox.ini relative to
- the workspace. The default uses tox.ini at the project root.
- (default: '.')
- :tox-envs: Tox environments to run. If blank run everything described
- in tox.ini. (default: '')
- :use-release-file: Whether to use the release file. (default: true)
-
-PyPI Release Merge
-------------------
-
-Publishes a Python library on merge of a patch set with a release yaml
-file. Runs tox, builds source and (optionally) binary distributions,
-checks the format of the version string, checks that the distribution
-file names contain the release version string, checks if the tag
-exists in the code repository for the release version, then if the tag
-does not exist, tags the code repository with the release version,
-signs the tag and pushes the tag to the git server. Finally this
-uploads the distributions to a PyPI repository.
-
-This job is similar to the PyPI merge job, but is only triggered by
-merge of a release yaml file, also this checks the version and tag
-before uploading to a public repository such as PyPI.
-
-See the PyPI Release Verify job above for documentation of the release
-yaml file format.
-
-The build node for PyPI release merge jobs must be CentOS, which
-supports the sigul client for accessing a signing server.
-
-A Jenkins user can also trigger this release job via the "Build with
-parameters" action, removing the need to merge a release yaml file.
-The user must enter parameters in the same way as a release yaml file,
-except for the special USE_RELEASE_FILE and DRY_RUN check boxes. The
-user must uncheck the USE_RELEASE_FILE check box if the job should run
-with a release file, while passing the required information as build
-parameters. Similarly, the user must uncheck the DRY_RUN check box to
-test the job while skipping upload of files to a repository.
-
-The special parameters are as follows::
-
- VERSION = 1.0.0
- USE_RELEASE_FILE = false
- DRY_RUN = false
-
-:Template Names:
+ - {project-name}-pypi-verify-{stream}
+ - gerrit-pypi-verify
+ - github-pypi-verify
- - {project-name}-pypi-release-merge-{stream}
- - gerrit-pypi-release-merge
- - github-pypi-release-merge
+:Comment Trigger: **recheck|reverify** post a comment with one of the
+ triggers to launch this job manually. Do not include any other
+ text or vote in the same comment.
:Required Parameters:
- :build-node: The node to run build on, which must be Centos.
+ :build-node: The node to run the build on.
:jenkins-ssh-credential: Credential to use for SSH. (Generally set
in defaults.yaml)
+ :mvn-settings: The settings file with credentials for the project
+ :project: Git repository name
+ :project-name: Jenkins job name prefix
:Optional Parameters:
:disable-job: Whether to disable the job (default: false)
:dist-binary: Whether to build a binary wheel distribution. (default: true)
:git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
- :parallel: Boolean indicator for tox to run tests in parallel or series.
- (default: false, in series)
- :pre-build-script: Shell script to execute before the tox builder.
- For example, install prerequisites or move files to the repo root.
- (default: a string with a shell comment)
- :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
- Release jobs should use a server like pypy.org. (default: pypi)
+ :mvn-opts: Sets MAVEN_OPTS to start up the JVM running Maven. (default: '')
+ :mvn-params: Parameters to pass to the mvn CLI. (default: '')
+ :mvn-version: Version of maven to use. (default: mvn35)
+ :parallel: If different from false, try pass this parameter to tox option
+ "--parallel" to parallelize jobs in the envlist (and then activate the
+ option "--parallel-live" to display output in logs).
+ Possible values are "auto" (equivalent to "true" for legacy),
+ "all" or any integer. Any other value is equivalent to "false".
+ (default: false, in series)
+ :pre-build-script: Shell script to execute before the tox builder. For
+ example, install system prerequisites. (default: a shell comment)
:python-version: Python version to invoke pip install of tox-pyenv
(default: python3)
:stream: Keyword representing a release code-name.
(default: '.')
:tox-envs: Tox environments to run. If blank run everything described
in tox.ini. (default: '')
- :use-release-file: Whether to use the release file. (default: true)
+ :gerrit_trigger_file_paths: Override file paths used to filter which file
+ modifications trigger a build. Refer to JJB documentation for "file-path" details.
+ https://jenkins-job-builder.readthedocs.io/en/latest/triggers.html#triggers.gerrit
Code Review / releng / global-jjb.git / blobdiff