8 .. include:: ../job-groups.rst
10 Below is a list of Python job groups:
12 .. literalinclude:: ../../jjb/lf-python-job-groups.yaml
22 Runs CLM scanning against a Python project.
26 :clm-project-name: Project name in Nexus IQ to send results to.
28 lf-infra-pypi-tag-release
29 -------------------------
31 Checks the format of the release version string and checks the git
32 repository for that tag. In a merge job, if the tag does not exist,
33 adds the tag to the repository, signs the tag, and pushes the tag
34 to the git server. Signing requires sigul, which is only available
35 on a CentOS build node.
40 Uploads distribution files from subdirectory "dist" to a PyPI repository
41 using a Python virtual enviroment to install required packages. The
42 Jenkins server must have a configuration file ".pypirc".
46 :pypi-repo: PyPI repository key in .pypirc configuration file;
47 e.g., "pypi-test" or "pypi".
52 Installs Tox into a virtualenv.
56 :python-version: Version of Python to invoke the pip install of the tox-pyenv
57 package that creates a virtual environment, either "python2" or "python3".
62 Creates a Tox virtual environment and invokes tox.
66 :parallel: Boolean. If true use detox (distributed tox);
76 CLM scans for Python based repos. This job will call the Nexus IQ CLI
77 directly to run the scans.
79 A new credential named "nexus-iq-xc-clm" needs to exist in the Jenkins
80 credentials. The credential should contain the username and password
81 to access Nexus IQ Server.
85 - {project-name}-python-clm-{stream}
86 - gerrit-python-xc-clm
87 - github-python-xc-clm
89 :Comment Trigger: run-clm
93 :build-node: The node to run build on.
94 :jenkins-ssh-credential: Credential to use for SSH. (Generally should
95 get configured in defaults.yaml)
99 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
100 :nexus-iq-cli-version: Nexus IQ CLI package version to download and use.
102 :nexus-iq-namespace: Insert a namespace to project AppID for projects that
103 share a Nexus IQ system to avoid project name collision. We recommend
104 inserting a trailing - dash if using this parameter.
105 For example 'odl-'. (default: '')
106 :build-timeout: Timeout in minutes before aborting build. (default: 60)
107 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
108 :java-version: Version of Java to use for the build. (default: openjdk8)
109 :pre-build-script: Shell script to execute before the CLM builder.
110 For example, install prerequisites or move files to the repo root.
111 (default: a string with a shell comment)
112 :stream: Keyword used to represent a release code-name.
113 Often the same as the branch. (default: master)
114 :submodule-recursive: Whether to checkout submodules recursively.
116 :submodule-timeout: Timeout (in minutes) for checkout operation.
118 :submodule-disable: Disable submodule checkout operation.
120 :gerrit_clm_triggers: Override Gerrit Triggers.
121 :gerrit_trigger_file_paths: Override file paths used to filter which file
122 modifications trigger a build. Refer to JJB documentation for "file-path" details.
123 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
126 Python Sonar with Tox
127 ---------------------
129 Sonar scans for Python based repos. This job invokes tox to run tests
130 and gather coverage statistics from the test results, then invokes
131 Maven to publish the results to either a Sonar server or SonarCloud.
133 To get the Sonar coverage results, file tox.ini must exist and contain
134 coverage commands to run.
136 The coverage commands define the code that gets executed by the test
137 suites. Checking coverage does not guarantee that the tests execute
138 properly, but it identifies code that is not executed by any test.
140 This job reuses the Sonar builders used for Java/Maven projects which
141 run maven twice. The first invocation does nothing for Python
142 projects, so the job uses the goal 'validate' by default. The second
143 invocation publishes results using the goal 'sonar:sonar' by default.
151 coverage run --module pytest --junitxml xunit-results.xml
152 coverage xml --omit=".tox/py27/*","tests/*"
153 coverage report --omit=".tox/py27/*","tests/*"
155 For more details refer to coverage and sonar documentation:
157 https://coverage.readthedocs.io/
159 https://docs.sonarqube.org/display/PLUG/Python+Coverage+Results+Import
163 - {project-name}-tox-sonar
167 :Comment Trigger: run-sonar
169 :Required parameters:
171 :build-node: The node to run build on.
172 :jenkins-ssh-credential: Credential to use for SSH. (Generally should
173 get configured in defaults.yaml)
174 :mvn-settings: The name of the settings file with credentials for the project.
176 :Optional parameters:
178 :branch: Git branch, should be master (default: master)
179 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
180 :build-timeout: Timeout in minutes before aborting build. (default: 60)
181 :cron: Cron schedule when to trigger the job. This parameter also
182 supports multiline input via YAML pipe | character in cases where
183 one may want to provide more than 1 cron timer. (default: H 11 * * *
185 :disable-job: Whether to disable the job (default: false)
186 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
187 :github-url: URL for Github. (default: https://github.com)
188 :java-version: Version of Java to use for the build. (default: openjdk8)
189 :mvn-global-settings: The name of the Maven global settings to use for
190 :mvn-goals: The Maven goal to run first. (default: validate)
191 :mvn-version: Version of maven to use. (default: mvn35)
192 :parallel: Boolean indicator for tox to run tests in parallel or series.
193 (default: false, in series)
194 :pre-build-script: Shell script to execute before the Sonar builder.
195 For example, install prerequisites or move files to the repo root.
196 (default: a string with a shell comment)
197 :python-version: Python version to invoke pip install of tox-pyenv
199 :sonarcloud: Whether or not to use SonarCloud ``true|false``.
201 :sonarcloud-project-key: SonarCloud project key. (default: '')
202 :sonarcloud-project-organization: SonarCloud project organization.
204 :sonarcloud-api-token: SonarCloud API Token. (default: '')
205 :sonar-mvn-goal: The Maven goal to run the Sonar plugin. (default: sonar:sonar)
206 :stream: Keyword used to represent a release code-name.
207 Often the same as the branch. (default: master)
208 :submodule-recursive: Whether to checkout submodules recursively.
210 :submodule-timeout: Timeout (in minutes) for checkout operation.
212 :submodule-disable: Disable submodule checkout operation.
214 :tox-dir: Directory containing the project's tox.ini relative to
215 the workspace. The default uses tox.ini at the project root.
217 :tox-envs: Tox environments to run. If blank run everything described
218 in tox.ini. (default: '')
219 :gerrit_sonar_triggers: Override Gerrit Triggers.
220 :gerrit_trigger_file_paths: Override file paths used to filter which file
221 modifications trigger a build. Refer to JJB documentation for "file-path" details.
222 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
228 Tox runner to verify a project on creation of a patch set. This job
229 is pyenv aware so if the image contains an installation of pyenv at
230 /opt/pyenv it will pick it up and run Python tests with the
231 appropriate Python versions. This job will set the following pyenv
232 variables before running.
236 export PYENV_ROOT="/opt/pyenv"
237 export PATH="$PYENV_ROOT/bin:$PATH"
241 - {project-name}-tox-verify-{stream}
245 :Comment Trigger: recheck|reverify
247 :Required Parameters:
249 :build-node: The node to run build on.
250 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
253 :Optional Parameters:
255 :branch: The branch to build against. (default: master)
256 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
257 :build-timeout: Timeout in minutes before aborting build. (default: 10)
258 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
259 :pre-build-script: Shell script to execute before the Tox builder.
260 For example, install prerequisites or move files to the repo root.
261 (default: a string with a shell comment)
262 :parallel: Boolean indicator for tox to run tests in parallel or series.
263 (default: false, in series)
264 :python-version: Python version to invoke pip install of tox-pyenv
266 :stream: Keyword representing a release code-name.
267 Often the same as the branch. (default: master)
268 :submodule-recursive: Whether to checkout submodules recursively.
270 :submodule-timeout: Timeout (in minutes) for checkout operation.
272 :submodule-disable: Disable submodule checkout operation.
274 :tox-dir: Directory containing the project's tox.ini relative to
275 the workspace. The default uses tox.ini at the project root.
277 :tox-envs: Tox environments to run. If blank run everything described
278 in tox.ini. (default: '')
279 :gerrit_trigger_file_paths: Override file paths used to filter which file
280 modifications trigger a build. Refer to JJB documentation for "file-path" details.
281 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
287 Tox runner to verify a project after merge of a patch set. This job
288 is pyenv aware so if the image contains an installation of pyenv at
289 /opt/pyenv it will pick it up and run Python tests with the
290 appropriate Python versions. This job will set the following pyenv
291 variables before running.
295 export PYENV_ROOT="/opt/pyenv"
296 export PATH="$PYENV_ROOT/bin:$PATH"
300 - {project-name}-tox-merge-{stream}
304 :Comment Trigger: remerge
306 :Required Parameters:
308 :build-node: The node to run build on.
309 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
312 :Optional Parameters:
314 :branch: The branch to build against. (default: master)
315 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
316 :build-timeout: Timeout in minutes before aborting build. (default: 10)
317 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
318 :pre-build-script: Shell script to execute before the CLM builder.
319 For example, install prerequisites or move files to the repo root.
320 (default: a string with only a comment)
321 :python-version: Python version to invoke pip install of tox-pyenv
323 :stream: Keyword representing a release code-name.
324 Often the same as the branch. (default: master)
325 :submodule-recursive: Whether to checkout submodules recursively.
327 :submodule-timeout: Timeout (in minutes) for checkout operation.
329 :submodule-disable: Disable submodule checkout operation.
331 :tox-dir: Directory containing the project's tox.ini relative to
332 the workspace. The default uses tox.ini at the project root.
334 :tox-envs: Tox environments to run. If blank run everything described
335 in tox.ini. (default: '')
336 :gerrit_trigger_file_paths: Override file paths used to filter which file
337 modifications trigger a build. Refer to JJB documentation for "file-path" details.
338 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
344 Verifies a Python library project on creation of a patch set. Runs tox
345 then builds a source distribution and (optionally) a binary
346 distribution. The project repository must have a setup.py file with
347 configuration for packaging the component.
349 The tox runner is pyenv aware so if the image contains an installation
350 of pyenv at /opt/pyenv it will pick it up and run Python tests with
351 the appropriate Python versions. The tox runner sets the following
352 pyenv variables before running.
356 export PYENV_ROOT="/opt/pyenv"
357 export PATH="$PYENV_ROOT/bin:$PATH"
361 - {project-name}-pypi-verify-{stream}
365 :Comment Trigger: recheck
367 :Required Parameters:
369 :build-node: The node to run the build on.
370 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
373 :Optional Parameters:
375 :branch: The branch to build against. (default: master)
376 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
377 :build-timeout: Timeout in minutes before aborting build. (default: 15)
378 :disable-job: Whether to disable the job (default: false)
379 :dist-binary: Whether to build a binary wheel distribution. (default: true)
380 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
381 :parallel: Boolean indicator for tox to run tests in parallel or series.
382 (default: false, in series)
383 :pre-build-script: Shell script to execute before the tox builder. For
384 example, install system prerequisites. (default: a shell comment)
385 :python-version: Python version to invoke pip install of tox-pyenv
387 :stream: Keyword representing a release code-name.
388 Often the same as the branch. (default: master)
389 :submodule-recursive: Whether to checkout submodules recursively.
391 :submodule-timeout: Timeout (in minutes) for checkout operation.
393 :submodule-disable: Disable submodule checkout operation.
395 :tox-dir: Directory containing the project's tox.ini relative to
396 the workspace. The default uses tox.ini at the project root.
398 :tox-envs: Tox environments to run. If blank run everything described
399 in tox.ini. (default: '')
400 :gerrit_trigger_file_paths: Override file paths used to filter which file
401 modifications trigger a build. Refer to JJB documentation for "file-path" details.
402 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
408 Creates and uploads distribution files on merge of a patch set. Runs
409 tox, builds a source distribution and (optionally) a binary
410 distribution, and uploads the distribution(s) to a PyPI repository.
411 This job should be configured to use a test PyPI repository like
412 testpypi.python.org, not a public release area like the global PyPI
413 repository. Like the verify job, this requires a setup.py file for
414 packaging the component.
416 The tox runner is pyenv aware so if the image contains an installation
417 of pyenv at /opt/pyenv it will pick it up and run Python tests with
418 the appropriate Python versions. The tox runner sets the following
419 pyenv variables before running.
423 export PYENV_ROOT="/opt/pyenv"
424 export PATH="$PYENV_ROOT/bin:$PATH"
427 Requires a .pypirc configuration file in the Jenkins builder home directory,
428 an example appears next that uses API tokens. No repository is needed in the
433 [distutils] # this tells distutils what package indexes you can push to
434 index-servers = pypi-test pypi
437 repository: https://test.pypi.org/legacy/
439 password: pypi-test-api-token-goes-here
443 password: pypi-api-token-goes-here
448 - {project-name}-pypi-merge-{stream}
452 :Comment Trigger: remerge
454 :Required Parameters:
456 :build-node: The node to run the build on.
457 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
460 :Optional Parameters:
462 :branch: The branch to build against. (default: master)
463 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
464 :build-timeout: Timeout in minutes before aborting build. (default: 15)
465 :cron: Cron schedule when to trigger the job. Supports daily builds.
466 This parameter also supports multiline input via YAML pipe | character in
467 cases where one may want to provide more than 1 cron timer. (default: empty)
468 :disable-job: Whether to disable the job (default: false)
469 :dist-binary: Whether to build a binary wheel distribution. (default: true)
470 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
471 :parallel: Boolean indicator for tox to run tests in parallel or series.
472 (default: false, in series)
473 :pre-build-script: Shell script to execute before the tox builder. For
474 example, install system prerequisites. (default: a shell comment)
475 :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
476 Merge jobs should use a server like testpypi.python.org. (default: pypi-test)
477 :python-version: Python version to invoke pip install of tox-pyenv
479 :stream: Keyword representing a release code-name.
480 Often the same as the branch. (default: master)
481 :submodule-recursive: Whether to checkout submodules recursively.
483 :submodule-timeout: Timeout (in minutes) for checkout operation.
485 :submodule-disable: Disable submodule checkout operation.
487 :tox-dir: Directory containing the project's tox.ini relative to
488 the workspace. The default uses tox.ini at the project root.
490 :tox-envs: Tox environments to run. If blank run everything described
491 in tox.ini. (default: '')
492 :gerrit_trigger_file_paths: Override file paths used to filter which file
493 modifications trigger a build. Refer to JJB documentation for "file-path" details.
494 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
500 Verifies a Python library project on creation of a patch set with a
501 release yaml file. Runs tox, builds source and (optionally) binary
502 distributions, checks the format of the version string, checks that
503 the distribution file names contain the release version string, and
504 checks if the tag exists in the code repository for the release
507 To initiate the release process, create a releases/ or .releases/
508 directory at the root of the project repository, add one release yaml
509 file to it, and submit a change set with that release yaml file. A
510 schema and and an example for the release yaml file appear below. The
511 version in the release yaml file must be a valid Semantic Versioning
512 (SemVer) string, matching either the pattern "v#.#.#" or "#.#.#" where
513 "#" is one or more digits.
515 This job is similar to the PyPI verify job, but is only triggered by a
516 patch set with a release yaml file.
518 The build node for PyPI release verify jobs must be CentOS, which
519 supports the sigul client for accessing a signing server.
523 The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml).
524 In words, the directory name can be ".releases" or "releases"; the file
525 name can be anything with suffix ".yaml".
527 The JSON schema for a pypi release file appears below.
532 $schema: "http://json-schema.org/schema#"
533 $id: "https://github.com/lfit/releng-global-jjb/blob/master/release-pypi-schema.yaml"
536 - "distribution_type"
549 An example of a pypi release file appears below.
553 $ cat releases/1.0.0-pypi.yaml
555 distribution_type: pypi
557 project: 'example-project'
562 - {project-name}-pypi-release-verify-{stream}
563 - gerrit-pypi-release-verify
564 - github-pypi-release-verify
566 :Comment Trigger: recheck
568 :Required Parameters:
570 :build-node: The node to run build on, which must be Centos.
571 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
574 :Optional Parameters:
576 :branch: The branch to build against. (default: master)
577 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
578 :build-timeout: Timeout in minutes before aborting build. (default: 15)
579 :disable-job: Whether to disable the job (default: false)
580 :dist-binary: Whether to build a binary wheel distribution. (default: true)
581 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
582 :parallel: Boolean indicator for tox to run tests in parallel or series.
583 (default: false, in series)
584 :pre-build-script: Shell script to execute before the tox builder.
585 For example, install prerequisites or move files to the repo root.
586 (default: a string with a shell comment)
587 :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
588 Release jobs should use a server like pypy.org. (default: pypi)
589 :python-version: Python version to invoke pip install of tox-pyenv
591 :stream: Keyword representing a release code-name.
592 Often the same as the branch. (default: master)
593 :submodule-recursive: Whether to checkout submodules recursively.
595 :submodule-timeout: Timeout (in minutes) for checkout operation.
597 :submodule-disable: Disable submodule checkout operation.
599 :tox-dir: Directory containing the project's tox.ini relative to
600 the workspace. The default uses tox.ini at the project root.
602 :tox-envs: Tox environments to run. If blank run everything described
603 in tox.ini. (default: '')
604 :use-release-file: Whether to use the release file. (default: true)
609 Publishes a Python library on merge of a patch set with a release yaml
610 file. Runs tox, builds source and (optionally) binary distributions,
611 checks the format of the version string, checks that the distribution
612 file names contain the release version string, checks if the tag
613 exists in the code repository for the release version, then if the tag
614 does not exist, tags the code repository with the release version,
615 signs the tag and pushes the tag to the git server. Finally this
616 uploads the distributions to a PyPI repository.
618 This job is similar to the PyPI merge job, but is only triggered by
619 merge of a release yaml file, also this checks the version and tag
620 before uploading to a public repository such as PyPI.
622 See the PyPI Release Verify job above for documentation of the release
625 The build node for PyPI release merge jobs must be CentOS, which
626 supports the sigul client for accessing a signing server.
628 A Jenkins user can also trigger this release job via the "Build with
629 parameters" action, removing the need to merge a release yaml file.
630 The user must enter parameters in the same way as a release yaml file,
631 except for the special USE_RELEASE_FILE and DRY_RUN check boxes. The
632 user must uncheck the USE_RELEASE_FILE check box if the job should run
633 with a release file, while passing the required information as build
634 parameters. Similarly, the user must uncheck the DRY_RUN check box to
635 test the job while skipping upload of files to a repository.
637 The special parameters are as follows::
640 USE_RELEASE_FILE = false
645 - {project-name}-pypi-release-merge-{stream}
646 - gerrit-pypi-release-merge
647 - github-pypi-release-merge
649 :Comment Trigger: remerge
651 :Required Parameters:
653 :build-node: The node to run build on, which must be Centos.
654 :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
657 :Optional Parameters:
659 :branch: The branch to build against. (default: master)
660 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
661 :build-timeout: Timeout in minutes before aborting build. (default: 15)
662 :disable-job: Whether to disable the job (default: false)
663 :dist-binary: Whether to build a binary wheel distribution. (default: true)
664 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
665 :parallel: Boolean indicator for tox to run tests in parallel or series.
666 (default: false, in series)
667 :pre-build-script: Shell script to execute before the tox builder.
668 For example, install prerequisites or move files to the repo root.
669 (default: a string with a shell comment)
670 :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
671 Release jobs should use a server like pypy.org. (default: pypi)
672 :python-version: Python version to invoke pip install of tox-pyenv
674 :stream: Keyword representing a release code-name.
675 Often the same as the branch. (default: master)
676 :submodule-recursive: Whether to checkout submodules recursively.
678 :submodule-timeout: Timeout (in minutes) for checkout operation.
680 :submodule-disable: Disable submodule checkout operation.
682 :tox-dir: Directory containing the project's tox.ini relative to
683 the workspace. The default uses tox.ini at the project root.
685 :tox-envs: Tox environments to run. If blank run everything described
686 in tox.ini. (default: '')
687 :use-release-file: Whether to use the release file. (default: true)