8 .. include:: ../job-groups.rst
10 Below is a list of Python job groups:
12 .. literalinclude:: ../../jjb/lf-python-job-groups.yaml
22 Runs CLM scanning against a Python project.
26 :clm-project-name: Project name in Nexus IQ to send results to.
28 lf-infra-pypi-tag-release
29 -------------------------
31 Checks the format of the release version string and checks the git
32 repository for that tag. In a merge job, continues to tag the repository
33 and push the tag to the git server. Also installs supporting tools
34 including Sigul and lftools. Sigul requires a CentOS build node.
39 Uploads distribution files from subdirectory "dist" to a PyPI repository
40 using a Python virtual enviroment to install required packages. The
41 Jenkins server must have a configuration file ".pypirc".
45 :pypi-repo: PyPI repository key in .pypirc configuration file;
46 e.g., "pypi-test" or "pypi".
51 Installs Tox into a virtualenv.
55 :python-version: Version of Python to invoke the pip install of the tox-pyenv
56 package that creates a virtual environment, either "python2" or "python3".
61 Creates a Tox virtual environment and invokes tox.
65 :parallel: Boolean. If true use detox (distributed tox);
75 CLM scans for Python based repos. This job will call the Nexus IQ CLI
76 directly to run the scans.
78 A new credential named "nexus-iq-xc-clm" needs to exist in the Jenkins
79 credentials. The credential should contain the username and password
80 to access Nexus IQ Server.
84 - {project-name}-python-clm-{stream}
85 - gerrit-python-xc-clm
86 - github-python-xc-clm
88 :Comment Trigger: run-clm
92 :build-node: The node to run build on.
93 :jenkins-ssh-credential: Credential to use for SSH. (Generally should
94 get configured in defaults.yaml)
98 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
99 :nexus-iq-cli-version: Nexus IQ CLI package version to download and use.
101 :nexus-iq-namespace: Insert a namespace to project AppID for projects that
102 share a Nexus IQ system to avoid project name collision. We recommend
103 inserting a trailing - dash if using this parameter.
104 For example 'odl-'. (default: '')
105 :build-timeout: Timeout in minutes before aborting build. (default: 60)
106 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
107 :java-version: Version of Java to use for the build. (default: openjdk8)
108 :pre-build-script: Shell script to execute before the CLM builder.
109 For example, install prerequisites or move files to the repo root.
110 (default: a string with a shell comment)
111 :stream: Keyword used to represent a release code-name.
112 Often the same as the branch. (default: master)
113 :submodule-recursive: Whether to checkout submodules recursively.
115 :submodule-timeout: Timeout (in minutes) for checkout operation.
117 :submodule-disable: Disable submodule checkout operation.
119 :gerrit_clm_triggers: Override Gerrit Triggers.
120 :gerrit_trigger_file_paths: Override file paths used to filter which file
121 modifications trigger a build. Refer to JJB documentation for "file-path" details.
122 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
125 Python Sonar with Tox
126 ---------------------
128 Sonar scans for Python based repos. This job invokes tox to run tests
129 and gather coverage statistics from the test results, then invokes
130 Maven to publish the results to a Sonar server.
132 To get the Sonar coverage results, file tox.ini must exist and contain
133 coverage commands to run.
135 The coverage commands define the code that gets executed by the test
136 suites. Checking coverage does not guarantee that the tests execute
137 properly, but it identifies code that is not executed by any test.
139 This job reuses the Sonar builder used in Java/Maven projects which
140 runs maven twice. The first invocation does nothing for Python
141 projects, so the job uses the goal 'validate' by default. The second
142 invocation publishes results using the goal 'sonar:sonar' by default.
150 coverage run --module pytest --junitxml xunit-results.xml
151 coverage xml --omit=".tox/py27/*","tests/*"
152 coverage report --omit=".tox/py27/*","tests/*"
154 For more details refer to coverage and sonar documentation:
156 https://coverage.readthedocs.io/
158 https://docs.sonarqube.org/display/PLUG/Python+Coverage+Results+Import
162 - {project-name}-tox-sonar
166 :Comment Trigger: run-sonar
168 :Required parameters:
170 :build-node: The node to run build on.
171 :jenkins-ssh-credential: Credential to use for SSH. (Generally should
172 get configured in defaults.yaml)
173 :mvn-settings: The name of the settings file with credentials for the project.
175 :Optional parameters:
177 :branch: Git branch, should be master (default: master)
178 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
179 :build-timeout: Timeout in minutes before aborting build. (default: 60)
180 :cron: Cron schedule when to trigger the job. This parameter also
181 supports multiline input via YAML pipe | character in cases where
182 one may want to provide more than 1 cron timer. (default: H 11 * * *
184 :disable-job: Whether to disable the job (default: false)
185 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
186 :github-url: URL for Github. (default: https://github.com)
187 :java-version: Version of Java to use for the build. (default: openjdk8)
188 :mvn-global-settings: The name of the Maven global settings to use for
189 :mvn-goals: The Maven goal to run first. (default: validate)
190 :mvn-version: Version of maven to use. (default: mvn35)
191 :parallel: Boolean indicator for tox to run tests in parallel or series.
192 (default: false, in series)
193 :pre-build-script: Shell script to execute before the Sonar builder.
194 For example, install prerequisites or move files to the repo root.
195 (default: a string with a shell comment)
196 :python-version: Python version to invoke pip install of tox-pyenv
198 :sonar-mvn-goal: The Maven goal to run the Sonar plugin. (default: sonar:sonar)
199 :stream: Keyword used to represent a release code-name.
200 Often the same as the branch. (default: master)
201 :submodule-recursive: Whether to checkout submodules recursively.
203 :submodule-timeout: Timeout (in minutes) for checkout operation.
205 :submodule-disable: Disable submodule checkout operation.
207 :gerrit_sonar_triggers: Override Gerrit Triggers.
208 :gerrit_trigger_file_paths: Override file paths used to filter which file
209 modifications trigger a build. Refer to JJB documentation for "file-path" details.
210 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
216 Tox runner to verify a project on creation of a patch set. This job
217 is pyenv aware so if the image contains an installation of pyenv at
218 /opt/pyenv it will pick it up and run Python tests with the
219 appropriate Python versions. This job will set the following pyenv
220 variables before running.
224 export PYENV_ROOT="/opt/pyenv"
225 export PATH="$PYENV_ROOT/bin:$PATH"
229 - {project-name}-tox-verify-{stream}
233 :Comment Trigger: recheck|reverify
235 :Required Parameters:
237 :build-node: The node to run build on.
238 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
241 :Optional Parameters:
243 :branch: The branch to build against. (default: master)
244 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
245 :build-timeout: Timeout in minutes before aborting build. (default: 10)
246 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
247 :pre-build-script: Shell script to execute before the Tox builder.
248 For example, install prerequisites or move files to the repo root.
249 (default: a string with a shell comment)
250 :parallel: Boolean indicator for tox to run tests in parallel or series.
251 (default: false, in series)
252 :python-version: Python version to invoke pip install of tox-pyenv
254 :stream: Keyword representing a release code-name.
255 Often the same as the branch. (default: master)
256 :submodule-recursive: Whether to checkout submodules recursively.
258 :submodule-timeout: Timeout (in minutes) for checkout operation.
260 :submodule-disable: Disable submodule checkout operation.
262 :tox-dir: Directory containing the project's tox.ini relative to
263 the workspace. The default uses tox.ini at the project root.
265 :tox-envs: Tox environments to run. If blank run everything described
266 in tox.ini. (default: '')
267 :gerrit_trigger_file_paths: Override file paths used to filter which file
268 modifications trigger a build. Refer to JJB documentation for "file-path" details.
269 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
275 Tox runner to verify a project after merge of a patch set. This job
276 is pyenv aware so if the image contains an installation of pyenv at
277 /opt/pyenv it will pick it up and run Python tests with the
278 appropriate Python versions. This job will set the following pyenv
279 variables before running.
283 export PYENV_ROOT="/opt/pyenv"
284 export PATH="$PYENV_ROOT/bin:$PATH"
288 - {project-name}-tox-merge-{stream}
292 :Comment Trigger: remerge
294 :Required Parameters:
296 :build-node: The node to run build on.
297 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
300 :Optional Parameters:
302 :branch: The branch to build against. (default: master)
303 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
304 :build-timeout: Timeout in minutes before aborting build. (default: 10)
305 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
306 :pre-build-script: Shell script to execute before the CLM builder.
307 For example, install prerequisites or move files to the repo root.
308 (default: a string with only a comment)
309 :python-version: Python version to invoke pip install of tox-pyenv
311 :stream: Keyword representing a release code-name.
312 Often the same as the branch. (default: master)
313 :submodule-recursive: Whether to checkout submodules recursively.
315 :submodule-timeout: Timeout (in minutes) for checkout operation.
317 :submodule-disable: Disable submodule checkout operation.
319 :tox-dir: Directory containing the project's tox.ini relative to
320 the workspace. The default uses tox.ini at the project root.
322 :tox-envs: Tox environments to run. If blank run everything described
323 in tox.ini. (default: '')
324 :gerrit_trigger_file_paths: Override file paths used to filter which file
325 modifications trigger a build. Refer to JJB documentation for "file-path" details.
326 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
332 Verifies a Python library project on creation of a patch set. Runs tox
333 then builds a source distribution and (optionally) a binary
334 distribution. The project repository must have a setup.py file with
335 configuration for packaging the component.
337 The tox runner is pyenv aware so if the image contains an installation
338 of pyenv at /opt/pyenv it will pick it up and run Python tests with
339 the appropriate Python versions. The tox runner sets the following
340 pyenv variables before running.
344 export PYENV_ROOT="/opt/pyenv"
345 export PATH="$PYENV_ROOT/bin:$PATH"
349 - {project-name}-pypi-verify-{stream}
353 :Comment Trigger: recheck
355 :Required Parameters:
357 :build-node: The node to run the build on.
358 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
361 :Optional Parameters:
363 :branch: The branch to build against. (default: master)
364 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
365 :build-timeout: Timeout in minutes before aborting build. (default: 15)
366 :disable-job: Whether to disable the job (default: false)
367 :dist-binary: Whether to build a binary wheel distribution. (default: true)
368 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
369 :parallel: Boolean indicator for tox to run tests in parallel or series.
370 (default: false, in series)
371 :pre-build-script: Shell script to execute before the tox builder. For
372 example, install system prerequisites. (default: a shell comment)
373 :python-version: Python version to invoke pip install of tox-pyenv
375 :stream: Keyword representing a release code-name.
376 Often the same as the branch. (default: master)
377 :submodule-recursive: Whether to checkout submodules recursively.
379 :submodule-timeout: Timeout (in minutes) for checkout operation.
381 :submodule-disable: Disable submodule checkout operation.
383 :tox-dir: Directory containing the project's tox.ini relative to
384 the workspace. The default uses tox.ini at the project root.
386 :tox-envs: Tox environments to run. If blank run everything described
387 in tox.ini. (default: '')
388 :gerrit_trigger_file_paths: Override file paths used to filter which file
389 modifications trigger a build. Refer to JJB documentation for "file-path" details.
390 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
396 Creates and uploads distribution files on merge of a patch set. Runs
397 tox, builds a source distribution and (optionally) a binary
398 distribution, and uploads the distribution(s) to a PyPI repository.
399 This job should be configured to use a test PyPI repository like
400 testpypi.python.org, not a public release area like the global PyPI
401 repository. Like the verify job, this requires a setup.py file for
402 packaging the component.
404 The tox runner is pyenv aware so if the image contains an installation
405 of pyenv at /opt/pyenv it will pick it up and run Python tests with
406 the appropriate Python versions. The tox runner sets the following
407 pyenv variables before running.
411 export PYENV_ROOT="/opt/pyenv"
412 export PATH="$PYENV_ROOT/bin:$PATH"
415 Requires a .pypirc configuration file in the Jenkins builder home directory,
416 an example appears next that uses API tokens. No repository is needed in the
421 [distutils] # this tells distutils what package indexes you can push to
422 index-servers = pypi-test pypi
425 repository: https://test.pypi.org/legacy/
427 password: pypi-test-api-token-goes-here
431 password: pypi-api-token-goes-here
436 - {project-name}-pypi-merge-{stream}
440 :Comment Trigger: pypi-remerge
442 :Required Parameters:
444 :build-node: The node to run the build on.
445 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
448 :Optional Parameters:
450 :branch: The branch to build against. (default: master)
451 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
452 :build-timeout: Timeout in minutes before aborting build. (default: 15)
453 :cron: Cron schedule when to trigger the job. Supports daily builds.
454 This parameter also supports multiline input via YAML pipe | character in
455 cases where one may want to provide more than 1 cron timer. (default: empty)
456 :disable-job: Whether to disable the job (default: false)
457 :dist-binary: Whether to build a binary wheel distribution. (default: true)
458 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
459 :parallel: Boolean indicator for tox to run tests in parallel or series.
460 (default: false, in series)
461 :pre-build-script: Shell script to execute before the tox builder. For
462 example, install system prerequisites. (default: a shell comment)
463 :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
464 Merge jobs should use a server like testpypi.python.org. (default: pypi-test)
465 :python-version: Python version to invoke pip install of tox-pyenv
467 :stream: Keyword representing a release code-name.
468 Often the same as the branch. (default: master)
469 :submodule-recursive: Whether to checkout submodules recursively.
471 :submodule-timeout: Timeout (in minutes) for checkout operation.
473 :submodule-disable: Disable submodule checkout operation.
475 :tox-dir: Directory containing the project's tox.ini relative to
476 the workspace. The default uses tox.ini at the project root.
478 :tox-envs: Tox environments to run. If blank run everything described
479 in tox.ini. (default: '')
480 :gerrit_trigger_file_paths: Override file paths used to filter which file
481 modifications trigger a build. Refer to JJB documentation for "file-path" details.
482 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
488 Verifies a Python library project on creation of a patch set with a
489 release yaml file. Runs tox, builds source and (optionally) binary
490 distributions, checks the format of the version string, checks that
491 the distribution file names contain the release version string, and
492 checks that no tag exists in the code repository for the release
495 To initiate the release process, create a releases/ or .releases/
496 directory at the root of the project repository, add one release yaml
497 file to it, and submit a change set with that release yaml file. A
498 schema and and an example for the release yaml file appear below. The
499 version in the release yaml file must be a valid Semantic Versioning
500 (SemVer) string, matching either the pattern "v#.#.#" or "#.#.#" where
501 "#" is one or more digits.
503 This job is similar to the PyPI verify job, but is only triggered by a
504 patch set with a release yaml file.
506 The build node for PyPI release verify jobs must be CentOS, which
507 supports the sigul client for accessing a signing server.
511 The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml).
512 In words, the directory name can be ".releases" or "releases"; the file
513 name can be anything with suffix ".yaml".
515 The JSON schema for a pypi release file appears below.
520 $schema: "http://json-schema.org/schema#"
521 $id: "https://github.com/lfit/releng-global-jjb/blob/master/release-pypi-schema.yaml"
524 - "distribution_type"
537 An example of a pypi release file appears below.
541 $ cat releases/1.0.0-pypi.yaml
543 distribution_type: pypi
545 project: 'example-project'
550 - {project-name}-pypi-release-verify-{stream}
551 - gerrit-pypi-release-verify
552 - github-pypi-release-verify
554 :Required Parameters:
556 :build-node: The node to run build on, which must be Centos.
557 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
560 :Optional Parameters:
562 :branch: The branch to build against. (default: master)
563 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
564 :build-timeout: Timeout in minutes before aborting build. (default: 15)
565 :disable-job: Whether to disable the job (default: false)
566 :dist-binary: Whether to build a binary wheel distribution. (default: true)
567 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
568 :parallel: Boolean indicator for tox to run tests in parallel or series.
569 (default: false, in series)
570 :pre-build-script: Shell script to execute before the tox builder.
571 For example, install prerequisites or move files to the repo root.
572 (default: a string with a shell comment)
573 :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
574 Release jobs should use a server like pypy.org. (default: pypi)
575 :python-version: Python version to invoke pip install of tox-pyenv
577 :stream: Keyword representing a release code-name.
578 Often the same as the branch. (default: master)
579 :submodule-recursive: Whether to checkout submodules recursively.
581 :submodule-timeout: Timeout (in minutes) for checkout operation.
583 :submodule-disable: Disable submodule checkout operation.
585 :tox-dir: Directory containing the project's tox.ini relative to
586 the workspace. The default uses tox.ini at the project root.
588 :tox-envs: Tox environments to run. If blank run everything described
589 in tox.ini. (default: '')
590 :use-release-file: Whether to use the release file. (default: true)
595 Publishes a Python library on merge of a patch set with a release yaml
596 file. Runs tox, builds source and (optionally) binary distributions,
597 checks the format of the version string, checks that the distribution
598 file names contain the release version string, checks that no tag
599 exists in the code repository for the release version, tags the code
600 repository with the release version, pushes the tag to the git server,
601 and uploads distributions to a PyPI repository.
603 This job is similar to the PyPI merge job, but is only triggered by
604 merge of a release yaml file and checks the version and tag before
605 uploading to a public repository such as PyPI.
607 See the PyPI Release Verify job above for documentation of the release
610 The build node for PyPI release merge jobs must be CentOS, which
611 supports the sigul client for accessing a signing server.
613 A Jenkins user can also trigger this release job via the "Build with
614 parameters" action, removing the need to merge a release yaml file.
615 The user must enter parameters in the same way as a release yaml file,
616 except for the special USE_RELEASE_FILE and DRY_RUN check boxes. The
617 user must uncheck the USE_RELEASE_FILE check box if the job should run
618 with a release file, while passing the required information as build
619 parameters. Similarly, the user must uncheck the DRY_RUN check box to
620 test the job while skipping upload of files to a repository.
622 The special parameters are as follows::
625 USE_RELEASE_FILE = false
630 - {project-name}-pypi-release-merge-{stream}
631 - gerrit-pypi-release-merge
632 - github-pypi-release-merge
634 :Required Parameters:
636 :build-node: The node to run build on, which must be Centos.
637 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
640 :Optional Parameters:
642 :branch: The branch to build against. (default: master)
643 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
644 :build-timeout: Timeout in minutes before aborting build. (default: 15)
645 :disable-job: Whether to disable the job (default: false)
646 :dist-binary: Whether to build a binary wheel distribution. (default: true)
647 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
648 :parallel: Boolean indicator for tox to run tests in parallel or series.
649 (default: false, in series)
650 :pre-build-script: Shell script to execute before the tox builder.
651 For example, install prerequisites or move files to the repo root.
652 (default: a string with a shell comment)
653 :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
654 Release jobs should use a server like pypy.org. (default: pypi)
655 :python-version: Python version to invoke pip install of tox-pyenv
657 :stream: Keyword representing a release code-name.
658 Often the same as the branch. (default: master)
659 :submodule-recursive: Whether to checkout submodules recursively.
661 :submodule-timeout: Timeout (in minutes) for checkout operation.
663 :submodule-disable: Disable submodule checkout operation.
665 :tox-dir: Directory containing the project's tox.ini relative to
666 the workspace. The default uses tox.ini at the project root.
668 :tox-envs: Tox environments to run. If blank run everything described
669 in tox.ini. (default: '')
670 :use-release-file: Whether to use the release file. (default: true)