8 .. include:: ../job-groups.rst
10 Below is a list of Python job groups:
12 .. literalinclude:: ../../jjb/lf-python-job-groups.yaml
22 Runs CLM scanning against a Python project.
26 :clm-project-name: Project name in Nexus IQ to send results to.
31 Installs Tox into a virtualenv.
35 :python-version: Version of Python to invoke the pip install of the tox-pyenv
36 package that creates a virtual environment, either "python2" or "python3".
41 Creates a Tox virtual environment and invokes tox.
45 :parallel: Boolean. If true use detox (distributed tox);
55 CLM scans for Python based repos. This job will call the Nexus IQ CLI
56 directly to run the scans.
58 A new credential named "nexus-iq-xc-clm" needs to exist in the Jenkins
59 credentials. The credential should contain the username and password
60 to access Nexus IQ Server.
64 - {project-name}-python-clm-{stream}
65 - gerrit-python-xc-clm
66 - github-python-xc-clm
68 :Comment Trigger: **run-clm** post a comment with the trigger to launch
69 this job manually. Do not include any other text or vote in the
74 :build-node: The node to run build on.
75 :jenkins-ssh-credential: Credential to use for SSH. (Generally should
76 get configured in defaults.yaml)
80 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
81 :nexus-iq-cli-version: Nexus IQ CLI package version to download and use.
83 :nexus-iq-namespace: Insert a namespace to project AppID for projects that
84 share a Nexus IQ system to avoid project name collision. We recommend
85 inserting a trailing - dash if using this parameter.
86 For example 'odl-'. (default: '')
87 :build-timeout: Timeout in minutes before aborting build. (default: 60)
88 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
89 :java-version: Version of Java to use for the build. (default: openjdk8)
90 :pre-build-script: Shell script to execute before the CLM builder.
91 For example, install prerequisites or move files to the repo root.
92 (default: a string with a shell comment)
93 :stream: Keyword used to represent a release code-name.
94 Often the same as the branch. (default: master)
95 :submodule-recursive: Whether to checkout submodules recursively.
97 :submodule-timeout: Timeout (in minutes) for checkout operation.
99 :submodule-disable: Disable submodule checkout operation.
101 :gerrit_clm_triggers: Override Gerrit Triggers.
102 :gerrit_trigger_file_paths: Override file paths used to filter which file
103 modifications trigger a build. Refer to JJB documentation for "file-path" details.
104 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
107 Python Sonar with Tox
108 ---------------------
110 Sonar scans for Python based repos. This job invokes tox to run tests
111 and gather coverage statistics from the test results, then invokes
112 Maven to publish the results to either a Sonar server or SonarCloud.
114 **Deprecated**, new projects should use Tox Sonarqube.
116 To get the Sonar coverage results, file tox.ini must exist and contain
117 coverage commands to run.
119 The coverage commands define the code that gets executed by the test
120 suites. Checking coverage does not guarantee that the tests execute
121 properly, but it identifies code that is not executed by any test.
123 .. comment Start ignoring WriteGoodLintBear
125 This job reuses the Sonar builders used for Java/Maven projects which
126 run maven twice. The first invocation does nothing for Python
127 projects, so the job uses the goal 'validate' by default. The second
128 invocation publishes results using the goal 'sonar:sonar' by default.
130 .. comment Stop ignoring
138 coverage run --module pytest --junitxml xunit-results.xml
139 coverage xml --omit=".tox/py27/*","tests/*"
140 coverage report --omit=".tox/py27/*","tests/*"
142 For more details refer to coverage and sonar documentation:
144 https://coverage.readthedocs.io/
146 https://docs.sonarqube.org/display/PLUG/Python+Coverage+Results+Import
150 - {project-name}-tox-sonar
154 :Comment Trigger: **run-sonar** post a comment with the trigger to launch
155 this job manually. Do not include any other text or vote in the
158 :Required parameters:
160 :build-node: The node to run build on.
161 :jenkins-ssh-credential: Credential to use for SSH. (Generally should
162 get configured in defaults.yaml)
163 :mvn-settings: The name of the settings file with credentials for the project.
165 .. comment Start ignoring WriteGoodLintBear
167 :Optional parameters:
169 :branch: Git branch, should be master (default: master)
170 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
171 :build-timeout: Timeout in minutes before aborting build. (default: 60)
172 :cron: Cron schedule when to trigger the job. This parameter also
173 supports multiline input via YAML pipe | character in cases where
174 one may want to provide more than 1 cron timer. (default: H 11 * * *
176 :disable-job: Whether to disable the job (default: false)
177 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
178 :github-url: URL for Github. (default: https://github.com)
179 :java-version: Version of Java to use for the build. (default: openjdk8)
180 :mvn-global-settings: The name of the Maven global settings to use
181 :mvn-goals: The Maven goal to run first. (default: validate)
182 :mvn-version: Version of maven to use. (default: mvn35)
183 :parallel: Boolean indicator for tox to run tests in parallel or series.
184 (default: false, in series)
185 :pre-build-script: Shell script to execute before the Sonar builder.
186 For example, install prerequisites or move files to the repo root.
187 (default: a string with a shell comment)
188 :python-version: Python version to invoke pip install of tox-pyenv
190 :sonarcloud: Boolean indicator to use SonarCloud ``true|false``.
192 :sonarcloud-project-key: SonarCloud project key. (default: '')
193 :sonarcloud-project-organization: SonarCloud project organization.
195 :sonarcloud-api-token: SonarCloud API Token. (default: '')
196 :sonar-mvn-goal: The Maven goal to run the Sonar plugin. (default: sonar:sonar)
197 :stream: Keyword used to represent a release code-name.
198 Often the same as the branch. (default: master)
199 :submodule-recursive: Whether to checkout submodules recursively.
201 :submodule-timeout: Timeout (in minutes) for checkout operation.
203 :submodule-disable: Disable submodule checkout operation.
205 :tox-dir: Directory containing the project's tox.ini relative to
206 the workspace. The default uses tox.ini at the project root.
208 :tox-envs: Tox environments to run. If blank run everything described
209 in tox.ini. (default: '')
210 :gerrit_sonar_triggers: Override Gerrit Triggers.
211 :gerrit_trigger_file_paths: Override file paths used to filter which file
212 modifications trigger a build. Refer to JJB documentation for "file-path" details.
213 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
215 .. comment Stop ignoring
221 The SonarQube job invokes tox to run tests and generate code-coverage
222 statistics, then runs the SonarQube Scanner Jenkins plug-in to analyze
223 code, gather coverage data, and upload the results to a SonarQube server
224 such as SonarCloud.io. Optionally runs a shell script before tox.
226 Requires ``SonarQube Scanner for Jenkins``
228 This job runs on the master branch because the basic Sonar configuration
229 does not support multi-branch.
231 Plug-in configurations
232 Manage Jenkins --> Configure System --> SonarQube servers
233 - Name: Sonar (fixed)
234 - Server URL: https://sonar.project.org/ or https://sonarcloud.io
235 - Server authentication token: none for local, API token (saved as
236 a "secret text" credential) for Sonarcloud
238 Manage Jenkins --> Global Tool Configuration --> SonarQube Scanner
239 - Name: SonarQube Scanner (fixed)
240 - Install automatically
241 - Select latest version
245 - {project-name}-tox-sonarqube
246 - gerrit-tox-sonarqube
247 - github-tox-sonarqube
249 :Comment Trigger: ``run-sonar``
251 :Required parameters:
253 :build-node: The node to run the build on.
254 (Commonly in defaults.yaml)
255 :jenkins-ssh-credential: Credential to use for SSH.
256 (Commonly in defaults.yaml)
257 :project: The git repository name.
258 :project-name: Prefix used to name jobs.
260 .. comment Start ignoring WriteGoodLintBear
262 :Optional Parameters:
264 :archive-artifacts: Pattern for files to archive to the logs server
265 (default: '\*\*/\*.log')
266 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
267 :build-timeout: Timeout in minutes before aborting build. (default: 15)
268 :cron: Cron schedule when to trigger the job. This parameter also
269 supports multiline input via YAML pipe | character in cases where
270 one may want to provide more than 1 cron timer. (default: @weekly)
271 :disable-job: Whether to disable the job (default: false)
272 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
273 :github-url: URL for Github. (default: https://github.com)
274 :parallel: Boolean indicator for tox to run tests in parallel or series.
275 (default: false, in series)
276 :pre-build-script: Shell script to run before tox. Useful for setting up
277 dependencies. (default: a string with a shell comment)
278 :python-version: Python version to invoke pip install of tox-pyenv
280 :sonar-additional-args: Command line arguments. (default: '')
281 :sonar-java-opts: JVM options. For example, use option -Xmx
282 to increase the memory size limit. (default: '')
283 :sonar-project-file: The file name with Sonar configuration properties
284 (default: sonar-project.properties)
285 :sonar-properties: Sonar configuration properties. (default: '')
286 :sonar-task: Sonar task to run. (default: '')
287 :tox-dir: Directory containing the project's tox.ini relative to
288 the workspace. The default uses tox.ini at the project root.
290 :tox-envs: Tox environments to run. If blank run everything described
291 in tox.ini. (default: '')
293 .. comment Stop ignoring
295 .. note:: A job definition must provide one of the optional parameters
296 ``sonar-project-file`` and ``sonar-properties``; they cannot both be
297 empty. Set Sonar properties directly in the job definition by setting
298 the ``sonar-project-file`` property to ``""`` and adding all properties
299 under ``sonar-properties``.
301 :Required Sonar Properties:
303 - sonar.login: The API token for authentication at SonarCloud.
304 Commonly defined as key "sonarcloud_api_token" in defaults.yaml.
305 - sonar.organization: The umbrella project name; e.g., "opendaylight".
306 Commonly defined as key "sonarcloud_project_organization" in defaults.yaml.
307 - sonar.projectName: The git repository name without slashes; e.g., "infrautils".
308 - sonar.projectKey: The globally unique key for the report in SonarCloud. Most
309 teams use the catenation of sonar.organization, an underscore, and
310 sonar.projectName; e.g., "opendaylight_infrautils".
312 :Optional Sonar Properties:
314 - sonar.cfamily.gcov.reportsPath: directory with GCOV output files
315 - Documentation of SonarQube properties is here:
316 https://docs.sonarqube.org/latest/analysis/overview/
319 Example job definition
320 ^^^^^^^^^^^^^^^^^^^^^^
322 The following example defines a job for a basic Python project. This definition
323 uses configuration parameters in the umbrella project's defaults.yaml file.
328 name: my-package-sonar
330 project-name: my-package
331 sonar-project-file: ""
333 sonar.login={sonarcloud_api_token}
334 sonar.projectKey={sonarcloud_project_organization}_{project-name}
335 sonar.projectName={project-name}
336 sonar.organization={sonarcloud_project_organization}
337 sonar.sourceEncoding=UTF-8
338 sonar.sources=mypackage
339 sonar.exclusions=tests/*,setup.py
340 sonar.python.coverage.reportPaths=coverage.xml
342 - gerrit-tox-sonarqube
347 Tox runner to verify a project on creation of a patch set. This job
348 is pyenv aware so if the image contains an installation of pyenv at
349 /opt/pyenv it will pick it up and run Python tests with the
350 appropriate Python versions. This job will set the following pyenv
351 variables before running.
355 export PYENV_ROOT="/opt/pyenv"
356 export PATH="$PYENV_ROOT/bin:$PATH"
360 - {project-name}-tox-verify-{stream}
364 :Comment Trigger: **recheck|reverify** post a comment with one of the
365 triggers to launch this job manually. Do not include any other
366 text or vote in the same comment.
368 :Required Parameters:
370 :build-node: The node to run build on.
371 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
374 :Optional Parameters:
376 :branch: The branch to build against. (default: master)
377 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
378 :build-timeout: Timeout in minutes before aborting build. (default: 10)
379 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
380 :pre-build-script: Shell script to execute before the Tox builder.
381 For example, install prerequisites or move files to the repo root.
382 (default: a string with a shell comment)
383 :parallel: Boolean indicator for tox to run tests in parallel or series.
384 (default: false, in series)
385 :python-version: Python version to invoke pip install of tox-pyenv
387 :stream: Keyword representing a release code-name.
388 Often the same as the branch. (default: master)
389 :submodule-recursive: Whether to checkout submodules recursively.
391 :submodule-timeout: Timeout (in minutes) for checkout operation.
393 :submodule-disable: Disable submodule checkout operation.
395 :tox-dir: Directory containing the project's tox.ini relative to
396 the workspace. The default uses tox.ini at the project root.
398 :tox-envs: Tox environments to run. If blank run everything described
399 in tox.ini. (default: '')
400 :gerrit_trigger_file_paths: Override file paths used to filter which file
401 modifications trigger a build. Refer to JJB documentation for "file-path" details.
402 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
408 Tox runner to verify a project after merge of a patch set. This job
409 is pyenv aware so if the image contains an installation of pyenv at
410 /opt/pyenv it will pick it up and run Python tests with the
411 appropriate Python versions. This job will set the following pyenv
412 variables before running.
416 export PYENV_ROOT="/opt/pyenv"
417 export PATH="$PYENV_ROOT/bin:$PATH"
421 - {project-name}-tox-merge-{stream}
425 :Comment Trigger: **remerge** post a comment with the trigger to launch
426 this job manually. Do not include any other text or vote in the
429 :Required Parameters:
431 :build-node: The node to run build on.
432 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
435 :Optional Parameters:
437 :branch: The branch to build against. (default: master)
438 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
439 :build-timeout: Timeout in minutes before aborting build. (default: 10)
440 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
441 :pre-build-script: Shell script to execute before the CLM builder.
442 For example, install prerequisites or move files to the repo root.
443 (default: a string with a shell comment)
444 :python-version: Python version to invoke pip install of tox-pyenv
446 :stream: Keyword representing a release code-name.
447 Often the same as the branch. (default: master)
448 :submodule-recursive: Whether to checkout submodules recursively.
450 :submodule-timeout: Timeout (in minutes) for checkout operation.
452 :submodule-disable: Disable submodule checkout operation.
454 :tox-dir: Directory containing the project's tox.ini relative to
455 the workspace. The default uses tox.ini at the project root.
457 :tox-envs: Tox environments to run. If blank run everything described
458 in tox.ini. (default: '')
459 :gerrit_trigger_file_paths: Override file paths used to filter which file
460 modifications trigger a build. Refer to JJB documentation for "file-path" details.
461 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
467 Creates and uploads distribution files on merge of a patch set. Runs
468 tox, builds a source distribution and (optionally) a binary
469 distribution, and uploads the distribution(s) to a PyPI repository.
470 The project git repository must have a setup.py file
471 with configuration for packaging the component.
473 This job should use a staging repository like testpypi.python.org,
474 which sets up use of release jobs to promote the distributions later.
475 This job can also use a public release area like the global PyPI
476 repository if the release process is not needed. These PyPI
477 repositories allow upload of a package at a specific version once,
478 they do not allow overwrite of a package. This means that a merge job
479 will fail in the upload step if the package version already exists in
480 the target repository.
482 The tox runner is pyenv aware so if the image contains an installation
483 of pyenv at /opt/pyenv it will pick it up and run Python tests with
484 the appropriate Python versions. The tox runner sets the following
485 pyenv variables before running.
489 export PYENV_ROOT="/opt/pyenv"
490 export PATH="$PYENV_ROOT/bin:$PATH"
492 Installable package projects should use the directory layout shown
493 below. All Python files are in a repo subdirectory separate from
494 non-Python files like documentation. This layout allows highly
495 specific build-job triggers in Jenkins using the subdirectory
496 paths. For example, a PyPI merge job should not run on a non-Python
497 file change such as documentation, because the job cannot upload the
500 To make the document files available for building a Python package
501 long description in setup.py, add a symbolic link "docs" in the
502 package subdirectory pointing to the top-level docs directory.
510 │ └── release-notes.rst
512 ├── helloworld-package/
516 │ │ ├── helloworld.py
520 │ │ ├── helloworld_tests.py
521 │ │ └── helloworld_mocks.py
523 │ ├── requirements.txt
528 │ └── pypi-helloworld.yaml
535 Jobs built from the PyPI templates depend on a .pypirc configuration file
536 in the Jenkins builder home directory. An example appears next that uses
537 API tokens. Note that in the [pypi] entry the repository key-value pair
538 is optional, it defaults to pypi.org.
542 [distutils] # this tells distutils what package indexes you can push to
543 index-servers = pypi-test pypi
546 repository: https://test.pypi.org/legacy/
548 password: pypi-test-api-token-goes-here
552 password: pypi-api-token-goes-here
557 - {project-name}-pypi-merge-{stream}
561 :Comment Trigger: **remerge** post a comment with the trigger to launch
562 this job manually. Do not include any other text or vote in the
565 :Required Parameters:
567 :build-node: The node to run the build on.
568 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
570 :mvn-settings: The settings file with credentials for the project
571 :project: Git repository name
572 :project-name: Jenkins job name prefix
574 :Optional Parameters:
576 :branch: The branch to build against. (default: master)
577 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
578 :build-timeout: Timeout in minutes before aborting build. (default: 15)
579 :cron: Cron schedule when to trigger the job. Supports daily builds.
580 This parameter also supports multiline input via YAML pipe | character in
581 cases where one may want to provide more than 1 cron timer. (default: empty)
582 :disable-job: Whether to disable the job (default: false)
583 :dist-binary: Whether to build a binary wheel distribution. (default: true)
584 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
585 :mvn-opts: Sets MAVEN_OPTS to start up the JVM running Maven. (default: '')
586 :mvn-params: Parameters to pass to the mvn CLI. (default: '')
587 :mvn-version: Version of maven to use. (default: mvn35)
588 :parallel: Boolean indicator for tox to run tests in parallel or series.
589 (default: false, in series)
590 :pre-build-script: Shell script to execute before the tox builder. For
591 example, install system prerequisites. (default: a shell comment)
592 :pypi-repo: Key for the PyPI target repository in the .pypirc file,
593 ideally a server like test.pypy.org. (default: pypi-test)
594 :python-version: Python version to invoke pip install of tox-pyenv
596 :stream: Keyword representing a release code-name.
597 Often the same as the branch. (default: master)
598 :submodule-recursive: Whether to checkout submodules recursively.
600 :submodule-timeout: Timeout (in minutes) for checkout operation.
602 :submodule-disable: Disable submodule checkout operation.
604 :tox-dir: Directory containing the project's tox.ini relative to
605 the workspace. The default uses tox.ini at the project root.
607 :tox-envs: Tox environments to run. If blank run everything described
608 in tox.ini. (default: '')
609 :gerrit_trigger_file_paths: Override file paths used to filter which file
610 modifications trigger a build. Refer to JJB documentation for "file-path" details.
611 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
617 Verifies a Python library project on creation of a patch set. Runs tox
618 then builds a source distribution and (optionally) a binary
619 distribution. The project repository must have a setup.py file with
620 configuration for packaging the component.
622 The tox runner is pyenv aware so if the image contains an installation
623 of pyenv at /opt/pyenv it will pick it up and run Python tests with
624 the appropriate Python versions. The tox runner sets the following
625 pyenv variables before running.
629 export PYENV_ROOT="/opt/pyenv"
630 export PATH="$PYENV_ROOT/bin:$PATH"
634 - {project-name}-pypi-verify-{stream}
638 :Comment Trigger: **recheck|reverify** post a comment with one of the
639 triggers to launch this job manually. Do not include any other
640 text or vote in the same comment.
642 :Required Parameters:
644 :build-node: The node to run the build on.
645 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
647 :mvn-settings: The settings file with credentials for the project
648 :project: Git repository name
649 :project-name: Jenkins job name prefix
651 :Optional Parameters:
653 :branch: The branch to build against. (default: master)
654 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
655 :build-timeout: Timeout in minutes before aborting build. (default: 15)
656 :disable-job: Whether to disable the job (default: false)
657 :dist-binary: Whether to build a binary wheel distribution. (default: true)
658 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
659 :mvn-opts: Sets MAVEN_OPTS to start up the JVM running Maven. (default: '')
660 :mvn-params: Parameters to pass to the mvn CLI. (default: '')
661 :mvn-version: Version of maven to use. (default: mvn35)
662 :parallel: Boolean indicator for tox to run tests in parallel or series.
663 (default: false, in series)
664 :pre-build-script: Shell script to execute before the tox builder. For
665 example, install system prerequisites. (default: a shell comment)
666 :python-version: Python version to invoke pip install of tox-pyenv
668 :stream: Keyword representing a release code-name.
669 Often the same as the branch. (default: master)
670 :submodule-recursive: Whether to checkout submodules recursively.
672 :submodule-timeout: Timeout (in minutes) for checkout operation.
674 :submodule-disable: Disable submodule checkout operation.
676 :tox-dir: Directory containing the project's tox.ini relative to
677 the workspace. The default uses tox.ini at the project root.
679 :tox-envs: Tox environments to run. If blank run everything described
680 in tox.ini. (default: '')
681 :gerrit_trigger_file_paths: Override file paths used to filter which file
682 modifications trigger a build. Refer to JJB documentation for "file-path" details.
683 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit