8 .. include:: ../job-groups.rst
10 Below is a list of Python job groups:
12 .. literalinclude:: ../../jjb/lf-python-job-groups.yaml
22 Runs CLM scanning against a Python project.
26 :clm-project-name: Project name in Nexus IQ to send results to.
31 Installs Tox into a virtualenv.
35 :python-version: Version of Python to invoke the pip install of the tox-pyenv
36 package that creates a virtual environment, either "python2" or "python3".
41 Creates a Tox virtual environment and invokes tox.
45 :parallel: Boolean. If true use detox (distributed tox);
55 CLM scans for Python based repos. This job will call the Nexus IQ CLI
56 directly to run the scans.
58 A new credential named "nexus-iq-xc-clm" needs to exist in the Jenkins
59 credentials. The credential should contain the username and password
60 to access Nexus IQ Server.
64 - {project-name}-python-clm-{stream}
65 - gerrit-python-xc-clm
66 - github-python-xc-clm
68 :Comment Trigger: **run-clm** post a comment with the trigger to launch
69 this job manually. Do not include any other text or vote in the
74 :build-node: The node to run build on.
75 :jenkins-ssh-credential: Credential to use for SSH. (Generally should
76 get configured in defaults.yaml)
80 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
81 :nexus-iq-cli-version: Nexus IQ CLI package version to download and use.
83 :nexus-iq-namespace: Insert a namespace to project AppID for projects that
84 share a Nexus IQ system to avoid project name collision. We recommend
85 inserting a trailing - dash if using this parameter.
86 For example 'odl-'. (default: '')
87 :build-timeout: Timeout in minutes before aborting build. (default: 60)
88 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
89 :java-version: Version of Java to use for the build. (default: openjdk8)
90 :pre-build-script: Shell script to execute before the CLM builder.
91 For example, install prerequisites or move files to the repo root.
92 (default: a string with a shell comment)
93 :stream: Keyword used to represent a release code-name.
94 Often the same as the branch. (default: master)
95 :submodule-recursive: Whether to checkout submodules recursively.
97 :submodule-timeout: Timeout (in minutes) for checkout operation.
99 :submodule-disable: Disable submodule checkout operation.
101 :gerrit_clm_triggers: Override Gerrit Triggers.
102 :gerrit_trigger_file_paths: Override file paths used to filter which file
103 modifications trigger a build. Refer to JJB documentation for "file-path" details.
104 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
107 Python Sonar with Tox
108 ---------------------
110 Sonar scans for Python based repos. This job invokes tox to run tests
111 and gather coverage statistics from the test results, then invokes
112 Maven to publish the results to either a Sonar server or SonarCloud.
114 To get the Sonar coverage results, file tox.ini must exist and contain
115 coverage commands to run.
117 The coverage commands define the code that gets executed by the test
118 suites. Checking coverage does not guarantee that the tests execute
119 properly, but it identifies code that is not executed by any test.
121 .. comment Start ignoring WriteGoodLintBear
123 This job reuses the Sonar builders used for Java/Maven projects which
124 run maven twice. The first invocation does nothing for Python
125 projects, so the job uses the goal 'validate' by default. The second
126 invocation publishes results using the goal 'sonar:sonar' by default.
128 .. comment Stop ignoring
136 coverage run --module pytest --junitxml xunit-results.xml
137 coverage xml --omit=".tox/py27/*","tests/*"
138 coverage report --omit=".tox/py27/*","tests/*"
140 For more details refer to coverage and sonar documentation:
142 https://coverage.readthedocs.io/
144 https://docs.sonarqube.org/display/PLUG/Python+Coverage+Results+Import
148 - {project-name}-tox-sonar
152 :Comment Trigger: **run-sonar** post a comment with the trigger to launch
153 this job manually. Do not include any other text or vote in the
156 :Required parameters:
158 :build-node: The node to run build on.
159 :jenkins-ssh-credential: Credential to use for SSH. (Generally should
160 get configured in defaults.yaml)
161 :mvn-settings: The name of the settings file with credentials for the project.
163 .. comment Start ignoring WriteGoodLintBear
165 :Optional parameters:
167 :branch: Git branch, should be master (default: master)
168 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
169 :build-timeout: Timeout in minutes before aborting build. (default: 60)
170 :cron: Cron schedule when to trigger the job. This parameter also
171 supports multiline input via YAML pipe | character in cases where
172 one may want to provide more than 1 cron timer. (default: H 11 * * *
174 :disable-job: Whether to disable the job (default: false)
175 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
176 :github-url: URL for Github. (default: https://github.com)
177 :java-version: Version of Java to use for the build. (default: openjdk8)
178 :mvn-global-settings: The name of the Maven global settings to use
179 :mvn-goals: The Maven goal to run first. (default: validate)
180 :mvn-version: Version of maven to use. (default: mvn35)
181 :parallel: Boolean indicator for tox to run tests in parallel or series.
182 (default: false, in series)
183 :pre-build-script: Shell script to execute before the Sonar builder.
184 For example, install prerequisites or move files to the repo root.
185 (default: a string with a shell comment)
186 :python-version: Python version to invoke pip install of tox-pyenv
188 :sonarcloud: Boolean indicator to use SonarCloud ``true|false``.
190 :sonarcloud-project-key: SonarCloud project key. (default: '')
191 :sonarcloud-project-organization: SonarCloud project organization.
193 :sonarcloud-api-token: SonarCloud API Token. (default: '')
194 :sonar-mvn-goal: The Maven goal to run the Sonar plugin. (default: sonar:sonar)
195 :stream: Keyword used to represent a release code-name.
196 Often the same as the branch. (default: master)
197 :submodule-recursive: Whether to checkout submodules recursively.
199 :submodule-timeout: Timeout (in minutes) for checkout operation.
201 :submodule-disable: Disable submodule checkout operation.
203 :tox-dir: Directory containing the project's tox.ini relative to
204 the workspace. The default uses tox.ini at the project root.
206 :tox-envs: Tox environments to run. If blank run everything described
207 in tox.ini. (default: '')
208 :gerrit_sonar_triggers: Override Gerrit Triggers.
209 :gerrit_trigger_file_paths: Override file paths used to filter which file
210 modifications trigger a build. Refer to JJB documentation for "file-path" details.
211 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
213 .. comment Stop ignoring
219 Tox runner to verify a project on creation of a patch set. This job
220 is pyenv aware so if the image contains an installation of pyenv at
221 /opt/pyenv it will pick it up and run Python tests with the
222 appropriate Python versions. This job will set the following pyenv
223 variables before running.
227 export PYENV_ROOT="/opt/pyenv"
228 export PATH="$PYENV_ROOT/bin:$PATH"
232 - {project-name}-tox-verify-{stream}
236 :Comment Trigger: **recheck|reverify** post a comment with one of the
237 triggers to launch this job manually. Do not include any other
238 text or vote in the same comment.
240 :Required Parameters:
242 :build-node: The node to run build on.
243 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
246 :Optional Parameters:
248 :branch: The branch to build against. (default: master)
249 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
250 :build-timeout: Timeout in minutes before aborting build. (default: 10)
251 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
252 :pre-build-script: Shell script to execute before the Tox builder.
253 For example, install prerequisites or move files to the repo root.
254 (default: a string with a shell comment)
255 :parallel: Boolean indicator for tox to run tests in parallel or series.
256 (default: false, in series)
257 :python-version: Python version to invoke pip install of tox-pyenv
259 :stream: Keyword representing a release code-name.
260 Often the same as the branch. (default: master)
261 :submodule-recursive: Whether to checkout submodules recursively.
263 :submodule-timeout: Timeout (in minutes) for checkout operation.
265 :submodule-disable: Disable submodule checkout operation.
267 :tox-dir: Directory containing the project's tox.ini relative to
268 the workspace. The default uses tox.ini at the project root.
270 :tox-envs: Tox environments to run. If blank run everything described
271 in tox.ini. (default: '')
272 :gerrit_trigger_file_paths: Override file paths used to filter which file
273 modifications trigger a build. Refer to JJB documentation for "file-path" details.
274 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
280 Tox runner to verify a project after merge of a patch set. This job
281 is pyenv aware so if the image contains an installation of pyenv at
282 /opt/pyenv it will pick it up and run Python tests with the
283 appropriate Python versions. This job will set the following pyenv
284 variables before running.
288 export PYENV_ROOT="/opt/pyenv"
289 export PATH="$PYENV_ROOT/bin:$PATH"
293 - {project-name}-tox-merge-{stream}
297 :Comment Trigger: **remerge** post a comment with the trigger to launch
298 this job manually. Do not include any other text or vote in the
301 :Required Parameters:
303 :build-node: The node to run build on.
304 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
307 :Optional Parameters:
309 :branch: The branch to build against. (default: master)
310 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
311 :build-timeout: Timeout in minutes before aborting build. (default: 10)
312 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
313 :pre-build-script: Shell script to execute before the CLM builder.
314 For example, install prerequisites or move files to the repo root.
315 (default: a string with a shell comment)
316 :python-version: Python version to invoke pip install of tox-pyenv
318 :stream: Keyword representing a release code-name.
319 Often the same as the branch. (default: master)
320 :submodule-recursive: Whether to checkout submodules recursively.
322 :submodule-timeout: Timeout (in minutes) for checkout operation.
324 :submodule-disable: Disable submodule checkout operation.
326 :tox-dir: Directory containing the project's tox.ini relative to
327 the workspace. The default uses tox.ini at the project root.
329 :tox-envs: Tox environments to run. If blank run everything described
330 in tox.ini. (default: '')
331 :gerrit_trigger_file_paths: Override file paths used to filter which file
332 modifications trigger a build. Refer to JJB documentation for "file-path" details.
333 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
339 Creates and uploads distribution files on merge of a patch set. Runs
340 tox, builds a source distribution and (optionally) a binary
341 distribution, and uploads the distribution(s) to a PyPI repository.
342 The project git repository must have a setup.py file
343 with configuration for packaging the component.
345 This job should use a staging repository like testpypi.python.org,
346 which sets up use of release jobs to promote the distributions later.
347 This job can also use a public release area like the global PyPI
348 repository if the release process is not needed. These PyPI
349 repositories allow upload of a package at a specific version once,
350 they do not allow overwrite of a package. This means that a merge job
351 will fail in the upload step if the package version already exists in
352 the target repository.
354 The tox runner is pyenv aware so if the image contains an installation
355 of pyenv at /opt/pyenv it will pick it up and run Python tests with
356 the appropriate Python versions. The tox runner sets the following
357 pyenv variables before running.
361 export PYENV_ROOT="/opt/pyenv"
362 export PATH="$PYENV_ROOT/bin:$PATH"
364 Installable package projects should use the directory layout shown
365 below. All Python files are in a repo subdirectory separate from
366 non-Python files like documentation. This layout allows highly
367 specific build-job triggers in Jenkins using the subdirectory
368 paths. For example, a PyPI merge job should not run on a non-Python
369 file change such as documentation, because the job cannot upload the
379 ├── helloworld-package/
383 │ │ ├── helloworld.py
387 │ │ ├── helloworld_tests.py
388 │ │ └── helloworld_mocks.py
390 │ ├── requirements.txt
395 │ └── pypi-helloworld.yaml
402 Use of these templates requires a .pypirc configuration file in the
403 Jenkins builder home directory. An example appears next that uses API
404 tokens. Note that in the [pypi] entry the repository key-value pair is
405 optional, it defaults to pypi.org.
409 [distutils] # this tells distutils what package indexes you can push to
410 index-servers = pypi-test pypi
413 repository: https://test.pypi.org/legacy/
415 password: pypi-test-api-token-goes-here
419 password: pypi-api-token-goes-here
424 - {project-name}-pypi-merge-{stream}
428 :Comment Trigger: **remerge** post a comment with the trigger to launch
429 this job manually. Do not include any other text or vote in the
432 :Required Parameters:
434 :build-node: The node to run the build on.
435 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
437 :project: Git repository name
438 :project-name: Jenkins job name prefix
440 :Optional Parameters:
442 :branch: The branch to build against. (default: master)
443 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
444 :build-timeout: Timeout in minutes before aborting build. (default: 15)
445 :cron: Cron schedule when to trigger the job. Supports daily builds.
446 This parameter also supports multiline input via YAML pipe | character in
447 cases where one may want to provide more than 1 cron timer. (default: empty)
448 :disable-job: Whether to disable the job (default: false)
449 :dist-binary: Whether to build a binary wheel distribution. (default: true)
450 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
451 :parallel: Boolean indicator for tox to run tests in parallel or series.
452 (default: false, in series)
453 :pre-build-script: Shell script to execute before the tox builder. For
454 example, install system prerequisites. (default: a shell comment)
455 :pypi-repo: Key for the PyPI target repository in the .pypirc file,
456 ideally a server like test.pypy.org. (default: pypi-test)
457 :python-version: Python version to invoke pip install of tox-pyenv
459 :stream: Keyword representing a release code-name.
460 Often the same as the branch. (default: master)
461 :submodule-recursive: Whether to checkout submodules recursively.
463 :submodule-timeout: Timeout (in minutes) for checkout operation.
465 :submodule-disable: Disable submodule checkout operation.
467 :tox-dir: Directory containing the project's tox.ini relative to
468 the workspace. The default uses tox.ini at the project root.
470 :tox-envs: Tox environments to run. If blank run everything described
471 in tox.ini. (default: '')
472 :gerrit_trigger_file_paths: Override file paths used to filter which file
473 modifications trigger a build. Refer to JJB documentation for "file-path" details.
474 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
480 Verifies a Python library project on creation of a patch set. Runs tox
481 then builds a source distribution and (optionally) a binary
482 distribution. The project repository must have a setup.py file with
483 configuration for packaging the component.
485 The tox runner is pyenv aware so if the image contains an installation
486 of pyenv at /opt/pyenv it will pick it up and run Python tests with
487 the appropriate Python versions. The tox runner sets the following
488 pyenv variables before running.
492 export PYENV_ROOT="/opt/pyenv"
493 export PATH="$PYENV_ROOT/bin:$PATH"
497 - {project-name}-pypi-verify-{stream}
501 :Comment Trigger: **recheck|reverify** post a comment with one of the
502 triggers to launch this job manually. Do not include any other
503 text or vote in the same comment.
505 :Required Parameters:
507 :build-node: The node to run the build on.
508 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
510 :project: Git repository name
511 :project-name: Jenkins job name prefix
513 :Optional Parameters:
515 :branch: The branch to build against. (default: master)
516 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
517 :build-timeout: Timeout in minutes before aborting build. (default: 15)
518 :disable-job: Whether to disable the job (default: false)
519 :dist-binary: Whether to build a binary wheel distribution. (default: true)
520 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
521 :parallel: Boolean indicator for tox to run tests in parallel or series.
522 (default: false, in series)
523 :pre-build-script: Shell script to execute before the tox builder. For
524 example, install system prerequisites. (default: a shell comment)
525 :python-version: Python version to invoke pip install of tox-pyenv
527 :stream: Keyword representing a release code-name.
528 Often the same as the branch. (default: master)
529 :submodule-recursive: Whether to checkout submodules recursively.
531 :submodule-timeout: Timeout (in minutes) for checkout operation.
533 :submodule-disable: Disable submodule checkout operation.
535 :tox-dir: Directory containing the project's tox.ini relative to
536 the workspace. The default uses tox.ini at the project root.
538 :tox-envs: Tox environments to run. If blank run everything described
539 in tox.ini. (default: '')
540 :gerrit_trigger_file_paths: Override file paths used to filter which file
541 modifications trigger a build. Refer to JJB documentation for "file-path" details.
542 https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit