X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=blobdiff_plain;f=docs%2Fjjb%2Flf-python-jobs.rst;h=117ab3875191f9ca2aa28f7e929318f245008d7b;hb=30fb028f8fe8376c690c2e0b4e82597a8a66ea05;hp=44a09ca242dcd9eea4619316d526a5cf9ca78d3d;hpb=7eea20901e957ef43de1918119c75688f5eb87ba;p=releng%2Fglobal-jjb.git diff --git a/docs/jjb/lf-python-jobs.rst b/docs/jjb/lf-python-jobs.rst index 44a09ca2..117ab387 100644 --- a/docs/jjb/lf-python-jobs.rst +++ b/docs/jjb/lf-python-jobs.rst @@ -5,23 +5,12 @@ Python Jobs Job Groups ========== -{project-name}-python-jobs --------------------------- +.. include:: ../job-groups.rst -Jobs for Python projects using Gerrit. +Below is a list of Python job groups: -:Includes: - - - gerrit-tox-verify - -{project-name}-github-python-jobs ---------------------------------- - -Jobs for Python projects using GitHub. - -:Includes: - - - github-tox-verify +.. literalinclude:: ../../jjb/lf-python-job-groups.yaml + :language: yaml Macros @@ -30,7 +19,7 @@ Macros lf-infra-clm-python ------------------- -Run CLM scanning against a Python project. +Runs CLM scanning against a Python project. :Required Parameters: @@ -39,32 +28,22 @@ Run CLM scanning against a Python project. lf-infra-tox-install -------------------- -Install Tox into a virtualenv. +Installs Tox into a virtualenv. :Required Parameters: - :python-version: Version of Python to install into the Tox virtualenv. - Eg. python2 / python3 + :python-version: Version of Python to invoke the pip install of the tox-pyenv + package that creates a virtual environment, either "python2" or "python3". -lf-infra-tox-sonar ------------------- +lf-infra-tox-run +---------------- -Runs Sonar scanning against a Python project. +Creates a Tox virtual environment and invokes tox. :Required Parameters: - :java-version: Version of Java to use to run Sonar. - :mvn-version: Version of Maven to use to run Sonar. - -lf-tox-install --------------- - -Runs a shell script that installs tox in a Python virtualenv. - -:Required Parameters: - - :python-version: Base Python version to use in the virtualenv. For example - python2 or python3. + :parallel: Boolean. If true use detox (distributed tox); + else use regular tox. Job Templates @@ -76,9 +55,9 @@ Python XC CLM CLM scans for Python based repos. This job will call the Nexus IQ CLI directly to run the scans. -A new credential named "nexus-iq-xc-clm" needs to exist in the Jenkins credentials. -The credential should contain the username and password to access Nexus -IQ Server. +A new credential named "nexus-iq-xc-clm" needs to exist in the Jenkins +credentials. The credential should contain the username and password +to access Nexus IQ Server. :Template Names: @@ -86,6 +65,10 @@ IQ Server. - gerrit-python-xc-clm - github-python-xc-clm +:Comment Trigger: **run-clm** post a comment with the trigger to launch + this job manually. Do not include any other text or vote in the + same comment. + :Required parameters: :build-node: The node to run build on. @@ -95,34 +78,54 @@ IQ Server. :Optional parameters: :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) - :nexus-iq-cli-version: Nexus IQ CLI package version to download and use. (default: 1.44.0-01) - :build-timeout: Timeout in seconds before aborting build. (default: 60) + :nexus-iq-cli-version: Nexus IQ CLI package version to download and use. + (default: 1.44.0-01) + :nexus-iq-namespace: Insert a namespace to project AppID for projects that + share a Nexus IQ system to avoid project name collision. We recommend + inserting a trailing - dash if using this parameter. + For example 'odl-'. (default: '') + :build-timeout: Timeout in minutes before aborting build. (default: 60) :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) :java-version: Version of Java to use for the build. (default: openjdk8) + :pre-build-script: Shell script to execute before the CLM builder. + For example, install prerequisites or move files to the repo root. + (default: a string with a shell comment) :stream: Keyword used to represent a release code-name. Often the same as the branch. (default: master) :submodule-recursive: Whether to checkout submodules recursively. (default: true) + :submodule-timeout: Timeout (in minutes) for checkout operation. + (default: 10) + :submodule-disable: Disable submodule checkout operation. + (default: false) :gerrit_clm_triggers: Override Gerrit Triggers. - :gerrit_trigger_file_paths: Override file paths which used to filter which - file modifications will trigger a build. Refer to JJB documentation for - "file-path" details. + :gerrit_trigger_file_paths: Override file paths used to filter which file + modifications trigger a build. Refer to JJB documentation for "file-path" details. https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit Python Sonar with Tox --------------------- -Sonar scans for Python based repos. This job will perform a tox call which -runs the coverage command to gather tests results. These test results get -published back into Sonar after running the Sonar goals. +Sonar scans for Python based repos. This job invokes tox to run tests +and gather coverage statistics from the test results, then invokes +Maven to publish the results to either a Sonar server or SonarCloud. -To get the Sonar coverage results, tox.ini needs to exist and configured -with coverage commands to run. +To get the Sonar coverage results, file tox.ini must exist and contain +coverage commands to run. -The coverage commands define the code that gets executed by the test suites. -It does not guarantee that the code tests executed properly, but it will help -pointing out the code that is not tested at all. +The coverage commands define the code that gets executed by the test +suites. Checking coverage does not guarantee that the tests execute +properly, but it identifies code that is not executed by any test. + +.. comment Start ignoring WriteGoodLintBear + +This job reuses the Sonar builders used for Java/Maven projects which +run maven twice. The first invocation does nothing for Python +projects, so the job uses the goal 'validate' by default. The second +invocation publishes results using the goal 'sonar:sonar' by default. + +.. comment Stop ignoring For example: @@ -146,44 +149,78 @@ https://docs.sonarqube.org/display/PLUG/Python+Coverage+Results+Import - gerrit-tox-sonar - github-tox-sonar +:Comment Trigger: **run-sonar** post a comment with the trigger to launch + this job manually. Do not include any other text or vote in the + same comment. + :Required parameters: :build-node: The node to run build on. :jenkins-ssh-credential: Credential to use for SSH. (Generally should get configured in defaults.yaml) - :mvn-settings: The name of settings file containing credentials for the project. + :mvn-settings: The name of the settings file with credentials for the project. + +.. comment Start ignoring WriteGoodLintBear :Optional parameters: + :branch: Git branch, should be master (default: master) :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) - :build-timeout: Timeout in seconds before aborting build. (default: 60) + :build-timeout: Timeout in minutes before aborting build. (default: 60) :cron: Cron schedule when to trigger the job. This parameter also supports multiline input via YAML pipe | character in cases where one may want to provide more than 1 cron timer. (default: H 11 * * * to run once a day) + :disable-job: Whether to disable the job (default: false) :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :github-url: URL for Github. (default: https://github.com) :java-version: Version of Java to use for the build. (default: openjdk8) - :mvn-global-settings: The name of the Maven global settings to use for - Maven configuration. (default: global-settings) + :mvn-global-settings: The name of the Maven global settings to use + :mvn-goals: The Maven goal to run first. (default: validate) :mvn-version: Version of maven to use. (default: mvn35) + :parallel: Boolean indicator for tox to run tests in parallel or series. + (default: false, in series) + :pre-build-script: Shell script to execute before the Sonar builder. + For example, install prerequisites or move files to the repo root. + (default: a string with a shell comment) + :python-version: Python version to invoke pip install of tox-pyenv + (default: python2) + :sonarcloud: Boolean indicator to use SonarCloud ``true|false``. + (default: false) + :sonarcloud-project-key: SonarCloud project key. (default: '') + :sonarcloud-project-organization: SonarCloud project organization. + (default: '') + :sonarcloud-api-token: SonarCloud API Token. (default: '') + :sonar-mvn-goal: The Maven goal to run the Sonar plugin. (default: sonar:sonar) :stream: Keyword used to represent a release code-name. Often the same as the branch. (default: master) :submodule-recursive: Whether to checkout submodules recursively. (default: true) + :submodule-timeout: Timeout (in minutes) for checkout operation. + (default: 10) + :submodule-disable: Disable submodule checkout operation. + (default: false) + :tox-dir: Directory containing the project's tox.ini relative to + the workspace. The default uses tox.ini at the project root. + (default: '.') + :tox-envs: Tox environments to run. If blank run everything described + in tox.ini. (default: '') :gerrit_sonar_triggers: Override Gerrit Triggers. - :gerrit_trigger_file_paths: Override file paths which used to filter which - file modifications will trigger a build. Refer to JJB documentation for - "file-path" details. + :gerrit_trigger_file_paths: Override file paths used to filter which file + modifications trigger a build. Refer to JJB documentation for "file-path" details. https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit +.. comment Stop ignoring + Tox Verify ---------- -Tox runner to verify a project. This job is pyenv aware so if the image -contains an installation of pyenv at /opt/pyenv it will pick it up and run -Python tests with the appropriate Python versions. This job will set the -following pyenv variables before running. +Tox runner to verify a project on creation of a patch set. This job +is pyenv aware so if the image contains an installation of pyenv at +/opt/pyenv it will pick it up and run Python tests with the +appropriate Python versions. This job will set the following pyenv +variables before running. .. code:: bash @@ -196,6 +233,71 @@ following pyenv variables before running. - gerrit-tox-verify - github-tox-verify +:Comment Trigger: **recheck|reverify** post a comment with one of the + triggers to launch this job manually. Do not include any other + text or vote in the same comment. + +:Required Parameters: + + :build-node: The node to run build on. + :jenkins-ssh-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + +:Optional Parameters: + + :branch: The branch to build against. (default: master) + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 10) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :pre-build-script: Shell script to execute before the Tox builder. + For example, install prerequisites or move files to the repo root. + (default: a string with a shell comment) + :parallel: Boolean indicator for tox to run tests in parallel or series. + (default: false, in series) + :python-version: Python version to invoke pip install of tox-pyenv + (default: python2) + :stream: Keyword representing a release code-name. + Often the same as the branch. (default: master) + :submodule-recursive: Whether to checkout submodules recursively. + (default: true) + :submodule-timeout: Timeout (in minutes) for checkout operation. + (default: 10) + :submodule-disable: Disable submodule checkout operation. + (default: false) + :tox-dir: Directory containing the project's tox.ini relative to + the workspace. The default uses tox.ini at the project root. + (default: '.') + :tox-envs: Tox environments to run. If blank run everything described + in tox.ini. (default: '') + :gerrit_trigger_file_paths: Override file paths used to filter which file + modifications trigger a build. Refer to JJB documentation for "file-path" details. + https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit + + +Tox Merge +--------- + +Tox runner to verify a project after merge of a patch set. This job +is pyenv aware so if the image contains an installation of pyenv at +/opt/pyenv it will pick it up and run Python tests with the +appropriate Python versions. This job will set the following pyenv +variables before running. + +.. code:: bash + + export PYENV_ROOT="/opt/pyenv" + export PATH="$PYENV_ROOT/bin:$PATH" + +:Template Names: + + - {project-name}-tox-merge-{stream} + - gerrit-tox-merge + - github-tox-merge + +:Comment Trigger: **remerge** post a comment with the trigger to launch + this job manually. Do not include any other text or vote in the + same comment. + :Required Parameters: :build-node: The node to run build on. @@ -206,20 +308,248 @@ following pyenv variables before running. :branch: The branch to build against. (default: master) :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) - :build-timeout: Timeout in seconds before aborting build. (default: 10) + :build-timeout: Timeout in minutes before aborting build. (default: 10) :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) - :python-version: Version of Python to configure as a base in virtualenv. + :pre-build-script: Shell script to execute before the CLM builder. + For example, install prerequisites or move files to the repo root. + (default: a string with a shell comment) + :python-version: Python version to invoke pip install of tox-pyenv + (default: python2) + :stream: Keyword representing a release code-name. + Often the same as the branch. (default: master) + :submodule-recursive: Whether to checkout submodules recursively. + (default: true) + :submodule-timeout: Timeout (in minutes) for checkout operation. + (default: 10) + :submodule-disable: Disable submodule checkout operation. + (default: false) + :tox-dir: Directory containing the project's tox.ini relative to + the workspace. The default uses tox.ini at the project root. + (default: '.') + :tox-envs: Tox environments to run. If blank run everything described + in tox.ini. (default: '') + :gerrit_trigger_file_paths: Override file paths used to filter which file + modifications trigger a build. Refer to JJB documentation for "file-path" details. + https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit + + +PyPI Merge +---------- + +Creates and uploads distribution files on merge of a patch set. Runs +tox, builds a source distribution and (optionally) a binary +distribution, and uploads the distribution(s) to a PyPI repository. +The project git repository must have a setup.py file +with configuration for packaging the component. + +This job should use a staging repository like testpypi.python.org, +which sets up use of release jobs to promote the distributions later. +This job can also use a public release area like the global PyPI +repository if the release process is not needed. These PyPI +repositories allow upload of a package at a specific version once, +they do not allow overwrite of a package. This means that a merge job +will fail in the upload step if the package version already exists in +the target repository. + +The tox runner is pyenv aware so if the image contains an installation +of pyenv at /opt/pyenv it will pick it up and run Python tests with +the appropriate Python versions. The tox runner sets the following +pyenv variables before running. + +.. code:: bash + + export PYENV_ROOT="/opt/pyenv" + export PATH="$PYENV_ROOT/bin:$PATH" + +Installable package projects should use the directory layout shown +below. All Python files are in a repo subdirectory separate from +non-Python files like documentation. This layout allows highly +specific build-job triggers in Jenkins using the subdirectory +paths. For example, a PyPI merge job should not run on a non-Python +file change such as documentation, because the job cannot upload the +same package twice. + +To make the document files available for building a Python package +long description in setup.py, add a symbolic link "docs" in the +package subdirectory pointing to the top-level docs directory. + +.. code-block:: bash + + git-repo-name/ + │ + ├── docs/ + │ ├── index.rst + │ └── release-notes.rst + │ + ├── helloworld-package/ + │ │ + │ └── helloworld/ + │ │ ├── __init__.py + │ │ ├── helloworld.py + │ │ └── helpers.py + │ │ + │ ├── tests/ + │ │ ├── helloworld_tests.py + │ │ └── helloworld_mocks.py + │ │ + │ ├── requirements.txt + │ └── setup.py + │ └── tox.ini + │ + ├── releases/ + │ └── pypi-helloworld.yaml + │ + ├── .gitignore + ├── LICENSE + └── README.md + + +Jobs built from the PyPI templates depend on a .pypirc configuration file +in the Jenkins builder home directory. An example appears next that uses +API tokens. Note that in the [pypi] entry the repository key-value pair +is optional, it defaults to pypi.org. + +.. code-block:: bash + + [distutils] # this tells distutils what package indexes you can push to + index-servers = pypi-test pypi + + [pypi-test] + repository: https://test.pypi.org/legacy/ + username: __token__ + password: pypi-test-api-token-goes-here + + [pypi] + username: __token__ + password: pypi-api-token-goes-here + + +:Template Names: + + - {project-name}-pypi-merge-{stream} + - gerrit-pypi-merge + - github-pypi-merge + +:Comment Trigger: **remerge** post a comment with the trigger to launch + this job manually. Do not include any other text or vote in the + same comment. + +:Required Parameters: + + :build-node: The node to run the build on. + :jenkins-ssh-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + :mvn-settings: The settings file with credentials for the project + :project: Git repository name + :project-name: Jenkins job name prefix + +:Optional Parameters: + + :branch: The branch to build against. (default: master) + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 15) + :cron: Cron schedule when to trigger the job. Supports daily builds. + This parameter also supports multiline input via YAML pipe | character in + cases where one may want to provide more than 1 cron timer. (default: empty) + :disable-job: Whether to disable the job (default: false) + :dist-binary: Whether to build a binary wheel distribution. (default: true) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :mvn-opts: Sets MAVEN_OPTS to start up the JVM running Maven. (default: '') + :mvn-params: Parameters to pass to the mvn CLI. (default: '') + :mvn-version: Version of maven to use. (default: mvn35) + :parallel: Boolean indicator for tox to run tests in parallel or series. + (default: false, in series) + :pre-build-script: Shell script to execute before the tox builder. For + example, install system prerequisites. (default: a shell comment) + :pypi-repo: Key for the PyPI target repository in the .pypirc file, + ideally a server like test.pypy.org. (default: pypi-test) + :python-version: Python version to invoke pip install of tox-pyenv (default: python3) :stream: Keyword representing a release code-name. Often the same as the branch. (default: master) :submodule-recursive: Whether to checkout submodules recursively. (default: true) + :submodule-timeout: Timeout (in minutes) for checkout operation. + (default: 10) + :submodule-disable: Disable submodule checkout operation. + (default: false) :tox-dir: Directory containing the project's tox.ini relative to - the workspace. Empty works if tox.ini is at project root. - (default: '') + the workspace. The default uses tox.ini at the project root. + (default: '.') + :tox-envs: Tox environments to run. If blank run everything described + in tox.ini. (default: '') + :gerrit_trigger_file_paths: Override file paths used to filter which file + modifications trigger a build. Refer to JJB documentation for "file-path" details. + https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit + + +PyPI Verify +----------- + +Verifies a Python library project on creation of a patch set. Runs tox +then builds a source distribution and (optionally) a binary +distribution. The project repository must have a setup.py file with +configuration for packaging the component. + +The tox runner is pyenv aware so if the image contains an installation +of pyenv at /opt/pyenv it will pick it up and run Python tests with +the appropriate Python versions. The tox runner sets the following +pyenv variables before running. + +.. code:: bash + + export PYENV_ROOT="/opt/pyenv" + export PATH="$PYENV_ROOT/bin:$PATH" + +:Template Names: + + - {project-name}-pypi-verify-{stream} + - gerrit-pypi-verify + - github-pypi-verify + +:Comment Trigger: **recheck|reverify** post a comment with one of the + triggers to launch this job manually. Do not include any other + text or vote in the same comment. + +:Required Parameters: + + :build-node: The node to run the build on. + :jenkins-ssh-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + :mvn-settings: The settings file with credentials for the project + :project: Git repository name + :project-name: Jenkins job name prefix + +:Optional Parameters: + + :branch: The branch to build against. (default: master) + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 15) + :disable-job: Whether to disable the job (default: false) + :dist-binary: Whether to build a binary wheel distribution. (default: true) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :mvn-opts: Sets MAVEN_OPTS to start up the JVM running Maven. (default: '') + :mvn-params: Parameters to pass to the mvn CLI. (default: '') + :mvn-version: Version of maven to use. (default: mvn35) + :parallel: Boolean indicator for tox to run tests in parallel or series. + (default: false, in series) + :pre-build-script: Shell script to execute before the tox builder. For + example, install system prerequisites. (default: a shell comment) + :python-version: Python version to invoke pip install of tox-pyenv + (default: python3) + :stream: Keyword representing a release code-name. + Often the same as the branch. (default: master) + :submodule-recursive: Whether to checkout submodules recursively. + (default: true) + :submodule-timeout: Timeout (in minutes) for checkout operation. + (default: 10) + :submodule-disable: Disable submodule checkout operation. + (default: false) + :tox-dir: Directory containing the project's tox.ini relative to + the workspace. The default uses tox.ini at the project root. + (default: '.') :tox-envs: Tox environments to run. If blank run everything described in tox.ini. (default: '') - :gerrit_trigger_file_paths: Override file paths which used to filter which - file modifications will trigger a build. Refer to JJB documentation for - "file-path" details. + :gerrit_trigger_file_paths: Override file paths used to filter which file + modifications trigger a build. Refer to JJB documentation for "file-path" details. https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit