From: Aric Gardner Date: Wed, 11 Sep 2019 15:22:00 +0000 (+0000) Subject: Merge "Add python templates to run tox on merge" X-Git-Tag: v0.43.0~1 X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=commitdiff_plain;ds=sidebyside;h=758df937f1010be8cc13fbeff0e727e2696ebeb6;hp=-c;p=releng%2Fglobal-jjb.git Merge "Add python templates to run tox on merge" --- 758df937f1010be8cc13fbeff0e727e2696ebeb6 diff --combined docs/jjb/lf-python-jobs.rst index 4972e696,8f940e8c..175f0e26 --- a/docs/jjb/lf-python-jobs.rst +++ b/docs/jjb/lf-python-jobs.rst @@@ -35,6 -35,16 +35,6 @@@ Install Tox into a virtualenv :python-version: Version of Python to install into the Tox virtualenv. Eg. python2 / python3 -lf-infra-tox-sonar ------------------- - -Runs Sonar scanning against a Python project. - -:Required Parameters: - - :java-version: Version of Java to use to run Sonar. - :mvn-version: Version of Maven to use to run Sonar. - lf-tox-install -------------- @@@ -87,7 -97,7 +87,7 @@@ IQ Server :java-version: Version of Java to use for the build. (default: openjdk8) :pre-build-script: Shell script to execute before the CLM builder. For example, install prerequisites or move files to the repo root. - (default: a string with only a comment) + (default: a string with a shell comment) :stream: Keyword used to represent a release code-name. Often the same as the branch. (default: master) :submodule-recursive: Whether to checkout submodules recursively. @@@ -106,21 -116,16 +106,21 @@@ Python Sonar with Tox --------------------- -Sonar scans for Python based repos. This job will perform a tox call which -runs the coverage command to gather tests results. These test results get -published back into Sonar after running the Sonar goals. +Sonar scans for Python based repos. This job invokes tox to run tests and +gather coverage statistics from the test results, then invokes Maven to +publish the results to a Sonar server. -To get the Sonar coverage results, tox.ini needs to exist and configured -with coverage commands to run. +To get the Sonar coverage results, file tox.ini must exist and contain coverage +commands to run. The coverage commands define the code that gets executed by the test suites. -It does not guarantee that the code tests executed properly, but it will help -pointing out the code that is not tested at all. +Checking coverage does not guarantee that the tests execute properly, but it +identifies code that is not executed by any test. + +This job reuses the Sonar builder used in Java/Maven projects which runs maven +twice. The first invocation does nothing for Python projects, so the job uses +the goal 'validate' by default. The second invocation publishes results using +the goal 'sonar:sonar' by default. For example: @@@ -151,29 -156,24 +151,29 @@@ https://docs.sonarqube.org/display/PLUG :build-node: The node to run build on. :jenkins-ssh-credential: Credential to use for SSH. (Generally should get configured in defaults.yaml) - :mvn-settings: The name of settings file containing credentials for the project. + :mvn-settings: The name of the settings file with credentials for the project. :Optional parameters: + :branch: Git branch, should be master (default: master) :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) :build-timeout: Timeout in minutes before aborting build. (default: 60) :cron: Cron schedule when to trigger the job. This parameter also supports multiline input via YAML pipe | character in cases where one may want to provide more than 1 cron timer. (default: H 11 * * * to run once a day) + :disable-job: Whether to disable the job (default: false) :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :github-url: URL for Github. (default: https://github.com) :java-version: Version of Java to use for the build. (default: openjdk8) :mvn-global-settings: The name of the Maven global settings to use for - Maven configuration. (default: global-settings) + :mvn-goals: The Maven goal to run first. (default: validate) :mvn-version: Version of maven to use. (default: mvn35) :pre-build-script: Shell script to execute before the Sonar builder. For example, install prerequisites or move files to the repo root. - (default: a string with only a comment) + (default: a string with a comment) + :python-version: Python version (default: python2) + :sonar-mvn-goal: The Maven goal to run the Sonar plugin. (default: sonar:sonar) :stream: Keyword used to represent a release code-name. Often the same as the branch. (default: master) :submodule-recursive: Whether to checkout submodules recursively. @@@ -192,10 -192,10 +192,10 @@@ Tox Verify ---------- - Tox runner to verify a project. This job is pyenv aware so if the image - contains an installation of pyenv at /opt/pyenv it will pick it up and run - Python tests with the appropriate Python versions. This job will set the - following pyenv variables before running. + Tox runner to verify a project on creation of a patch set. + This job is pyenv aware so if the image contains an installation of pyenv + at /opt/pyenv it will pick it up and run Python tests with the appropriate + Python versions. This job will set the following pyenv variables before running. .. code:: bash @@@ -224,7 -224,7 +224,7 @@@ :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) :pre-build-script: Shell script to execute before the Tox builder. For example, install prerequisites or move files to the repo root. - (default: a string with only a comment) + (default: a string with a shell comment) :python-version: Version of Python to configure as a base in virtualenv. (default: python3) :stream: Keyword representing a release code-name. @@@ -240,7 -240,64 +240,64 @@@ (default: '') :tox-envs: Tox environments to run. If blank run everything described in tox.ini. (default: '') - :gerrit_trigger_file_paths: Override file paths which used to filter which + :gerrit_trigger_file_paths: Override file paths used to filter which + file modifications will trigger a build. Refer to JJB documentation for + "file-path" details. + https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit + + + Tox Merge + --------- + + Tox runner to verify a project after merge of a patch set. + This job is pyenv aware so if the image contains an installation of pyenv + at /opt/pyenv it will pick it up and run Python tests with the appropriate + Python versions. This job will set the following pyenv variables before running. + + .. code:: bash + + export PYENV_ROOT="/opt/pyenv" + export PATH="$PYENV_ROOT/bin:$PATH" + + :Template Names: + + - {project-name}-tox-merge-{stream} + - gerrit-tox-merge + - github-tox-merge + + :Comment Trigger: remerge + + :Required Parameters: + + :build-node: The node to run build on. + :jenkins-ssh-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + + :Optional Parameters: + + :branch: The branch to build against. (default: master) + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 10) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :pre-build-script: Shell script to execute before the CLM builder. + For example, install prerequisites or move files to the repo root. + (default: a string with only a comment) + :python-version: Version of Python to configure as a base in virtualenv. + (default: python3) + :stream: Keyword representing a release code-name. + Often the same as the branch. (default: master) + :submodule-recursive: Whether to checkout submodules recursively. + (default: true) + :submodule-timeout: Timeout (in minutes) for checkout operation. + (default: 10) + :submodule-disable: Disable submodule checkout operation. + (default: false) + :tox-dir: Directory containing the project's tox.ini relative to + the workspace. Empty works if tox.ini is at project root. + (default: '') + :tox-envs: Tox environments to run. If blank run everything described + in tox.ini. (default: '') + :gerrit_trigger_file_paths: Override file paths used to filter which file modifications will trigger a build. Refer to JJB documentation for "file-path" details. https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit diff --combined jjb/lf-python-jobs.yaml index ffdc381b,9cd6d798..451eb635 --- a/jjb/lf-python-jobs.yaml +++ b/jjb/lf-python-jobs.yaml @@@ -7,7 -7,7 +7,7 @@@ name: lf-infra-tox-install builders: - inject: - properties-content: "PYTHON_VERSION={python-version}" + properties-content: "PYTHON={python-version}" - shell: !include-raw-escape: ../shell/tox-install.sh - builder: @@@ -206,9 -206,9 +206,9 @@@ - "{branch}" included-regions: "{obj:github_included_regions}" -######################## +######################### # Python Sonar with Tox # -######################## +######################### - lf_tox_sonar: &lf_tox_sonar name: lf-tox_sonar @@@ -225,12 -225,11 +225,12 @@@ git-url: "$GIT_URL/$PROJECT" github-url: "https://github.com" java-version: openjdk8 - pre-build-script: "# pre-build script goes here" - python-version: python2 mvn-global-settings: global-settings + mvn-goals: validate mvn-settings: "{mvn-settings}" mvn-version: mvn35 + pre-build-script: "# pre-build script goes here" + python-version: python2 sonar-mvn-goal: "sonar:sonar" stream: master submodule-recursive: true @@@ -284,15 -283,30 +284,15 @@@ - lf-provide-maven-settings: global-settings-file: "{mvn-global-settings}" settings-file: "{mvn-settings}" - - lf-infra-tox-sonar: + - lf-infra-maven-sonar: java-version: "{java-version}" + mvn-goals: "{mvn-goals}" mvn-settings: "{mvn-settings}" mvn-version: "{mvn-version}" publishers: - lf-infra-publish -- builder: - name: lf-infra-tox-sonar - # Run a Sonar build with Maven - builders: - - lf-maven-install: - mvn-version: "{mvn-version}" - - lf-update-java-alternatives: - java-version: "{java-version}" - - inject: - # TODO: Switch this to the sonar wrapper when JJB 2.0 is available - properties-content: SONAR_HOST_URL=$SONAR_URL - - shell: !include-raw-escape: - - ../shell/common-variables.sh - - ../shell/maven-sonar.sh - - lf-provide-maven-settings-cleanup - - job-template: name: "{project-name}-tox-sonar" id: gerrit-tox-sonar @@@ -377,11 -391,11 +377,11 @@@ included-regions: "{obj:github_included_regions}" ############## - # Tox Verify # + # Tox Common # ############## - - lf_tox_verify: &lf_tox_verify - name: lf-tox-verify + - lf_tox_common: &lf_tox_common + name: lf-tox-common ###################### # Default parameters # @@@ -456,7 -470,7 +456,7 @@@ # Python projects typically use tox to run testing. name: "{project-name}-tox-verify-{stream}" id: gerrit-tox-verify - <<: *lf_tox_verify + <<: *lf_tox_common ###################### # Default parameters # @@@ -508,7 -522,7 +508,7 @@@ # Python projects typically use tox to run testing. name: "{project-name}-tox-verify-{stream}" id: github-tox-verify - <<: *lf_tox_verify + <<: *lf_tox_common properties: - lf-infra-properties: @@@ -537,3 -551,83 +537,83 @@@ white-list-target-branches: - "{branch}" included-regions: "{obj:github_included_regions}" + + - job-template: + # Run tox after merge of gerrit change set + name: "{project-name}-tox-merge-{stream}" + id: gerrit-tox-merge + <<: *lf_tox_common + + ###################### + # Default parameters # + ###################### + + gerrit_merge_triggers: + - change-merged-event + - comment-added-contains-event: + comment-contains-value: '^Patch Set\s+\d+:\s+remerge\s*$' + + ##################### + # Job Configuration # + ##################### + + scm: + - lf-infra-gerrit-scm: + jenkins-ssh-credential: "{jenkins-ssh-credential}" + git-url: "{git-url}" + refspec: "$GERRIT_REFSPEC" + branch: "$GERRIT_BRANCH" + submodule-recursive: "{submodule-recursive}" + submodule-timeout: "{submodule-timeout}" + submodule-disable: "{submodule-disable}" + choosing-strategy: gerrit + + triggers: + - gerrit: + server-name: "{gerrit-server-name}" + trigger-on: "{obj:gerrit_merge_triggers}" + projects: + - project-compare-type: ANT + project-pattern: "{project}" + branches: + - branch-compare-type: ANT + branch-pattern: "**/{branch}" + file-paths: "{obj:gerrit_trigger_file_paths}" + + - job-template: + # Run tox after merge of github pull request + name: "{project-name}-tox-merge-{stream}" + id: github-tox-merge + <<: *lf_tox_common + + properties: + - lf-infra-properties: + build-days-to-keep: "{build-days-to-keep}" + - github: + url: "{github-url}/{github-org}/{project}" + + scm: + - lf-infra-github-scm: + url: "{git-clone-url}{github-org}/{project}" + refspec: "" + branch: "refs/heads/{branch}" + submodule-recursive: "{submodule-recursive}" + submodule-timeout: "{submodule-timeout}" + submodule-disable: "{submodule-disable}" + choosing-strategy: default + jenkins-ssh-credential: "{jenkins-ssh-credential}" + + triggers: + - github-pull-request: + trigger-phrase: "^remerge$" + only-trigger-phrase: true + status-context: "Tox Merge" + permit-all: true + github-hooks: true + org-list: + - "{github-org}" + white-list: "{obj:github_pr_whitelist}" + admin-list: "{obj:github_pr_admin_list}" + white-list-target-branches: + - "{branch}" + included-regions: "{obj:github_included_regions}"