From: Lott, Christopher (cl778h) Date: Mon, 23 Mar 2020 21:14:30 +0000 (-0400) Subject: Add Tox Nexus IQ (CLM) templates for Python X-Git-Tag: v0.53.1~1^2 X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=commitdiff_plain;h=refs%2Fchanges%2F87%2F63487%2F14;p=releng%2Fglobal-jjb.git Add Tox Nexus IQ (CLM) templates for Python Add new templates gerrit-tox-nexus-iq-clm/github-tox-nexus-iq-clm to analyze Python package requirements. Add script nexus-iq-python-cli.sh with correct invocation and arguments for the latest version of the Nexus IQ scanner. Drop broken templates g*it-python-xc-clm, unused builder lf-infra-clm-python and shell script nexus-iq-cli.sh. Change-Id: I5c34ad02e3e37a0e04dfa7e64b1013ebc26119fc Issue-ID: RELENG-2821 Signed-off-by: Lott, Christopher (cl778h) --- diff --git a/.jjb-test/lf-python-jobs.yaml b/.jjb-test/lf-python-jobs.yaml index 657a1d80..fcae63a7 100644 --- a/.jjb-test/lf-python-jobs.yaml +++ b/.jjb-test/lf-python-jobs.yaml @@ -8,6 +8,7 @@ - gerrit-pypi-release-verify - gerrit-pypi-stage - gerrit-pypi-verify + - gerrit-tox-nexus-iq-clm - gerrit-tox-sonar - gerrit-tox-sonarqube @@ -22,6 +23,7 @@ - github-pypi-release-verify - github-pypi-stage - github-pypi-verify + - github-tox-nexus-iq-clm - github-tox-sonar - github-tox-sonarqube diff --git a/docs/jjb/lf-python-jobs.rst b/docs/jjb/lf-python-jobs.rst index 475bc2ab..410e30da 100644 --- a/docs/jjb/lf-python-jobs.rst +++ b/docs/jjb/lf-python-jobs.rst @@ -16,14 +16,15 @@ Below is a list of Python job groups: Macros ====== -lf-infra-clm-python -------------------- +lf-infra-nexus-iq-python-cli +---------------------------- -Runs CLM scanning against a Python project. +Runs Nexus IQ command-line interface CLM scan on Python package requirements. :Required Parameters: - :clm-project-name: Project name in Nexus IQ to send results to. + :nexus-iq-project-name: Project name in Nexus IQ to send results to. + :requirements-file: File name with output of pip freeze. lf-infra-tox-install -------------------- @@ -49,59 +50,82 @@ Creates a Tox virtual environment and invokes tox. Job Templates ============= -Python XC CLM -------------- +Tox Nexus IQ CLM +---------------- + +The Nexus IQ job invokes tox and the Nexus IQ scanner to analyze packages for +component lifecycle management (CLM). Runs tox to discover the required packages, +downloads the command-line interface (CLI) scanner, runs the scanner on the package +list, then uploads the results to a Nexus IQ server. The project's tox.ini file must +define a test environment that runs 'pip freeze' and captures the output; that +environment does not need to execute any tests. For example: + +.. code-block:: bash + + [testenv:clm] + # use pip to report dependencies with versions + whitelist_externals = sh + commands = sh -c 'pip freeze > requirements.txt' -CLM scans for Python based repos. This job will call the Nexus IQ CLI -directly to run the scans. -A new credential named "nexus-iq-xc-clm" needs to exist in the Jenkins -credentials. The credential should contain the username and password -to access Nexus IQ Server. +This job runs on the master branch because the basic Nexus IQ configuration +does not support multi-branch. :Template Names: - - {project-name}-python-clm-{stream} - - gerrit-python-xc-clm - - github-python-xc-clm + - {project-name}-tox-nexus-iq-clm + - gerrit-tox-nexus-iq-clm + - github-tox-nexus-iq-clm -:Comment Trigger: **run-clm** post a comment with the trigger to launch - this job manually. Do not include any other text or vote in the - same comment. +:Comment Trigger: ``run-clm`` :Required parameters: - :build-node: The node to run build on. - :jenkins-ssh-credential: Credential to use for SSH. (Generally should - get configured in defaults.yaml) + :build-node: The node to run the build on. + (Commonly in defaults.yaml) + :jenkins-ssh-credential: Credential to use for SSH. + (Commonly in defaults.yaml) + :project: The git repository name. + :project-name: Prefix used to name jobs. -:Optional parameters: +:Optional Parameters: + :archive-artifacts: Pattern for files to archive to the logs server + (default: '\*\*/\*.log') + :branch: Git branch, should be master (default: master) :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 15) + :cron: Cron schedule when to trigger the job. This parameter also + supports multiline input via the YAML pipe | character to allow + more than 1 cron timer. (default: @weekly) + :disable-job: Whether to disable the job (default: false) + :gerrit_nexusiq_triggers: Override Gerrit Triggers. + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :github-url: URL for Github. (default: https://github.com) + :java-version: Version of Java to use for the scan. (default: openjdk8) :nexus-iq-cli-version: Nexus IQ CLI package version to download and use. - (default: 1.44.0-01) + (default is a string like 1.89.0-02, see file lf-python-jobs.yaml) :nexus-iq-namespace: Insert a namespace to project AppID for projects that share a Nexus IQ system to avoid project name collision. We recommend inserting a trailing - dash if using this parameter. For example 'odl-'. (default: '') - :build-timeout: Timeout in minutes before aborting build. (default: 60) - :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) - :java-version: Version of Java to use for the build. (default: openjdk8) - :pre-build-script: Shell script to execute before the CLM builder. - For example, install prerequisites or move files to the repo root. - (default: a string with a shell comment) - :stream: Keyword used to represent a release code-name. - Often the same as the branch. (default: master) + :pre-build-script: Shell script to run before tox. Useful for setting up + dependencies. (default: a string with a shell comment) + :python-version: Python version to invoke pip install of tox-pyenv + (default: python3) + :requirements-file: Name of file with output of pip freeze. + (default: requirements.txt) :submodule-recursive: Whether to checkout submodules recursively. (default: true) :submodule-timeout: Timeout (in minutes) for checkout operation. (default: 10) :submodule-disable: Disable submodule checkout operation. (default: false) - :gerrit_clm_triggers: Override Gerrit Triggers. - :gerrit_trigger_file_paths: Override file paths used to filter which file - modifications trigger a build. Refer to JJB documentation for "file-path" details. - https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit + :tox-dir: Directory containing the project's tox.ini relative to + the workspace. The default uses tox.ini at the project root. + (default: '.') + :tox-envs: Tox environment with the appropriate pip freeze invocation. + (default: 'clm') Python Sonar with Tox diff --git a/jjb/lf-python-jobs.yaml b/jjb/lf-python-jobs.yaml index c3206e59..c29dd5d8 100644 --- a/jjb/lf-python-jobs.yaml +++ b/jjb/lf-python-jobs.yaml @@ -4,11 +4,13 @@ ########## - builder: - name: lf-infra-clm-python + name: lf-infra-nexus-iq-python-cli builders: - inject: - properties-content: "CLM_PROJECT_NAME={clm-project-name}" - - shell: !include-raw-escape: ../shell/nexus-iq-cli.sh + properties-content: | + NEXUS_IQ_PROJECT_NAME={nexus-iq-project-name} + REQUIREMENTS_FILE={requirements-file} + - shell: !include-raw-escape: ../shell/nexus-iq-python-cli.sh - builder: name: lf-infra-tox-install @@ -63,39 +65,36 @@ publishers: - lf-infra-publish -################# -# Python XC CLM # -################# +#################### +# Tox Nexus IQ CLM # +#################### -- lf_python_clm_xc: &lf_python_xc_clm - name: lf-python-xc-clm +- lf_tox_nexus_iq_clm: &lf_tox_nexus_iq_clm + name: lf-tox-nexus-iq-clm ###################### # Default parameters # ###################### branch: master - build-days-to-keep: 30 # 30 days for troubleshooting purposes - build-timeout: 60 + build-days-to-keep: 7 + build-timeout: 15 + cron: "@weekly" disable-job: false git-url: "$GIT_URL/$PROJECT" github-url: "https://github.com" - java-version: openjdk8 - nexus-iq-cli-version: 1.44.0-01 + java-version: openjdk8 # Scanner is a jar + nexus-iq-cli-version: 1.89.0-02 nexus-iq-namespace: "" # Recommend a trailing dash when set. Example: odl- - pre-build-script: " # pre-build script goes here" + pre-build-script: "# pre-build script goes here" + python-version: python3 + requirements-file: requirements.txt stream: master submodule-recursive: true submodule-timeout: 10 submodule-disable: false - - gerrit_trigger_file_paths: - - compare-type: REG_EXP - pattern: ".*" - - # github_included_regions MUST match gerrit_trigger_file_paths - github_included_regions: - - ".*" + tox-dir: "." + tox-envs: "clm" ##################### # Job Configuration # @@ -108,35 +107,52 @@ project: "{project}" branch: "{branch}" stream: "{stream}" + - lf-infra-tox-parameters: + tox-dir: "{tox-dir}" + tox-envs: "{tox-envs}" + - string: + name: ARCHIVE_ARTIFACTS + default: "{archive-artifacts}" + description: Artifacts to archive to the logs server. - string: name: NEXUS_IQ_CLI_VERSION default: "{nexus-iq-cli-version}" - description: Nexus IQ CLI package to download and use. + description: Nexus IQ CLI jar to download and run. wrappers: + - lf-infra-wrappers: + build-timeout: "{build-timeout}" + jenkins-ssh-credential: "{jenkins-ssh-credential}" - credentials-binding: - username-password-separated: credential-id: nexus-iq-xc-clm - username: CLM_USER - password: CLM_PASSWORD + username: NEXUS_IQ_USER + password: NEXUS_IQ_PASSWORD + builders: + - lf-infra-pre-build - lf-update-java-alternatives: java-version: "{java-version}" + - lf-infra-tox-install: + python-version: "{python-version}" - shell: "{pre-build-script}" - - lf-infra-clm-python: - clm-project-name: "{nexus-iq-namespace}{project-name}" + - lf-infra-tox-run: + parallel: false + - lf-infra-nexus-iq-python-cli: + nexus-iq-project-name: "{nexus-iq-namespace}{project-name}" + requirements-file: "{requirements-file}" - job-template: - name: "{project-name}-python-clm-{stream}" - id: gerrit-python-xc-clm + name: "{project-name}-tox-nexus-iq-clm" + id: gerrit-tox-nexus-iq-clm <<: *lf_python_common - <<: *lf_python_xc_clm + <<: *lf_tox_nexus_iq_clm ###################### # Default parameters # ###################### - gerrit_clm_triggers: + gerrit_nexus_iq_triggers: - comment-added-contains-event: comment-contains-value: '^Patch Set\s+\d+:\s+run-clm\s*$' @@ -156,18 +172,16 @@ choosing-strategy: default triggers: - # Build weekly on Saturdays - - timed: "H H * * 6" + - timed: "{obj:cron}" - gerrit: server-name: "{gerrit-server-name}" - trigger-on: "{obj:gerrit_clm_triggers}" + trigger-on: "{obj:gerrit_nexus_iq_triggers}" projects: - project-compare-type: ANT project-pattern: "{project}" branches: - branch-compare-type: ANT branch-pattern: "**/{branch}" - file-paths: "{obj:gerrit_trigger_file_paths}" skip-vote: successful: true failed: true @@ -175,10 +189,10 @@ notbuilt: true - job-template: - name: "{project-name}-python-clm-{stream}" - id: github-python-xc-clm + name: "{project-name}-tox-nexus-iq" + id: github-tox-nexus-iq-clm <<: *lf_python_common - <<: *lf_python_xc_clm + <<: *lf_tox_nexus_iq_clm properties: - lf-infra-properties: @@ -198,8 +212,7 @@ jenkins-ssh-credential: "{jenkins-ssh-credential}" triggers: - # Build weekly on Saturdays - - timed: "H H * * 6" + - timed: "{obj:cron}" - github-pull-request: trigger-phrase: "^run-clm$" only-trigger-phrase: false @@ -208,7 +221,6 @@ github-hooks: true white-list-target-branches: - "{branch}" - included-regions: "{obj:github_included_regions}" ######################### # Python Sonar with Tox # diff --git a/releasenotes/notes/tox-nexus-iq-python-jobs-3fda2e30d7f3e5ef.yaml b/releasenotes/notes/tox-nexus-iq-python-jobs-3fda2e30d7f3e5ef.yaml new file mode 100644 index 00000000..ab79ee6b --- /dev/null +++ b/releasenotes/notes/tox-nexus-iq-python-jobs-3fda2e30d7f3e5ef.yaml @@ -0,0 +1,10 @@ +--- +features: + - | + Add new templates gerrit-tox-nexus-iq/github-tox-nexus-iq to + analyze Python package requirements. Add script + nexus-iq-python-cli.sh with correct invocation and arguments + for the latest version of the Nexus IQ scanner. + + Drop broken templates g*it-python-xc-clm, unused builder + lf-infra-clm-python and shell script nexus-iq-cli.sh. diff --git a/shell/nexus-iq-cli.sh b/shell/nexus-iq-python-cli.sh similarity index 54% rename from shell/nexus-iq-cli.sh rename to shell/nexus-iq-python-cli.sh index b4f8cef5..b47bf74d 100644 --- a/shell/nexus-iq-cli.sh +++ b/shell/nexus-iq-python-cli.sh @@ -1,31 +1,36 @@ #!/bin/bash # SPDX-License-Identifier: EPL-1.0 ############################################################################## -# Copyright (c) 2018 The Linux Foundation and others. +# Copyright (c) 2020 The Linux Foundation and others. # # All rights reserved. This program and the accompanying materials # are made available under the terms of the Eclipse Public License v1.0 # which accompanies this distribution, and is available at # http://www.eclipse.org/legal/epl-v10.html ############################################################################## -echo "---> nexus-iq-cli.sh" +echo "---> nexus-iq-python-cli.sh" # This script downloads the specified version of the nexus-iq-cli jar, uses it -# to perform an XC Evaluation or extended report which provides a scan of python -# files within the repo starting at the root, then publishes the result to an LF -# server using the specified credentials. +# to analyze the Python project dependencies from the specified requirements file, +# then publishes the result to an LF server using the specified credentials. # stop on error or unbound variable set -eu # do not print commands, credentials should not be logged set +x +# check if tox established the prereqs for this analysis +if [[ ! -f $REQUIREMENTS_FILE ]]; then + echo "ERROR: failed to find file $REQUIREMENTS_FILE" + exit 1 +fi CLI_LOCATION="/tmp/nexus-iq-cli-${NEXUS_IQ_CLI_VERSION}.jar" echo "INFO: downloading nexus-iq-cli version $NEXUS_IQ_CLI_VERSION" wget -nv "https://download.sonatype.com/clm/scanner/nexus-iq-cli-${NEXUS_IQ_CLI_VERSION}.jar" -O "${CLI_LOCATION}" echo "-a" > cli-auth.txt -echo "${CLM_USER}:${CLM_PASSWORD}" >> cli-auth.txt -echo "INFO: running nexus-iq-cli scan on project $CLM_PROJECT_NAME" -java -jar "${CLI_LOCATION}" @cli-auth.txt -xc -i "${CLM_PROJECT_NAME}" -s https://nexus-iq.wl.linuxfoundation.org -t build . +echo "${NEXUS_IQ_USER}:${NEXUS_IQ_PASSWORD}" >> cli-auth.txt +echo "INFO: running nexus-iq-cli on project $NEXUS_IQ_PROJECT_NAME and file $REQUIREMENTS_FILE" +# result.json is a mystery +java -jar "${CLI_LOCATION}" @cli-auth.txt -s https://nexus-iq.wl.linuxfoundation.org -i "${NEXUS_IQ_PROJECT_NAME}" -t build -r result.json "${REQUIREMENTS_FILE}" rm cli-auth.txt rm "${CLI_LOCATION}" -echo "---> nexus-iq-cli.sh ends" +echo "---> nexus-iq-python-cli.sh ends"