- gerrit-pypi-release-verify
- gerrit-pypi-stage
- gerrit-pypi-verify
+ - gerrit-tox-nexus-iq-clm
- gerrit-tox-sonar
- gerrit-tox-sonarqube
- github-pypi-release-verify
- github-pypi-stage
- github-pypi-verify
+ - github-tox-nexus-iq-clm
- github-tox-sonar
- github-tox-sonarqube
Macros
======
-lf-infra-clm-python
--------------------
+lf-infra-nexus-iq-python-cli
+----------------------------
-Runs CLM scanning against a Python project.
+Runs Nexus IQ command-line interface CLM scan on Python package requirements.
:Required Parameters:
- :clm-project-name: Project name in Nexus IQ to send results to.
+ :nexus-iq-project-name: Project name in Nexus IQ to send results to.
+ :requirements-file: File name with output of pip freeze.
lf-infra-tox-install
--------------------
Job Templates
=============
-Python XC CLM
--------------
+Tox Nexus IQ CLM
+----------------
+
+The Nexus IQ job invokes tox and the Nexus IQ scanner to analyze packages for
+component lifecycle management (CLM). Runs tox to discover the required packages,
+downloads the command-line interface (CLI) scanner, runs the scanner on the package
+list, then uploads the results to a Nexus IQ server. The project's tox.ini file must
+define a test environment that runs 'pip freeze' and captures the output; that
+environment does not need to execute any tests. For example:
+
+.. code-block:: bash
+
+ [testenv:clm]
+ # use pip to report dependencies with versions
+ whitelist_externals = sh
+ commands = sh -c 'pip freeze > requirements.txt'
-CLM scans for Python based repos. This job will call the Nexus IQ CLI
-directly to run the scans.
-A new credential named "nexus-iq-xc-clm" needs to exist in the Jenkins
-credentials. The credential should contain the username and password
-to access Nexus IQ Server.
+This job runs on the master branch because the basic Nexus IQ configuration
+does not support multi-branch.
:Template Names:
- - {project-name}-python-clm-{stream}
- - gerrit-python-xc-clm
- - github-python-xc-clm
+ - {project-name}-tox-nexus-iq-clm
+ - gerrit-tox-nexus-iq-clm
+ - github-tox-nexus-iq-clm
-:Comment Trigger: **run-clm** post a comment with the trigger to launch
- this job manually. Do not include any other text or vote in the
- same comment.
+:Comment Trigger: ``run-clm``
:Required parameters:
- :build-node: The node to run build on.
- :jenkins-ssh-credential: Credential to use for SSH. (Generally should
- get configured in defaults.yaml)
+ :build-node: The node to run the build on.
+ (Commonly in defaults.yaml)
+ :jenkins-ssh-credential: Credential to use for SSH.
+ (Commonly in defaults.yaml)
+ :project: The git repository name.
+ :project-name: Prefix used to name jobs.
-:Optional parameters:
+:Optional Parameters:
+ :archive-artifacts: Pattern for files to archive to the logs server
+ (default: '\*\*/\*.log')
+ :branch: Git branch, should be master (default: master)
:build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+ :build-timeout: Timeout in minutes before aborting build. (default: 15)
+ :cron: Cron schedule when to trigger the job. This parameter also
+ supports multiline input via the YAML pipe | character to allow
+ more than 1 cron timer. (default: @weekly)
+ :disable-job: Whether to disable the job (default: false)
+ :gerrit_nexusiq_triggers: Override Gerrit Triggers.
+ :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+ :github-url: URL for Github. (default: https://github.com)
+ :java-version: Version of Java to use for the scan. (default: openjdk8)
:nexus-iq-cli-version: Nexus IQ CLI package version to download and use.
- (default: 1.44.0-01)
+ (default is a string like 1.89.0-02, see file lf-python-jobs.yaml)
:nexus-iq-namespace: Insert a namespace to project AppID for projects that
share a Nexus IQ system to avoid project name collision. We recommend
inserting a trailing - dash if using this parameter.
For example 'odl-'. (default: '')
- :build-timeout: Timeout in minutes before aborting build. (default: 60)
- :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
- :java-version: Version of Java to use for the build. (default: openjdk8)
- :pre-build-script: Shell script to execute before the CLM builder.
- For example, install prerequisites or move files to the repo root.
- (default: a string with a shell comment)
- :stream: Keyword used to represent a release code-name.
- Often the same as the branch. (default: master)
+ :pre-build-script: Shell script to run before tox. Useful for setting up
+ dependencies. (default: a string with a shell comment)
+ :python-version: Python version to invoke pip install of tox-pyenv
+ (default: python3)
+ :requirements-file: Name of file with output of pip freeze.
+ (default: requirements.txt)
:submodule-recursive: Whether to checkout submodules recursively.
(default: true)
:submodule-timeout: Timeout (in minutes) for checkout operation.
(default: 10)
:submodule-disable: Disable submodule checkout operation.
(default: false)
- :gerrit_clm_triggers: Override Gerrit Triggers.
- :gerrit_trigger_file_paths: Override file paths used to filter which file
- modifications trigger a build. Refer to JJB documentation for "file-path" details.
- https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+ :tox-dir: Directory containing the project's tox.ini relative to
+ the workspace. The default uses tox.ini at the project root.
+ (default: '.')
+ :tox-envs: Tox environment with the appropriate pip freeze invocation.
+ (default: 'clm')
Python Sonar with Tox
##########
- builder:
- name: lf-infra-clm-python
+ name: lf-infra-nexus-iq-python-cli
builders:
- inject:
- properties-content: "CLM_PROJECT_NAME={clm-project-name}"
- - shell: !include-raw-escape: ../shell/nexus-iq-cli.sh
+ properties-content: |
+ NEXUS_IQ_PROJECT_NAME={nexus-iq-project-name}
+ REQUIREMENTS_FILE={requirements-file}
+ - shell: !include-raw-escape: ../shell/nexus-iq-python-cli.sh
- builder:
name: lf-infra-tox-install
publishers:
- lf-infra-publish
-#################
-# Python XC CLM #
-#################
+####################
+# Tox Nexus IQ CLM #
+####################
-- lf_python_clm_xc: &lf_python_xc_clm
- name: lf-python-xc-clm
+- lf_tox_nexus_iq_clm: &lf_tox_nexus_iq_clm
+ name: lf-tox-nexus-iq-clm
######################
# Default parameters #
######################
branch: master
- build-days-to-keep: 30 # 30 days for troubleshooting purposes
- build-timeout: 60
+ build-days-to-keep: 7
+ build-timeout: 15
+ cron: "@weekly"
disable-job: false
git-url: "$GIT_URL/$PROJECT"
github-url: "https://github.com"
- java-version: openjdk8
- nexus-iq-cli-version: 1.44.0-01
+ java-version: openjdk8 # Scanner is a jar
+ nexus-iq-cli-version: 1.89.0-02
nexus-iq-namespace: "" # Recommend a trailing dash when set. Example: odl-
- pre-build-script: " # pre-build script goes here"
+ pre-build-script: "# pre-build script goes here"
+ python-version: python3
+ requirements-file: requirements.txt
stream: master
submodule-recursive: true
submodule-timeout: 10
submodule-disable: false
-
- gerrit_trigger_file_paths:
- - compare-type: REG_EXP
- pattern: ".*"
-
- # github_included_regions MUST match gerrit_trigger_file_paths
- github_included_regions:
- - ".*"
+ tox-dir: "."
+ tox-envs: "clm"
#####################
# Job Configuration #
project: "{project}"
branch: "{branch}"
stream: "{stream}"
+ - lf-infra-tox-parameters:
+ tox-dir: "{tox-dir}"
+ tox-envs: "{tox-envs}"
+ - string:
+ name: ARCHIVE_ARTIFACTS
+ default: "{archive-artifacts}"
+ description: Artifacts to archive to the logs server.
- string:
name: NEXUS_IQ_CLI_VERSION
default: "{nexus-iq-cli-version}"
- description: Nexus IQ CLI package to download and use.
+ description: Nexus IQ CLI jar to download and run.
wrappers:
+ - lf-infra-wrappers:
+ build-timeout: "{build-timeout}"
+ jenkins-ssh-credential: "{jenkins-ssh-credential}"
- credentials-binding:
- username-password-separated:
credential-id: nexus-iq-xc-clm
- username: CLM_USER
- password: CLM_PASSWORD
+ username: NEXUS_IQ_USER
+ password: NEXUS_IQ_PASSWORD
+
builders:
+ - lf-infra-pre-build
- lf-update-java-alternatives:
java-version: "{java-version}"
+ - lf-infra-tox-install:
+ python-version: "{python-version}"
- shell: "{pre-build-script}"
- - lf-infra-clm-python:
- clm-project-name: "{nexus-iq-namespace}{project-name}"
+ - lf-infra-tox-run:
+ parallel: false
+ - lf-infra-nexus-iq-python-cli:
+ nexus-iq-project-name: "{nexus-iq-namespace}{project-name}"
+ requirements-file: "{requirements-file}"
- job-template:
- name: "{project-name}-python-clm-{stream}"
- id: gerrit-python-xc-clm
+ name: "{project-name}-tox-nexus-iq-clm"
+ id: gerrit-tox-nexus-iq-clm
<<: *lf_python_common
- <<: *lf_python_xc_clm
+ <<: *lf_tox_nexus_iq_clm
######################
# Default parameters #
######################
- gerrit_clm_triggers:
+ gerrit_nexus_iq_triggers:
- comment-added-contains-event:
comment-contains-value: '^Patch Set\s+\d+:\s+run-clm\s*$'
choosing-strategy: default
triggers:
- # Build weekly on Saturdays
- - timed: "H H * * 6"
+ - timed: "{obj:cron}"
- gerrit:
server-name: "{gerrit-server-name}"
- trigger-on: "{obj:gerrit_clm_triggers}"
+ trigger-on: "{obj:gerrit_nexus_iq_triggers}"
projects:
- project-compare-type: ANT
project-pattern: "{project}"
branches:
- branch-compare-type: ANT
branch-pattern: "**/{branch}"
- file-paths: "{obj:gerrit_trigger_file_paths}"
skip-vote:
successful: true
failed: true
notbuilt: true
- job-template:
- name: "{project-name}-python-clm-{stream}"
- id: github-python-xc-clm
+ name: "{project-name}-tox-nexus-iq"
+ id: github-tox-nexus-iq-clm
<<: *lf_python_common
- <<: *lf_python_xc_clm
+ <<: *lf_tox_nexus_iq_clm
properties:
- lf-infra-properties:
jenkins-ssh-credential: "{jenkins-ssh-credential}"
triggers:
- # Build weekly on Saturdays
- - timed: "H H * * 6"
+ - timed: "{obj:cron}"
- github-pull-request:
trigger-phrase: "^run-clm$"
only-trigger-phrase: false
github-hooks: true
white-list-target-branches:
- "{branch}"
- included-regions: "{obj:github_included_regions}"
#########################
# Python Sonar with Tox #
--- /dev/null
+---
+features:
+ - |
+ Add new templates gerrit-tox-nexus-iq/github-tox-nexus-iq to
+ analyze Python package requirements. Add script
+ nexus-iq-python-cli.sh with correct invocation and arguments
+ for the latest version of the Nexus IQ scanner.
+
+ Drop broken templates g*it-python-xc-clm, unused builder
+ lf-infra-clm-python and shell script nexus-iq-cli.sh.
#!/bin/bash
# SPDX-License-Identifier: EPL-1.0
##############################################################################
-# Copyright (c) 2018 The Linux Foundation and others.
+# Copyright (c) 2020 The Linux Foundation and others.
#
# All rights reserved. This program and the accompanying materials
# are made available under the terms of the Eclipse Public License v1.0
# which accompanies this distribution, and is available at
# http://www.eclipse.org/legal/epl-v10.html
##############################################################################
-echo "---> nexus-iq-cli.sh"
+echo "---> nexus-iq-python-cli.sh"
# This script downloads the specified version of the nexus-iq-cli jar, uses it
-# to perform an XC Evaluation or extended report which provides a scan of python
-# files within the repo starting at the root, then publishes the result to an LF
-# server using the specified credentials.
+# to analyze the Python project dependencies from the specified requirements file,
+# then publishes the result to an LF server using the specified credentials.
# stop on error or unbound variable
set -eu
# do not print commands, credentials should not be logged
set +x
+# check if tox established the prereqs for this analysis
+if [[ ! -f $REQUIREMENTS_FILE ]]; then
+ echo "ERROR: failed to find file $REQUIREMENTS_FILE"
+ exit 1
+fi
CLI_LOCATION="/tmp/nexus-iq-cli-${NEXUS_IQ_CLI_VERSION}.jar"
echo "INFO: downloading nexus-iq-cli version $NEXUS_IQ_CLI_VERSION"
wget -nv "https://download.sonatype.com/clm/scanner/nexus-iq-cli-${NEXUS_IQ_CLI_VERSION}.jar" -O "${CLI_LOCATION}"
echo "-a" > cli-auth.txt
-echo "${CLM_USER}:${CLM_PASSWORD}" >> cli-auth.txt
-echo "INFO: running nexus-iq-cli scan on project $CLM_PROJECT_NAME"
-java -jar "${CLI_LOCATION}" @cli-auth.txt -xc -i "${CLM_PROJECT_NAME}" -s https://nexus-iq.wl.linuxfoundation.org -t build .
+echo "${NEXUS_IQ_USER}:${NEXUS_IQ_PASSWORD}" >> cli-auth.txt
+echo "INFO: running nexus-iq-cli on project $NEXUS_IQ_PROJECT_NAME and file $REQUIREMENTS_FILE"
+# result.json is a mystery
+java -jar "${CLI_LOCATION}" @cli-auth.txt -s https://nexus-iq.wl.linuxfoundation.org -i "${NEXUS_IQ_PROJECT_NAME}" -t build -r result.json "${REQUIREMENTS_FILE}"
rm cli-auth.txt
rm "${CLI_LOCATION}"
-echo "---> nexus-iq-cli.sh ends"
+echo "---> nexus-iq-python-cli.sh ends"
Code Review / releng / global-jjb.git / commitdiff