From c1ab7d08db987962c6c5c73abfaade4b5445d028 Mon Sep 17 00:00:00 2001 From: "Lott, Christopher (cl778h)" Date: Thu, 27 Feb 2020 14:45:39 -0500 Subject: [PATCH] Add templates Cmake + SonarQube Jenkins plugin New templates gerrit-cmake-sonarqube and github-cmake-sonarqube use the SonarQube Jenkins plug-in to analyze CXX code and publish the results. Modeled after the generic gerrit/github-sonar templates. The new templates lift the limitations of the existing templates gerrit/github-cmake-sonar which download and install a Sonar scanner, and cannot report unit-test code coverage statistics. Change-Id: I2457ab0a40e226b8b5b2bbe18d2b9f5cde3281e1 Issue-ID: RELENG-2760 Signed-off-by: Lott, Christopher (cl778h) --- .jjb-test/lf-c-cpp-jobs.yaml | 34 +++++ docs/jjb/lf-c-cpp-jobs.rst | 147 ++++++++++++++++++-- jjb/lf-c-cpp-jobs.yaml | 148 ++++++++++++++++++++- .../notes/cmake-sonarqube-3b43bea721775986.yaml | 9 ++ shell/cmake-sonarqube.sh | 39 ++++++ 5 files changed, 365 insertions(+), 12 deletions(-) create mode 100644 .jjb-test/lf-c-cpp-jobs.yaml create mode 100644 releasenotes/notes/cmake-sonarqube-3b43bea721775986.yaml create mode 100644 shell/cmake-sonarqube.sh diff --git a/.jjb-test/lf-c-cpp-jobs.yaml b/.jjb-test/lf-c-cpp-jobs.yaml new file mode 100644 index 00000000..a8236a71 --- /dev/null +++ b/.jjb-test/lf-c-cpp-jobs.yaml @@ -0,0 +1,34 @@ +--- +- project: + name: gerrit-cmake-jobs + jobs: + - gerrit-cmake-sonar + - gerrit-cmake-sonarqube + - gerrit-cmake-stage + - gerrit-cmake-verify + + project-name: gerrit-cmake + javadoc-path: org.example.project/$STREAM + staging-profile-id: uuddlrlrba + mvn-settings: gerrit-cmake-project-settings + mvn-snapshot-id: fakeproject-snapshots + nexus-group-id: fakeproject-group-id + nexus-snapshot-repo: fakeproject-snapshots + lftools-version: bogus-lftools-version + +- project: + name: github-cmake-jobs + jobs: + - github-cmake-sonar + - github-cmake-sonarqube + - github-cmake-stage + - github-cmake-verify + + project-name: github-cmake + javadoc-path: org.example.project/$STREAM + staging-profile-id: uuddlrlrba + mvn-settings: github-cmake-project-settings + mvn-snapshot-id: fakeproject-snapshots + nexus-group-id: fakeproject-group-id + nexus-snapshot-repo: fakeproject-snapshots + lftools-version: bogus-lftools-version diff --git a/docs/jjb/lf-c-cpp-jobs.rst b/docs/jjb/lf-c-cpp-jobs.rst index 302fe03f..362d7146 100644 --- a/docs/jjb/lf-c-cpp-jobs.rst +++ b/docs/jjb/lf-c-cpp-jobs.rst @@ -9,10 +9,12 @@ CMake Sonar ----------- The Sonar job installs the SonarQube CXX build wrapper and scanner tools, -uses the build wrapper to invoke cmake && make, runs the scanner to -analyze code files, and publishes the results to SonarCloud. Does not -support reporting of code coverage. Optionally runs a shell script before -the build to install prerequisites. +runs cmake, uses the build wrapper to invoke make, runs the scanner to +analyze code files, then publishes the results to a SonarQube server like +SonarCloud. Optionally runs a shell script before the build to install +prerequisites. Does not support code coverage reporting. + +**Deprecated**; new projects should use a CMake SonarQube template. This job purposely runs on the master branch because the basic SonarCloud configuration does not support multi-branch. @@ -59,6 +61,128 @@ configuration does not support multi-branch. :gerrit_sonar_triggers: Override Gerrit Triggers. +CMake SonarQube +--------------- + +The SonarQube job installs the SonarQube CXX build wrapper, runs cmake, uses +the build wrapper to invoke make, then runs the SonarQube Scanner Jenkins +plug-in to analyze code for bugs, code smells and security vulnerabilities, +and to upload the result (possibly including code-coverage statistics) to +a SonarQube server or to SonarCloud.io. Optionally runs a shell script +before the build to install prerequisites. + +Requires ``SonarQube Scanner for Jenkins`` + +This job runs on the master branch because the basic Sonar configuration +does not support multi-branch. + +Plug-in configurations + Manage Jenkins --> Configure System --> SonarQube servers + - Name: Sonar (fixed) + - Server URL: https://sonar.project.org/ or https://sonarcloud.io + - Server authentication token: none for local, API token (saved as + a "secret text" credential) for Sonarcloud + + Manage Jenkins --> Global Tool Configuration --> SonarQube Scanner + - Name: SonarQube Scanner (fixed) + - Install automatically + - Select latest version + +:Template Names: + + - {project-name}-cmake-sonarqube + - gerrit-cmake-sonarqube + - github-cmake-sonarqube + +:Comment Trigger: ``run-sonar`` + +:Required parameters: + + :build-node: The node to run the build on. + (Commonly in defaults.yaml) + :jenkins-ssh-credential: Credential to use for SSH. + (Commonly in defaults.yaml) + :project: The git repository name. + :project-name: Prefix used to name jobs. + +.. comment Start ignoring WriteGoodLintBear + +:Optional Parameters: + + :archive-artifacts: Pattern for files to archive to the logs server + (default: '\*\*/\*.log') + :build-wrap-dir: Build wrapper output subdirectory name. + (default: $WORKSPACE/bw-output) + :cmake-opts: Parameters to pass to cmake. (default: '') + :cron: Cron schedule when to trigger the job. This parameter also + supports multiline input via YAML pipe | character in cases where + one may want to provide more than 1 cron timer. (default: @weekly) + :install-prefix: CMAKE_INSTALL_PREFIX to use for install. + (default: $BUILD_DIR/output) + :make-opts: Parameters to pass to make. (default: '') + :sonar-additional-args: Command line arguments. (default: '') + :sonar-java-opts: JVM options. For example, use option -Xmx + to increase the memory size limit. (default: '') + :sonar-prescan-script: A shell script to run before the build and scan. + Useful for setting up dependencies. (default: '') + :sonar-project-file: The file name with Sonar configuration properties + (default: sonar-project.properties) + :sonar-properties: Sonar configuration properties. (default: '') + :sonar-task: Sonar task to run. (default: '') + +.. comment Stop ignoring + +.. note:: A job definition must provide one of the optional parameters + ``sonar-project-file`` and ``sonar-properties``; they cannot both be + empty. Set Sonar properties directly in the job definition by setting + the ``sonar-project-file`` property to ``""`` and adding all properties + under ``sonar-properties``. + +:Required Sonar Properties: + + - sonar.login: The API token for authentication at SonarCloud. + Commonly defined as key "sonarcloud_api_token" in defaults.yaml. + - sonar.organization: The umbrella project name; e.g., "opendaylight". + Commonly defined as key "sonarcloud_project_organization" in defaults.yaml. + - sonar.projectName: The git repository name without slashes; e.g., "infrautils". + - sonar.projectKey: The globally unique key for the report in SonarCloud. Most + teams use the catenation of sonar.organization, an underscore, and + sonar.projectName; e.g., "opendaylight_infrautils". + +:Optional Sonar Properties: + + - sonar.cfamily.gcov.reportsPath: directory with GCOV output files + - Documentation of SonarQube properties is here: + https://docs.sonarqube.org/latest/analysis/overview/ + + +Example job definition +^^^^^^^^^^^^^^^^^^^^^^ + +The following example defines a job for a project with CXX source files +in the "src" directory, and unit tests that write coverage files in +GCOV format to the "test" directory. This definition uses configuration +parameters in the umbrella project's defaults.yaml file. + +.. code-block:: yaml + + - project: + name: my-project-sonar + project: my/project + project-name: my-project + sonar-project-file: "" + sonar-properties: | + sonar.login={sonarcloud_api_token} + sonar.projectKey={sonarcloud_project_organization}_{project-name} + sonar.projectName={project-name} + sonar.organization={sonarcloud_project_organization} + sonar.sourceEncoding=UTF-8 + sonar.sources=src + sonar.cfamily.build-wrapper-output=$WORKSPACE/bw-output + sonar.cfamily.gcov.reportsPath=test + jobs: + - gerrit-cmake-sonarqube + CMake Stage ----------- @@ -112,15 +236,16 @@ project into a tar.xz tarball to produce a release candidate. Use method 2 in conjunction with 'pre-build' configuration to generate the artifact_version automatically from files in the - project's repository. For example with pre-build script: + project's repository. An example pre-build script appears below. + - .. code-block:: bash +.. code-block:: bash - #!/bin/bash - MAJOR_VERSION="$(grep 'set(OCIO_VERSION_MAJOR' CMakeLists.txt | awk '{{print $NF}}' | awk -F')' '{{print $1}}')" - MINOR_VERSION="$(grep 'set(OCIO_VERSION_MINOR' CMakeLists.txt | awk '{{print $NF}}' | awk -F')' '{{print $1}}')" - PATCH_VERSION="$(grep 'set(OCIO_VERSION_PATCH' CMakeLists.txt | awk '{{print $NF}}' | awk -F')' '{{print $1}}')" - echo "${{MAJOR_VERSION}}.${{MINOR_VERSION}}.${{PATCH_VERSION}}" > /tmp/artifact_version + #!/bin/bash + MAJOR_VERSION="$(grep 'set(OCIO_VERSION_MAJOR' CMakeLists.txt | awk '{{print $NF}}' | awk -F')' '{{print $1}}')" + MINOR_VERSION="$(grep 'set(OCIO_VERSION_MINOR' CMakeLists.txt | awk '{{print $NF}}' | awk -F')' '{{print $1}}')" + PATCH_VERSION="$(grep 'set(OCIO_VERSION_PATCH' CMakeLists.txt | awk '{{print $NF}}' | awk -F')' '{{print $1}}')" + echo "${{MAJOR_VERSION}}.${{MINOR_VERSION}}.${{PATCH_VERSION}}" > /tmp/artifact_version CMake Verify ------------ diff --git a/jjb/lf-c-cpp-jobs.yaml b/jjb/lf-c-cpp-jobs.yaml index bded1c79..fa4e35b0 100644 --- a/jjb/lf-c-cpp-jobs.yaml +++ b/jjb/lf-c-cpp-jobs.yaml @@ -1,6 +1,6 @@ --- ############### -# CMAKE Sonar # +# CMAKE SONAR # ############### - lf_cmake_sonar: &lf_cmake_sonar @@ -134,6 +134,152 @@ triggers: - timed: "{obj:cron}" +################### +# CMAKE SONARQUBE # +################### + +- lf_cmake_sonarqube: &lf_cmake_sonarqube + name: lf-cmake-sonarqube + + ###################### + # Default parameters # + ###################### + + archive-artifacts: > + **/*.log + branch: master # Sonar should always be run on master branch + build-days-to-keep: 7 + build-dir: "$WORKSPACE/build" + build-timeout: 15 + build-wrap-dir: "$WORKSPACE/bw-output" + cmake-opts: "" + cron: "@weekly" + disable-job: false + git-url: "$GIT_URL/$PROJECT" + github-url: "https://github.com" + install-prefix: "$BUILD_DIR/output" + make-opts: "" + stream: master + submodule-recursive: true + submodule-timeout: 10 + submodule-disable: false + + # Sonar properties + sonar-additional-args: "" + sonar-java-opts: "" + sonar-prescan-script: "" + sonar-project-file: "sonar-project.properties" + sonar-properties: "" + sonar-task: "" + + ##################### + # Job Configuration # + ##################### + + project-type: freestyle + node: "{build-node}" + disabled: "{disable-job}" + + properties: + - lf-infra-properties: + build-days-to-keep: "{build-days-to-keep}" + + wrappers: + - lf-infra-wrappers: + build-timeout: "{build-timeout}" + jenkins-ssh-credential: "{jenkins-ssh-credential}" + + parameters: + - lf-infra-parameters: + project: "{project}" + branch: "{branch}" + stream: "{stream}" + - lf-cmake-parameters: + build-dir: "{build-dir}" + cmake-opts: "{cmake-opts}" + install-prefix: "{install-prefix}" + make-opts: "{make-opts}" + - string: + name: ARCHIVE_ARTIFACTS + default: "{archive-artifacts}" + description: Artifacts to archive to the logs server. + - string: + name: BUILD_WRAP_DIR + default: "{build-wrap-dir}" + description: SonarQube build wrapper output directory. + + builders: + - lf-infra-pre-build + - shell: "{sonar-prescan-script}" + - shell: !include-raw-escape: ../shell/cmake-sonarqube.sh + - lf-infra-sonar: + sonar-task: "{sonar-task}" + sonar-project-file: "{sonar-project-file}" + sonar-properties: "{sonar-properties}" + sonar-java-opts: "{sonar-java-opts}" + sonar-additional-args: "{sonar-additional-args}" + + publishers: + - lf-infra-publish + +- job-template: + name: "{project-name}-cmake-sonarqube" + id: gerrit-cmake-sonarqube + concurrent: false + <<: *lf_cmake_sonarqube + + scm: + - lf-infra-gerrit-scm: + branch: "$GERRIT_BRANCH" + jenkins-ssh-credential: "{jenkins-ssh-credential}" + git-url: "{git-url}" + refspec: "$GERRIT_REFSPEC" + submodule-recursive: "{submodule-recursive}" + submodule-timeout: "{submodule-timeout}" + submodule-disable: "{submodule-disable}" + choosing-strategy: default + + triggers: + - timed: "{obj:cron}" + - gerrit: + server-name: "{gerrit-server-name}" + trigger-on: + - comment-added-contains-event: + comment-contains-value: '^Patch Set\s+\d+:\s+run-sonar\s*$' + projects: + - project-compare-type: "ANT" + project-pattern: "{project}" + branches: + - branch-compare-type: "ANT" + branch-pattern: "**/{branch}" + +- job-template: + name: "{project-name}-cmake-sonarqube" + id: github-cmake-sonarqube + concurrent: false + <<: *lf_cmake_sonarqube + + scm: + - lf-infra-github-scm: + branch: "$sha1" + jenkins-ssh-credential: "{jenkins-ssh-credential}" + url: "{git-clone-url}{github-org}/{project}" + refspec: "+refs/pull/*:refs/remotes/origin/pr/*" + submodule-recursive: "{submodule-recursive}" + submodule-timeout: "{submodule-timeout}" + submodule-disable: "{submodule-disable}" + choosing-strategy: default + + triggers: + - github-pull-request: + trigger-phrase: "^run-sonar$" + only-trigger-phrase: false + status-context: "CMake Sonarqube" + permit-all: true + github-hooks: true + white-list-target-branches: + - "{branch}" + ############### # CMAKE STAGE # ############### diff --git a/releasenotes/notes/cmake-sonarqube-3b43bea721775986.yaml b/releasenotes/notes/cmake-sonarqube-3b43bea721775986.yaml new file mode 100644 index 00000000..cc5fc698 --- /dev/null +++ b/releasenotes/notes/cmake-sonarqube-3b43bea721775986.yaml @@ -0,0 +1,9 @@ +--- +features: + - | + New templates gerrit-cmake-sonarqube and github-cmake-sonarqube + use the SonarQube Jenkins plug-in to analyze CXX code and publish + the results. Modeled after the generic gerrit/github-sonar templates. + The new templates lift the limitations of the existing templates + gerrit/github-cmake-sonar which download and install a Sonar scanner, + and cannot report unit-test code coverage statistics. diff --git a/shell/cmake-sonarqube.sh b/shell/cmake-sonarqube.sh new file mode 100644 index 00000000..0dd2cf55 --- /dev/null +++ b/shell/cmake-sonarqube.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# SPDX-License-Identifier: EPL-1.0 +############################################################################## +# Copyright (c) 2020 The Linux Foundation and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Eclipse Public License v1.0 +# which accompanies this distribution, and is available at +# http://www.eclipse.org/legal/epl-v10.html +############################################################################## +echo "---> cmake-sonarqube.sh" + +# Runs cmake then make using a wrapper for the SonarQube Jenkins plug-in +# https://docs.sonarqube.org/latest/analysis/languages/cfamily/ + +set -eux -o pipefail + +build_dir="${BUILD_DIR:-$WORKSPACE/build}" +build_wrap_dir="${BUILD_WRAP_DIR:-$WORKSPACE/bw-output}" +cmake_opts="${CMAKE_OPTS:-}" +make_opts="${MAKE_OPTS:-}" + +cd /tmp || exit 1 +wget -q -O bw.zip https://sonarcloud.io/static/cpp/build-wrapper-linux-x86.zip +unzip -q bw.zip +sudo mv build-wrapper-* /opt/build-wrapper + +mkdir -p "$build_dir" +cd "$build_dir" || exit 1 + +# $cmake_opts needs to wordsplit to pass options. +# shellcheck disable=SC2086 +eval cmake -DCMAKE_INSTALL_PREFIX="$INSTALL_PREFIX" $cmake_opts .. + +# $make_opts may be empty. +# shellcheck disable=SC2086 +/opt/build-wrapper/build-wrapper-linux-x86-64 --out-dir "$build_wrap_dir" make $make_opts + +echo "---> cmake-sonarqube.sh ends" -- 2.16.6