Add packer-verify-build jobs

[releng/global-jjb.git] / docs / jjb / lf-ci-jobs.rst

diff --git a/docs/jjb/lf-ci-jobs.rst b/docs/jjb/lf-ci-jobs.rst
index 37071d3..1ec5485 100644 (file)
--- a/docs/jjb/lf-ci-jobs.rst
+++ b/docs/jjb/lf-ci-jobs.rst
@@ -107,10 +107,13 @@ Job Templates
 Gerrit Branch Lock
 ------------------
 
 Gerrit Branch Lock
 ------------------
 

-Job submits a patch to lock or unlock a project's branch.
+Job submits a patch to lock or unlock a project's branch. This should only be
+loaded once, as "ci-management-gerrit-branch-lock" (or "ci-management"
+equivalent). That job will process lock/unlock requests for all projects and
+all branches.

 
 :Template Names:
 
 :Template Names:

-    - {project-name}-gerrit-branch-lock-{stream}
+    - {project-name}-gerrit-branch-lock

     - gerrit-branch-lock
 
 :Comment Trigger:
     - gerrit-branch-lock
 
 :Comment Trigger:


@@ -126,14 +129,9 @@ Job submits a patch to lock or unlock a project's branch.
 
 :Optional parameters:
 
 
 :Optional parameters:
 

-    :branch: Git branch to build against. (default: master)

     :git-url: URL to clone project from. (default: $GIT_URL/$GERRIT_PROJECT)
     :git-url: URL to clone project from. (default: $GIT_URL/$GERRIT_PROJECT)

-    :stream: Keyword that can be used to represent a release code-name.
-        Often the same as the branch. (default: master)

     :submodule-timeout: Timeout (in minutes) for checkout operation.
         (default: 10)
     :submodule-timeout: Timeout (in minutes) for checkout operation.
         (default: 10)

-    :submodule-disable: Disable submodule checkout operation.
-        (default: false)

     :gerrit_merge_triggers: Override Gerrit Triggers.
 
 .. _lf-global-jjb-jenkins-cfg-merge:
     :gerrit_merge_triggers: Override Gerrit Triggers.
 
 .. _lf-global-jjb-jenkins-cfg-merge:


@@ -245,11 +243,12 @@ configuration in the format ``KEY=value``.
        In the case of template definitions of a parameter below is not passed
        the one defined in default clouds will be inherited.
 
        In the case of template definitions of a parameter below is not passed
        the one defined in default clouds will be inherited.
 

-    :IMAGE_NAME: The image name to use for this template.
-        (required)
+    :IMAGE_NAME: The image name to use for this template. (required)

     :HARDWARE_ID: OpenStack flavor to use. (required)
 
     :LABELS: Labels to assign to the vm. (default: FILE_NAME)
     :HARDWARE_ID: OpenStack flavor to use. (required)
 
     :LABELS: Labels to assign to the vm. (default: FILE_NAME)

+    :VOLUME_SIZE: Volume size to assign to vm. (default: "")
+    :HARDWARE_ID: Hardware Id to assign to vm. (default: "")

     :NETWORK_ID: OpenStack network to use. (default: "")
     :USER_DATA_ID: User Data to pass into the instance.
         (default: jenkins-init-script)
     :NETWORK_ID: OpenStack network to use. (default: "")
     :USER_DATA_ID: User Data to pass into the instance.
         (default: jenkins-init-script)


@@ -266,16 +265,18 @@ configuration in the format ``KEY=value``.
     :START_TIMEOUT: Number of milliseconds to wait for the agent to be
         provisioned and connected. (default: 600000)
     :KEY_PAIR_NAME: SSH Public Key Pair to use for authentication.
     :START_TIMEOUT: Number of milliseconds to wait for the agent to be
         provisioned and connected. (default: 600000)
     :KEY_PAIR_NAME: SSH Public Key Pair to use for authentication.

-        (default: jenkins)
+        (default: jenkins-ssh)

     :NUM_EXECUTORS: Number of executors to enable for the instance.
         (default: 1)
     :NUM_EXECUTORS: Number of executors to enable for the instance.
         (default: 1)

-    :JVM_OPTIONS: JVM Options to pass to Java. (default: "")
+    :JVM_OPTIONS: JVM Options to pass to Java. (default: null)

     :FS_ROOT: File system root for the workspace. (default: "/w")
     :FS_ROOT: File system root for the workspace. (default: "/w")

+    :NODE_PROPERTIES: Node properties. (default: null)

     :RETENTION_TIME: Number of minutes to wait for an idle slave to be used
         again before it's removed. If set to -1, the slave will be kept
         forever. (default: 0)
     :CONNECTION_TYPE: The connection type for Jenkins to connect to the build
         minion. Valid options: JNLP, SSH. (default: "SSH")
     :RETENTION_TIME: Number of minutes to wait for an idle slave to be used
         again before it's removed. If set to -1, the slave will be kept
         forever. (default: 0)
     :CONNECTION_TYPE: The connection type for Jenkins to connect to the build
         minion. Valid options: JNLP, SSH. (default: "SSH")

+    :CONFIG_TYPE: Configuration drive. (default: null)

 
 For a live example see the OpenDaylight project jenkins-config directory.
 https://github.com/opendaylight/releng-builder/tree/master/jenkins-config
 
 For a live example see the OpenDaylight project jenkins-config directory.
 https://github.com/opendaylight/releng-builder/tree/master/jenkins-config


@@ -442,7 +443,8 @@ Runs `jenkins-jobs update` to update production job configuration
 JJB Verify
 ----------
 
 JJB Verify
 ----------
 

-Runs `jenkins-jobs test` to validate JJB syntax
+Runs `jenkins-jobs test` to validate JJB syntax. Optionally validates
+build-node labels used in templates and job definitions.

 
 :Template Names:
     - {project-name}-jjb-verify
 
 :Template Names:
     - {project-name}-jjb-verify


@@ -463,6 +465,10 @@ Runs `jenkins-jobs test` to validate JJB syntax
     :build-concurrent: Whether or not to allow this job to run multiple jobs
         simultaneously. (default: true)
     :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
     :build-concurrent: Whether or not to allow this job to run multiple jobs
         simultaneously. (default: true)
     :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)

+    :build-node-label-check: Whether to check build-node labels in jobs
+        against node names in cloud config files (default: false)
+    :build-node-label-list: Space-separated list of external build-node
+        labels not present in cloud config files (default: "")

     :build-timeout: Timeout in minutes before aborting build. (default: 10)
     :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
     :jjb-cache: JJB cache location. (default: $HOME/.cache/jenkins_jobs)
     :build-timeout: Timeout in minutes before aborting build. (default: 10)
     :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
     :jjb-cache: JJB cache location. (default: $HOME/.cache/jenkins_jobs)


@@ -529,7 +535,7 @@ Info YAML Verify
 
 Info YAML Verify job validates that INFO.yaml file changes are kept isolated from
 other file changes. Verifies INFO.yaml files follow the schema defined in
 
 Info YAML Verify job validates that INFO.yaml file changes are kept isolated from
 other file changes. Verifies INFO.yaml files follow the schema defined in

-`global-jjb/info-schema`.
+`lfit/releng-global-jjb/schema/info-schema.yaml`.

 
 :Template Names:
     - {project-name}-info-yaml-verify
 
 :Template Names:
     - {project-name}-info-yaml-verify


@@ -650,6 +656,71 @@ Full Example:
 
 .. literalinclude:: ../../.jjb-test/lf-ci-jobs/openstack-cron-full.yaml
 
 
 .. literalinclude:: ../../.jjb-test/lf-ci-jobs/openstack-cron-full.yaml
 

+.. _gjjb-openstack-update-cloud-image:
+
+OpenStack Update Cloud Image
+----------------------------
+
+This job finds and updates OpenStack cloud images on the ci-management source
+repository.
+
+The job is triggered in two ways:
+
+1. When packer merge job completes, the new image name created is passed
+   down to the job.
+2. When the job is triggered manually to update all new images.
+
+When the job is triggered through an upstream packer merge job, this only
+generates a change request for the new image built.
+
+When the job is triggered manually, this job finds the latest images on
+OpenStack cloud and compares them with the images currently used in the source
+ci-management source repository. If the compared images have newer
+time stamps are **all** updated through a change request.
+
+This job requires a Jenkins configuration merge and verify job setup and
+working on Jenkins.
+
+:Template Names:
+    - {project-name}-openstack-update-cloud-image
+    - gerrit-openstack-update-cloud-image
+    - github-openstack-update-cloud-image
+
+:Required parameters:
+
+    :build-node: The node to run build on.
+    :jenkins-ssh-credential: Credential to use for SSH. (Generally should
+        be configured in defaults.yaml)
+    :new-image-name: Name of new image name passed from packer merge job or
+        set to 'all' to update all images. (default: all)
+
+:Optional parameters:
+
+    :branch: Git branch to fetch for the build. (default: master)
+    :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+    :build-timeout: Timeout in minutes before aborting build. (default: 90)
+    :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+    :openstack-cloud: OS_CLOUD setting to pass to openstack client.
+        (default: vex)
+    :stream: Keyword that can be used to represent a release code-name.
+        Often the same as the branch. (default: master)
+    :submodule-recursive: Whether to checkout submodules recursively.
+        (default: true)
+    :submodule-timeout: Timeout (in minutes) for checkout operation.
+        (default: 10)
+    :submodule-disable: Disable submodule checkout operation.
+        (default: false)
+    :update-cloud-image: Submit a change request to update new built cloud
+        image to Jenkins. (default: false)
+
+Minimal Example:
+
+.. literalinclude:: ../../.jjb-test/lf-ci-jobs/openstack-update-cloud-image-minimal.yaml
+
+Full Example:
+
+.. literalinclude:: ../../.jjb-test/lf-ci-jobs/openstack-update-cloud-image-full.yaml
+

 
 .. _gjjb-packer-merge:
 
 
 .. _gjjb-packer-merge:
 


@@ -673,9 +744,9 @@ Packer Merge job runs `packer build` to build system images in the cloud.
     :mvn-settings: The name of settings file containing credentials for
         the project.
     :platforms: Platform or distribution to build. Typically json file
     :mvn-settings: The name of settings file containing credentials for
         the project.
     :platforms: Platform or distribution to build. Typically json file

-        found in the packer/vars directory. (Example: centos)
-    :template: System template to build. Typically shell script found in
-        the packer/provision directory. (Example: java-builder)
+        found in the packer/vars directory. (Example: centos-7)
+    :templates: System template to build. Typically a yaml file or shell script
+        found in the packer/provision directory. (Example: docker)

 
 :Optional parameters:
 
 
 :Optional parameters:
 


@@ -683,6 +754,7 @@ Packer Merge job runs `packer build` to build system images in the cloud.
     :branch: Git branch to fetch for the build. (default: master)
     :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
     :build-timeout: Timeout in minutes before aborting build. (default: 90)
     :branch: Git branch to fetch for the build. (default: master)
     :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
     :build-timeout: Timeout in minutes before aborting build. (default: 90)

+    :gerrit_verify_triggers: Override Gerrit Triggers.

     :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
     :openstack: Packer template uses an OpenStack builder (default: true).
     :openstack-cloud: Sets OS_CLOUD variable to the value of this parameter.
     :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
     :openstack: Packer template uses an OpenStack builder (default: true).
     :openstack-cloud: Sets OS_CLOUD variable to the value of this parameter.


@@ -698,15 +770,16 @@ Packer Merge job runs `packer build` to build system images in the cloud.
         (default: 10)
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
         (default: 10)
     :submodule-disable: Disable submodule checkout operation.
         (default: false)

+    :update-cloud-image: Submit a change request to update newly built cloud
+        image to Jenkins. (default: false)

 
 

-    :gerrit_verify_triggers: Override Gerrit Triggers.

 
 Test an in-progress patch
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
 
 Test an in-progress patch
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 

-To test an in-progress patch from a GitHub Pull Request. Upload this
+To test an in-progress patch from a GitHub Pull Request, upload this

 job to the :doc:`Jenkins Sandbox <lfdocs:jenkins-sandbox>`. Then when manually
 job to the :doc:`Jenkins Sandbox <lfdocs:jenkins-sandbox>`. Then when manually

-building the job replace the GERRIT_REFSPEC parameter with the GitHub Pull
+building the job, replace the GERRIT_REFSPEC parameter with the GitHub Pull

 Request number of the patch you would like to test.
 
 Example GitHub:
 Request number of the patch you would like to test.
 
 Example GitHub:


@@ -715,12 +788,15 @@ Example GitHub:
 
    GERRIT_REFSPEC: origin/pr/49/merge
 
 
    GERRIT_REFSPEC: origin/pr/49/merge
 

+

 .. _gjjb-packer-verify:
 
 Packer Verify
 -------------
 
 .. _gjjb-packer-verify:
 
 Packer Verify
 -------------
 

-Packer Verify job runs `packer validate` to verify packer configuration.
+Packer Verify job runs `packer validate` to verify packer configuration. The
+verify job checks superficial syntax of the template and other files. It does
+not attempt to build an image, and cannot detect all possible build issues.

 
 :Template Names:
     - {project-name}-packer-verify
 
 :Template Names:
     - {project-name}-packer-verify


@@ -742,6 +818,9 @@ Packer Verify job runs `packer validate` to verify packer configuration.
     :branch: Git branch to fetch for the build. (default: master)
     :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
     :build-timeout: Timeout in minutes before aborting build. (default: 10)
     :branch: Git branch to fetch for the build. (default: master)
     :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
     :build-timeout: Timeout in minutes before aborting build. (default: 10)

+    :gerrit_trigger_file_paths: Override file paths which can be used to
+        filter which file modifications will trigger a build.
+    :gerrit_verify_triggers: Override Gerrit Triggers.

     :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
     :openstack: Packer template uses an OpenStack builder (default: true).
     :openstack-cloud: Sets OS_CLOUD variable to the value of this parameter.
     :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
     :openstack: Packer template uses an OpenStack builder (default: true).
     :openstack-cloud: Sets OS_CLOUD variable to the value of this parameter.


@@ -758,9 +837,61 @@ Packer Verify job runs `packer validate` to verify packer configuration.
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
 
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
 

-    :gerrit_verify_triggers: Override Gerrit Triggers.
-    :gerrit_trigger_file_paths: Override file paths which can be used to
-        filter which file modifications will trigger a build.
+
+.. _gjjb-packer-verify-build:
+
+Packer Verify Build
+-------------------
+
+Packer Verify Build job is essentially the same as the
+:ref:`Packer Merge job<gjjb-packer-merge>`. It is triggered only by its keyword,
+and will build a useable image. If the last patch set before a merge has a
+successful verify build, the merge job will not build the same image.
+
+:Template Names:
+   - {project-name}-packer-verify-build-{platforms}-{templates}
+   - gerrit-packer-verify-build
+   - github-packer-verify-build
+
+:Comment Trigger: verify-build|packer-build
+
+:Required parameters:
+
+   :build-node: The node to run build on.
+   :jenkins-ssh-credential: Credential to use for SSH. (Generally should
+       be configured in defaults.yaml)
+   :mvn-settings: The name of settings file containing credentials for
+       the project.
+   :platforms: Platform or distribution to build. Typically json file
+       found in the packer/vars directory. (Example: centos-7)
+   :templates: System template to build. Typically a yaml file or shell script
+       found in the packer/provision directory. (Example: docker)
+
+:Optional parameters:
+
+   :branch: Git branch to fetch for the build. (default: master)
+   :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+   :build-timeout: Timeout in minutes before aborting build. (default: 10)
+   :gerrit_trigger_file_paths: Override file paths which can be used to
+       filter which file modifications will trigger a build.
+   :gerrit_verify_triggers: Override Gerrit Triggers.
+   :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+   :openstack: Packer template uses an OpenStack builder (default: true).
+   :openstack-cloud: Sets OS_CLOUD variable to the value of this parameter.
+       (default: vex).
+   :packer-cloud-settings: Name of settings file containing credentials
+       for the cloud that packer will build on. (default: packer-cloud-env)
+   :packer-version: Version of packer to install / use in build. (default: 1.0.2)
+   :stream: Keyword that can be used to represent a release code-name.
+       Often the same as the branch. (default: master)
+   :submodule-recursive: Whether to checkout submodules recursively.
+       (default: true)
+   :submodule-timeout: Timeout (in minutes) for checkout operation.
+       (default: 10)
+   :submodule-disable: Disable submodule checkout operation.
+       (default: false)
+   :update-cloud-image: Submit a change request to update new built cloud
+       image to Jenkins. (default: false)

 
 
 Puppet Verify
 
 
 Puppet Verify


@@ -806,3 +937,104 @@ the base directory is usually the best place to run from.
         (default: 10)
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
         (default: 10)
     :submodule-disable: Disable submodule checkout operation.
         (default: false)

+
+
+Sonar
+-----
+
+Runs the Jenkins SonarQube Scanner plug-in to analyze code for bugs,
+code smells and security vulnerabilities, and to upload the result
+(possibly including code-coverage statistics) to a SonarQube server
+or to SonarCloud.io.
+
+Requires ``SonarQube Scanner for Jenkins``
+
+One of the optional parameters sonar-project-file and sonar-properties
+must be supplied; they cannot both be empty.
+
+Plug-in configurations
+    Manage Jenkins --> Configure System --> SonarQube servers
+        - Name: Sonar (fixed)
+        - Server URL: https://sonar.server.org/ or https://sonarcloud.io
+        - Server authentication token: none for local, API token (saved as a
+          "secret text" credential) for Sonarcloud
+
+    Manage Jenkins --> Global Tool Configuration --> SonarQube Scanner
+        - Name: SonarQube Scanner (fixed)
+        - Install automatically
+        - Select latest version
+
+.. note:: Sonar properties can be set directly in the job definition by
+   setting the sonar-project-file to ``""`` and adding all properties under
+   ``sonar-properties``.
+
+:Template Names:
+
+    - {project-name}-sonar
+    - gerrit-sonar
+    - github-sonar
+
+:Optional Parameters:
+    :sonar-task: Sonar task to run. (default: "")
+    :sonar-project-file: The filename for the project's properties
+        (default: "sonar-project.properties")
+    :sonar-properties: Sonar configuration properties. (default: "")
+    :sonar-java-opts: JVM options. (default: "")
+    :sonar-additional-args: Additional command line arguments. (default: "")
+
+
+Sonar with Prescan
+------------------
+
+The same as the Sonar job above, except the caller also defines a builder
+called ``lf-sonar-prescan``, in which they can put any builders that they want
+to run prior to the Sonar scan.
+
+.. code-block:: yaml
+
+   - builder:
+       name: lf-sonar-prescan
+       builders:
+         - shell: "# Pre-scan shell script"
+
+:Template Names:
+
+    - {project-name}-sonar-prescan
+    - gerrit-sonar-prescan
+    - github-sonar-prescan
+
+:Required Parameters:
+    :lf-sonar-prescan: A builder that will run prior to the Sonar scan.
+
+:Optional Parameters:
+    :sonar-task: Sonar task to run. (default: "")
+    :sonar-project-file: The filename for the project's properties
+        (default: "sonar-project.properties")
+    :sonar-properties: Sonar configuration properties. (default: "")
+    :sonar-java-opts: JVM options. (default: "")
+    :sonar-additional-args: Additional command line arguments. (default: "")
+
+
+Sonar with Prescan Script
+-------------------------
+
+The same as the Sonar job above, except the caller must supply a shell script
+to run prior to the Sonar scan. This is commonly used to install prerequisites,
+build the project, execute unit tests and generate a code-coverage report.
+
+:Template Names:
+
+    - {project-name}-sonar-prescan-script
+    - gerrit-sonar-prescan-script
+    - github-sonar-prescan-script
+
+:Required Parameters:
+    :sonar-prescan-script: A shell script that will run prior to the Sonar scan.
+
+:Optional Parameters:
+    :sonar-task: Sonar task to run. (default: "")
+    :sonar-project-file: The filename for the project's properties.
+        (default: "sonar-project.properties")
+    :sonar-properties: Sonar configuration properties. (default: "")
+    :sonar-java-opts: JVM options. (default: "")
+    :sonar-additional-args: Additional command line arguments. (default: "")