X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=blobdiff_plain;f=docs%2Fjjb%2Flf-ci-jobs.rst;h=1ec548515022e8b74c748dbc3cbab4770c8d0600;hb=eccb60c2014c1dfed187ad0d783a57664fac3086;hp=37071d30bfa6356d2048e0c14d0a495903cfa1d1;hpb=fa4b6a88db1698921a1e382bf6a9b256da3abd4d;p=releng%2Fglobal-jjb.git diff --git a/docs/jjb/lf-ci-jobs.rst b/docs/jjb/lf-ci-jobs.rst index 37071d30..1ec54851 100644 --- a/docs/jjb/lf-ci-jobs.rst +++ b/docs/jjb/lf-ci-jobs.rst @@ -107,10 +107,13 @@ Job Templates Gerrit Branch Lock ------------------ -Job submits a patch to lock or unlock a project's branch. +Job submits a patch to lock or unlock a project's branch. This should only be +loaded once, as "ci-management-gerrit-branch-lock" (or "ci-management" +equivalent). That job will process lock/unlock requests for all projects and +all branches. :Template Names: - - {project-name}-gerrit-branch-lock-{stream} + - {project-name}-gerrit-branch-lock - gerrit-branch-lock :Comment Trigger: @@ -126,14 +129,9 @@ Job submits a patch to lock or unlock a project's branch. :Optional parameters: - :branch: Git branch to build against. (default: master) :git-url: URL to clone project from. (default: $GIT_URL/$GERRIT_PROJECT) - :stream: Keyword that can be used to represent a release code-name. - Often the same as the branch. (default: master) :submodule-timeout: Timeout (in minutes) for checkout operation. (default: 10) - :submodule-disable: Disable submodule checkout operation. - (default: false) :gerrit_merge_triggers: Override Gerrit Triggers. .. _lf-global-jjb-jenkins-cfg-merge: @@ -245,11 +243,12 @@ configuration in the format ``KEY=value``. In the case of template definitions of a parameter below is not passed the one defined in default clouds will be inherited. - :IMAGE_NAME: The image name to use for this template. - (required) + :IMAGE_NAME: The image name to use for this template. (required) :HARDWARE_ID: OpenStack flavor to use. (required) :LABELS: Labels to assign to the vm. (default: FILE_NAME) + :VOLUME_SIZE: Volume size to assign to vm. (default: "") + :HARDWARE_ID: Hardware Id to assign to vm. (default: "") :NETWORK_ID: OpenStack network to use. (default: "") :USER_DATA_ID: User Data to pass into the instance. (default: jenkins-init-script) @@ -266,16 +265,18 @@ configuration in the format ``KEY=value``. :START_TIMEOUT: Number of milliseconds to wait for the agent to be provisioned and connected. (default: 600000) :KEY_PAIR_NAME: SSH Public Key Pair to use for authentication. - (default: jenkins) + (default: jenkins-ssh) :NUM_EXECUTORS: Number of executors to enable for the instance. (default: 1) - :JVM_OPTIONS: JVM Options to pass to Java. (default: "") + :JVM_OPTIONS: JVM Options to pass to Java. (default: null) :FS_ROOT: File system root for the workspace. (default: "/w") + :NODE_PROPERTIES: Node properties. (default: null) :RETENTION_TIME: Number of minutes to wait for an idle slave to be used again before it's removed. If set to -1, the slave will be kept forever. (default: 0) :CONNECTION_TYPE: The connection type for Jenkins to connect to the build minion. Valid options: JNLP, SSH. (default: "SSH") + :CONFIG_TYPE: Configuration drive. (default: null) For a live example see the OpenDaylight project jenkins-config directory. https://github.com/opendaylight/releng-builder/tree/master/jenkins-config @@ -442,7 +443,8 @@ Runs `jenkins-jobs update` to update production job configuration JJB Verify ---------- -Runs `jenkins-jobs test` to validate JJB syntax +Runs `jenkins-jobs test` to validate JJB syntax. Optionally validates +build-node labels used in templates and job definitions. :Template Names: - {project-name}-jjb-verify @@ -463,6 +465,10 @@ Runs `jenkins-jobs test` to validate JJB syntax :build-concurrent: Whether or not to allow this job to run multiple jobs simultaneously. (default: true) :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-node-label-check: Whether to check build-node labels in jobs + against node names in cloud config files (default: false) + :build-node-label-list: Space-separated list of external build-node + labels not present in cloud config files (default: "") :build-timeout: Timeout in minutes before aborting build. (default: 10) :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) :jjb-cache: JJB cache location. (default: $HOME/.cache/jenkins_jobs) @@ -529,7 +535,7 @@ Info YAML Verify Info YAML Verify job validates that INFO.yaml file changes are kept isolated from other file changes. Verifies INFO.yaml files follow the schema defined in -`global-jjb/info-schema`. +`lfit/releng-global-jjb/schema/info-schema.yaml`. :Template Names: - {project-name}-info-yaml-verify @@ -650,6 +656,71 @@ Full Example: .. literalinclude:: ../../.jjb-test/lf-ci-jobs/openstack-cron-full.yaml +.. _gjjb-openstack-update-cloud-image: + +OpenStack Update Cloud Image +---------------------------- + +This job finds and updates OpenStack cloud images on the ci-management source +repository. + +The job is triggered in two ways: + +1. When packer merge job completes, the new image name created is passed + down to the job. +2. When the job is triggered manually to update all new images. + +When the job is triggered through an upstream packer merge job, this only +generates a change request for the new image built. + +When the job is triggered manually, this job finds the latest images on +OpenStack cloud and compares them with the images currently used in the source +ci-management source repository. If the compared images have newer +time stamps are **all** updated through a change request. + +This job requires a Jenkins configuration merge and verify job setup and +working on Jenkins. + +:Template Names: + - {project-name}-openstack-update-cloud-image + - gerrit-openstack-update-cloud-image + - github-openstack-update-cloud-image + +:Required parameters: + + :build-node: The node to run build on. + :jenkins-ssh-credential: Credential to use for SSH. (Generally should + be configured in defaults.yaml) + :new-image-name: Name of new image name passed from packer merge job or + set to 'all' to update all images. (default: all) + +:Optional parameters: + + :branch: Git branch to fetch for the build. (default: master) + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 90) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :openstack-cloud: OS_CLOUD setting to pass to openstack client. + (default: vex) + :stream: Keyword that can be used to represent a release code-name. + Often the same as the branch. (default: master) + :submodule-recursive: Whether to checkout submodules recursively. + (default: true) + :submodule-timeout: Timeout (in minutes) for checkout operation. + (default: 10) + :submodule-disable: Disable submodule checkout operation. + (default: false) + :update-cloud-image: Submit a change request to update new built cloud + image to Jenkins. (default: false) + +Minimal Example: + +.. literalinclude:: ../../.jjb-test/lf-ci-jobs/openstack-update-cloud-image-minimal.yaml + +Full Example: + +.. literalinclude:: ../../.jjb-test/lf-ci-jobs/openstack-update-cloud-image-full.yaml + .. _gjjb-packer-merge: @@ -673,9 +744,9 @@ Packer Merge job runs `packer build` to build system images in the cloud. :mvn-settings: The name of settings file containing credentials for the project. :platforms: Platform or distribution to build. Typically json file - found in the packer/vars directory. (Example: centos) - :template: System template to build. Typically shell script found in - the packer/provision directory. (Example: java-builder) + found in the packer/vars directory. (Example: centos-7) + :templates: System template to build. Typically a yaml file or shell script + found in the packer/provision directory. (Example: docker) :Optional parameters: @@ -683,6 +754,7 @@ Packer Merge job runs `packer build` to build system images in the cloud. :branch: Git branch to fetch for the build. (default: master) :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) :build-timeout: Timeout in minutes before aborting build. (default: 90) + :gerrit_verify_triggers: Override Gerrit Triggers. :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) :openstack: Packer template uses an OpenStack builder (default: true). :openstack-cloud: Sets OS_CLOUD variable to the value of this parameter. @@ -698,15 +770,16 @@ Packer Merge job runs `packer build` to build system images in the cloud. (default: 10) :submodule-disable: Disable submodule checkout operation. (default: false) + :update-cloud-image: Submit a change request to update newly built cloud + image to Jenkins. (default: false) - :gerrit_verify_triggers: Override Gerrit Triggers. Test an in-progress patch ^^^^^^^^^^^^^^^^^^^^^^^^^ -To test an in-progress patch from a GitHub Pull Request. Upload this +To test an in-progress patch from a GitHub Pull Request, upload this job to the :doc:`Jenkins Sandbox `. Then when manually -building the job replace the GERRIT_REFSPEC parameter with the GitHub Pull +building the job, replace the GERRIT_REFSPEC parameter with the GitHub Pull Request number of the patch you would like to test. Example GitHub: @@ -715,12 +788,15 @@ Example GitHub: GERRIT_REFSPEC: origin/pr/49/merge + .. _gjjb-packer-verify: Packer Verify ------------- -Packer Verify job runs `packer validate` to verify packer configuration. +Packer Verify job runs `packer validate` to verify packer configuration. The +verify job checks superficial syntax of the template and other files. It does +not attempt to build an image, and cannot detect all possible build issues. :Template Names: - {project-name}-packer-verify @@ -742,6 +818,9 @@ Packer Verify job runs `packer validate` to verify packer configuration. :branch: Git branch to fetch for the build. (default: master) :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) :build-timeout: Timeout in minutes before aborting build. (default: 10) + :gerrit_trigger_file_paths: Override file paths which can be used to + filter which file modifications will trigger a build. + :gerrit_verify_triggers: Override Gerrit Triggers. :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) :openstack: Packer template uses an OpenStack builder (default: true). :openstack-cloud: Sets OS_CLOUD variable to the value of this parameter. @@ -758,9 +837,61 @@ Packer Verify job runs `packer validate` to verify packer configuration. :submodule-disable: Disable submodule checkout operation. (default: false) - :gerrit_verify_triggers: Override Gerrit Triggers. - :gerrit_trigger_file_paths: Override file paths which can be used to - filter which file modifications will trigger a build. + +.. _gjjb-packer-verify-build: + +Packer Verify Build +------------------- + +Packer Verify Build job is essentially the same as the +:ref:`Packer Merge job`. It is triggered only by its keyword, +and will build a useable image. If the last patch set before a merge has a +successful verify build, the merge job will not build the same image. + +:Template Names: + - {project-name}-packer-verify-build-{platforms}-{templates} + - gerrit-packer-verify-build + - github-packer-verify-build + +:Comment Trigger: verify-build|packer-build + +:Required parameters: + + :build-node: The node to run build on. + :jenkins-ssh-credential: Credential to use for SSH. (Generally should + be configured in defaults.yaml) + :mvn-settings: The name of settings file containing credentials for + the project. + :platforms: Platform or distribution to build. Typically json file + found in the packer/vars directory. (Example: centos-7) + :templates: System template to build. Typically a yaml file or shell script + found in the packer/provision directory. (Example: docker) + +:Optional parameters: + + :branch: Git branch to fetch for the build. (default: master) + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 10) + :gerrit_trigger_file_paths: Override file paths which can be used to + filter which file modifications will trigger a build. + :gerrit_verify_triggers: Override Gerrit Triggers. + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :openstack: Packer template uses an OpenStack builder (default: true). + :openstack-cloud: Sets OS_CLOUD variable to the value of this parameter. + (default: vex). + :packer-cloud-settings: Name of settings file containing credentials + for the cloud that packer will build on. (default: packer-cloud-env) + :packer-version: Version of packer to install / use in build. (default: 1.0.2) + :stream: Keyword that can be used to represent a release code-name. + Often the same as the branch. (default: master) + :submodule-recursive: Whether to checkout submodules recursively. + (default: true) + :submodule-timeout: Timeout (in minutes) for checkout operation. + (default: 10) + :submodule-disable: Disable submodule checkout operation. + (default: false) + :update-cloud-image: Submit a change request to update new built cloud + image to Jenkins. (default: false) Puppet Verify @@ -806,3 +937,104 @@ the base directory is usually the best place to run from. (default: 10) :submodule-disable: Disable submodule checkout operation. (default: false) + + +Sonar +----- + +Runs the Jenkins SonarQube Scanner plug-in to analyze code for bugs, +code smells and security vulnerabilities, and to upload the result +(possibly including code-coverage statistics) to a SonarQube server +or to SonarCloud.io. + +Requires ``SonarQube Scanner for Jenkins`` + +One of the optional parameters sonar-project-file and sonar-properties +must be supplied; they cannot both be empty. + +Plug-in configurations + Manage Jenkins --> Configure System --> SonarQube servers + - Name: Sonar (fixed) + - Server URL: https://sonar.server.org/ or https://sonarcloud.io + - Server authentication token: none for local, API token (saved as a + "secret text" credential) for Sonarcloud + + Manage Jenkins --> Global Tool Configuration --> SonarQube Scanner + - Name: SonarQube Scanner (fixed) + - Install automatically + - Select latest version + +.. note:: Sonar properties can be set directly in the job definition by + setting the sonar-project-file to ``""`` and adding all properties under + ``sonar-properties``. + +:Template Names: + + - {project-name}-sonar + - gerrit-sonar + - github-sonar + +:Optional Parameters: + :sonar-task: Sonar task to run. (default: "") + :sonar-project-file: The filename for the project's properties + (default: "sonar-project.properties") + :sonar-properties: Sonar configuration properties. (default: "") + :sonar-java-opts: JVM options. (default: "") + :sonar-additional-args: Additional command line arguments. (default: "") + + +Sonar with Prescan +------------------ + +The same as the Sonar job above, except the caller also defines a builder +called ``lf-sonar-prescan``, in which they can put any builders that they want +to run prior to the Sonar scan. + +.. code-block:: yaml + + - builder: + name: lf-sonar-prescan + builders: + - shell: "# Pre-scan shell script" + +:Template Names: + + - {project-name}-sonar-prescan + - gerrit-sonar-prescan + - github-sonar-prescan + +:Required Parameters: + :lf-sonar-prescan: A builder that will run prior to the Sonar scan. + +:Optional Parameters: + :sonar-task: Sonar task to run. (default: "") + :sonar-project-file: The filename for the project's properties + (default: "sonar-project.properties") + :sonar-properties: Sonar configuration properties. (default: "") + :sonar-java-opts: JVM options. (default: "") + :sonar-additional-args: Additional command line arguments. (default: "") + + +Sonar with Prescan Script +------------------------- + +The same as the Sonar job above, except the caller must supply a shell script +to run prior to the Sonar scan. This is commonly used to install prerequisites, +build the project, execute unit tests and generate a code-coverage report. + +:Template Names: + + - {project-name}-sonar-prescan-script + - gerrit-sonar-prescan-script + - github-sonar-prescan-script + +:Required Parameters: + :sonar-prescan-script: A shell script that will run prior to the Sonar scan. + +:Optional Parameters: + :sonar-task: Sonar task to run. (default: "") + :sonar-project-file: The filename for the project's properties. + (default: "sonar-project.properties") + :sonar-properties: Sonar configuration properties. (default: "") + :sonar-java-opts: JVM options. (default: "") + :sonar-additional-args: Additional command line arguments. (default: "")