X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=blobdiff_plain;f=docs%2Fjjb%2Flf-release-jobs.rst;fp=docs%2Fjjb%2Flf-release-jobs.rst;h=60ec72c1f23fe4bfb9d678dbab99c9e083a23639;hb=c53e11db14ab7abede74c74801bec9ca381c8cc6;hp=d0f5340feb34f7d005ba23b161f7a377dadf79b3;hpb=d0dfd24055995c4a2ed27fb7bc9e3dee65ba0d16;p=releng%2Fglobal-jjb.git diff --git a/docs/jjb/lf-release-jobs.rst b/docs/jjb/lf-release-jobs.rst index d0f5340f..60ec72c1 100644 --- a/docs/jjb/lf-release-jobs.rst +++ b/docs/jjb/lf-release-jobs.rst @@ -1,32 +1,43 @@ .. _lf-global-jjb-release: -####################### Self-Serve Release Jobs -####################### - -Self-serve release jobs allow a project team to direct Jenkins to -promote a jar file or container image from a staging area to a release -area. To trigger the action, create a releases/ or .releases/ -directory, add a release yaml file to it, and submit a change set with -one release yaml file to Gerrit. Upon merge of the change, Jenkins will -sign the reference extrapolated by log_dir and promote the artifact. The -expected format of the release yaml file appears in schemas and examples -below. - -The build node for maven and container release jobs must be CentOS, -which supports the sigul client for accessing a signing server. The -build node for container release jobs must have Docker installed. - -A Jenkins user can also trigger a release job via the "Build with -parameters" action, removing the need for a release yaml file. The -user must enter parameters in the same way as a release yaml file, -except for the special USE_RELEASE_FILE and DRY_RUN check boxes. The -user must uncheck the USE_RELEASE_FILE check box if the job should -run with a release file, while passing the required information as -build parameters. Similarly, the user must uncheck the DRY_RUN check -box to test the job while skipping repository promotion to Nexus. - -The special parameters are as follows:: +======================= + +Self-serve release jobs allow project committers to direct Jenkins to +promote a jar file, container image or Python package from a staging +area to a release area. + +To use the self-release process, create a releases/ or .releases/ +directory at the root of the project repository, add one release yaml +file to it, and submit a change set with that release yaml file. The +required contents of the release yaml file are different for each type +of release, see the schemas and examples shown below. The version in +the release yaml file must be a valid Semantic Versioning (SemVer) +string, matching either the pattern "v#.#.#" or "#.#.#" where "#" is +one or more digits. Upon merge of the change, Jenkins will sign the +reference extrapolated by log_dir and promote the artifact. + +.. note:: + + The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml). + In words, the directory name can be ".releases" or "releases"; the file + name can be anything with suffix ".yaml". + +The build node for all release jobs must be CentOS, which supports the +sigul client for accessing a signing server. The build node for +container release jobs must have Docker installed. + +A Jenkins admin user can also trigger a release job via the "Build +with parameters" action, removing the need to create and merge a +release yaml file. The user must enter parameters in the same way as +a release yaml file, except for the special USE_RELEASE_FILE and +DRY_RUN check boxes. The user must uncheck the USE_RELEASE_FILE check +box if the job should run without a release file, instead passing the +required information as build parameters. The user can check the +DRY_RUN check box to test the job while skipping upload of files to +the release repository. + +For example, the parameters for a Maven release are as follows:: GERRIT_BRANCH = master VERSION = 1.0.0 @@ -35,13 +46,35 @@ The special parameters are as follows:: USE_RELEASE_FILE = false DRY_RUN = false -.. note:: +Maven Release Files +------------------- - The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml). - In words, the directory name can be ".releases" or "releases"; the file - name can be anything with suffix ".yaml". +An example of a maven release file appears below. + +.. code-block:: none -The JSON schema for a maven release job appears below. + $ cat releases/maven-release.yaml + --- + distribution_type: maven + log_dir: example-project-maven-stage-master/17/ + project: example-project + version: 1.0.0 + + +The following parameters must appear in a maven release yaml file. + +:Required Parameters: + + :distribution_type: Must be "maven". + :log_dir: The suffix of the logs URL reported on completion by the + Jenkins stage job that created and pushed the artifact + to the staging repository. For example, use value + "example-project-maven-stage-master/17" for the logs URL + https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-maven-stage-master/17 + :project: The name of the project. + :version: The semantic version string used for the artifact. + +The JSON schema for a maven release file appears below. .. code-block:: none @@ -66,19 +99,46 @@ The JSON schema for a maven release job appears below. type: "string" -Example of a maven release file: +Container Release Files +----------------------- -.. code-block:: bash +An example of a container release file appears below. - $ cat releases/1.0.0-maven.yaml +.. code-block:: none + + $ cat releases/container-release.yaml --- - distribution_type: 'maven' - version: '1.0.0' - project: 'example-project' - log_dir: 'example-project-maven-stage-master/17/' + distribution_type: container + container_release_tag: 1.0.0 + container_pull_registry: nexus.onap.org:10003 + container_push_registry: nexus.onap.org:10002 + project: test + ref: d1b9cd2dd345fbeec0d3e2162e008358b8b663b2 + containers: + - name: test-backend + version: 1.0.0-20190806T184921Z + - name: test-frontend + version: 1.0.0-20190806T184921Z + +The following parameters must appear in a container release yaml file. + +:Required Parameters: -The JSON schema for a container release job appears below. + :distribution_type: Must be "container". + :container_release_tag: The string to use as a Docker tag on all + released containers. + :container_pull_registry: The Nexus registry that supplies the staged + image(s). + :container_push_registry: The Nexus registry that receives the released + image(s). + :project: The name of the project. + :ref: The git commit reference (SHA-1 code) to tag with the version string. + :containers: A list of name and version (tag) pairs that specify the + Docker images in the container-pull registry to promote to the + container-push registry. + +The JSON schema for a container release file appears below. .. code-block:: none @@ -116,30 +176,74 @@ The JSON schema for a container release job appears below. type: "string" -An example of a container release file appears below. The job tags the -git repository at the specified commit reference. The job applies the -container_release_tag string to all released containers. The job uses the -per-container version strings to pull images from the container registry. +PyPI Release Files +------------------ -.. code-block:: bash +An example of a PyPI release file appears below. + +.. code-block:: none - $ cat releases/1.0.0-container.yaml + $ cat releases/pypi-release.yaml --- - distribution_type: 'container' - container_release_tag: '1.0.0' - container_pull_registry: 'nexus.onap.org:10003" - container_push_registry: 'nexus.onap.org:10002" - project: 'test' - ref: d1b9cd2dd345fbeec0d3e2162e008358b8b663b2 - containers: - - name: test-backend - version: 1.0.0-20190806T184921Z - - name: test-frontend - version: 1.0.0-20190806T184921Z + distribution_type: pypi + pypi_project: mypackage + python_version: 3.4 + version: 1.0.0 + +The following parameters must appear in the PyPI release yaml file. +These are not part of the Jenkins job definition to allow independent +self-release of a package maintained in a git repository with other +packages. -Example of a Jenkins job configuration that uses the global-jjb -templates for Gerrit: +:Required Parameters: + + :distribution_type: Must be "pypi". + :log_dir: The suffix of the logs URL reported on completion by the + Jenkins merge job that created and pushed the distribution files + to the staging repository. For example, use value + "example-project-pypi-merge-master/17" for the logs URL + https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-pypi-merge-master/17 + :pypi_project: The PyPI project name at the staging and + release repositories, for example "mypackage". + :python_version: The Python interpreter version to use for pip + "Requires-Python" compatibility checks, for example "3" or "3.7.0". + :version: The semantic version string used for the package in the + setup.py file. + +The JSON schema for a PyPI release file appears below. + +.. code-block:: none + + --- + $schema: "http://json-schema.org/schema#" + $id: "https://github.com/lfit/releng-global-jjb/blob/master/release-pypi-schema.yaml" + + required: + - "distribution_type" + - "log_dir" + - "pypi_project" + - "python_version" + - "version" + + properties: + distribution_type: + type: "string" + log_dir: + type: "string" + pypi_project: + type: "string" + python_version: + type: "string" + version: + type: "string" + + +Jenkins Jobs +------------ + +An example of a Jenkins job configuration that uses the global-jjb +templates for maven and container release jobs appears next. .. code-block:: none @@ -158,11 +262,167 @@ templates for Gerrit: Release Engineers: please follow the setup guide below before adding the job definition. +JJB Macros +---------- + +lf-release +~~~~~~~~~~ + +Release verify and merge jobs are the same except for their scm, +trigger, and builders definition. This anchor is the common template. + +JJB Templates +------------- + +Release Merge +~~~~~~~~~~~~~ + +This template supports Maven and Container release jobs. + +:Template Name: {project-name}-release-merge + +:Comment Trigger: remerge + +:Required parameters: + + :build-node: The node to run build on. + :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + :project: Git repository name + :project-name: Jenkins job name prefix + +:Optional parameters: + + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 15) + + :gerrit_merge_triggers: Override Gerrit Triggers. + :gerrit_trigger_file_paths: Override file paths filter which checks which + file modifications will trigger a build. + **default**:: + + - compare-type: REG_EXP + pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' + + +Release Verify +~~~~~~~~~~~~~~ + +This template supports Maven and Container release jobs. + +:Template Name: {project-name}-release-verify + +:Comment Trigger: recheck|reverify + +:Required Parameters: + + :build-node: The node to run build on. + :jenkins-ssh-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + :project: Git repository name + :project-name: Jenkins job name prefix + +:Optional Parameters: + + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-node: The node to run build on. + :build-timeout: Timeout in minutes before aborting build. (default: 15) + :gerrit-skip-vote: Skip voting for this job. (default: false) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + + :gerrit_verify_triggers: Override Gerrit Triggers. + :gerrit_trigger_file_paths: Override file paths filter which checks which + file modifications will trigger a build. + **default**:: + + - compare-type: REG_EXP + pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' + + +PyPI Release Merge +~~~~~~~~~~~~~~~~~~ + +Publishes a Python package on merge of a patch set with a release yaml +file. Checks the format of the version string, downloads the package +artifacts from the PyPI staging repository, uploads the package +artifacts to the PyPI release repository, tags the git repository, +signs the tag and pushes the tag to the git server. The release verify +template accepts neither a branch nor a stream parameter. + +:Template Names: + + - {project-name}-pypi-release-merge + - gerrit-pypi-release-merge + - github-pypi-release-merge + +:Comment Trigger: remerge + +:Required Parameters: + + :build-node: The node to run build on, which must be Centos. + :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + :project: Git repository name + :project-name: Jenkins job name prefix + +:Optional Parameters: + + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 15) + :disable-job: Whether to disable the job (default: false) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :pypi-stage-index: Base URL of the PyPI staging repository. + (default https://test.pypi.org/simple) + :pypi-repo: Key for the PyPI release repository in the .pypirc file, + should be the repository pypy.org. (default: pypi) + :use-release-file: Whether to use the release file. (default: true) + + +PyPI Release Verify +~~~~~~~~~~~~~~~~~~~ + +Verifies a Python package project on creation of a patch set with a +release yaml file. Checks the contents of the release yaml file, +checks the format of the version string, and downloads the release +artifacts from the specified PyPI staging repository. The release +verify template accepts neither a branch nor a stream parameter. + +:Template Names: + + - {project-name}-pypi-release-verify + - gerrit-pypi-release-verify + - github-pypi-release-verify + +:Comment Trigger: recheck + +:Required Parameters: + + :build-node: The node to run build on, which must be Centos. + :jenkins-ssh-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + :project: Git repository name + :project-name: Jenkins job name prefix + +:Optional Parameters: + + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 15) + :disable-job: Whether to disable the job (default: false) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :pypi-stage-index: Base URL of the PyPI staging repository. + (default https://test.pypi.org/simple) + :pypi-repo: Key for the PyPI release repository in the .pypirc file, + should be the repository pypy.org (default: pypi) + :use-release-file: Whether to use the release file. (default: true) + + Setup for LFID, Nexus, Jenkins and Gerrit -========================================= +----------------------------------------- + +This section is for the Linux Foundation release engineering team. LFID -==== +~~~~ Create an ``lfid`` and an ``ssh-key`` @@ -181,14 +441,14 @@ ssh-key example: Nexus -===== +~~~~~ Create a Nexus account called ``'jenkins-release'`` with promote privileges. .. image:: ../_static/nexus-promote-privs.png Gerrit -====== +~~~~~~ Log into your Gerrit with ``YOUR_RELEASE_USERNAME``, upload the public part of the ``ssh-key`` you created earlier. Log out of Gerrit and log @@ -214,17 +474,17 @@ In All project, grant group self-serve-release the following: Jenkins -======= +~~~~~~~ Add a global credential to Jenkins called ``jenkins-release`` and set the ID: ``'jenkins-release'`` as its value insert the private half of the ``ssh-key`` that you created for your Gerrit user. -Add Global vars in Jenkins: -Jenkins configure -> Global properties -> Environment variables +Add Global variables in Jenkins: +Jenkins configure -> Global properties -> Environment variables:: -``RELEASE_USERNAME = YOUR_RELEASE_USERNAME`` -``RELEASE_EMAIL = YOUR_RELEASE_EMAIL`` + RELEASE_USERNAME = YOUR_RELEASE_USERNAME + RELEASE_EMAIL = YOUR_RELEASE_EMAIL .. note:: @@ -234,12 +494,14 @@ Jenkins configure -> Global properties -> Environment variables Jenkins configure -> Managed Files -> Add a New Config -> Custom File -id: signing-pubkey -Name: SIGNING_PUBKEY (optional) -Comment: SIGNING_PUBKEY (optional) +.. code-block:: none + + id: signing-pubkey + Name: SIGNING_PUBKEY (optional) + Comment: SIGNING_PUBKEY (optional) -Content: (Ask Andy for the public signing key) ------BEGIN PGP PUBLIC KEY BLOCK----- + Content: (Ask Andy for the public signing key) + -----BEGIN PGP PUBLIC KEY BLOCK----- Add or edit the managed file in Jenkins called ``lftoolsini``, @@ -253,87 +515,11 @@ appending a nexus section: Jenkins Settings -> Managed files -> Add password= Ci-management -============= +~~~~~~~~~~~~~ Upgrade your project's global-jjb if needed, then add the following to your global defaults file (e.g., jjb/defaults.yaml). .. code-block:: none - jenkins-ssh-release-credential: 'jenkins-release' - -Macros -====== - -lf-release ----------- - -Release verify and merge jobs are the same except for their scm, -trigger, and builders definition. This anchor is the common template. - -Job Templates -============= - -Release Merge -------------- - -:Template Name: {project-name}-release-merge - -:Comment Trigger: remerge - -:Required parameters: - - :build-node: The node to run build on. - :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set - in defaults.yaml) - :stream: run this job against: ** - -:Optional parameters: - - :branch: Git branch to fetch for the build. (default: all) - :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) - :build-timeout: Timeout in minutes before aborting build. (default: 15) - :project-pattern: Project to trigger build against. (default: \*\*) - - :gerrit_merge_triggers: Override Gerrit Triggers. - :gerrit_trigger_file_paths: Override file paths filter which checks which - file modifications will trigger a build. - **default**:: - - - compare-type: REG_EXP - pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' - - -Release Verify ------------------- - -:Template Name: {project-name}-release-verify - -:Comment Trigger: recheck|reverify - -:Required Parameters: - - :build-node: The node to run build on. - :jenkins-ssh-credential: Credential to use for SSH. (Generally set - in defaults.yaml) - :stream: run this job against: ** - -:Optional Parameters: - - :branch: Git branch to fetch for the build. (default: all) - :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) - :build-node: The node to run build on. - :build-timeout: Timeout in minutes before aborting build. (default: 15) - :doc-dir: Directory where tox will place built docs. - as defined in the tox.ini (default: docs/_build/html) - :gerrit-skip-vote: Skip voting for this job. (default: false) - :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) - :project-pattern: Project to trigger build against. (default: \*\*) - - :gerrit_verify_triggers: Override Gerrit Triggers. - :gerrit_trigger_file_paths: Override file paths filter which checks which - file modifications will trigger a build. - **default**:: - - - compare-type: REG_EXP - pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' + jenkins-ssh-release-credential: jenkins-release