X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=blobdiff_plain;f=docs%2Fjjb%2Flf-release-jobs.rst;h=2ddc6eec004db31a272f4fa34182d4f18c3950ff;hb=6b1b8ed30a5a553965664d84b2a028534e63ee59;hp=0a9cc09ec9fbbf9a3daa481059f9e5580e016b16;hpb=5c49acecbc25ed26a243ac39885895ee3955fab9;p=releng%2Fglobal-jjb.git diff --git a/docs/jjb/lf-release-jobs.rst b/docs/jjb/lf-release-jobs.rst index 0a9cc09e..2ddc6eec 100644 --- a/docs/jjb/lf-release-jobs.rst +++ b/docs/jjb/lf-release-jobs.rst @@ -1,22 +1,47 @@ .. _lf-global-jjb-release: -####################### Self-Serve Release Jobs -####################### +======================= + +Self-serve release jobs allow project committers to promote a jar +file, container image, Python package or PackageCloud artifact from a +staging area to a release area. A release yaml file controls the +process, and Jenkins promotes the artifact when a project committer +merges the release yaml file in Gerrit. + +To use the self-release process, create a releases/ or .releases/ +directory at the root of the project repository, add one release yaml +file to it, and submit a change set with that release yaml file. The +required contents of the release yaml file are different for each type +of release, see the schemas and examples shown below. The version +string in the release yaml file should be a valid Semantic Versioning +(SemVer) string, matching the pattern "#.#.#" where "#" is one or more +digits. A version string matching the pattern "v#.#.#" is also +accepted. Upon merge of the change, a Jenkins job promotes the +artifact and pushes a gpg-signed tag to the repository. -Self-serve release jobs allow a project team to direct Jenkins to promote jar files or container -images from staging areas to release areas. To trigger the action, create a releases/ or .releases/ -directory, place one or more release yaml files in it, and submit the change to Gerrit. Upon merge -of the change, Jenkins will sign the reference(s) extrapolated by log_dir and promote the artifact(s). +.. note:: + + The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml). + In words, the directory name can be ".releases" or "releases"; the file + name can be anything with suffix ".yaml". Some release jobs require + a specific prefix on the file, as described below. + +The build node for all release jobs must be CentOS, which supports the +sigul client for accessing a signing server to sign a tag. The build +node for container release jobs must have Docker installed. -Release jobs can also be triggered from Jenkins via the "Build with parameters" action, removing the -need for a release yaml file. The parameters must be filled out in the same way as a release file, -except for the special USE_RELEASE_FILE and DRY_RUN check boxes. The USE_RELEASE_FILE check box must -be unchecked if the job is expected to run with a release file, while passing the required information -as build parameters. Similarly, the DRY_RUN check box must be unchecked if the job needs to be tested -while skipping repository promotion to Nexus. +A Jenkins admin user can also trigger a release job via the "Build +with parameters" action, removing the need to create and merge a +release yaml file. The user must enter parameters in the same way as +a release yaml file, except for the special USE_RELEASE_FILE and +DRY_RUN check boxes. The user must uncheck the USE_RELEASE_FILE check +box if the job should run without a release file, instead passing the +required information as build parameters. The user can check the +DRY_RUN check box to test the job while skipping upload of files to +the release repository. -The special parameters are as follows:: +For example, the parameters for a Maven release are as follows:: GERRIT_BRANCH = master VERSION = 1.0.0 @@ -25,168 +50,238 @@ The special parameters are as follows:: USE_RELEASE_FILE = false DRY_RUN = false -.. note:: - - The release files regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml). - In words, the directory name can be ".releases" or "releases"; the file - name can be any name with suffix ".yaml". +Maven Release Files +------------------- -Example of a maven release file: +An example of a maven release file appears below. -.. code-block:: bash +.. code-block:: none - $ cat releases/maven-1.0.0.yaml + $ cat releases/maven-release.yaml --- - distribution_type: 'maven' - version: '1.0.0' - project: 'example-project' - log_dir: 'example-project-maven-stage-master/17/' + distribution_type: maven + log_dir: example-project-maven-stage-master/17/ + project: example-project + version: 1.0.0 -Example of a container release file: +The following parameters must appear in a maven release yaml file. -.. code-block:: bash +:Required Parameters: - $ cat releases/container-1.0.0.yaml - --- - distribution_type: 'container' - version: '1.0.0' - project: 'test' - containers: - - name: test-backend - version: 1.0.0-20190806T184921Z - - name: test-frontend - version: 1.0.0-20190806T184921Z + :distribution_type: Must be "maven". + :log_dir: The suffix of the logs URL reported on completion by the + Jenkins stage job that created and pushed the artifact + to the staging repository. For example, use value + "example-project-maven-stage-master/17" for the logs URL + https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-maven-stage-master/17 + :project: The name of the project. + :version: The semantic version string used for the artifact. + +:Optional Parameters: + :git_tag: The tag string to sign and push to the Git repository. + (default: the semantic version string) -.. note:: +The JSON schema for a maven release file appears below. - Job should be appended under gerrit-maven-stage +.. literalinclude:: ../../schema/release-schema.yaml + :language: yaml -Example of a terse Jenkins job to call the global-jjb macro: + +Container Release Files +----------------------- + +An example of a container release file appears below. .. code-block:: none - - gerrit-maven-stage: - sign-artifacts: true - build-node: centos7-docker-8c-8g - maven-versions-plugin: true - - '{project-name}-gerrit-release-jobs': - build-node: centos7-docker-8c-8g + $ cat releases/container-release.yaml + --- + distribution_type: container + container_release_tag: 1.0.0 + container_pull_registry: nexus.onap.org:10003 + container_push_registry: nexus.onap.org:10002 + project: test + ref: d1b9cd2dd345fbeec0d3e2162e008358b8b663b2 + containers: + - name: test-backend + version: 1.0.0-20190806T184921Z + - name: test-frontend + version: 1.0.0-20190806T184921Z -.. note:: - Release Engineers: please follow the setup guide below before adding the job definition. +The following parameters must appear in a container release yaml file. +:Required Parameters: -Setup for LFID, Nexus, Jenkins and Gerrit -========================================= + :distribution_type: Must be "container". + :container_release_tag: The string to use as a Docker tag on all + released containers. + :container_pull_registry: The Nexus registry that supplies the staged + image(s). + :container_push_registry: The Nexus registry that receives the released + image(s). + :project: The name of the project. + :ref: The git commit reference (SHA-1 code) to tag with the version string. + :containers: A list of name and version (tag) pairs that specify the + Docker images in the container-pull registry to promote to the + container-push registry. -LFID -==== +:Optional Parameters: -Create an ``lfid`` and an ``ssh-key`` + :git_tag: The tag string to sign and push to the Git repository. + (default: the semantic version string) -``YOUR_RELEASE_USERNAME`` for example: onap-release +The JSON schema for a container release file appears below. -``YOUR_RELEASE_EMAIL`` for example: collab-it+onap-release@linuxfoundation.org +.. literalinclude:: ../../schema/release-container-schema.yaml + :language: yaml -ssh-key example: -.. code-block:: bash +PyPI Release Files +------------------ - ssh-keygen -t rsa -C "collab-it+odl-release@linuxfoundation.org" -f /tmp/odl-release +An example of a PyPI release file appears below. Name of the release file must +start with "pypi". For example releases/pypi-1.0.0-mypackage.yaml +.. code-block:: none -`Create an LFID with the above values `_ + $ cat releases/pypi-1.0.0-mypackage.yaml + --- + pypi_project: mypackage + python_version: '3.4' + version: 1.0.0 + log_dir: example-project-pypi-merge-master/17 -Nexus -===== +The following parameters must appear in the PyPI release yaml file. +These are not part of the Jenkins job definition to allow independent +self-release of a package maintained in a git repository with other +packages. -Create a Nexus account called ``'jenkins-release'`` with promote privileges. +:Required Parameters: -.. image:: ../_static/nexus-promote-privs.png + :log_dir: The suffix of the logs URL reported on completion by the + Jenkins merge job that created and pushed the distribution files + to the staging repository. For example, use value + "example-project-pypi-merge-master/17" for the logs URL + https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-pypi-merge-master/17 + :pypi_project: The PyPI project name at the staging and + release repositories, for example "mypackage". + :python_version: The Python interpreter version to use for pip + "Requires-Python" compatibility checks, for example '3', '3.7' or 3.7.4. + Put valid decimal values such as 3 or 3.7 in quotes to pass schema validation. + :version: The semantic version string used for the package in the + setup.py file. -Gerrit -====== +:Optional Parameters: -Log into your Gerrit with ``YOU_RELEASE_USERNAME``, upload the public part of the ``ssh-key`` you -created earlier. Log out of Gerrit and log in again with your normal account for the next steps. + :git_tag: The tag string to sign and push to the Git repository. + (default: the semantic version string) +The JSON schema for a PyPI release file appears below. -In Gerrit create a new group called ``self-serve-release`` and give it direct push rights via ``All-Projects`` -Add ``YOUR_RELEASE_USERNAME`` to group ``self-serve-release`` and group ``Non-Interactive Users`` +.. literalinclude:: ../../schema/release-pypi-schema.yaml + :language: yaml -In All project, grant group self-serve-release the following: +PackageCloud Release Files +-------------------------- -.. code-block:: none +An example of a PackageCloud release file appears below. Name of release file +must start with "packagecloud". For example releases/packagecloud-1.6-tree.yaml - [access "refs/heads/*"] - push = group self-serve-release - [access "refs/tags/*"] - createTag = group self-serve-release - createSignedTag = group self-serve-release - forgeCommitter = group self-serve-release - push = group self-serve-release +.. code-block:: none + $ cat releases/packagecloud-1.6-tree.yaml + --- + package_name: tree + packages: + - name: tree_1.6.0_amd64.deb + - name: tree-dev_1.6.0_amd64.deb + - name: tree-devel-1.6.0-1.x86_64.rpm + - name: tree-1.6.0-1.x86_64.rpm + ref: 5555cd2dd345fbeec0d3e2162e00835852342cda + log_dir: example-project-packagecloud-merge/21 + version: 1.6.0 + +The following parameters must appear in the PackageCloud release yaml file. +These are not part of the Jenkins job definition to allow independent +self-release of a package maintained in a git repository with other +packages. -Jenkins -======= +:Required Parameters: -Add a global credential to Jenkins called ``jenkins-release`` and set the ID: ``'jenkins-release'`` -as its value insert the private half of the ``ssh-key`` that you created for your Gerrit user. + :package_name: Name of the release package. + :packages: A list of names that specify the packages to promote. + Found in jenkins console log when using gem to push package eg. + "Pushing /path/of/package/name-of-package.rpm... success!" + OR using rest api call to query packagecloud.io repo + "curl https://packagecloud.io/api/v1/repos/test_user/test_repo/search?q= + | yq -r .[].filename" + :ref: The git commit reference (SHA-1 code) to tag with the version string. + :log_dir: The suffix of the logs URL reported on completion by the + Jenkins merge job that created and pushed the distribution files + to the staging repository. For example, use value + "example-project-packagecloud-merge-/21" for the logs URL + https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-packagecloud-merge/21 + :version: The semantic version string used for the package. -Add Global vars in Jenkins: -Jenkins configure -> Global properties -> Environment variables +:Optional Parameters: -``RELEASE_USERNAME = YOUR_RELEASE_USERNAME`` -``RELEASE_EMAIL = YOUR_RELEASE_EMAIL`` + :git_tag: The tag string to sign and push to the Git repository. + (default: the semantic version string) -Jenkins configure -> Managed Files -> Add a New Config -> Custom File +The JSON schema for a PackageCloud release file appears below. -id: signing-pubkey -Name: SIGNING_PUBKEY (optional) -Comment: SIGNING_PUBKEY (optional) +.. literalinclude:: ../../schema/release-packagecloud-schema.yaml + :language: yaml -Content: (Ask Andy for the public signing key) ------BEGIN PGP PUBLIC KEY BLOCK----- +Jenkins Jobs +------------ -Add or edit the managed file in Jenkins called ``lftoolsini``, appending a nexus section: -Jenkins Settings -> Managed files -> Add (or edit) -> Custom file +An example of a Jenkins job configuration that uses the global-jjb +templates for maven and container release jobs appears next. .. code-block:: none - [nexus.example.com] - username=jenkins-release - password= + - project: + name: my-project-release + project: my-project + project-name: my-project + build-node: centos7-docker-4c-4g + mvn-settings: my-project-settings + jobs: + - '{project-name}-gerrit-release-jobs' -Ci-management -============= -Upgrade your project's global-jjb if needed, then add the following to your global defaults -file (e.g., jjb/defaults.yaml). +.. note:: -.. code-block:: bash + Release Engineers: please follow the setup guide below before adding the job definition. - jenkins-ssh-release-credential: 'jenkins-release' -Macros -====== +JJB Macros +---------- lf-release ----------- +~~~~~~~~~~ -Release verify and merge jobs are the same except for their scm, trigger, and -builders definition. This anchor is the common template. +Release verify and merge jobs are the same except for their scm, +trigger, and builders definition. This anchor is the common template. -Job Templates -============= +JJB Templates +------------- Release Merge -------------- +~~~~~~~~~~~~~ + +This template supports Maven and Container release jobs. + +This template uses a git commit choosing strategy that builds the merged +commit with the release yaml file, not the tip of the target branch, so +projects can repeat the release action in case of merge job failure. :Template Name: {project-name}-release-merge @@ -197,14 +292,13 @@ Release Merge :build-node: The node to run build on. :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set in defaults.yaml) - :stream: run this job against: ** + :project: Git repository name + :project-name: Jenkins job name prefix :Optional parameters: - :branch: Git branch to fetch for the build. (default: all) :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) :build-timeout: Timeout in minutes before aborting build. (default: 15) - :project-pattern: Project to trigger build against. (default: \*\*) :gerrit_merge_triggers: Override Gerrit Triggers. :gerrit_trigger_file_paths: Override file paths filter which checks which @@ -216,7 +310,9 @@ Release Merge Release Verify ------------------- +~~~~~~~~~~~~~~ + +This template supports Maven and Container release jobs. :Template Name: {project-name}-release-verify @@ -227,19 +323,16 @@ Release Verify :build-node: The node to run build on. :jenkins-ssh-credential: Credential to use for SSH. (Generally set in defaults.yaml) - :stream: run this job against: ** + :project: Git repository name + :project-name: Jenkins job name prefix :Optional Parameters: - :branch: Git branch to fetch for the build. (default: all) :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) :build-node: The node to run build on. :build-timeout: Timeout in minutes before aborting build. (default: 15) - :doc-dir: Directory where tox will place built docs. - as defined in the tox.ini (default: docs/_build/html) :gerrit-skip-vote: Skip voting for this job. (default: false) :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) - :project-pattern: Project to trigger build against. (default: \*\*) :gerrit_verify_triggers: Override Gerrit Triggers. :gerrit_trigger_file_paths: Override file paths filter which checks which @@ -248,3 +341,277 @@ Release Verify - compare-type: REG_EXP pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)' + + +PyPI Release Merge +~~~~~~~~~~~~~~~~~~ + +Publishes a Python package on merge of a patch set with a release yaml +file. Checks the format of the version string, downloads the package +artifacts from the PyPI staging repository, uploads the package +artifacts to the PyPI release repository, tags the git repository, +signs the tag and pushes the tag to the git server. The release merge +template accepts neither a branch nor a stream parameter. + +These templates use a git commit choosing strategy that builds the merged +commit with the release yaml file, not the tip of the target branch, so +projects can repeat the release action in case of merge job failure. + +:Template Names: + + - {project-name}-pypi-release-merge + - gerrit-pypi-release-merge + - github-pypi-release-merge + +:Comment Trigger: remerge + +:Required Parameters: + + :build-node: The node to run build on, which must be Centos. + :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + :project: Git repository name + :project-name: Jenkins job name prefix + +:Optional Parameters: + + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 15) + :disable-job: Whether to disable the job (default: false) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :pypi-stage-index: Base URL of the PyPI staging repository. + (default https://test.pypi.org/simple) + :pypi-repo: Key for the PyPI release repository in the .pypirc file, + should be the repository pypy.org. (default: pypi) + :use-release-file: Whether to use the release file. (default: true) + + :gerrit_trigger_file_paths: Override file paths filter which checks which + file modifications will trigger a build. + **default**:: + + - compare-type: REG_EXP + pattern: '(releases\/pypi.*\.yaml|\.releases\/pypi.*\.yaml)' + +PyPI Release Verify +~~~~~~~~~~~~~~~~~~~ + +Verifies a Python package project on creation of a patch set with a +release yaml file. Checks the contents of the release yaml file, +checks the format of the version string, and downloads the release +artifacts from the specified PyPI staging repository. The release +verify template accepts neither a branch nor a stream parameter. + +:Template Names: + + - {project-name}-pypi-release-verify + - gerrit-pypi-release-verify + - github-pypi-release-verify + +:Comment Trigger: recheck + +:Required Parameters: + + :build-node: The node to run build on, which must be Centos. + :jenkins-ssh-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + :project: Git repository name + :project-name: Jenkins job name prefix + +:Optional Parameters: + + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 15) + :disable-job: Whether to disable the job (default: false) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :pypi-stage-index: Base URL of the PyPI staging repository. + (default https://test.pypi.org/simple) + :pypi-repo: Key for the PyPI release repository in the .pypirc file, + should be the repository pypy.org (default: pypi) + :use-release-file: Whether to use the release file. (default: true) + + :gerrit_trigger_file_paths: Override file paths filter which checks which + file modifications will trigger a build. + **default**:: + + - compare-type: REG_EXP + pattern: '(releases\/pypi.*\.yaml|\.releases\/pypi.*\.yaml)' + +PackageCloud Release Verify +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This template supports PackageCloud release jobs. Checks that the specified +packages are present in the staging repository and absent from the release +repository. + +:Template Name: {project-name}-packagecloud-release-verify + +:Comment Trigger: recheck|reverify + +:Required Parameters: + + :build-node: The node to run build on. + :jenkins-ssh-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + :project: Git repository name + :project-name: Jenkins job name prefix + +:Optional Parameters: + + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-node: The node to run build on. + :build-timeout: Timeout in minutes before aborting build. (default: 15) + :gerrit-skip-vote: Skip voting for this job. (default: false) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + + :gerrit_verify_triggers: Override Gerrit Triggers. + :gerrit_trigger_file_paths: Override file paths filter which checks which + file modifications will trigger a build. + **default**:: + + - compare-type: REG_EXP + pattern: '(releases\/packagecloud.*\.yaml|\.releases\/packagecloud.*\.yaml)' + + +PackageCloud Release Merge +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This template supports PackageCloud release jobs. Promotes the specified +packages from the staging repository to the release repository. + +This template uses a git commit choosing strategy that builds the merged +commit with the release yaml file, not the tip of the target branch, so +projects can repeat the release action in case of merge job failure. + +:template name: {project-name}-packagecloud-release-merge + +:comment trigger: remerge + +:required parameters: + + :build-node: the node to run build on. + :jenkins-ssh-release-credential: credential to use for ssh. (generally set + in defaults.yaml) + :project: git repository name + :project-name: jenkins job name prefix + +:optional parameters: + + :build-days-to-keep: days to keep build logs in jenkins. (default: 7) + :build-timeout: timeout in minutes before aborting build. (default: 15) + + :gerrit_merge_triggers: override gerrit triggers. + :gerrit_trigger_file_paths: override file paths filter which checks which + file modifications will trigger a build. + **default**:: + + - compare-type: reg_exp + pattern: '(releases\/packagecloud.*\.yaml|\.releases\/packagecloud.*\.yaml)' + + +Setup for LFID, Nexus, Jenkins and Gerrit +----------------------------------------- + +This section is for the Linux Foundation release engineering team. + +LFID +~~~~ + +Create an ``lfid`` and an ``ssh-key`` + +``YOUR_RELEASE_USERNAME`` for example: onap-release + +``YOUR_RELEASE_EMAIL`` for example: collab-it+onap-release@linuxfoundation.org + +ssh-key example: + +.. code-block:: bash + + ssh-keygen -t rsa -C "collab-it+odl-release@linuxfoundation.org" -f /tmp/odl-release + + +`Create an LFID with the above values <https://identity.linuxfoundation.org>`_ + + +Nexus +~~~~~ + +Create a Nexus account called ``'jenkins-release'`` with promote privileges. + +.. image:: ../_static/nexus-promote-privs.png + +Gerrit +~~~~~~ + +Log into your Gerrit with ``YOUR_RELEASE_USERNAME``, upload the public +part of the ``ssh-key`` you created earlier. Log out of Gerrit and log +in again with your normal account for the next steps. + + +In Gerrit create a new group called ``self-serve-release`` and give it +direct push rights via ``All-Projects`` Add ``YOUR_RELEASE_USERNAME`` +to group ``self-serve-release`` and group ``Non-Interactive Users`` + + +In All project, grant group self-serve-release the following: + +.. code-block:: none + + [access "refs/heads/*"] + push = group self-serve-release + [access "refs/tags/*"] + createTag = group self-serve-release + createSignedTag = group self-serve-release + forgeCommitter = group self-serve-release + push = group self-serve-release + + +Jenkins +~~~~~~~ + +Add a global credential to Jenkins called ``jenkins-release`` and set +the ID: ``'jenkins-release'`` as its value insert the private half of +the ``ssh-key`` that you created for your Gerrit user. + +Add Global variables in Jenkins: +Jenkins configure -> Global properties -> Environment variables:: + + RELEASE_USERNAME = YOUR_RELEASE_USERNAME + RELEASE_EMAIL = YOUR_RELEASE_EMAIL + + +.. note:: + + Add these variables to your global-vars-$SILO.sh file or they will + be overwritten. + +Jenkins configure -> Managed Files -> Add a New Config -> Custom File + +.. code-block:: none + + id: signing-pubkey + Name: SIGNING_PUBKEY (optional) + Comment: SIGNING_PUBKEY (optional) + + Content: (Ask Andy for the public signing key) + -----BEGIN PGP PUBLIC KEY BLOCK----- + + +Add or edit the managed file in Jenkins called ``lftoolsini``, +appending a nexus section: Jenkins Settings -> Managed files -> Add +(or edit) -> Custom file + +.. code-block:: none + + [nexus.example.com] + username=jenkins-release + password=<plaintext password> + +Ci-management +~~~~~~~~~~~~~ + +Upgrade your project's global-jjb if needed, then add the following to +your global defaults file (e.g., jjb/defaults.yaml). + +.. code-block:: none + + jenkins-ssh-release-credential: jenkins-release