1 .. _lf-global-jjb-release:
3 Self-Serve Release Jobs
4 =======================
6 Self-serve release jobs allow project committers to promote a jar
7 file, container image, Python package or PackageCloud artifact from a
8 staging area to a release area. A release yaml file controls the
9 process, and Jenkins promotes the artifact when a project committer
10 merges the release yaml file in Gerrit.
12 To use the self-release process, create a releases/ or .releases/
13 directory at the root of the project repository, add one release yaml
14 file to it, and submit a change set with that release yaml file. The
15 required contents of the release yaml file are different for each type
16 of release, see the schemas and examples shown below. The version
17 string in the release yaml file should be a valid Semantic Versioning
18 (SemVer) string, matching the pattern "#.#.#" where "#" is one or more
19 digits. A version string matching the pattern "v#.#.#" is also
20 accepted. Upon merge of the change, a Jenkins job promotes the
21 artifact and pushes a gpg-signed tag to the repository.
25 The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml).
26 In words, the directory name can be ".releases" or "releases"; the file
27 name can be anything with suffix ".yaml". Some release jobs require
28 a specific prefix on the file, as described below.
30 The build node for all release jobs must be CentOS, which supports the
31 sigul client for accessing a signing server to sign a tag. The build
32 node for container release jobs must have Docker installed.
34 A Jenkins admin user can also trigger a release job via the "Build
35 with parameters" action, removing the need to create and merge a
36 release yaml file. The user must enter parameters in the same way as
37 a release yaml file, except for the special USE_RELEASE_FILE and
38 DRY_RUN check boxes. The user must uncheck the USE_RELEASE_FILE check
39 box if the job should run without a release file, instead passing the
40 required information as build parameters. The user can check the
41 DRY_RUN check box to test the job while skipping upload of files to
42 the release repository.
44 For example, the parameters for a Maven release are as follows::
46 GERRIT_BRANCH = master
48 LOG_DIR = example-project-maven-stage-master/17/
49 DISTRIBUTION_TYPE = maven
50 USE_RELEASE_FILE = false
53 It's recommended to use Semantic Versions (SemVer) for releases. Refer to
54 https://semver.org for more details on SemVer. For projects that do not
55 follow SemVer can use a build parameter (OVERRIDE_SEMVER_REGEX) with the
56 release job. This build param overrides the default SemVer regex.
62 An example of a maven release file appears below.
66 $ cat releases/maven-release.yaml
68 distribution_type: maven
69 log_dir: example-project-maven-stage-master/17/
70 project: example-project
74 The following parameters must appear in a maven release yaml file.
78 :distribution_type: Must be "maven".
79 :log_dir: The suffix of the logs URL reported on successful completion
80 by the Jenkins stage job that created and pushed the artifact
81 to the staging repository. For example, use value
82 "example-project-maven-stage-master/17" for the logs URL
83 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-maven-stage-master/17
84 :project: The name of the project.
85 :version: The semantic version string used for the artifact.
89 :git_tag: The tag string to sign and push to the Git repository.
90 (default: the semantic version string)
91 :tag_release: Tag Gerrit Repo during the release process.
94 The JSON schema for a maven release file appears below.
96 .. literalinclude:: ../../schema/release-schema.yaml
100 Container Release Files
101 -----------------------
103 An example of a container release file appears below.
107 $ cat releases/container-release.yaml
109 distribution_type: container
110 container_release_tag: 1.0.0
111 container_pull_registry: nexus.onap.org:10003
112 container_push_registry: nexus.onap.org:10002
114 ref: d1b9cd2dd345fbeec0d3e2162e008358b8b663b2
117 version: 1.0.0-20190806T184921Z
118 - name: test-frontend
119 version: 1.0.0-20190806T184921Z
122 The following parameters must appear in a container release yaml file.
124 :Required Parameters:
126 :distribution_type: Must be "container".
127 :container_release_tag: The string to use as a Docker tag on all
129 :container_pull_registry: The Nexus registry that supplies the staged
131 :container_push_registry: The Nexus registry that receives the released
133 :project: The name of the project.
134 :ref: The git commit reference (SHA-1 code) to tag with the version string.
135 :containers: A list of name and version (tag) pairs that specify the
136 Docker images in the container-pull registry to promote to the
137 container-push registry.
139 :Optional Parameters:
141 :git_tag: The tag string to sign and push to the Git repository.
142 (default: the semantic version string)
143 :tag_release: Tag Gerrit Repo during the release process.
146 The JSON schema for a container release file appears below.
148 .. literalinclude:: ../../schema/release-container-schema.yaml
155 An example of a PyPI release file appears below. Name of the release file must
156 start with "pypi". For example releases/pypi-1.0.0-mypackage.yaml
160 $ cat releases/pypi-1.0.0-mypackage.yaml
162 pypi_project: mypackage
163 python_version: '3.4'
165 log_dir: example-project-pypi-merge-master/17
168 The following parameters must appear in the PyPI release yaml file.
169 These are not part of the Jenkins job definition to allow independent
170 self-release of a package maintained in a git repository with other
173 :Required Parameters:
175 :log_dir: The suffix of the logs URL reported on successful completion
176 by the Jenkins merge job that created and pushed the distribution
177 files to the staging repository. For example, use value
178 "example-project-pypi-merge-master/17" for the logs URL
179 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-pypi-merge-master/17
180 :pypi_project: The PyPI project name at the staging and
181 release repositories, for example "mypackage".
182 :python_version: The Python interpreter version to use for pip
183 "Requires-Python" compatibility checks, for example '3', '3.7' or 3.7.4.
184 Put valid decimal values such as 3 or 3.7 in quotes to pass schema validation.
185 :version: The semantic version string used for the package in the
188 :Optional Parameters:
190 :git_tag: The tag string to sign and push to the Git repository.
191 (default: the semantic version string)
192 :tag_release: Tag Gerrit Repo during the release process.
195 The JSON schema for a PyPI release file appears below.
197 .. literalinclude:: ../../schema/release-pypi-schema.yaml
201 PackageCloud Release Files
202 --------------------------
204 An example of a PackageCloud release file appears below. Name of release file
205 must start with "packagecloud". For example releases/packagecloud-1.6-tree.yaml
209 $ cat releases/packagecloud-1.6-tree.yaml
213 - name: tree_1.6.0_amd64.deb
214 - name: tree-dev_1.6.0_amd64.deb
215 - name: tree-devel-1.6.0-1.x86_64.rpm
216 - name: tree-1.6.0-1.x86_64.rpm
217 ref: 5555cd2dd345fbeec0d3e2162e00835852342cda
218 log_dir: example-project-packagecloud-merge/21
221 The following parameters must appear in the PackageCloud release yaml file.
222 These are not part of the Jenkins job definition to allow independent
223 self-release of a package maintained in a git repository with other
226 :Required Parameters:
228 :package_name: Name of the release package.
229 :packages: A list of names that specify the packages to promote.
230 Found in jenkins console log when using gem to push package eg.
231 "Pushing /path/of/package/name-of-package.rpm... success!"
232 OR using rest api call to query packagecloud.io repo
233 "curl https://packagecloud.io/api/v1/repos/test_user/test_repo/search?q=
234 | yq -r .[].filename"
235 :ref: The git commit reference (SHA-1 code) to tag with the version string.
236 :log_dir: The suffix of the logs URL reported on successful completion
237 by the Jenkins merge job that created and pushed the distribution
238 files to the staging repository. For example, use value
239 "example-project-packagecloud-merge-/21" for the logs URL
240 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-packagecloud-merge/21
241 :version: The semantic version string used for the package.
243 :Optional Parameters:
245 :git_tag: The tag string to sign and push to the Git repository.
246 (default: the semantic version string)
247 :tag_release: Tag Gerrit Repo during the release process.
250 The JSON schema for a PackageCloud release file appears below.
252 .. literalinclude:: ../../schema/release-packagecloud-schema.yaml
259 An example of a Jenkins job configuration that uses the global-jjb
260 templates for maven and container release jobs appears next.
265 name: my-project-release
267 project-name: my-project
268 build-node: centos7-docker-4c-4g
269 mvn-settings: my-project-settings
271 - '{project-name}-gerrit-release-jobs'
276 Release Engineers: please follow the setup guide below before adding the job definition.
285 Release verify and merge jobs are the same except for their scm,
286 trigger, and builders definition. This anchor is the common template.
294 This template supports Maven and Container release jobs.
296 This template uses a git commit choosing strategy that builds the merged
297 commit with the release yaml file, not the tip of the target branch, so
298 projects can repeat the release action in case of merge job failure.
300 :Template Name: {project-name}-release-merge
302 :Comment Trigger: remerge
304 :Required parameters:
306 :build-node: The node to run build on.
307 :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
309 :project: Git repository name
310 :project-name: Jenkins job name prefix
312 :Optional parameters:
314 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
315 :build-timeout: Timeout in minutes before aborting build. (default: 15)
317 :gerrit_merge_triggers: Override Gerrit Triggers.
318 :gerrit_trigger_file_paths: Override file paths filter which checks which
319 file modifications will trigger a build. The default pattern is the
320 regular expression ``(releases\/.*\.yaml|\.releases\/.*\.yaml)``
326 This template supports Maven and Container release jobs.
328 :Template Name: {project-name}-release-verify
330 :Comment Trigger: recheck|reverify
332 :Required Parameters:
334 :build-node: The node to run build on.
335 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
337 :project: Git repository name
338 :project-name: Jenkins job name prefix
340 :Optional Parameters:
342 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
343 :build-node: The node to run build on.
344 :build-timeout: Timeout in minutes before aborting build. (default: 15)
345 :gerrit-skip-vote: Skip voting for this job. (default: false)
346 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
348 :gerrit_verify_triggers: Override Gerrit Triggers.
349 :gerrit_trigger_file_paths: Override file paths filter which checks which
350 file modifications will trigger a build. The default pattern is the
351 regular expression ``(releases\/.*\.yaml|\.releases\/.*\.yaml)``
357 Publishes a Python package on merge of a patch set with a release yaml
358 file. Checks the format of the version string, downloads the package
359 artifacts from the PyPI staging repository, uploads the package
360 artifacts to the PyPI release repository, tags the git repository,
361 signs the tag and pushes the tag to the git server. The release merge
362 template accepts neither a branch nor a stream parameter.
364 These templates use a git commit choosing strategy that builds the merged
365 commit with the release yaml file, not the tip of the target branch, so
366 projects can repeat the release action in case of merge job failure.
370 - {project-name}-pypi-release-merge
371 - gerrit-pypi-release-merge
372 - github-pypi-release-merge
374 :Comment Trigger: remerge
376 :Required Parameters:
378 :build-node: The node to run build on, which must be Centos.
379 :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
381 :project: Git repository name
382 :project-name: Jenkins job name prefix
384 :Optional Parameters:
386 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
387 :build-timeout: Timeout in minutes before aborting build. (default: 15)
388 :disable-job: Whether to disable the job (default: false)
389 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
390 :pypi-stage-index: Base URL of the PyPI staging repository.
391 (default https://test.pypi.org/simple)
392 :pypi-repo: Key for the PyPI release repository in the .pypirc file,
393 should be the repository pypy.org. (default: pypi)
394 :use-release-file: Whether to use the release file. (default: true)
396 :gerrit_release_trigger_file_paths: Override file paths filter which checks
397 which file modifications will trigger a build. The default pattern is the
398 regular expression ``(releases\/pypi.*\.yaml|\.releases\/pypi.*\.yaml)``
403 Verifies a Python package project on creation of a patch set with a
404 release yaml file. Checks the contents of the release yaml file,
405 checks the format of the version string, and downloads the release
406 artifacts from the specified PyPI staging repository. The release
407 verify template accepts neither a branch nor a stream parameter.
411 - {project-name}-pypi-release-verify
412 - gerrit-pypi-release-verify
413 - github-pypi-release-verify
415 :Comment Trigger: recheck
417 :Required Parameters:
419 :build-node: The node to run build on, which must be Centos.
420 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
422 :project: Git repository name
423 :project-name: Jenkins job name prefix
425 :Optional Parameters:
427 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
428 :build-timeout: Timeout in minutes before aborting build. (default: 15)
429 :disable-job: Whether to disable the job (default: false)
430 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
431 :pypi-stage-index: Base URL of the PyPI staging repository.
432 (default https://test.pypi.org/simple)
433 :pypi-repo: Key for the PyPI release repository in the .pypirc file,
434 should be the repository pypy.org (default: pypi)
435 :use-release-file: Whether to use the release file. (default: true)
437 :gerrit_release_trigger_file_paths: Override file paths filter which checks
438 which file modifications will trigger a build. The default pattern is the
439 regular expression ``(releases\/pypi.*\.yaml|\.releases\/pypi.*\.yaml)``
441 PackageCloud Release Verify
442 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
444 This template supports PackageCloud release jobs. Checks that the specified
445 packages are present in the staging repository and absent from the release
446 repository. The file path trigger uses the regular expression
447 ``(releases\/packagecloud.*\.yaml|\.releases\/packagecloud.*\.yaml)``
449 :Template Name: {project-name}-packagecloud-release-verify
451 :Comment Trigger: recheck|reverify
453 :Required Parameters:
455 :build-node: The node to run build on.
456 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
458 :project: Git repository name
459 :project-name: Jenkins job name prefix
461 :Optional Parameters:
463 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
464 :build-node: The node to run build on.
465 :build-timeout: Timeout in minutes before aborting build. (default: 15)
466 :gerrit-skip-vote: Skip voting for this job. (default: false)
467 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
469 PackageCloud Release Merge
470 ~~~~~~~~~~~~~~~~~~~~~~~~~~
472 This template supports PackageCloud release jobs. Promotes the specified
473 packages from the staging repository to the release repository.
474 The file path trigger uses the regular expression
475 ``(releases\/packagecloud.*\.yaml|\.releases\/packagecloud.*\.yaml)``
477 This template uses a git commit choosing strategy that builds the merged
478 commit with the release yaml file, not the tip of the target branch, so
479 projects can repeat the release action in case of merge job failure.
481 :Template Name: {project-name}-packagecloud-release-merge
483 :Comment Trigger: remerge
485 :Required Parameters:
487 :build-node: the node to run build on.
488 :jenkins-ssh-release-credential: credential to use for ssh. (generally set
490 :project: git repository name
491 :project-name: jenkins job name prefix
493 :Optional Parameters:
495 :build-days-to-keep: days to keep build logs in jenkins. (default: 7)
496 :build-timeout: timeout in minutes before aborting build. (default: 15)
498 Setup for LFID, Nexus, Jenkins and Gerrit
499 -----------------------------------------
501 This section is for the Linux Foundation release engineering team.
506 Create an ``lfid`` and an ``ssh-key``
508 ``YOUR_RELEASE_USERNAME`` for example: onap-release
510 ``YOUR_RELEASE_EMAIL`` for example: collab-it+onap-release@linuxfoundation.org
516 ssh-keygen -t rsa -C "collab-it+odl-release@linuxfoundation.org" -f /tmp/odl-release
519 `Create an LFID with the above values <https://identity.linuxfoundation.org>`_
525 Create a Nexus account called ``'jenkins-release'`` with promote privileges.
527 .. image:: ../_static/nexus-promote-privs.png
532 Log into your Gerrit with ``YOUR_RELEASE_USERNAME``, upload the public
533 part of the ``ssh-key`` you created earlier. Log out of Gerrit and log
534 in again with your normal account for the next steps.
537 In Gerrit create a new group called ``self-serve-release`` and give it
538 direct push rights via ``All-Projects`` Add ``YOUR_RELEASE_USERNAME``
539 to group ``self-serve-release`` and group ``Non-Interactive Users``
542 In All project, grant group self-serve-release the following:
546 [access "refs/heads/*"]
547 push = group self-serve-release
548 [access "refs/tags/*"]
549 createTag = group self-serve-release
550 createSignedTag = group self-serve-release
551 forgeCommitter = group self-serve-release
552 push = group self-serve-release
558 Add a global credential to Jenkins called ``jenkins-release`` and set
559 the ID: ``'jenkins-release'`` as its value insert the private half of
560 the ``ssh-key`` that you created for your Gerrit user.
562 Add Global variables in Jenkins:
563 Jenkins configure -> Global properties -> Environment variables::
565 RELEASE_USERNAME = YOUR_RELEASE_USERNAME
566 RELEASE_EMAIL = YOUR_RELEASE_EMAIL
571 Add these variables to your global-vars-$SILO.sh file or they will
574 Jenkins configure -> Managed Files -> Add a New Config -> Custom File
579 Name: SIGNING_PUBKEY (optional)
580 Comment: SIGNING_PUBKEY (optional)
582 Content: (Ask Andy for the public signing key)
583 -----BEGIN PGP PUBLIC KEY BLOCK-----
586 Add or edit the managed file in Jenkins called ``lftoolsini``,
587 appending a nexus section: Jenkins Settings -> Managed files -> Add
588 (or edit) -> Custom file
593 username=jenkins-release
594 password=<plaintext password>
599 Upgrade your project's global-jjb if needed, then add the following to
600 your global defaults file (e.g., jjb/defaults.yaml).
604 jenkins-ssh-release-credential: jenkins-release