1 .. _lf-global-jjb-release:
3 Self-Serve Release Jobs
4 =======================
6 Self-serve release jobs allow project committers to promote a jar
7 file, container image, Python package or PackageCloud artifact from a
8 staging area to a release area. A release yaml file controls the
9 process, and Jenkins promotes the artifact when a project committer
10 merges the release yaml file in Gerrit.
12 To use the self-release process, create a releases/ or .releases/ directory at
13 the root of the project repository, add one release yaml file to it, and submit
14 a change set with that release yaml file. The required contents of the release
15 yaml file are different for each release, see the schemas and examples shown
16 below. The version string in the release yaml file should be a valid Semantic
17 Versioning (SemVer) string, matching the pattern "#.#.#" where "#" is one or
18 more digits. A version string matching the pattern "v#.#.#" is also accepted.
19 Upon merge of the change, a Jenkins job promotes the artifact and pushes a
20 gpg-signed tag to the repository.
24 The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml).
25 In words, the directory name can be ".releases" or "releases"; the file
26 name can be anything with suffix ".yaml". Some release jobs require
27 a specific prefix on the file, as described below.
29 The build node for all release jobs must be CentOS, which supports the
30 sigul client for accessing a signing server to sign a tag. The build
31 node for container release jobs must have Docker installed.
33 A Jenkins admin user can also trigger a release job via the "Build
34 with parameters" action, removing the need to create and merge a
35 release yaml file. The user must enter parameters in the same way as
36 a release yaml file, except for the special USE_RELEASE_FILE and
37 DRY_RUN check boxes. The user must uncheck the USE_RELEASE_FILE check
38 box if the job should run without a release file, instead passing the
39 required information as build parameters. The user can check the
40 DRY_RUN check box to test the job while skipping upload of files to
41 the release repository.
43 For example, the parameters for a Maven release are as follows::
45 GERRIT_BRANCH = master
47 LOG_DIR = example-project-maven-stage-master/17/
48 DISTRIBUTION_TYPE = maven
49 USE_RELEASE_FILE = false
52 It's recommended to use Semantic Versions (SemVer) for releases. Refer to
53 https://semver.org for more details on SemVer. For projects that do not
54 follow SemVer can use a build parameter (OVERRIDE_SEMVER_REGEX) with the
55 release job. This build param overrides the default SemVer regex.
61 An example of a maven release file appears below.
65 $ cat releases/maven-release.yaml
67 distribution_type: maven
68 log_dir: example-project-maven-stage-master/17/
69 project: example-project
73 The following parameters must appear in a maven release yaml file.
77 :distribution_type: Must be "maven".
78 :log_dir: The suffix of the logs URL reported on successful completion
79 by the Jenkins stage job that created and pushed the artifact
80 to the staging repository. For example, use value
81 "example-project-maven-stage-master/17" for the logs URL
82 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-maven-stage-master/17
83 :project: The name of the project.
84 :version: The semantic version string used for the artifact.
88 :git_tag: The tag string to sign and push to the Git repository.
89 (default: the semantic version string)
90 :tag_release: Tag Gerrit Repo during the release process.
93 The JSON schema for a maven release file appears below.
95 .. literalinclude:: ../../schema/release-schema.yaml
99 Container Release Files
100 -----------------------
102 An example of a container release file appears below.
106 $ cat releases/container-release.yaml
108 distribution_type: container
109 container_release_tag: 1.0.0
110 container_pull_registry: nexus.onap.org:10003
111 container_push_registry: nexus.onap.org:10002
113 ref: d1b9cd2dd345fbeec0d3e2162e008358b8b663b2
116 version: 1.0.0-20190806T184921Z
117 - name: test-frontend
118 version: 1.0.0-20190806T184921Z
121 The following parameters must appear in a container release yaml file.
123 :Required Parameters:
125 :distribution_type: Must be "container".
126 :container_release_tag: The string to use as a Docker tag on all
128 :container_pull_registry: The Nexus registry that supplies the staged
130 :container_push_registry: The Nexus registry that receives the released
132 :project: The name of the project.
133 :ref: The git commit reference (SHA-1 code) to tag with the version string.
134 :containers: A list of name and version (tag) pairs that specify the
135 Docker images in the container-pull registry to promote to the
136 container-push registry.
138 :Optional Parameters:
140 :git_tag: The tag string to sign and push to the Git repository.
141 (default: the semantic version string)
142 :tag_release: Tag Gerrit Repo during the release process.
145 The JSON schema for a container release file appears below.
147 .. literalinclude:: ../../schema/release-container-schema.yaml
154 An example of a PyPI release file appears below. Name of the release file must
155 start with "pypi". For example releases/pypi-1.0.0-mypackage.yaml
159 $ cat releases/pypi-1.0.0-mypackage.yaml
161 pypi_project: mypackage
162 python_version: '3.4'
164 log_dir: example-project-pypi-merge-master/17
167 The following parameters must appear in the PyPI release yaml file.
168 These are not part of the Jenkins job definition to allow independent
169 self-release of a package maintained in a git repository with other
172 :Required Parameters:
174 :log_dir: The suffix of the logs URL reported on successful completion
175 by the Jenkins merge job that created and pushed the distribution
176 files to the staging repository. For example, use value
177 "example-project-pypi-merge-master/17" for the logs URL
178 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-pypi-merge-master/17
179 :pypi_project: The PyPI project name at the staging and
180 release repositories, for example "mypackage".
181 :python_version: The Python interpreter version to use for pip
182 "Requires-Python" compatibility checks, for example '3', '3.7' or 3.7.4.
183 Put valid decimal values such as 3 or 3.7 in quotes to pass schema validation.
184 :version: The semantic version string used for the package in the
187 :Optional Parameters:
189 :git_tag: The tag string to sign and push to the Git repository.
190 (default: the semantic version string)
191 :tag_release: Tag Gerrit Repo during the release process.
194 The JSON schema for a PyPI release file appears below.
196 .. literalinclude:: ../../schema/release-pypi-schema.yaml
200 PackageCloud Release Files
201 --------------------------
203 An example of a PackageCloud release file appears below. Name of release file
204 must start with "packagecloud". For example releases/packagecloud-1.6-tree.yaml
208 $ cat releases/packagecloud-1.6-tree.yaml
212 - name: tree_1.6.0_amd64.deb
213 - name: tree-dev_1.6.0_amd64.deb
214 - name: tree-devel-1.6.0-1.x86_64.rpm
215 - name: tree-1.6.0-1.x86_64.rpm
216 ref: 5555cd2dd345fbeec0d3e2162e00835852342cda
217 log_dir: example-project-packagecloud-merge/21
220 The following parameters must appear in the PackageCloud release yaml file.
221 These are not part of the Jenkins job definition to allow independent
222 self-release of a package maintained in a git repository with other
225 :Required Parameters:
227 :package_name: Name of the release package.
228 :packages: A list of names that specify the packages to promote.
229 Found in jenkins console log when using gem to push package eg.
230 "Pushing /path/of/package/name-of-package.rpm... success!"
231 OR using rest api call to query packagecloud.io repo
232 "curl https://packagecloud.io/api/v1/repos/test_user/test_repo/search?q=
233 | yq -r .[].filename"
234 :ref: The git commit reference (SHA-1 code) to tag with the version string.
235 :log_dir: The suffix of the logs URL reported on successful completion
236 by the Jenkins merge job that created and pushed the distribution
237 files to the staging repository. For example, use value
238 "example-project-packagecloud-merge-/21" for the logs URL
239 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-packagecloud-merge/21
240 :version: The semantic version string used for the package.
242 :Optional Parameters:
244 :git_tag: The tag string to sign and push to the Git repository.
245 (default: the semantic version string)
246 :tag_release: Tag Gerrit Repo during the release process.
249 The JSON schema for a PackageCloud release file appears below.
251 .. literalinclude:: ../../schema/release-packagecloud-schema.yaml
258 Below is a list of Release job groups:
260 .. literalinclude:: ../../jjb/lf-release-job-groups.yaml
267 An example of a Jenkins job configuration that uses the global-jjb
268 templates for maven and container release jobs appears next.
273 name: my-project-release
275 project-name: my-project
276 build-node: centos7-docker-4c-4g
277 mvn-settings: my-project-settings
279 - '{project-name}-gerrit-release-jobs'
284 Release Engineers: please follow the setup guide below before adding the job definition.
293 Release verify and merge jobs are the same except for their scm,
294 trigger, and builders definition. This anchor is the common template.
302 This template supports Maven and Container release jobs.
304 This template uses a git commit choosing strategy that builds the merged
305 commit with the release yaml file, not the tip of the target branch, so
306 projects can repeat the release action in case of merge job failure.
308 :Template Name: {project-name}-release-merge-{stream}
310 :Comment Trigger: remerge
312 :Required parameters:
314 :build-node: The node to run build on.
315 :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
317 :project: Git repository name
318 :project-name: Jenkins job name prefix
320 :Optional parameters:
322 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
323 :build-timeout: Timeout in minutes before aborting build. (default: 15)
324 :stream: Keyword that represents a release code-name.
325 Often the same as the branch. (default: master)
327 :gerrit_merge_triggers: Override Gerrit Triggers.
328 :gerrit_trigger_file_paths: Override file paths filter which checks which
329 file modifications will trigger a build. The default pattern is the
330 regular expression ``(releases\/.*\.yaml|\.releases\/.*\.yaml)``
336 This template supports Maven and Container release jobs.
338 :Template Name: {project-name}-release-verify-{stream}
340 :Comment Trigger: recheck|reverify
342 :Required Parameters:
344 :build-node: The node to run build on.
345 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
347 :project: Git repository name
348 :project-name: Jenkins job name prefix
350 :Optional Parameters:
352 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
353 :build-node: The node to run build on.
354 :build-timeout: Timeout in minutes before aborting build. (default: 15)
355 :gerrit-skip-vote: Skip voting for this job. (default: false)
356 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
357 :stream: Keyword that represents a release code-name.
358 Often the same as the branch. (default: master)
360 :gerrit_verify_triggers: Override Gerrit Triggers.
361 :gerrit_trigger_file_paths: Override file paths filter which checks which
362 file modifications will trigger a build. The default pattern is the
363 regular expression ``(releases\/.*\.yaml|\.releases\/.*\.yaml)``
369 Publishes a Python package on merge of a patch set with a release yaml
370 file. Checks the format of the version string, downloads the package
371 artifacts from the PyPI staging repository, uploads the package
372 artifacts to the PyPI release repository, tags the git repository,
373 signs the tag and pushes the tag to the git server. The release merge
374 template accepts neither a branch nor a stream parameter.
376 These templates use a git commit choosing strategy that builds the merged
377 commit with the release yaml file, not the tip of the target branch, so
378 projects can repeat the release action in case of merge job failure.
382 - {project-name}-pypi-release-merge
383 - gerrit-pypi-release-merge
384 - github-pypi-release-merge
386 :Comment Trigger: remerge
388 :Required Parameters:
390 :build-node: The node to run build on, which must be Centos.
391 :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
393 :project: Git repository name
394 :project-name: Jenkins job name prefix
396 :Optional Parameters:
398 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
399 :build-timeout: Timeout in minutes before aborting build. (default: 15)
400 :disable-job: Whether to disable the job (default: false)
401 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
402 :pypi-stage-index: Base URL of the PyPI staging repository.
403 (default https://test.pypi.org/simple)
404 :pypi-repo: Key for the PyPI release repository in the .pypirc file,
405 should be the repository pypy.org. (default: pypi)
406 :use-release-file: Whether to use the release file. (default: true)
408 :gerrit_release_trigger_file_paths: Override file paths filter which checks
409 which file modifications will trigger a build. The default pattern is the
410 regular expression ``(releases\/pypi.*\.yaml|\.releases\/pypi.*\.yaml)``
415 Verifies a Python package project on creation of a patch set with a
416 release yaml file. Checks the contents of the release yaml file,
417 checks the format of the version string, and downloads the release
418 artifacts from the specified PyPI staging repository. The release
419 verify template accepts neither a branch nor a stream parameter.
423 - {project-name}-pypi-release-verify
424 - gerrit-pypi-release-verify
425 - github-pypi-release-verify
427 :Comment Trigger: recheck
429 :Required Parameters:
431 :build-node: The node to run build on, which must be Centos.
432 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
434 :project: Git repository name
435 :project-name: Jenkins job name prefix
437 :Optional Parameters:
439 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
440 :build-timeout: Timeout in minutes before aborting build. (default: 15)
441 :disable-job: Whether to disable the job (default: false)
442 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
443 :pypi-stage-index: Base URL of the PyPI staging repository.
444 (default https://test.pypi.org/simple)
445 :pypi-repo: Key for the PyPI release repository in the .pypirc file,
446 should be the repository pypy.org (default: pypi)
447 :use-release-file: Whether to use the release file. (default: true)
449 :gerrit_release_trigger_file_paths: Override file paths filter which checks
450 which file modifications will trigger a build. The default pattern is the
451 regular expression ``(releases\/pypi.*\.yaml|\.releases\/pypi.*\.yaml)``
453 PackageCloud Release Verify
454 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
456 This template supports PackageCloud release jobs. Checks that the specified
457 packages are present in the staging repository and absent from the release
458 repository. The file path trigger uses the regular expression
459 ``(releases\/packagecloud.*\.yaml|\.releases\/packagecloud.*\.yaml)``
461 :Template Name: {project-name}-packagecloud-release-verify
463 :Comment Trigger: recheck|reverify
465 :Required Parameters:
467 :build-node: The node to run build on.
468 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
470 :project: Git repository name
471 :project-name: Jenkins job name prefix
473 :Optional Parameters:
475 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
476 :build-node: The node to run build on.
477 :build-timeout: Timeout in minutes before aborting build. (default: 15)
478 :gerrit-skip-vote: Skip voting for this job. (default: false)
479 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
481 PackageCloud Release Merge
482 ~~~~~~~~~~~~~~~~~~~~~~~~~~
484 This template supports PackageCloud release jobs. Promotes the specified
485 packages from the staging repository to the release repository.
486 The file path trigger uses the regular expression
487 ``(releases\/packagecloud.*\.yaml|\.releases\/packagecloud.*\.yaml)``
489 This template uses a git commit choosing strategy that builds the merged
490 commit with the release yaml file, not the tip of the target branch, so
491 projects can repeat the release action in case of merge job failure.
493 :Template Name: {project-name}-packagecloud-release-merge
495 :Comment Trigger: remerge
497 :Required Parameters:
499 :build-node: the node to run build on.
500 :jenkins-ssh-release-credential: credential to use for ssh. (generally set
502 :project: git repository name
503 :project-name: jenkins job name prefix
505 :Optional Parameters:
507 :build-days-to-keep: days to keep build logs in jenkins. (default: 7)
508 :build-timeout: timeout in minutes before aborting build. (default: 15)
510 Setup for LFID, Nexus, Jenkins and Gerrit
511 -----------------------------------------
513 This section is for the Linux Foundation release engineering team.
518 Create an ``lfid`` and an ``ssh-key``
520 ``YOUR_RELEASE_USERNAME`` for example: onap-release
522 ``YOUR_RELEASE_EMAIL`` for example: collab-it+onap-release@linuxfoundation.org
528 ssh-keygen -t rsa -C "collab-it+odl-release@linuxfoundation.org" -f /tmp/odl-release
531 `Create an LFID with the above values <https://identity.linuxfoundation.org>`_
537 Create a Nexus account called ``'jenkins-release'`` with promote privileges.
539 .. image:: ../_static/nexus-promote-privs.png
544 Log into your Gerrit with ``YOUR_RELEASE_USERNAME``, upload the public
545 part of the ``ssh-key`` you created earlier. Log out of Gerrit and log
546 in again with your normal account for the next steps.
549 In Gerrit create a new group called ``self-serve-release`` and give it
550 direct push rights via ``All-Projects`` Add ``YOUR_RELEASE_USERNAME``
551 to group ``self-serve-release`` and group ``Non-Interactive Users``
554 In All project, grant group self-serve-release the following:
558 [access "refs/heads/*"]
559 push = group self-serve-release
560 [access "refs/tags/*"]
561 createTag = group self-serve-release
562 createSignedTag = group self-serve-release
563 forgeCommitter = group self-serve-release
564 push = group self-serve-release
570 Add a global credential to Jenkins called ``jenkins-release`` and set
571 the ID: ``'jenkins-release'`` as its value insert the private half of
572 the ``ssh-key`` that you created for your Gerrit user.
574 Add Global variables in Jenkins:
575 Jenkins configure -> Global properties -> Environment variables::
577 RELEASE_USERNAME = YOUR_RELEASE_USERNAME
578 RELEASE_EMAIL = YOUR_RELEASE_EMAIL
583 Add these variables to your global-vars-$SILO.sh file or they will
586 Jenkins configure -> Managed Files -> Add a New Config -> Custom File
591 Name: SIGNING_PUBKEY (optional)
592 Comment: SIGNING_PUBKEY (optional)
594 Content: (Ask Andy for the public signing key)
595 -----BEGIN PGP PUBLIC KEY BLOCK-----
598 Add or edit the managed file in Jenkins called ``lftoolsini``,
599 appending a nexus section: Jenkins Settings -> Managed files -> Add
600 (or edit) -> Custom file
605 username=jenkins-release
606 password=<plaintext password>
611 Upgrade your project's global-jjb if needed, then add the following to
612 your global defaults file (e.g., jjb/defaults.yaml).
616 jenkins-ssh-release-credential: jenkins-release