1 .. _lf-global-jjb-release:
3 Self-Serve Release Jobs
4 =======================
6 Self-serve release jobs allow project committers to promote a jar
7 file, container image, Python package or PackageCloud artifact from a
8 staging area to a release area. A release yaml file controls the
9 process, and Jenkins promotes the artifact when a project committer
10 merges the release yaml file in Gerrit.
12 To use the self-release process, create a releases/ or .releases/
13 directory at the root of the project repository, add one release yaml
14 file to it, and submit a change set with that release yaml file. The
15 required contents of the release yaml file are different for each type
16 of release, see the schemas and examples shown below. The version
17 string in the release yaml file should be a valid Semantic Versioning
18 (SemVer) string, matching the pattern "#.#.#" where "#" is one or more
19 digits. A version string matching the pattern "v#.#.#" is also
20 accepted. Upon merge of the change, a Jenkins job promotes the
21 artifact and pushes a gpg-signed tag to the repository.
25 The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml).
26 In words, the directory name can be ".releases" or "releases"; the file
27 name can be anything with suffix ".yaml". Some release jobs require
28 a specific prefix on the file, as described below.
30 The build node for all release jobs must be CentOS, which supports the
31 sigul client for accessing a signing server to sign a tag. The build
32 node for container release jobs must have Docker installed.
34 A Jenkins admin user can also trigger a release job via the "Build
35 with parameters" action, removing the need to create and merge a
36 release yaml file. The user must enter parameters in the same way as
37 a release yaml file, except for the special USE_RELEASE_FILE and
38 DRY_RUN check boxes. The user must uncheck the USE_RELEASE_FILE check
39 box if the job should run without a release file, instead passing the
40 required information as build parameters. The user can check the
41 DRY_RUN check box to test the job while skipping upload of files to
42 the release repository.
44 For example, the parameters for a Maven release are as follows::
46 GERRIT_BRANCH = master
48 LOG_DIR = example-project-maven-stage-master/17/
49 DISTRIBUTION_TYPE = maven
50 USE_RELEASE_FILE = false
53 It's recommended to use Semantic Versions (SemVer) for releases. Refer to
54 https://semver.org for more details on SemVer. For projects that do not
55 follow SemVer can use a build parameter (OVERRIDE_SEMVER_REGEX) with the
56 release job. This build param overrides the default SemVer regex.
62 An example of a maven release file appears below.
66 $ cat releases/maven-release.yaml
68 distribution_type: maven
69 log_dir: example-project-maven-stage-master/17/
70 project: example-project
74 The following parameters must appear in a maven release yaml file.
78 :distribution_type: Must be "maven".
79 :log_dir: The suffix of the logs URL reported on successful completion
80 by the Jenkins stage job that created and pushed the artifact
81 to the staging repository. For example, use value
82 "example-project-maven-stage-master/17" for the logs URL
83 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-maven-stage-master/17
84 :project: The name of the project.
85 :version: The semantic version string used for the artifact.
89 :git_tag: The tag string to sign and push to the Git repository.
90 (default: the semantic version string)
91 :tag_release: Tag Gerrit Repo during the release process.
94 The JSON schema for a maven release file appears below.
96 .. literalinclude:: ../../schema/release-schema.yaml
100 Container Release Files
101 -----------------------
103 An example of a container release file appears below.
107 $ cat releases/container-release.yaml
109 distribution_type: container
110 container_release_tag: 1.0.0
111 container_pull_registry: nexus.onap.org:10003
112 container_push_registry: nexus.onap.org:10002
114 ref: d1b9cd2dd345fbeec0d3e2162e008358b8b663b2
117 version: 1.0.0-20190806T184921Z
118 - name: test-frontend
119 version: 1.0.0-20190806T184921Z
122 The following parameters must appear in a container release yaml file.
124 :Required Parameters:
126 :distribution_type: Must be "container".
127 :container_release_tag: The string to use as a Docker tag on all
129 :container_pull_registry: The Nexus registry that supplies the staged
131 :container_push_registry: The Nexus registry that receives the released
133 :project: The name of the project.
134 :ref: The git commit reference (SHA-1 code) to tag with the version string.
135 :containers: A list of name and version (tag) pairs that specify the
136 Docker images in the container-pull registry to promote to the
137 container-push registry.
139 :Optional Parameters:
141 :git_tag: The tag string to sign and push to the Git repository.
142 (default: the semantic version string)
143 :tag_release: Tag Gerrit Repo during the release process.
146 The JSON schema for a container release file appears below.
148 .. literalinclude:: ../../schema/release-container-schema.yaml
155 An example of a PyPI release file appears below. Name of the release file must
156 start with "pypi". For example releases/pypi-1.0.0-mypackage.yaml
160 $ cat releases/pypi-1.0.0-mypackage.yaml
162 pypi_project: mypackage
163 python_version: '3.4'
165 log_dir: example-project-pypi-merge-master/17
168 The following parameters must appear in the PyPI release yaml file.
169 These are not part of the Jenkins job definition to allow independent
170 self-release of a package maintained in a git repository with other
173 :Required Parameters:
175 :log_dir: The suffix of the logs URL reported on successful completion
176 by the Jenkins merge job that created and pushed the distribution
177 files to the staging repository. For example, use value
178 "example-project-pypi-merge-master/17" for the logs URL
179 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-pypi-merge-master/17
180 :pypi_project: The PyPI project name at the staging and
181 release repositories, for example "mypackage".
182 :python_version: The Python interpreter version to use for pip
183 "Requires-Python" compatibility checks, for example '3', '3.7' or 3.7.4.
184 Put valid decimal values such as 3 or 3.7 in quotes to pass schema validation.
185 :version: The semantic version string used for the package in the
188 :Optional Parameters:
190 :git_tag: The tag string to sign and push to the Git repository.
191 (default: the semantic version string)
192 :tag_release: Tag Gerrit Repo during the release process.
195 The JSON schema for a PyPI release file appears below.
197 .. literalinclude:: ../../schema/release-pypi-schema.yaml
201 PackageCloud Release Files
202 --------------------------
204 An example of a PackageCloud release file appears below. Name of release file
205 must start with "packagecloud". For example releases/packagecloud-1.6-tree.yaml
209 $ cat releases/packagecloud-1.6-tree.yaml
213 - name: tree_1.6.0_amd64.deb
214 - name: tree-dev_1.6.0_amd64.deb
215 - name: tree-devel-1.6.0-1.x86_64.rpm
216 - name: tree-1.6.0-1.x86_64.rpm
217 ref: 5555cd2dd345fbeec0d3e2162e00835852342cda
218 log_dir: example-project-packagecloud-merge/21
221 The following parameters must appear in the PackageCloud release yaml file.
222 These are not part of the Jenkins job definition to allow independent
223 self-release of a package maintained in a git repository with other
226 :Required Parameters:
228 :package_name: Name of the release package.
229 :packages: A list of names that specify the packages to promote.
230 Found in jenkins console log when using gem to push package eg.
231 "Pushing /path/of/package/name-of-package.rpm... success!"
232 OR using rest api call to query packagecloud.io repo
233 "curl https://packagecloud.io/api/v1/repos/test_user/test_repo/search?q=
234 | yq -r .[].filename"
235 :ref: The git commit reference (SHA-1 code) to tag with the version string.
236 :log_dir: The suffix of the logs URL reported on successful completion
237 by the Jenkins merge job that created and pushed the distribution
238 files to the staging repository. For example, use value
239 "example-project-packagecloud-merge-/21" for the logs URL
240 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-packagecloud-merge/21
241 :version: The semantic version string used for the package.
243 :Optional Parameters:
245 :git_tag: The tag string to sign and push to the Git repository.
246 (default: the semantic version string)
247 :tag_release: Tag Gerrit Repo during the release process.
250 The JSON schema for a PackageCloud release file appears below.
252 .. literalinclude:: ../../schema/release-packagecloud-schema.yaml
259 Below is a list of Release job groups:
261 .. literalinclude:: ../../jjb/lf-release-job-groups.yaml
268 An example of a Jenkins job configuration that uses the global-jjb
269 templates for maven and container release jobs appears next.
274 name: my-project-release
276 project-name: my-project
277 build-node: centos7-docker-4c-4g
278 mvn-settings: my-project-settings
280 - '{project-name}-gerrit-release-jobs'
285 Release Engineers: please follow the setup guide below before adding the job definition.
294 Release verify and merge jobs are the same except for their scm,
295 trigger, and builders definition. This anchor is the common template.
303 This template supports Maven and Container release jobs.
305 This template uses a git commit choosing strategy that builds the merged
306 commit with the release yaml file, not the tip of the target branch, so
307 projects can repeat the release action in case of merge job failure.
309 :Template Name: {project-name}-release-merge
311 :Comment Trigger: remerge
313 :Required parameters:
315 :build-node: The node to run build on.
316 :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
318 :project: Git repository name
319 :project-name: Jenkins job name prefix
321 :Optional parameters:
323 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
324 :build-timeout: Timeout in minutes before aborting build. (default: 15)
326 :gerrit_merge_triggers: Override Gerrit Triggers.
327 :gerrit_trigger_file_paths: Override file paths filter which checks which
328 file modifications will trigger a build. The default pattern is the
329 regular expression ``(releases\/.*\.yaml|\.releases\/.*\.yaml)``
335 This template supports Maven and Container release jobs.
337 :Template Name: {project-name}-release-verify
339 :Comment Trigger: recheck|reverify
341 :Required Parameters:
343 :build-node: The node to run build on.
344 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
346 :project: Git repository name
347 :project-name: Jenkins job name prefix
349 :Optional Parameters:
351 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
352 :build-node: The node to run build on.
353 :build-timeout: Timeout in minutes before aborting build. (default: 15)
354 :gerrit-skip-vote: Skip voting for this job. (default: false)
355 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
357 :gerrit_verify_triggers: Override Gerrit Triggers.
358 :gerrit_trigger_file_paths: Override file paths filter which checks which
359 file modifications will trigger a build. The default pattern is the
360 regular expression ``(releases\/.*\.yaml|\.releases\/.*\.yaml)``
366 Publishes a Python package on merge of a patch set with a release yaml
367 file. Checks the format of the version string, downloads the package
368 artifacts from the PyPI staging repository, uploads the package
369 artifacts to the PyPI release repository, tags the git repository,
370 signs the tag and pushes the tag to the git server. The release merge
371 template accepts neither a branch nor a stream parameter.
373 These templates use a git commit choosing strategy that builds the merged
374 commit with the release yaml file, not the tip of the target branch, so
375 projects can repeat the release action in case of merge job failure.
379 - {project-name}-pypi-release-merge
380 - gerrit-pypi-release-merge
381 - github-pypi-release-merge
383 :Comment Trigger: remerge
385 :Required Parameters:
387 :build-node: The node to run build on, which must be Centos.
388 :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
390 :project: Git repository name
391 :project-name: Jenkins job name prefix
393 :Optional Parameters:
395 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
396 :build-timeout: Timeout in minutes before aborting build. (default: 15)
397 :disable-job: Whether to disable the job (default: false)
398 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
399 :pypi-stage-index: Base URL of the PyPI staging repository.
400 (default https://test.pypi.org/simple)
401 :pypi-repo: Key for the PyPI release repository in the .pypirc file,
402 should be the repository pypy.org. (default: pypi)
403 :use-release-file: Whether to use the release file. (default: true)
405 :gerrit_release_trigger_file_paths: Override file paths filter which checks
406 which file modifications will trigger a build. The default pattern is the
407 regular expression ``(releases\/pypi.*\.yaml|\.releases\/pypi.*\.yaml)``
412 Verifies a Python package project on creation of a patch set with a
413 release yaml file. Checks the contents of the release yaml file,
414 checks the format of the version string, and downloads the release
415 artifacts from the specified PyPI staging repository. The release
416 verify template accepts neither a branch nor a stream parameter.
420 - {project-name}-pypi-release-verify
421 - gerrit-pypi-release-verify
422 - github-pypi-release-verify
424 :Comment Trigger: recheck
426 :Required Parameters:
428 :build-node: The node to run build on, which must be Centos.
429 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
431 :project: Git repository name
432 :project-name: Jenkins job name prefix
434 :Optional Parameters:
436 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
437 :build-timeout: Timeout in minutes before aborting build. (default: 15)
438 :disable-job: Whether to disable the job (default: false)
439 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
440 :pypi-stage-index: Base URL of the PyPI staging repository.
441 (default https://test.pypi.org/simple)
442 :pypi-repo: Key for the PyPI release repository in the .pypirc file,
443 should be the repository pypy.org (default: pypi)
444 :use-release-file: Whether to use the release file. (default: true)
446 :gerrit_release_trigger_file_paths: Override file paths filter which checks
447 which file modifications will trigger a build. The default pattern is the
448 regular expression ``(releases\/pypi.*\.yaml|\.releases\/pypi.*\.yaml)``
450 PackageCloud Release Verify
451 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
453 This template supports PackageCloud release jobs. Checks that the specified
454 packages are present in the staging repository and absent from the release
455 repository. The file path trigger uses the regular expression
456 ``(releases\/packagecloud.*\.yaml|\.releases\/packagecloud.*\.yaml)``
458 :Template Name: {project-name}-packagecloud-release-verify
460 :Comment Trigger: recheck|reverify
462 :Required Parameters:
464 :build-node: The node to run build on.
465 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
467 :project: Git repository name
468 :project-name: Jenkins job name prefix
470 :Optional Parameters:
472 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
473 :build-node: The node to run build on.
474 :build-timeout: Timeout in minutes before aborting build. (default: 15)
475 :gerrit-skip-vote: Skip voting for this job. (default: false)
476 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
478 PackageCloud Release Merge
479 ~~~~~~~~~~~~~~~~~~~~~~~~~~
481 This template supports PackageCloud release jobs. Promotes the specified
482 packages from the staging repository to the release repository.
483 The file path trigger uses the regular expression
484 ``(releases\/packagecloud.*\.yaml|\.releases\/packagecloud.*\.yaml)``
486 This template uses a git commit choosing strategy that builds the merged
487 commit with the release yaml file, not the tip of the target branch, so
488 projects can repeat the release action in case of merge job failure.
490 :Template Name: {project-name}-packagecloud-release-merge
492 :Comment Trigger: remerge
494 :Required Parameters:
496 :build-node: the node to run build on.
497 :jenkins-ssh-release-credential: credential to use for ssh. (generally set
499 :project: git repository name
500 :project-name: jenkins job name prefix
502 :Optional Parameters:
504 :build-days-to-keep: days to keep build logs in jenkins. (default: 7)
505 :build-timeout: timeout in minutes before aborting build. (default: 15)
507 Setup for LFID, Nexus, Jenkins and Gerrit
508 -----------------------------------------
510 This section is for the Linux Foundation release engineering team.
515 Create an ``lfid`` and an ``ssh-key``
517 ``YOUR_RELEASE_USERNAME`` for example: onap-release
519 ``YOUR_RELEASE_EMAIL`` for example: collab-it+onap-release@linuxfoundation.org
525 ssh-keygen -t rsa -C "collab-it+odl-release@linuxfoundation.org" -f /tmp/odl-release
528 `Create an LFID with the above values <https://identity.linuxfoundation.org>`_
534 Create a Nexus account called ``'jenkins-release'`` with promote privileges.
536 .. image:: ../_static/nexus-promote-privs.png
541 Log into your Gerrit with ``YOUR_RELEASE_USERNAME``, upload the public
542 part of the ``ssh-key`` you created earlier. Log out of Gerrit and log
543 in again with your normal account for the next steps.
546 In Gerrit create a new group called ``self-serve-release`` and give it
547 direct push rights via ``All-Projects`` Add ``YOUR_RELEASE_USERNAME``
548 to group ``self-serve-release`` and group ``Non-Interactive Users``
551 In All project, grant group self-serve-release the following:
555 [access "refs/heads/*"]
556 push = group self-serve-release
557 [access "refs/tags/*"]
558 createTag = group self-serve-release
559 createSignedTag = group self-serve-release
560 forgeCommitter = group self-serve-release
561 push = group self-serve-release
567 Add a global credential to Jenkins called ``jenkins-release`` and set
568 the ID: ``'jenkins-release'`` as its value insert the private half of
569 the ``ssh-key`` that you created for your Gerrit user.
571 Add Global variables in Jenkins:
572 Jenkins configure -> Global properties -> Environment variables::
574 RELEASE_USERNAME = YOUR_RELEASE_USERNAME
575 RELEASE_EMAIL = YOUR_RELEASE_EMAIL
580 Add these variables to your global-vars-$SILO.sh file or they will
583 Jenkins configure -> Managed Files -> Add a New Config -> Custom File
588 Name: SIGNING_PUBKEY (optional)
589 Comment: SIGNING_PUBKEY (optional)
591 Content: (Ask Andy for the public signing key)
592 -----BEGIN PGP PUBLIC KEY BLOCK-----
595 Add or edit the managed file in Jenkins called ``lftoolsini``,
596 appending a nexus section: Jenkins Settings -> Managed files -> Add
597 (or edit) -> Custom file
602 username=jenkins-release
603 password=<plaintext password>
608 Upgrade your project's global-jjb if needed, then add the following to
609 your global defaults file (e.g., jjb/defaults.yaml).
613 jenkins-ssh-release-credential: jenkins-release