1 .. _lf-global-jjb-release:
3 Self-Serve Release Jobs
4 =======================
6 Self-serve release jobs allow project committers to promote a jar
7 file, container image, Python package or PackageCloud artifact from a
8 staging area to a release area. A release yaml file controls the
9 process, and Jenkins promotes the artifact when a project committer
10 merges the release yaml file in Gerrit.
12 To use the self-release process, create a releases/ or .releases/
13 directory at the root of the project repository, add one release yaml
14 file to it, and submit a change set with that release yaml file. The
15 required contents of the release yaml file are different for each type
16 of release, see the schemas and examples shown below. The version
17 string in the release yaml file should be a valid Semantic Versioning
18 (SemVer) string, matching the pattern "#.#.#" where "#" is one or more
19 digits. A version string matching the pattern "v#.#.#" is also
20 accepted. Upon merge of the change, a Jenkins job promotes the
21 artifact and pushes a gpg-signed tag to the repository.
25 The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml).
26 In words, the directory name can be ".releases" or "releases"; the file
27 name can be anything with suffix ".yaml". Some release jobs require
28 a specific prefix on the file, as described below.
30 The build node for all release jobs must be CentOS, which supports the
31 sigul client for accessing a signing server to sign a tag. The build
32 node for container release jobs must have Docker installed.
34 A Jenkins admin user can also trigger a release job via the "Build
35 with parameters" action, removing the need to create and merge a
36 release yaml file. The user must enter parameters in the same way as
37 a release yaml file, except for the special USE_RELEASE_FILE and
38 DRY_RUN check boxes. The user must uncheck the USE_RELEASE_FILE check
39 box if the job should run without a release file, instead passing the
40 required information as build parameters. The user can check the
41 DRY_RUN check box to test the job while skipping upload of files to
42 the release repository.
44 For example, the parameters for a Maven release are as follows::
46 GERRIT_BRANCH = master
48 LOG_DIR = example-project-maven-stage-master/17/
49 DISTRIBUTION_TYPE = maven
50 USE_RELEASE_FILE = false
56 An example of a maven release file appears below.
60 $ cat releases/maven-release.yaml
62 distribution_type: maven
63 log_dir: example-project-maven-stage-master/17/
64 project: example-project
68 The following parameters must appear in a maven release yaml file.
72 :distribution_type: Must be "maven".
73 :log_dir: The suffix of the logs URL reported on successful completion
74 by the Jenkins stage job that created and pushed the artifact
75 to the staging repository. For example, use value
76 "example-project-maven-stage-master/17" for the logs URL
77 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-maven-stage-master/17
78 :project: The name of the project.
79 :version: The semantic version string used for the artifact.
83 :git_tag: The tag string to sign and push to the Git repository.
84 (default: the semantic version string)
85 :tag_release: Tag Gerrit Repo during the release process.
88 The JSON schema for a maven release file appears below.
90 .. literalinclude:: ../../schema/release-schema.yaml
94 Container Release Files
95 -----------------------
97 An example of a container release file appears below.
101 $ cat releases/container-release.yaml
103 distribution_type: container
104 container_release_tag: 1.0.0
105 container_pull_registry: nexus.onap.org:10003
106 container_push_registry: nexus.onap.org:10002
108 ref: d1b9cd2dd345fbeec0d3e2162e008358b8b663b2
111 version: 1.0.0-20190806T184921Z
112 - name: test-frontend
113 version: 1.0.0-20190806T184921Z
116 The following parameters must appear in a container release yaml file.
118 :Required Parameters:
120 :distribution_type: Must be "container".
121 :container_release_tag: The string to use as a Docker tag on all
123 :container_pull_registry: The Nexus registry that supplies the staged
125 :container_push_registry: The Nexus registry that receives the released
127 :project: The name of the project.
128 :ref: The git commit reference (SHA-1 code) to tag with the version string.
129 :containers: A list of name and version (tag) pairs that specify the
130 Docker images in the container-pull registry to promote to the
131 container-push registry.
133 :Optional Parameters:
135 :git_tag: The tag string to sign and push to the Git repository.
136 (default: the semantic version string)
137 :tag_release: Tag Gerrit Repo during the release process.
140 The JSON schema for a container release file appears below.
142 .. literalinclude:: ../../schema/release-container-schema.yaml
149 An example of a PyPI release file appears below. Name of the release file must
150 start with "pypi". For example releases/pypi-1.0.0-mypackage.yaml
154 $ cat releases/pypi-1.0.0-mypackage.yaml
156 pypi_project: mypackage
157 python_version: '3.4'
159 log_dir: example-project-pypi-merge-master/17
162 The following parameters must appear in the PyPI release yaml file.
163 These are not part of the Jenkins job definition to allow independent
164 self-release of a package maintained in a git repository with other
167 :Required Parameters:
169 :log_dir: The suffix of the logs URL reported on successful completion
170 by the Jenkins merge job that created and pushed the distribution
171 files to the staging repository. For example, use value
172 "example-project-pypi-merge-master/17" for the logs URL
173 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-pypi-merge-master/17
174 :pypi_project: The PyPI project name at the staging and
175 release repositories, for example "mypackage".
176 :python_version: The Python interpreter version to use for pip
177 "Requires-Python" compatibility checks, for example '3', '3.7' or 3.7.4.
178 Put valid decimal values such as 3 or 3.7 in quotes to pass schema validation.
179 :version: The semantic version string used for the package in the
182 :Optional Parameters:
184 :git_tag: The tag string to sign and push to the Git repository.
185 (default: the semantic version string)
186 :tag_release: Tag Gerrit Repo during the release process.
189 The JSON schema for a PyPI release file appears below.
191 .. literalinclude:: ../../schema/release-pypi-schema.yaml
195 PackageCloud Release Files
196 --------------------------
198 An example of a PackageCloud release file appears below. Name of release file
199 must start with "packagecloud". For example releases/packagecloud-1.6-tree.yaml
203 $ cat releases/packagecloud-1.6-tree.yaml
207 - name: tree_1.6.0_amd64.deb
208 - name: tree-dev_1.6.0_amd64.deb
209 - name: tree-devel-1.6.0-1.x86_64.rpm
210 - name: tree-1.6.0-1.x86_64.rpm
211 ref: 5555cd2dd345fbeec0d3e2162e00835852342cda
212 log_dir: example-project-packagecloud-merge/21
215 The following parameters must appear in the PackageCloud release yaml file.
216 These are not part of the Jenkins job definition to allow independent
217 self-release of a package maintained in a git repository with other
220 :Required Parameters:
222 :package_name: Name of the release package.
223 :packages: A list of names that specify the packages to promote.
224 Found in jenkins console log when using gem to push package eg.
225 "Pushing /path/of/package/name-of-package.rpm... success!"
226 OR using rest api call to query packagecloud.io repo
227 "curl https://packagecloud.io/api/v1/repos/test_user/test_repo/search?q=
228 | yq -r .[].filename"
229 :ref: The git commit reference (SHA-1 code) to tag with the version string.
230 :log_dir: The suffix of the logs URL reported on successful completion
231 by the Jenkins merge job that created and pushed the distribution
232 files to the staging repository. For example, use value
233 "example-project-packagecloud-merge-/21" for the logs URL
234 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-packagecloud-merge/21
235 :version: The semantic version string used for the package.
237 :Optional Parameters:
239 :git_tag: The tag string to sign and push to the Git repository.
240 (default: the semantic version string)
241 :tag_release: Tag Gerrit Repo during the release process.
244 The JSON schema for a PackageCloud release file appears below.
246 .. literalinclude:: ../../schema/release-packagecloud-schema.yaml
253 An example of a Jenkins job configuration that uses the global-jjb
254 templates for maven and container release jobs appears next.
259 name: my-project-release
261 project-name: my-project
262 build-node: centos7-docker-4c-4g
263 mvn-settings: my-project-settings
265 - '{project-name}-gerrit-release-jobs'
270 Release Engineers: please follow the setup guide below before adding the job definition.
279 Release verify and merge jobs are the same except for their scm,
280 trigger, and builders definition. This anchor is the common template.
288 This template supports Maven and Container release jobs.
290 This template uses a git commit choosing strategy that builds the merged
291 commit with the release yaml file, not the tip of the target branch, so
292 projects can repeat the release action in case of merge job failure.
294 :Template Name: {project-name}-release-merge
296 :Comment Trigger: remerge
298 :Required parameters:
300 :build-node: The node to run build on.
301 :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
303 :project: Git repository name
304 :project-name: Jenkins job name prefix
306 :Optional parameters:
308 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
309 :build-timeout: Timeout in minutes before aborting build. (default: 15)
311 :gerrit_merge_triggers: Override Gerrit Triggers.
312 :gerrit_trigger_file_paths: Override file paths filter which checks which
313 file modifications will trigger a build. The default pattern is the
314 regular expression ``(releases\/.*\.yaml|\.releases\/.*\.yaml)``
320 This template supports Maven and Container release jobs.
322 :Template Name: {project-name}-release-verify
324 :Comment Trigger: recheck|reverify
326 :Required Parameters:
328 :build-node: The node to run build on.
329 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
331 :project: Git repository name
332 :project-name: Jenkins job name prefix
334 :Optional Parameters:
336 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
337 :build-node: The node to run build on.
338 :build-timeout: Timeout in minutes before aborting build. (default: 15)
339 :gerrit-skip-vote: Skip voting for this job. (default: false)
340 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
342 :gerrit_verify_triggers: Override Gerrit Triggers.
343 :gerrit_trigger_file_paths: Override file paths filter which checks which
344 file modifications will trigger a build. The default pattern is the
345 regular expression ``(releases\/.*\.yaml|\.releases\/.*\.yaml)``
351 Publishes a Python package on merge of a patch set with a release yaml
352 file. Checks the format of the version string, downloads the package
353 artifacts from the PyPI staging repository, uploads the package
354 artifacts to the PyPI release repository, tags the git repository,
355 signs the tag and pushes the tag to the git server. The release merge
356 template accepts neither a branch nor a stream parameter.
358 These templates use a git commit choosing strategy that builds the merged
359 commit with the release yaml file, not the tip of the target branch, so
360 projects can repeat the release action in case of merge job failure.
364 - {project-name}-pypi-release-merge
365 - gerrit-pypi-release-merge
366 - github-pypi-release-merge
368 :Comment Trigger: remerge
370 :Required Parameters:
372 :build-node: The node to run build on, which must be Centos.
373 :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
375 :project: Git repository name
376 :project-name: Jenkins job name prefix
378 :Optional Parameters:
380 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
381 :build-timeout: Timeout in minutes before aborting build. (default: 15)
382 :disable-job: Whether to disable the job (default: false)
383 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
384 :pypi-stage-index: Base URL of the PyPI staging repository.
385 (default https://test.pypi.org/simple)
386 :pypi-repo: Key for the PyPI release repository in the .pypirc file,
387 should be the repository pypy.org. (default: pypi)
388 :use-release-file: Whether to use the release file. (default: true)
390 :gerrit_release_trigger_file_paths: Override file paths filter which checks
391 which file modifications will trigger a build. The default pattern is the
392 regular expression ``(releases\/pypi.*\.yaml|\.releases\/pypi.*\.yaml)``
397 Verifies a Python package project on creation of a patch set with a
398 release yaml file. Checks the contents of the release yaml file,
399 checks the format of the version string, and downloads the release
400 artifacts from the specified PyPI staging repository. The release
401 verify template accepts neither a branch nor a stream parameter.
405 - {project-name}-pypi-release-verify
406 - gerrit-pypi-release-verify
407 - github-pypi-release-verify
409 :Comment Trigger: recheck
411 :Required Parameters:
413 :build-node: The node to run build on, which must be Centos.
414 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
416 :project: Git repository name
417 :project-name: Jenkins job name prefix
419 :Optional Parameters:
421 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
422 :build-timeout: Timeout in minutes before aborting build. (default: 15)
423 :disable-job: Whether to disable the job (default: false)
424 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
425 :pypi-stage-index: Base URL of the PyPI staging repository.
426 (default https://test.pypi.org/simple)
427 :pypi-repo: Key for the PyPI release repository in the .pypirc file,
428 should be the repository pypy.org (default: pypi)
429 :use-release-file: Whether to use the release file. (default: true)
431 :gerrit_release_trigger_file_paths: Override file paths filter which checks
432 which file modifications will trigger a build. The default pattern is the
433 regular expression ``(releases\/pypi.*\.yaml|\.releases\/pypi.*\.yaml)``
435 PackageCloud Release Verify
436 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
438 This template supports PackageCloud release jobs. Checks that the specified
439 packages are present in the staging repository and absent from the release
440 repository. The file path trigger uses the regular expression
441 ``(releases\/packagecloud.*\.yaml|\.releases\/packagecloud.*\.yaml)``
443 :Template Name: {project-name}-packagecloud-release-verify
445 :Comment Trigger: recheck|reverify
447 :Required Parameters:
449 :build-node: The node to run build on.
450 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
452 :project: Git repository name
453 :project-name: Jenkins job name prefix
455 :Optional Parameters:
457 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
458 :build-node: The node to run build on.
459 :build-timeout: Timeout in minutes before aborting build. (default: 15)
460 :gerrit-skip-vote: Skip voting for this job. (default: false)
461 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
463 PackageCloud Release Merge
464 ~~~~~~~~~~~~~~~~~~~~~~~~~~
466 This template supports PackageCloud release jobs. Promotes the specified
467 packages from the staging repository to the release repository.
468 The file path trigger uses the regular expression
469 ``(releases\/packagecloud.*\.yaml|\.releases\/packagecloud.*\.yaml)``
471 This template uses a git commit choosing strategy that builds the merged
472 commit with the release yaml file, not the tip of the target branch, so
473 projects can repeat the release action in case of merge job failure.
475 :Template Name: {project-name}-packagecloud-release-merge
477 :Comment Trigger: remerge
479 :Required Parameters:
481 :build-node: the node to run build on.
482 :jenkins-ssh-release-credential: credential to use for ssh. (generally set
484 :project: git repository name
485 :project-name: jenkins job name prefix
487 :Optional Parameters:
489 :build-days-to-keep: days to keep build logs in jenkins. (default: 7)
490 :build-timeout: timeout in minutes before aborting build. (default: 15)
492 Setup for LFID, Nexus, Jenkins and Gerrit
493 -----------------------------------------
495 This section is for the Linux Foundation release engineering team.
500 Create an ``lfid`` and an ``ssh-key``
502 ``YOUR_RELEASE_USERNAME`` for example: onap-release
504 ``YOUR_RELEASE_EMAIL`` for example: collab-it+onap-release@linuxfoundation.org
510 ssh-keygen -t rsa -C "collab-it+odl-release@linuxfoundation.org" -f /tmp/odl-release
513 `Create an LFID with the above values <https://identity.linuxfoundation.org>`_
519 Create a Nexus account called ``'jenkins-release'`` with promote privileges.
521 .. image:: ../_static/nexus-promote-privs.png
526 Log into your Gerrit with ``YOUR_RELEASE_USERNAME``, upload the public
527 part of the ``ssh-key`` you created earlier. Log out of Gerrit and log
528 in again with your normal account for the next steps.
531 In Gerrit create a new group called ``self-serve-release`` and give it
532 direct push rights via ``All-Projects`` Add ``YOUR_RELEASE_USERNAME``
533 to group ``self-serve-release`` and group ``Non-Interactive Users``
536 In All project, grant group self-serve-release the following:
540 [access "refs/heads/*"]
541 push = group self-serve-release
542 [access "refs/tags/*"]
543 createTag = group self-serve-release
544 createSignedTag = group self-serve-release
545 forgeCommitter = group self-serve-release
546 push = group self-serve-release
552 Add a global credential to Jenkins called ``jenkins-release`` and set
553 the ID: ``'jenkins-release'`` as its value insert the private half of
554 the ``ssh-key`` that you created for your Gerrit user.
556 Add Global variables in Jenkins:
557 Jenkins configure -> Global properties -> Environment variables::
559 RELEASE_USERNAME = YOUR_RELEASE_USERNAME
560 RELEASE_EMAIL = YOUR_RELEASE_EMAIL
565 Add these variables to your global-vars-$SILO.sh file or they will
568 Jenkins configure -> Managed Files -> Add a New Config -> Custom File
573 Name: SIGNING_PUBKEY (optional)
574 Comment: SIGNING_PUBKEY (optional)
576 Content: (Ask Andy for the public signing key)
577 -----BEGIN PGP PUBLIC KEY BLOCK-----
580 Add or edit the managed file in Jenkins called ``lftoolsini``,
581 appending a nexus section: Jenkins Settings -> Managed files -> Add
582 (or edit) -> Custom file
587 username=jenkins-release
588 password=<plaintext password>
593 Upgrade your project's global-jjb if needed, then add the following to
594 your global defaults file (e.g., jjb/defaults.yaml).
598 jenkins-ssh-release-credential: jenkins-release