1 .. _lf-global-jjb-release:
3 Self-Serve Release Jobs
4 =======================
6 Self-serve release jobs allow project committers to promote a jar
7 file, container image, Python package or PackageCloud artifact from a
8 staging area to a release area. A release yaml file controls the
9 process, and Jenkins promotes the artifact when a project committer
10 merges the release yaml file in Gerrit.
12 To use the self-release process, create a releases/ or .releases/
13 directory at the root of the project repository, add one release yaml
14 file to it, and submit a change set with that release yaml file. The
15 required contents of the release yaml file are different for each type
16 of release, see the schemas and examples shown below. The version
17 string in the release yaml file should be a valid Semantic Versioning
18 (SemVer) string, matching the pattern "#.#.#" where "#" is one or more
19 digits. A version string matching the pattern "v#.#.#" is also
20 accepted. Upon merge of the change, a Jenkins job promotes the
21 artifact and pushes a gpg-signed tag to the repository.
25 The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml).
26 In words, the directory name can be ".releases" or "releases"; the file
27 name can be anything with suffix ".yaml". Some release jobs require
28 a specific prefix on the file, as described below.
30 The build node for all release jobs must be CentOS, which supports the
31 sigul client for accessing a signing server to sign a tag. The build
32 node for container release jobs must have Docker installed.
34 A Jenkins admin user can also trigger a release job via the "Build
35 with parameters" action, removing the need to create and merge a
36 release yaml file. The user must enter parameters in the same way as
37 a release yaml file, except for the special USE_RELEASE_FILE and
38 DRY_RUN check boxes. The user must uncheck the USE_RELEASE_FILE check
39 box if the job should run without a release file, instead passing the
40 required information as build parameters. The user can check the
41 DRY_RUN check box to test the job while skipping upload of files to
42 the release repository.
44 For example, the parameters for a Maven release are as follows::
46 GERRIT_BRANCH = master
48 LOG_DIR = example-project-maven-stage-master/17/
49 DISTRIBUTION_TYPE = maven
50 USE_RELEASE_FILE = false
56 An example of a maven release file appears below.
60 $ cat releases/maven-release.yaml
62 distribution_type: maven
63 log_dir: example-project-maven-stage-master/17/
64 project: example-project
68 The following parameters must appear in a maven release yaml file.
72 :distribution_type: Must be "maven".
73 :log_dir: The suffix of the logs URL reported on completion by the
74 Jenkins stage job that created and pushed the artifact
75 to the staging repository. For example, use value
76 "example-project-maven-stage-master/17" for the logs URL
77 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-maven-stage-master/17
78 :project: The name of the project.
79 :version: The semantic version string used for the artifact.
83 :git_tag: The tag string to sign and push to the Git repository.
84 (default: the semantic version string)
86 The JSON schema for a maven release file appears below.
88 .. literalinclude:: ../../schema/release-schema.yaml
92 Container Release Files
93 -----------------------
95 An example of a container release file appears below.
99 $ cat releases/container-release.yaml
101 distribution_type: container
102 container_release_tag: 1.0.0
103 container_pull_registry: nexus.onap.org:10003
104 container_push_registry: nexus.onap.org:10002
106 ref: d1b9cd2dd345fbeec0d3e2162e008358b8b663b2
109 version: 1.0.0-20190806T184921Z
110 - name: test-frontend
111 version: 1.0.0-20190806T184921Z
114 The following parameters must appear in a container release yaml file.
116 :Required Parameters:
118 :distribution_type: Must be "container".
119 :container_release_tag: The string to use as a Docker tag on all
121 :container_pull_registry: The Nexus registry that supplies the staged
123 :container_push_registry: The Nexus registry that receives the released
125 :project: The name of the project.
126 :ref: The git commit reference (SHA-1 code) to tag with the version string.
127 :containers: A list of name and version (tag) pairs that specify the
128 Docker images in the container-pull registry to promote to the
129 container-push registry.
131 :Optional Parameters:
133 :git_tag: The tag string to sign and push to the Git repository.
134 (default: the semantic version string)
136 The JSON schema for a container release file appears below.
138 .. literalinclude:: ../../schema/release-container-schema.yaml
145 An example of a PyPI release file appears below. Name of the release file must
146 start with "pypi". For example releases/pypi-1.0.0-mypackage.yaml
150 $ cat releases/pypi-1.0.0-mypackage.yaml
152 pypi_project: mypackage
153 python_version: '3.4'
155 log_dir: example-project-pypi-merge-master/17
158 The following parameters must appear in the PyPI release yaml file.
159 These are not part of the Jenkins job definition to allow independent
160 self-release of a package maintained in a git repository with other
163 :Required Parameters:
165 :log_dir: The suffix of the logs URL reported on completion by the
166 Jenkins merge job that created and pushed the distribution files
167 to the staging repository. For example, use value
168 "example-project-pypi-merge-master/17" for the logs URL
169 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-pypi-merge-master/17
170 :pypi_project: The PyPI project name at the staging and
171 release repositories, for example "mypackage".
172 :python_version: The Python interpreter version to use for pip
173 "Requires-Python" compatibility checks, for example '3', '3.7' or 3.7.4.
174 Put valid decimal values such as 3 or 3.7 in quotes to pass schema validation.
175 :version: The semantic version string used for the package in the
178 :Optional Parameters:
180 :git_tag: The tag string to sign and push to the Git repository.
181 (default: the semantic version string)
183 The JSON schema for a PyPI release file appears below.
185 .. literalinclude:: ../../schema/release-pypi-schema.yaml
189 PackageCloud Release Files
190 --------------------------
192 An example of a PackageCloud release file appears below. Name of release file
193 must start with "packagecloud". For example releases/packagecloud-1.6-tree.yaml
197 $ cat releases/packagecloud-1.6-tree.yaml
201 - name: tree_1.6.0_amd64.deb
202 - name: tree-dev_1.6.0_amd64.deb
203 - name: tree-devel-1.6.0-1.x86_64.rpm
204 - name: tree-1.6.0-1.x86_64.rpm
205 ref: 5555cd2dd345fbeec0d3e2162e00835852342cda
206 log_dir: example-project-packagecloud-merge/21
209 The following parameters must appear in the PackageCloud release yaml file.
210 These are not part of the Jenkins job definition to allow independent
211 self-release of a package maintained in a git repository with other
214 :Required Parameters:
216 :package_name: Name of the release package.
217 :packages: A list of names that specify the packages to promote.
218 Found in jenkins console log when using gem to push package eg.
219 "Pushing /path/of/package/name-of-package.rpm... success!"
220 OR using rest api call to query packagecloud.io repo
221 "curl https://packagecloud.io/api/v1/repos/test_user/test_repo/search?q=
222 | yq -r .[].filename"
223 :ref: The git commit reference (SHA-1 code) to tag with the version string.
224 :log_dir: The suffix of the logs URL reported on completion by the
225 Jenkins merge job that created and pushed the distribution files
226 to the staging repository. For example, use value
227 "example-project-packagecloud-merge-/21" for the logs URL
228 https://logs.lf-project.org/production/vex-sjc-lfp-jenkins-prod-1/example-project-packagecloud-merge/21
229 :version: The semantic version string used for the package.
231 :Optional Parameters:
233 :git_tag: The tag string to sign and push to the Git repository.
234 (default: the semantic version string)
236 The JSON schema for a PackageCloud release file appears below.
238 .. literalinclude:: ../../schema/release-packagecloud-schema.yaml
245 An example of a Jenkins job configuration that uses the global-jjb
246 templates for maven and container release jobs appears next.
251 name: my-project-release
253 project-name: my-project
254 build-node: centos7-docker-4c-4g
255 mvn-settings: my-project-settings
257 - '{project-name}-gerrit-release-jobs'
262 Release Engineers: please follow the setup guide below before adding the job definition.
271 Release verify and merge jobs are the same except for their scm,
272 trigger, and builders definition. This anchor is the common template.
280 This template supports Maven and Container release jobs.
282 :Template Name: {project-name}-release-merge
284 :Comment Trigger: remerge
286 :Required parameters:
288 :build-node: The node to run build on.
289 :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
291 :project: Git repository name
292 :project-name: Jenkins job name prefix
294 :Optional parameters:
296 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
297 :build-timeout: Timeout in minutes before aborting build. (default: 15)
299 :gerrit_merge_triggers: Override Gerrit Triggers.
300 :gerrit_trigger_file_paths: Override file paths filter which checks which
301 file modifications will trigger a build.
304 - compare-type: REG_EXP
305 pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)'
311 This template supports Maven and Container release jobs.
313 :Template Name: {project-name}-release-verify
315 :Comment Trigger: recheck|reverify
317 :Required Parameters:
319 :build-node: The node to run build on.
320 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
322 :project: Git repository name
323 :project-name: Jenkins job name prefix
325 :Optional Parameters:
327 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
328 :build-node: The node to run build on.
329 :build-timeout: Timeout in minutes before aborting build. (default: 15)
330 :gerrit-skip-vote: Skip voting for this job. (default: false)
331 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
333 :gerrit_verify_triggers: Override Gerrit Triggers.
334 :gerrit_trigger_file_paths: Override file paths filter which checks which
335 file modifications will trigger a build.
338 - compare-type: REG_EXP
339 pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)'
345 Publishes a Python package on merge of a patch set with a release yaml
346 file. Checks the format of the version string, downloads the package
347 artifacts from the PyPI staging repository, uploads the package
348 artifacts to the PyPI release repository, tags the git repository,
349 signs the tag and pushes the tag to the git server. The release merge
350 template accepts neither a branch nor a stream parameter.
354 - {project-name}-pypi-release-merge
355 - gerrit-pypi-release-merge
356 - github-pypi-release-merge
358 :Comment Trigger: remerge
360 :Required Parameters:
362 :build-node: The node to run build on, which must be Centos.
363 :jenkins-ssh-release-credential: Credential to use for SSH. (Generally set
365 :project: Git repository name
366 :project-name: Jenkins job name prefix
368 :Optional Parameters:
370 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
371 :build-timeout: Timeout in minutes before aborting build. (default: 15)
372 :disable-job: Whether to disable the job (default: false)
373 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
374 :pypi-stage-index: Base URL of the PyPI staging repository.
375 (default https://test.pypi.org/simple)
376 :pypi-repo: Key for the PyPI release repository in the .pypirc file,
377 should be the repository pypy.org. (default: pypi)
378 :use-release-file: Whether to use the release file. (default: true)
380 :gerrit_trigger_file_paths: Override file paths filter which checks which
381 file modifications will trigger a build.
384 - compare-type: REG_EXP
385 pattern: '(releases\/pypi.*\.yaml|\.releases\/pypi.*\.yaml)'
390 Verifies a Python package project on creation of a patch set with a
391 release yaml file. Checks the contents of the release yaml file,
392 checks the format of the version string, and downloads the release
393 artifacts from the specified PyPI staging repository. The release
394 verify template accepts neither a branch nor a stream parameter.
398 - {project-name}-pypi-release-verify
399 - gerrit-pypi-release-verify
400 - github-pypi-release-verify
402 :Comment Trigger: recheck
404 :Required Parameters:
406 :build-node: The node to run build on, which must be Centos.
407 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
409 :project: Git repository name
410 :project-name: Jenkins job name prefix
412 :Optional Parameters:
414 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
415 :build-timeout: Timeout in minutes before aborting build. (default: 15)
416 :disable-job: Whether to disable the job (default: false)
417 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
418 :pypi-stage-index: Base URL of the PyPI staging repository.
419 (default https://test.pypi.org/simple)
420 :pypi-repo: Key for the PyPI release repository in the .pypirc file,
421 should be the repository pypy.org (default: pypi)
422 :use-release-file: Whether to use the release file. (default: true)
424 :gerrit_trigger_file_paths: Override file paths filter which checks which
425 file modifications will trigger a build.
428 - compare-type: REG_EXP
429 pattern: '(releases\/pypi.*\.yaml|\.releases\/pypi.*\.yaml)'
431 PackageCloud Release Verify
432 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
434 This template supports PackageCloud release jobs.
436 :Template Name: {project-name}-packagecloud-release-verify
438 :Comment Trigger: recheck|reverify
440 :Required Parameters:
442 :build-node: The node to run build on.
443 :jenkins-ssh-credential: Credential to use for SSH. (Generally set
445 :project: Git repository name
446 :project-name: Jenkins job name prefix
448 :Optional Parameters:
450 :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
451 :build-node: The node to run build on.
452 :build-timeout: Timeout in minutes before aborting build. (default: 15)
453 :gerrit-skip-vote: Skip voting for this job. (default: false)
454 :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
456 :gerrit_verify_triggers: Override Gerrit Triggers.
457 :gerrit_trigger_file_paths: Override file paths filter which checks which
458 file modifications will trigger a build.
461 - compare-type: REG_EXP
462 pattern: '(releases\/packagecloud.*\.yaml|\.releases\/packagecloud.*\.yaml)'
465 PackageCloud Release Merge
466 ~~~~~~~~~~~~~~~~~~~~~~~~~~
468 This template supports PackageCloud release jobs.
470 :template name: {project-name}-packagecloud-release-merge
472 :comment trigger: remerge
474 :required parameters:
476 :build-node: the node to run build on.
477 :jenkins-ssh-release-credential: credential to use for ssh. (generally set
479 :project: git repository name
480 :project-name: jenkins job name prefix
482 :optional parameters:
484 :build-days-to-keep: days to keep build logs in jenkins. (default: 7)
485 :build-timeout: timeout in minutes before aborting build. (default: 15)
487 :gerrit_merge_triggers: override gerrit triggers.
488 :gerrit_trigger_file_paths: override file paths filter which checks which
489 file modifications will trigger a build.
492 - compare-type: reg_exp
493 pattern: '(releases\/packagecloud.*\.yaml|\.releases\/packagecloud.*\.yaml)'
496 Setup for LFID, Nexus, Jenkins and Gerrit
497 -----------------------------------------
499 This section is for the Linux Foundation release engineering team.
504 Create an ``lfid`` and an ``ssh-key``
506 ``YOUR_RELEASE_USERNAME`` for example: onap-release
508 ``YOUR_RELEASE_EMAIL`` for example: collab-it+onap-release@linuxfoundation.org
514 ssh-keygen -t rsa -C "collab-it+odl-release@linuxfoundation.org" -f /tmp/odl-release
517 `Create an LFID with the above values <https://identity.linuxfoundation.org>`_
523 Create a Nexus account called ``'jenkins-release'`` with promote privileges.
525 .. image:: ../_static/nexus-promote-privs.png
530 Log into your Gerrit with ``YOUR_RELEASE_USERNAME``, upload the public
531 part of the ``ssh-key`` you created earlier. Log out of Gerrit and log
532 in again with your normal account for the next steps.
535 In Gerrit create a new group called ``self-serve-release`` and give it
536 direct push rights via ``All-Projects`` Add ``YOUR_RELEASE_USERNAME``
537 to group ``self-serve-release`` and group ``Non-Interactive Users``
540 In All project, grant group self-serve-release the following:
544 [access "refs/heads/*"]
545 push = group self-serve-release
546 [access "refs/tags/*"]
547 createTag = group self-serve-release
548 createSignedTag = group self-serve-release
549 forgeCommitter = group self-serve-release
550 push = group self-serve-release
556 Add a global credential to Jenkins called ``jenkins-release`` and set
557 the ID: ``'jenkins-release'`` as its value insert the private half of
558 the ``ssh-key`` that you created for your Gerrit user.
560 Add Global variables in Jenkins:
561 Jenkins configure -> Global properties -> Environment variables::
563 RELEASE_USERNAME = YOUR_RELEASE_USERNAME
564 RELEASE_EMAIL = YOUR_RELEASE_EMAIL
569 Add these variables to your global-vars-$SILO.sh file or they will
572 Jenkins configure -> Managed Files -> Add a New Config -> Custom File
577 Name: SIGNING_PUBKEY (optional)
578 Comment: SIGNING_PUBKEY (optional)
580 Content: (Ask Andy for the public signing key)
581 -----BEGIN PGP PUBLIC KEY BLOCK-----
584 Add or edit the managed file in Jenkins called ``lftoolsini``,
585 appending a nexus section: Jenkins Settings -> Managed files -> Add
586 (or edit) -> Custom file
591 username=jenkins-release
592 password=<plaintext password>
597 Upgrade your project's global-jjb if needed, then add the following to
598 your global defaults file (e.g., jjb/defaults.yaml).
602 jenkins-ssh-release-credential: jenkins-release