1 .. _global-jjb-best-practices:
12 While some of this applies to the Global JJB project other recommendations
13 are generally useful to projects that might be defining JJB templates.
15 The Global JJB project is a useful example project to look at so we recommend
16 referring to the Maven job definitions as an example as you read the
19 https://github.com/lfit/releng-global-jjb/blob/master/jjb/lf-maven-jobs.yaml
21 We recommend sectioning off the template into 3 general sections in order:
23 1. Job Groups (optional)
25 3. Job Template Definitions
27 In section 1) not all configurations need this so is an optional section. Job
28 groups are useful in cases where there are jobs that are generally useful
29 together. For example the OpenDaylight uses a lot of Merge and Verify job
30 combinations so every new project will want both job types defined in their
33 In section 2) we want to define all common functions (anchors, aliases, macros)
34 that are generally useful to all jobs in the file. This allows job template
35 developers to look at the top of the file to see if there are useful functions
36 already defined that they can reuse.
38 In section 3) we can declare our job definitions. In the Global JJB project we
39 create Gerrit and GitHub versions of the jobs so the format we use here
40 might look strange at first but is well layed out for code reuse if we need to
41 define 2 or more versions of the same job template for different systems. We
42 will define this in more detail in the next section.
47 1. Comment of Job Template Name
48 2. Macro containing build definition of the job
49 a. Macro named after job
50 b. Complete documentation of the job parameters
51 c. Default parameters defined by the job
53 3. job-template definition containing build triggers
55 In section 1) we need to declare a in large comment text to identify the job
58 In section 2) we declare the actual job definition. This is so that we have a
59 single macro that we call in all the real job-template sections that is
60 reusable and not duplicating any code. First we declare the macro as the job
61 name. Then in 2.b) we provide the complete documentation of the job parameters
62 this is so that we can link users of the job to this file and they can
63 understand fully what options they can configure for this particular job.
64 Then we define defaults for any parameters that are optional. The last section
65 we define the job configuration which completes the macro.
67 In section 3) we declare the actual job-template. Because of all the
68 preparations above job template definitions should be small and simple. It
69 needs to define the scm and job triggers. The Global JJB project needs to
70 support both Gerrit and GitHub versions of the same job so the job definitions
71 there have 2 templates for each job defined.
74 Passing parameters to shell scripts
75 ===================================
77 There are 2 ways to pass parameters into scripts:
79 1. JJB variables in the format {var}
80 2. Environment variables in the format ${VAR}
82 We recommend avoiding using method 1 (Pass JJB variables) into shell scripts
83 and instead always use method 2 (Environment variables). This makes
84 troubleshooting JJB errors easier and does not require escaping curly braces.
86 This method requires 3 steps:
88 1. Declare a parameter section or inject the variable as properties-content.
89 2. Invoke the shell script with `include-raw-escape` instead of `include-raw`.
90 3. Use the shell variable in shell script.
93 The benefit of this method is that parameters will always be at the top
94 of the job page and when clicking the Build with Parameters button in Jenkins
95 we can see the parameters before running the job. We can review the
96 parameters retro-actively by visiting the job parameters page
97 ``job/lastSuccessfulBuild/parameters/``. Injecting variables as
98 properties-content makes the variable local to the specific macro, while
99 declaring it as parameter makes the variable global.
103 When a macro which invokes a shell script has no JJB parameters defined
104 `!include-raw-escape` will insert extra curly braces, in such cases its
105 recommended to use `!include-raw`.
107 Usage of config-file-provider
108 =============================
110 When using the config-file-provider plugin in Jenkins to provide a config file.
111 We recommend using a macro so that we can configure the builder to
112 remove the config file as a last step. This ensures
113 that credentials do not exist on the system for longer than it needs to.
122 - config-file-provider:
124 - file-id: jenkins-log-archives-settings
125 variable: SETTINGS_FILE
126 - shell: !include-raw:
127 - ../shell/logs-get-credentials.sh
128 - shell: !include-raw:
129 - ../shell/lftools-install.sh
130 - ../shell/logs-deploy.sh
131 - shell: !include-raw:
132 - ../shell/logs-clear-credentials.sh
133 - description-setter:
134 regexp: '^Build logs: .*'
136 In this example the script logs-deploy requires a config file to authenticate
137 with Nexus to push logs up. We declare a macro here so that we can ensure that
138 we remove credentials from the system after the scripts
139 complete running via the logs-clear-credentials.sh script. This script contains
142 1. Provide credentials via config-file-provider
143 2. Run the build scripts in this case lftools-install.sh and logs-deploy.sh
144 3. Remove credentials provided by config-file-provider
146 .. _preserve-variable-refs:
148 Preserving Objects in Variable References
149 =========================================
151 JJB has an option to preserve a data structure object when you want to pass
153 https://docs.openstack.org/infra/jenkins-job-builder/definition.html#variable-references
155 One thing that is not explicitly covered is the format of the variable name
156 that you pass the object to. When you use the `{obj:key}` notation to preserve
157 the original data structure object, it will not work if the variable name has a
158 dash `-` in it. The standard that we follow, and recommend, is to use an underscore
159 `_` instead of a dash.
165 .. literalinclude:: _static/github-pr-trigger.example
167 In the above example note the use of underscores in ``github_pr_whitelist``,
168 ``github_pr_admin_list``, and ``github_included_regions``.
170 Using single quotes around variables
171 ====================================
173 Its recommended to use single quotes around JJB variables '{variable}-field'
174 during variable substitution or when using a variable in a string field, in
175 other cases its recommended to drop the single quotes.
185 properties-content: |
188 settings: '{settings-file}'
189 file-version: '{file-version}'
192 Variable expansion and Defaults
193 ===============================
195 .. This documentation uses work originally provided by Thanh Ha on
196 .. the OpenDaylight dev mailing list.
197 .. https://lists.opendaylight.org/pipermail/dev/2017-October/004184.html
199 JJB has a concept called :ref:`Defaults <defaults>` which is what JJB will
200 replace a variable with if unset. Variables can configure dynamic content
201 in :ref:`job-template <job-template>` sections and allow certain options in
202 these sections to be configurable.
204 The section that expands Defaults is :ref:`Job Templates <job-template>` no
205 other sections will expand a default. This documentation will explain how
206 variables and defaults expansion works and which take precedence in JJB's
207 variable expansion logic for the following configuration sections.
217 :ref:`Macro <macro>` sections can contain variables but do **NOT** support
218 default values getting filled in both at the macro definition level and at the
219 defaults configuration level. :ref:`Macros <macro>` and
220 :ref:`Job Templates <job-template>` can use Macros but any variables defined in
221 a Macro needs to pass a value or a new variable redefined in the
222 :ref:`Job Template <job-template>` if you want to pass on the configuration.
224 So for example if you have a macro that has a '{msg}' variable:
233 - shell: "echo {msg}"
235 Any downstream job-templates or macros that use this macro **MUST** pass in a
236 `msg: Hello` definition or redefine the msg variable `msg: {msg}`.
239 Job Template sections
240 ---------------------
242 :ref:`Job Template <job-template>` sections can use defaults in two ways.
244 1. Configure the message:
249 name: echo-hello-world
254 2) Re-define '{msg}' variable
264 In option 2, we redefine the variable msg as `{message}` which a user of the
265 job-template can now pass into the job their own custom message which is
266 different than option 1, where we set a static message to pass in. We purposely
267 redefined the **{msg}** to **{message}** here to show that you do not need to
268 redefine it with the same name but we could have used the same name `{msg}` in
269 the template too if we wanted to keep it the same.
271 Job Templates can also default a default variable for the variables it defines.
279 message: 'Hello World'
284 This creates a job template variable called '{message}' which will default to
285 "Hello World" if the user of the template does not explicitly pass in a message.
287 We should be aware of 2 Defaults concepts:
289 1. Default as defined in the :ref:`job-template <job-template>`
290 2. Default as defined in a :ref:`defaults <defaults>` configuration
291 (typically defaults.yaml)
293 In this case there is a default '{message}' set in the
294 :ref:`job-template <job-template>`. JJB will use this default if the user
295 (project section) does not declare a {message}.
297 If we do not declare a default in the :ref:`job-template <job-template>` then
298 JJB will fallback to checking the "defaults configuration".
300 This means that the precendence of defaults is as follows:
309 :ref:`Project <project>` sections define real jobs and pass in variables as
310 necessary. Projects sections do NOT expand defaults.yaml. So you cannot
311 configure a setting with {var} in here and expect defaults.yaml to fill it in
312 for you. Define required configuration here.
328 :ref:`Defaults <defaults>` sections are the absolute last thing JJB checks if a
329 variable is not configured in a job-template and user did not pass in the
330 variable. JJB will fill in whatever is in the defaults configuration.
332 Variable expansion order of precedence seems to be:
334 1. job-group section definition
335 2. project section definition
336 3. job-template variable definition
337 4. defaults.yaml variable definition
339 .. note:: Defaults set variables in job-templates and are NOT used in Macros.
341 global-jjb should not provide job-group definitions and leave it up to users of
342 global-jjb to create their own as a job-group as a variable defined in a job
343 group the highest precendence. Global JJB should strive to be purely a
344 job-template and macro library for downstream consumers.
349 For any :ref:`Basic Job Configuration <job>` for example "concurrent", "jdk",
350 "node" etc... we cannot set defaults with the same name as JJB will not expand
351 them. To use "node" we need to give the variable for that setting a
352 different name such as "build-node" instead, if we want JJB to perform
353 expansion for those settings. This issue affects top level job configuration,
354 it does not appear to affect items below the top level such as calling a
355 builder, wrapper or parameter.