From: Thanh Ha Date: Wed, 25 Oct 2017 23:52:19 +0000 (+1000) Subject: Add note on JJB defaults X-Git-Tag: v0.12.0~7^2 X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=commitdiff_plain;h=cabe8986aefedccb82124fa360a3e89bcfbd81db;p=releng%2Fglobal-jjb.git Add note on JJB defaults This patch provides additional clarification on how defaults are used in JJB and when to expect the prioritization of variable expansion with JJB. Additionally add in intersphinx linking to jjb so that we can link directly to JJB upstream documentation. This documentation is based on details provided by Thanh Ha on the OpenDaylight dev mailing list. https://lists.opendaylight.org/pipermail/dev/2017-October/004184.html Issue: RELENG-591 Change-Id: I65aaf933b88519dfbd1daaf8e00661146bc8c38f Co-Authored-By: Anil Belur Signed-off-by: Anil Belur Signed-off-by: Thanh Ha --- diff --git a/docs/best-practices.rst b/docs/best-practices.rst index 129b9c3a..a9d4d20b 100644 --- a/docs/best-practices.rst +++ b/docs/best-practices.rst @@ -76,8 +76,8 @@ Passing parameters to shell scripts There are 2 ways to pass parameters into scripts: -1) JJB variables in the format {var} -2) Environment variables in the format ${VAR} +1. JJB variables in the format {var} +2. Environment variables in the format ${VAR} We recommend avoiding using method 1 (Pass JJB variables) into shell scripts and instead always use method 2 (Environment variables). This makes @@ -85,9 +85,9 @@ troubleshooting JJB errors easier and does not require escaping curly braces. This method requires 3 steps: -1) Declare a parameter section or inject the variable as properties-content. -2) Invoke the shell script with `include-raw-escape` instead of `include-raw`. -3) Use the shell variable in shell script. +1. Declare a parameter section or inject the variable as properties-content. +2. Invoke the shell script with `include-raw-escape` instead of `include-raw`. +3. Use the shell variable in shell script. The benefit of this method is that parameters will always be at the top @@ -193,3 +193,163 @@ Example: - build-file: settings: '{settings-file}' file-version: '{file-version}' + + +Variable expansion and Defaults +=============================== + +.. This documentation uses work originally provided by Thanh Ha on +.. the OpenDaylight dev mailing list. +.. https://lists.opendaylight.org/pipermail/dev/2017-October/004184.html + +JJB has a concept called :ref:`Defaults ` which is what JJB will +replace a variable with if unset. Variables can configure dynamic content +in :ref:`job-template ` sections and allow certain options in +these sections to be configurable. + +The section that expands Defaults is :ref:`Job Templates ` no +other sections will expand a default. This documentation will explain how +variables and defaults expansion works and which take precedence in JJB's +variable expansion logic for the following configuration sections. + +- macro +- job-template +- project +- default + +Macro sections +-------------- + +:ref:`Macro ` sections can contain variables but do **NOT** support +default values getting filled in both at the macro definition level and at the +defaults configuration level. :ref:`Macros ` and +:ref:`Job Templates ` can use Macros but any variables defined in +a Macro needs to pass a value or a new variable redefined in the +:ref:`Job Template ` if you want to pass on the configuration. + +So for example if you have a macro that has a '{msg}' variable: + +Example: + +.. code-block:: yaml + + - builder: + name: echo-msg + builders: + - shell: "echo {msg}" + +Any downstream job-templates or macros that use this macro **MUST** pass in a +`msg: Hello` definition or redefine the msg variable `msg: {msg}`. + + +Job Template sections +--------------------- + +:ref:`Job Template ` sections can use defaults in two ways. + +1. Configure the message: + + .. code-block:: yaml + + - job-template: + name: echo-hello-world + builders: + - echo-msg: + msg: 'Hello World' + +2) Re-define '{msg}' variable + + .. code-block:: yaml + + - job-template: + name: echo-message + builders: + - echo-msg: + msg: '{message}' + +In option 2, we redefine the variable msg as `{message}` which a user of the +job-template can now pass into the job their own custom message which is +different than option 1, where we set a static message to pass in. We purposely +redefined the **{msg}** to **{message}** here to show that you do not need to +redefine it with the same name but we could have used the same name `{msg}` in +the template too if we wanted to keep it the same. + +Job Templates can also default a default variable for the variables it defines. + +Example: + +.. code-block:: yaml + + - job-template: + name: echo-message + message: 'Hello World' + builders: + - echo-msg: + msg: '{message}' + +This creates a job template variable called '{message}' which will default to +"Hello World" if the user of the template does not explicitly pass in a message. + +We should be aware of 2 Defaults concepts: + +1. Default as defined in the :ref:`job-template ` +2. Default as defined in a :ref:`defaults ` configuration + (typically defaults.yaml) + +In this case there is a default '{message}' set in the +:ref:`job-template `. JJB will use this default if the user +(project section) does not declare a {message}. + +If we do not declare a default in the :ref:`job-template ` then +JJB will fallback to checking the "defaults configuration". + +This means that the precendence of defaults is as follows: + +1. User-provided +2. Job Template +3. Defaults.yaml + +Project sections +---------------- + +:ref:`Project ` sections define real jobs and pass in variables as +necessary. Projects sections do NOT expand defaults.yaml. So you cannot +configure a setting with {var} in here and expect defaults.yaml to fill it in +for you. Define required configuration here. + +Example: + +.. code-block:: yaml + + - project + name: foo + jobs: + - 'echo-message' + message: 'I am foo' + + +Defaults sections +----------------- + +:ref:`Defaults ` sections are the absolute last thing JJB checks if a +variable is not configured in a job-template and user did not pass in the +variable. JJB will fill in whatever is in the defaults configuration. + +Variable expansion order of precedence seems to be: + +1. project section definition +2. job-template variable definition +3. defaults.yaml variable definition + +.. note:: Defaults set variables in job-templates and are NOT used in Macros. + +Final thoughts +-------------- + +For any :ref:`Basic Job Configuration ` for example "concurrent", "jdk", +"node" etc... we cannot set defaults with the same name as JJB will not expand +them. To use "node" we need to give the variable for that setting a +different name such as "build-node" instead, if we want JJB to perform +expansion for those settings. This issue affects top level job configuration, +it does not appear to affect items below the top level such as calling a +builder, wrapper or parameter. diff --git a/docs/conf.py b/docs/conf.py index dea371d8..0359ed94 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -255,6 +255,7 @@ htmlhelp_basename = 'LinuxFoundationReleaseEngineeringGJJBdoc' # Example configuration for intersphinx: refer to the Python standard library. intersphinx_mapping = { + 'jjb': ('https://docs.openstack.org/infra/jenkins-job-builder/', None), 'lftools': ('http://lftools.releng.linuxfoundation.org/en/latest/', None), 'python': ('https://docs.python.org/', None), }