X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=blobdiff_plain;f=docs%2Fbest-practices.rst;h=85ee7f7d763b73eac93fe53295968c2466b34d28;hb=dc757e1ab24c2d99258b46bb604d1db0fc37ce1d;hp=129b9c3a52e78a28c949161c40aa1767f61b228d;hpb=c410c7c0863f27c332bb054a5bfc6332872dcb36;p=releng%2Fglobal-jjb.git diff --git a/docs/best-practices.rst b/docs/best-practices.rst index 129b9c3a..85ee7f7d 100644 --- a/docs/best-practices.rst +++ b/docs/best-practices.rst @@ -76,8 +76,8 @@ Passing parameters to shell scripts There are 2 ways to pass parameters into scripts: -1) JJB variables in the format {var} -2) Environment variables in the format ${VAR} +1. JJB variables in the format {var} +2. Environment variables in the format ${VAR} We recommend avoiding using method 1 (Pass JJB variables) into shell scripts and instead always use method 2 (Environment variables). This makes @@ -85,9 +85,9 @@ troubleshooting JJB errors easier and does not require escaping curly braces. This method requires 3 steps: -1) Declare a parameter section or inject the variable as properties-content. -2) Invoke the shell script with `include-raw-escape` instead of `include-raw`. -3) Use the shell variable in shell script. +1. Declare a parameter section or inject the variable as properties-content. +2. Invoke the shell script with `include-raw-escape` instead of `include-raw`. +3. Use the shell variable in shell script. The benefit of this method is that parameters will always be at the top @@ -104,6 +104,38 @@ declaring it as parameter makes the variable global. `!include-raw-escape` will insert extra curly braces, in such cases its recommended to use `!include-raw`. +.. _shell-scripts: + +Shell scripts +============= + +When developing shell scripts for JJB we recommend to create shell scripts as +a separate file instead of inlining in YAML. This way we can ensure that the +ShellCheck linter can catch potential issues with the scripts. + +When writing the script itself, we recommend to redeclare all expected +inputs at the top of the file using lowercase variable names before setting +``set -u`` after the inputs section. This ensures that all variables the +script expects are at the top of the file which is useful for others to review +and debug the script at a later stage. The ``set -u`` configuration before the +start of the script code ensures that we catch any of these undeclared +variables at the top of the file. + +Example: + +.. code-block:: bash + + #!/bin/bash + + # Inputs + tox_dir="${TOX_DIR:-$WORKSPACE}" + tox_envs="${TOX_ENVS:-}" + + # Script start + set -eux -o pipefail + + # ... script code goes here + Usage of config-file-provider ============================= @@ -143,6 +175,8 @@ complete running via the logs-clear-credentials.sh script. This script contains 2. Run the build scripts in this case lftools-install.sh and logs-deploy.sh 3. Remove credentials provided by config-file-provider +.. _preserve-variable-refs: + Preserving Objects in Variable References ========================================= @@ -160,18 +194,10 @@ Example: .. code-block:: yaml - - triggers: - - lf-infra-github-pr-trigger: - trigger-phrase: ^remerge$ - status-context: JJB Merge - permit-all: false - github-hooks: true - github-org: '{github-org}' - github_pr_whitelist: '{obj:github_pr_whitelist}' - github_pr_admin_list: '{obj:github_pr_admin_list}' + .. literalinclude:: _static/github-pr-trigger.example -In the above example note the use of underscores in `github_pr_admin_list` and -`github_pr_admin_list`. +In the above example note the use of underscores in ``github_pr_whitelist``, +``github_pr_admin_list``, and ``github_included_regions``. Using single quotes around variables ==================================== @@ -193,3 +219,169 @@ Example: - build-file: settings: '{settings-file}' file-version: '{file-version}' + + +Variable expansion and Defaults +=============================== + +.. This documentation uses work originally provided by Thanh Ha on +.. the OpenDaylight dev mailing list. +.. https://lists.opendaylight.org/pipermail/dev/2017-October/004184.html + +JJB has a concept called :ref:`Defaults ` which is what JJB will +replace a variable with if unset. Variables can configure dynamic content +in :ref:`job-template ` sections and allow certain options in +these sections to be configurable. + +The section that expands Defaults is :ref:`Job Templates ` no +other sections will expand a default. This documentation will explain how +variables and defaults expansion works and which take precedence in JJB's +variable expansion logic for the following configuration sections. + +- macro +- job-template +- project +- default + +Macro sections +-------------- + +:ref:`Macro ` sections can contain variables but do **NOT** support +default values getting filled in both at the macro definition level and at the +defaults configuration level. :ref:`Macros ` and +:ref:`Job Templates ` can use Macros but any variables defined in +a Macro needs to pass a value or a new variable redefined in the +:ref:`Job Template ` if you want to pass on the configuration. + +So for example if you have a macro that has a '{msg}' variable: + +Example: + +.. code-block:: yaml + + - builder: + name: echo-msg + builders: + - shell: "echo {msg}" + +Any downstream job-templates or macros that use this macro **MUST** pass in a +`msg: Hello` definition or redefine the msg variable `msg: {msg}`. + + +Job Template sections +--------------------- + +:ref:`Job Template ` sections can use defaults in two ways. + +1. Configure the message: + + .. code-block:: yaml + + - job-template: + name: echo-hello-world + builders: + - echo-msg: + msg: 'Hello World' + +2) Re-define '{msg}' variable + + .. code-block:: yaml + + - job-template: + name: echo-message + builders: + - echo-msg: + msg: '{message}' + +In option 2, we redefine the variable msg as `{message}` which a user of the +job-template can now pass into the job their own custom message which is +different than option 1, where we set a static message to pass in. We purposely +redefined the **{msg}** to **{message}** here to show that you do not need to +redefine it with the same name but we could have used the same name `{msg}` in +the template too if we wanted to keep it the same. + +Job Templates can also default a default variable for the variables it defines. + +Example: + +.. code-block:: yaml + + - job-template: + name: echo-message + message: 'Hello World' + builders: + - echo-msg: + msg: '{message}' + +This creates a job template variable called '{message}' which will default to +"Hello World" if the user of the template does not explicitly pass in a message. + +We should be aware of 2 Defaults concepts: + +1. Default as defined in the :ref:`job-template ` +2. Default as defined in a :ref:`defaults ` configuration + (typically defaults.yaml) + +In this case there is a default '{message}' set in the +:ref:`job-template `. JJB will use this default if the user +(project section) does not declare a {message}. + +If we do not declare a default in the :ref:`job-template ` then +JJB will fallback to checking the "defaults configuration". + +This means that the precedence of defaults is as follows: + +1. User-provided +2. Job Template +3. Defaults.yaml + +Project sections +---------------- + +:ref:`Project ` sections define real jobs and pass in variables as +necessary. Projects sections do NOT expand defaults.yaml. So you cannot +configure a setting with {var} in here and expect defaults.yaml to fill it in +for you. Define required configuration here. + +Example: + +.. code-block:: yaml + + - project + name: foo + jobs: + - 'echo-message' + message: 'I am foo' + + +Defaults sections +----------------- + +:ref:`Defaults ` sections are the absolute last thing JJB checks if a +variable is not configured in a job-template and user did not pass in the +variable. JJB will fill in whatever is in the defaults configuration. + +Variable expansion order of precedence seems to be: + +1. job-group section definition +2. project section definition +3. job-template variable definition +4. defaults.yaml variable definition + +.. note:: Defaults set variables in job-templates and are NOT used in Macros. + +global-jjb should not provide job-group definitions and leave it up to users of +global-jjb to create their own as a job-group as a variable defined in a job +group the highest precedence. Global JJB should strive to be purely a +job-template and macro library for downstream consumers. + +Final thoughts +-------------- + +For any :ref:`Basic Job Configuration ` for example "concurrent", "jdk", +"node" etc... we cannot set defaults with the same name as JJB will not expand +them. To use "node" we need to give the variable for that setting a +different name such as "build-node" instead, if we want JJB to perform +expansion for those settings. This issue affects top level job configuration, +it does not appear to affect items below the top level such as calling a +builder, wrapper or parameter.