releng/global-jjb.git
3 years agoMerge "Remove all refs to python2"
Eric Ball [Tue, 30 Mar 2021 21:14:51 +0000 (21:14 +0000)]
Merge "Remove all refs to python2"

3 years agoProvision settings for WS scan 72/67272/2
Jessica Wagantall [Tue, 30 Mar 2021 17:04:40 +0000 (10:04 -0700)]
Provision settings for WS scan

Provision global-settings to replace the
default used by the Unified Agent.

This will allow dependencies to be downloaded the
same way the maven build does.

Issue: RELENG-3430
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I2b751c7d7d966b0b9b07a90405ec61b2254bed6d

3 years agoRemove all refs to python2 63/67263/1
Anil Belur [Tue, 30 Mar 2021 04:13:59 +0000 (14:13 +1000)]
Remove all refs to python2

Python 2.7 is EOL and no longer supported. Some of the
lf-macros and docs still refer to py2 which could
mislead users.

Change-Id: I9f39948cec30fb573f25ee7ecb3bed8bf3600ab9
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoMerge "Update prettier pre-commit to ignore stage error"
Andrew Grimberg [Fri, 26 Mar 2021 14:59:24 +0000 (14:59 +0000)]
Merge "Update prettier pre-commit to ignore stage error"

3 years agoAdd sonar-prescan-script jobs for maven 18/67218/2 v0.62.0
Eric Ball [Thu, 25 Mar 2021 19:58:09 +0000 (12:58 -0700)]
Add sonar-prescan-script jobs for maven

We added these for the more generic Sonar jobs in lf-ci-jobs, but had
not added them for maven. We now have a need for this job type (it can
be used to ensure that a Sonarcloud-compatible version of nodejs is on
the system), so they are being added now.

Issue: IT-21602
Change-Id: Ic32f418a042839ab277dc1b85777d0eef475d620
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoMerge "Pin git review to 1.78"
Andrew Grimberg [Tue, 23 Mar 2021 21:58:05 +0000 (21:58 +0000)]
Merge "Pin git review to 1.78"

3 years agoUpdate WS Unified Agent version 86/67086/1
Jessica Wagantall [Fri, 12 Mar 2021 19:53:19 +0000 (11:53 -0800)]
Update WS Unified Agent version

Update WhiteSource Unified Agent to
version 21.2.1

Issue: RELENG-3430
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I5c4ae6d49b1822a04fad1ce9af955bf9e06d3851

3 years agoUpdate prettier pre-commit to ignore stage error 25/67025/1
Anil Belur [Wed, 10 Mar 2021 07:20:54 +0000 (17:20 +1000)]
Update prettier pre-commit to ignore stage error

Set stages to 'commit' to overide the default_stages.

This fixes the issue seen while using git-commmit:
[error] No matching files. Patterns: .git/COMMIT_EDITMSG

Change-Id: I959744b89e6cb22e84fca9119a1405e354734cbb
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoPin git review to 1.78 14/67014/1
Anil Belur [Tue, 9 Mar 2021 04:34:53 +0000 (14:34 +1000)]
Pin git review to 1.78

The latest version of module trys to look for git hook recursively within
the submodules.

Error:
    Entering 'global-jjb'
    cannot stat '.git/hooks/commit-msg': Not a directory
    fatal: run_command returned non-zero status for global-jjb

Remove workaround that has been resolved in v1.28 and use lf-activate-venv
to install git-review

Issue: RELENG-3435
Change-Id: I67b3730dcdf5a410020c63312897bb67b42017bd
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoMerge "Update the create script with V3 flavors."
Bengt Thuree [Fri, 5 Mar 2021 03:29:41 +0000 (03:29 +0000)]
Merge "Update the create script with V3 flavors."

3 years agoReorder release-job script 37/66937/2
Jessica Wagantall [Tue, 2 Mar 2021 19:25:28 +0000 (11:25 -0800)]
Reorder release-job script

Reorder functions and add function labels to make
release-job.sh easier to read.
Fix "Double quote to prevent globbing" line 34

Issue: RELENG-3376
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: Ic8349f692a50a645a99e884c852013f75332cccd

3 years agoAdd support for artifact releases 47/66647/6
Jessica Wagantall [Wed, 20 Jan 2021 21:10:01 +0000 (13:10 -0800)]
Add support for artifact releases

Currently Dent is the only customer for artifact
releases where they need a particular binary to be
pulled from Snapshots and pushed manually into Releases.

These are artifacts built using make.

Issue: RELENG-3376
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I5a8ec8e2906757b1700505ad10dd2ba8ac844f9a

3 years agoPin cryptography to 3.3.2 35/66835/7 v0.61.1
Anil Belur [Sat, 20 Feb 2021 01:00:50 +0000 (11:00 +1000)]
Pin cryptography to 3.3.2

The latest version of module breaks compatibility with the
latest version of pip.

Error:

 Traceback (most recent call last):
   File "<string>", line 1, in <module>
   File "/tmp/pip-build-vqk6fya9/cryptography/setup.py", line 14, i
   in <module> from setuptools_rust import RustExtension
 ModuleNotFoundError: No module named 'setuptools_rust'

Issue-Id: IT-21538 IT-21509
Change-Id: I83db995ae8321377b861cabcf6ac53485778e471
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoFix reno lint error in sention names 36/66836/1
Anil Belur [Mon, 22 Feb 2021 08:39:51 +0000 (18:39 +1000)]
Fix reno lint error in sention names

Error:
unrecognized section name

Update the version of reno as per the requirements.txt

Change-Id: Iadc07de41934c936fc82147bc8ebcd4f6135764e
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoUpdate pre-commit hook shellcheck to 2.1.5 10/66810/1 v0.61.0
Anil Belur [Thu, 18 Feb 2021 02:06:33 +0000 (12:06 +1000)]
Update pre-commit hook shellcheck to 2.1.5

The workaround with 4d088120467 is no longer
required with the latest 2.1.5 release.

Change-Id: Idf059bb7e184beb7c5ca9c5c5c97f5244ecbcec5
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoMerge "Add workaround for pre-commit 2.10.0"
Anil Belur [Wed, 17 Feb 2021 21:43:16 +0000 (21:43 +0000)]
Merge "Add workaround for pre-commit 2.10.0"

3 years agoAdd workaround for pre-commit 2.10.0 95/66795/1
Eric Ball [Wed, 17 Feb 2021 00:40:46 +0000 (16:40 -0800)]
Add workaround for pre-commit 2.10.0

The jumanjihouse shellcheck hook errors without
additional_dependencies being defined. While there is a PR open to
add this fix, it has not yet been merged and released. Until it is,
this workaround will enable using the latest versions of pre-commit
and the jumanjihouse hooks.
Github PR: https://github.com/jumanjihouse/pre-commit-hooks/pull/80

Change-Id: I0ec16952009128e25cccb6833698bf1bd864af2d
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoMerge "Fix rtd-verify submodule update command"
Andrew Grimberg [Tue, 16 Feb 2021 22:35:58 +0000 (22:35 +0000)]
Merge "Fix rtd-verify submodule update command"

3 years agoFix rtd-verify submodule update command 42/66742/3
Eric Ball [Fri, 5 Feb 2021 00:11:09 +0000 (16:11 -0800)]
Fix rtd-verify submodule update command

The old version is missing the "--init" flag, which causes it to not
add new submodules. This did not show up as an issue until a new
submodule was added in a production environment.
Additionally, the "--recursive" flag has been included to ensure
proper loading of recursive submodules.

Issue: IT-21490
Change-Id: Ie882980d3721ae0e1fea9fa1d3321b86d18be484
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoMerge "Add schema for artifact releases"
Jessica Wagantall [Wed, 10 Feb 2021 22:29:25 +0000 (22:29 +0000)]
Merge "Add schema for artifact releases"

3 years agoRemove python2 tox installation 97/66697/2
Aric Gardner [Fri, 29 Jan 2021 16:33:45 +0000 (11:33 -0500)]
Remove python2 tox installation

Update path to ensure that tox from tox-install.sh is run

Should still work for python2 tox
as pyenv will continue to provide a py2 env

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I568ebf679bbce20b76a0a686699702f81e7e7777

3 years agoSkip Metadata Capture on unsupported Clouds 84/66684/1
Trevor Bramwell [Thu, 28 Jan 2021 17:53:35 +0000 (09:53 -0800)]
Skip Metadata Capture on unsupported Clouds

Updates the 'capture-instance-metadata.sh' script to skip attempting to
capture instance metadata needed for job-cost.sh if the build is being
run on an unsupported cloud or platform.

Change-Id: I8bfea1416d11de1caeed2d414b90513f63de9b6f
Signed-off-by: Trevor Bramwell <tbramwell@linuxfoundation.org>
3 years agoUpdate the create script with V3 flavors. 31/66431/3
Anil Belur [Fri, 8 Jan 2021 23:15:36 +0000 (09:15 +1000)]
Update the create script with V3 flavors.

Issue: RELENG-3329
Change-Id: I46f8b87a60a6f4538f5b76168a51ec242b5963c2
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoMerge "Upgrade packer version 1.6.6"
Anil Belur [Thu, 28 Jan 2021 12:28:09 +0000 (12:28 +0000)]
Merge "Upgrade packer version 1.6.6"

3 years agoFix Sudo Log Ownership for non-'jenkins' users 51/66651/4
Trevor Bramwell [Wed, 27 Jan 2021 22:37:51 +0000 (14:37 -0800)]
Fix Sudo Log Ownership for non-'jenkins' users

If builds are run with a user other than 'jenkins', chown'ing of the
sudoers log will fail. Switching the command to use the current
user (and user's default login group) fixes this.

Note: See chown(1) DESCRIPTION for an explanation of using a colon after
the username while omitting the group

Change-Id: Ia9b96e93a250fd22eb36c94471a06c2e211dc9e3
Signed-off-by: Trevor Bramwell <tbramwell@linuxfoundation.org>
3 years agoAdd schema for artifact releases 38/66638/4
Jessica Wagantall [Wed, 27 Jan 2021 19:24:19 +0000 (11:24 -0800)]
Add schema for artifact releases

Add initial schema for artifact self
releases.

Issue: RELENG-3376
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I78fe9b602b34b9f062599a6ef823f177975f26e4

3 years agoPin pre-commit to v2.9.3 in tox.ini 56/66656/1
Eric Ball [Wed, 27 Jan 2021 23:08:25 +0000 (15:08 -0800)]
Pin pre-commit to v2.9.3 in tox.ini

A change in v2.10.0 of pre-commit introduces an incompatibility with
jumanjihouse/pre-commit-hooks, particularly the shellcheck hook. It
isn't clear whether this is intentional or not, but until either
pre-commit or the jumanjihouse hooks are updated, we need to pin to
the working version of pre-commit.

Change-Id: I2aa74f9310f55603d2e57d83f79c6f1a51726994
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoSkip Job Cost Calulations on Non-Cloud Platforms 42/66642/1
Trevor Bramwell [Wed, 27 Jan 2021 20:05:06 +0000 (12:05 -0800)]
Skip Job Cost Calulations on Non-Cloud Platforms

When 'job-cost.sh' is ran on a VM not initialized with cloud-init, or
running in OpenStack, attempts to read '/run/cloud-init/result.json'
will fail causing the job to become unsable.

This change adds a check for any relevant files to the beginning of
'job-cost.sh' in order to skip the calculation on VMs not running on
either AWS or OpenStack.

Change-Id: I878adee1ba3ceeffa574a1b1c7ae103811344123
Signed-off-by: Trevor Bramwell <tbramwell@linuxfoundation.org>
3 years agoAdd release examples GitHub 14/66614/5
Jessica Wagantall [Tue, 26 Jan 2021 21:17:49 +0000 (13:17 -0800)]
Add release examples GitHub

Add release-job examples for Github
projects.

Issue: RELENG-3346
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I41ff49f1d36f724fae060f243eed0521a1fc8859

3 years agoAdd self release jobs for GitHub 68/66568/6
Jessica Wagantall [Wed, 20 Jan 2021 21:55:55 +0000 (13:55 -0800)]
Add self release jobs for GitHub

Add self release support for GitHub based
repos.

Issue: RELENG-3366
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I84e56c8f63459bf0ecd69f6521a580383df1d6fd

3 years agoMerge "Add regex for common-packer var and templates" v0.60.5
Eric Ball [Mon, 25 Jan 2021 20:21:52 +0000 (20:21 +0000)]
Merge "Add regex for common-packer var and templates"

3 years agoMerge "Fix release job script"
Andrew Grimberg [Mon, 25 Jan 2021 18:13:50 +0000 (18:13 +0000)]
Merge "Fix release job script"

3 years agoRemove python 2.7 support 00/66600/1 v0.60.4
Anil Belur [Mon, 25 Jan 2021 04:00:59 +0000 (14:00 +1000)]
Remove python 2.7 support

As per the deprecation notice python 2.7 is not long supported.
This causing job failures since the dependencies install
are not maintained.

DEPRECATION: Python 2.7 reached the end of its life on January 1st,
2020. Please upgrade your Python as Python 2.7 is no longer
maintained. pip 21.0 will drop support for Python 2.7 in January
2021. More details about Python 2 support in pip can be found at
https://pip.pypa.io/en/latest/development/release-process/#python-2-support
pip 21.0 will remove support for this functionality.

Issue: RELENG-3375
Change-Id: I48cc4b91b9916918d222794534db5ade9b58dda0
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoAdd regex for common-packer var and templates 99/66599/2
Anil Belur [Sat, 23 Jan 2021 09:55:11 +0000 (19:55 +1000)]
Add regex for common-packer var and templates

Add regex to trigger packer jobs when common-packer templates
are updated.

Change-Id: Ie9d79e0c2e4e525cd19c3ab10066b5df5ae65742
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoUpgrade packer version 1.6.6 98/66598/1
Anil Belur [Sat, 23 Jan 2021 07:43:52 +0000 (17:43 +1000)]
Upgrade packer version 1.6.6

Change-Id: I3d5c36e6e9452f09784883e86eab4eb37dccf13f
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoFix release job script 97/66597/1
Anil Belur [Sat, 23 Jan 2021 07:33:06 +0000 (17:33 +1000)]
Fix release job script

Fix the release job script to handle LOG_DIR unbound variable
and condition to check if the LOGS_SERVER or CDN_URL is being used.

Issue: IT-21396
Change-Id: I3960a49aa12858af2ed4f47a1ec714ff793b70fd
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoMerge "Revert "Remove the -only CLI option for packer build"" v0.60.3
Anil Belur [Sat, 23 Jan 2021 02:24:04 +0000 (02:24 +0000)]
Merge "Revert "Remove the -only CLI option for packer build""

3 years agoMerge "Revert "Pass the builder type to verify jobs""
Anil Belur [Sat, 23 Jan 2021 02:23:15 +0000 (02:23 +0000)]
Merge "Revert "Pass the builder type to verify jobs""

3 years agoRevert "Remove the -only CLI option for packer build" 94/66594/1
Anil Belur [Sat, 23 Jan 2021 01:41:25 +0000 (11:41 +1000)]
Revert "Remove the -only CLI option for packer build"

This reverts commit c6d9571b076faf5d02305bd93faeda7bf23fe766.

Based on the input provided on the Gihub packer issue
hard-coded for our parser to be able to decode the rest of the config
properly."

Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Change-Id: I58debd1a961b92c3f7705a3f65a8bc883722596a

3 years agoRevert "Pass the builder type to verify jobs" 93/66593/1
Anil Belur [Sat, 23 Jan 2021 01:37:14 +0000 (11:37 +1000)]
Revert "Pass the builder type to verify jobs"

This reverts commit e0a9d10d3d6b488ed3a3b00050b34d9faee4a6f8.

Based on the input provided on the Gihub packer issue
hard-coded for our parser to be able to decode the rest of the config
properly."

Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Change-Id: If7162ced328d58b24ab9339f7e81d7dff76c90bb

3 years agoMerge "Add repo tag support for GitHub"
Jessica Wagantall [Fri, 22 Jan 2021 20:10:50 +0000 (20:10 +0000)]
Merge "Add repo tag support for GitHub"

3 years agoPass the builder type to verify jobs 76/66576/2 v0.60.2
Anil Belur [Fri, 22 Jan 2021 02:04:03 +0000 (12:04 +1000)]
Pass the builder type to verify jobs

The default builder type must be passed to packer verify jobs. For
packer jobs build on aws, set the `packer-builder: aws` in downstream
CI-management.

Change-Id: I599d167a957189584ef5b796c049d82e5a85ec7a
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoAdd repo tag support for GitHub 70/66570/3
Jessica Wagantall [Wed, 20 Jan 2021 22:54:19 +0000 (14:54 -0800)]
Add repo tag support for GitHub

Rename "tag-gerrit-repo" to "tag-git-repo" in
preparation to support self releases for Github
based projects.

Issue: RELENG-3368
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I6ceb6675a2f37cbfd1d008046aecb734665db6d3

3 years agoRemove the -only CLI option for packer build 29/66529/2 v0.60.1
Anil Belur [Sat, 16 Jan 2021 05:21:59 +0000 (15:21 +1000)]
Remove the -only CLI option for packer build

Remove the -only CLI option for packer build

The #CR 90dd675df482a8 submitted with common-packer v0.7.0
fails to build packer jobs because of a bug in the packer build CLI.

The `-only` option packer build CLI does not set the cloud provider
name/type as per the documentation. The alternative is to use an environment
variable that is passed through the packer job and can be reference in the
packer templates.

Issue: JIRA IT-21300
Github Issue: https://github.com/hashicorp/packer/issues/10495
Ref: https://gerrit.linuxfoundation.org/infra/c/releng/common-packer/+/66528
Change-Id: I97bc479ddac9a8f1f0dc83201735b0a084a8b3e4
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoMerge "Handle S3_BUCKET & CDN_URL in release jobs" v0.60.0
Andrew Grimberg [Wed, 30 Dec 2020 23:07:43 +0000 (23:07 +0000)]
Merge "Handle S3_BUCKET & CDN_URL in release jobs"

3 years agoHandle S3_BUCKET & CDN_URL in release jobs 15/66115/9
Anil Belur [Sun, 22 Nov 2020 22:26:30 +0000 (08:26 +1000)]
Handle S3_BUCKET & CDN_URL in release jobs

Fix release merge jobs to work with AWS S3_BUCKET when CDN_URL
is set.

Log shipping is being migrated from Nexus2 log server to AWS S3
Buckets. This requires the release job scripts to handle CDN_URL
and set the correct value for LOG_SERVER.

Issue: RELENG-3307
Change-Id: I508dd15508d9c9dd3741edf220d64526abed644f
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoMerge "Add pipeline-verify jobs to lint and check agents"
Andrew Grimberg [Mon, 28 Dec 2020 19:56:04 +0000 (19:56 +0000)]
Merge "Add pipeline-verify jobs to lint and check agents"

4 years agoAdd pipeline-verify jobs to lint and check agents 58/66258/4
Eric Ball [Wed, 9 Dec 2020 16:19:51 +0000 (08:19 -0800)]
Add pipeline-verify jobs to lint and check agents

This adds a simple pipeline verification job that will lint any
pipelines found, and check to make sure that they are not set up to
run on master.

Issue: RELENG-2185
Change-Id: I76d400d145857887a6d28d7c2a9869ec4acf414d
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
4 years agoMerge "Use write-good release tag v1.0.4"
Andrew Grimberg [Mon, 14 Dec 2020 17:55:38 +0000 (17:55 +0000)]
Merge "Use write-good release tag v1.0.4"

4 years agoCreate the archives directory in deploy-logs 06/66206/2
James E. Blair [Wed, 2 Dec 2020 22:24:39 +0000 (14:24 -0800)]
Create the archives directory in deploy-logs

The 'lftools deploy archives' command expects $WORKSPACE/archives
to already exist.  Normally it would be created by lf-infra-sysstat,
but to support the use of the deploy-logs macro in jobs which do not
run sysstat (and generally to make this macro entirely self-contained),
ensure the archives directory exists at the start of the script.

Signed-off-by: James E. Blair <jeblair@redhat.com>
Change-Id: Id90c77c3f2a6a9590b9b17f9223279d21066a3cf

4 years agoActivate virtual env for logs-deploy 59/66259/3 v0.59.0
Jessica Wagantall [Wed, 9 Dec 2020 19:08:39 +0000 (11:08 -0800)]
Activate virtual env for logs-deploy

Activate lftools env before processing the
logs-deploy script.

This will allow us to fetch the latest version of
pyOpenSSL and cryptography to resolve the following
issue:
https://github.com/pyca/pyopenssl/issues/973

Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: Ib8872cad64630573cf6209ce1012dd5d5eb7cf69

4 years agoUse write-good release tag v1.0.4 45/66245/1
Anil Belur [Tue, 8 Dec 2020 09:35:13 +0000 (19:35 +1000)]
Use write-good release tag v1.0.4

Issue: RELENG-3257
Change-Id: Ibf51a3f144ea9ce4c235659e7cd51661deddfb8f
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoSwitch write-good to upstream hook 69/65869/13
Anil Belur [Wed, 28 Oct 2020 13:43:45 +0000 (23:43 +1000)]
Switch write-good to upstream hook

Upstream PR [1.] is merged fixes issues with the
write-good npm dependencies.

Issue: RELENG-3257
[1.] https://github.com/btford/write-good/pull/126
Change-Id: I12ca7d37785da093be977072ceeb4fc226a1cd39
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoHandle "name" in openstack.name 69/66169/2
Aric Gardner [Wed, 25 Nov 2020 21:48:09 +0000 (16:48 -0500)]
Handle "name" in openstack.name

I thought this was 'cattle' everywhere.

Issue-Id: RELENG-2951
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I859aa4a53b73486f9c03869f05e3f45d7f951ec3

4 years agoMake sandbox a switch 90/66090/1
Aric Gardner [Tue, 17 Nov 2020 19:45:48 +0000 (14:45 -0500)]
Make sandbox a switch

This was Andys suggestion previously
it makes sense.

Issue-Id: RELENG-3036
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I106737c6c766fb566a2e0e74436f5bb5da36eac5

4 years agoconfigure global env vars 75/66075/9
Aric Gardner [Mon, 16 Nov 2020 19:59:20 +0000 (14:59 -0500)]
configure global env vars

create_jenkins_global_env_vars.py
thakes jenkins_global_envvars.yaml from
global-vars-production|sandbox.sh
and puts them in the casc.d directory
(or a custom one)

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I95c3f7f268ba7146198c791bba5378e233f9b913

4 years agoPin idna to 2.9 to resolve package incompatibility 98/65998/2 v0.58.1
Anil Belur [Tue, 10 Nov 2020 04:08:21 +0000 (14:08 +1000)]
Pin idna to 2.9 to resolve package incompatibility

Issue: IT-21003
Change-Id: I93ce5e108370b8ef490141f49d329192e8f07c02
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoOverride semver regex for projects 77/65977/10 v0.58.0
Anil Belur [Wed, 4 Nov 2020 23:36:59 +0000 (09:36 +1000)]
Override semver regex for projects

Allow project to override the semver regex to use versions
as exception that do not conform with semver standards.

Issue: IT-20974
Change-Id: If0a13ecf2394183d6e22e5348dff2a6f0a99c37a
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoMerge "Create managed config files yaml from config dir"
Aric Gardner [Tue, 3 Nov 2020 19:02:14 +0000 (19:02 +0000)]
Merge "Create managed config files yaml from config dir"

4 years agoVexxhost has added new flavors 27/65927/1
Aric Gardner [Fri, 30 Oct 2020 19:03:15 +0000 (15:03 -0400)]
Vexxhost has added new flavors

cloud odlci
openstack flavor list \
-c="ID" -c="Name" \
-f value | \
awk '{print "flavors[\"",$2,"\"]=","\"",$1,"\""}'| \
sort | sed 's/ //g'

Of note, these new flavors
were not seen when checking oranci

Issue-id: RELENG-3259
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I69a90129598e035ed196c3a174bcad064a4a8cbc

4 years agoCreate managed config files yaml from config dir 89/65789/5
Andrew Grimberg [Fri, 16 Oct 2020 21:53:04 +0000 (14:53 -0700)]
Create managed config files yaml from config dir

This script requires ruamel.yaml to be installed in the environment that
it is run in

Will take a configuration tree and build up the JCasC managed files
section and output to stdout the resultant yaml

Expected configuration of the tree is as follows:

top
|- config_type
    |- file-id
        |- config-params.yaml
        |- content
        |- ??CredentialMappings.yaml
        |- ??CredentialMappings.sandbox.yaml
...

The ??CredentialMappins.yaml will be used for production systems and the
sandbox version for sandbox systems (pass a -s or --sandbox)

Change-Id: I196e99f1db40223a29e892eb86efd70ecaa328f0
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
4 years agoPreliminary jcasc yaml generator 72/65672/15
Aric Gardner [Thu, 1 Oct 2020 20:07:53 +0000 (16:07 -0400)]
Preliminary jcasc yaml generator

generator that takes values from a projects builder/jenkins-config/*.cfg
files and creates a yaml file compatible with jcasc

Idea is to replace the groovy scripts in this same directory
with something less brittle.

A few problems, we can't validate the schema.
as none has been written.

so loading whatever this generates will be risky.

example output:
https://gist.github.com/Aricg/4366e4e54e6ab63905ecb7aaac56646f

Issue-Id: RELENG-3035
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I8fbb6a49f8d13ba36c32b30469410c3a69c0f5f0

4 years agoUpdate the version of prettier being used 78/65878/2
Andrew Grimberg [Wed, 28 Oct 2020 17:29:43 +0000 (10:29 -0700)]
Update the version of prettier being used

Per https://github.com/prettier/prettier/issues/8925 prettier has
created a new pre-commit repository for pre-commit integration. This
repository also uses a slightly different tag format.

Lastly, we're updating to the latest version version available at this
time

Adds a .prettierignore to keep it from reformatting the
jenkins-init-scripts/README that shouldn't be touched

Change-Id: I0a50fc4fc372f43c6cd323aa1d6a2ec2054aeeda
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
4 years agoMerge "Fix rtd-verify.sh to better validate submodules" v0.57.4
Eric Ball [Fri, 2 Oct 2020 17:45:03 +0000 (17:45 +0000)]
Merge "Fix rtd-verify.sh to better validate submodules"

4 years agoFix rtd-verify.sh to better validate submodules 78/65678/2
Eric Ball [Fri, 2 Oct 2020 00:52:22 +0000 (17:52 -0700)]
Fix rtd-verify.sh to better validate submodules

rtd-verify.sh was using "git submodule" commands to validate
submodules, but Jenkins reads the .gitmodules file and executes a
"git config" command for each submodule. Because of this, if a bad
submodule was added to .gitmodules, it would pass verify but cause
failures on subsequent builds after it was merged.

This change closes that loophole by reading the .gitmodules file, and
then running through the same "git config" command for each submodule
that Jenkins runs when pulling in the main branch.

Issue: RELENG-2932
Change-Id: Ic0e17d23a4763bd52172dad6b98146444bf4c165
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
4 years agoFix protect images 06/65606/5
Aric Gardner [Wed, 23 Sep 2020 18:52:41 +0000 (14:52 -0400)]
Fix protect images

Projects that do not have images in jjb
are failing this step of the build

fix code so we don't need to mess with set -u
or shellcheck disable
also only merge arrays if needed

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I119b68e344a34bb5bbe8baea2cc310497b7ee629

4 years agoMerge "Add release note for capture-instance-metadata"
Andrew Grimberg [Wed, 23 Sep 2020 17:54:17 +0000 (17:54 +0000)]
Merge "Add release note for capture-instance-metadata"

4 years agoDo not fail config management verify if no cloud 04/65604/2
Andrew Grimberg [Wed, 23 Sep 2020 16:39:20 +0000 (09:39 -0700)]
Do not fail config management verify if no cloud

Not having any openstack clouds defined is not a failure condition for
the config management. We already support skipping configuration (the
merge job) if there isn't a clouds config, but for some reason our
verify is trying to force that. This fixes the problem.

Change-Id: Ic48b91ef07808403b6214c6272bcd930e21a0af8
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
4 years agoAdd release note for capture-instance-metadata 03/65603/1
Andrew Grimberg [Wed, 23 Sep 2020 16:21:23 +0000 (09:21 -0700)]
Add release note for capture-instance-metadata

In our rush to get a fix for the capture-instance-metadata script not
properly handling AWS instances in change
59faa9977d726858902dcf513999616aaf6de7a5 we failed to put a release note
in place.

Issue: RELENG-3163
Change-Id: I17e7ff8a87ae8f23ee994c4312caa91903e54588
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
4 years agoFix capture-instance-metadata.sh 80/65580/3
Aric Gardner [Tue, 22 Sep 2020 20:51:59 +0000 (16:51 -0400)]
Fix capture-instance-metadata.sh

bios_vendor file now says Xen, no amazon string
to match on any longer.

cat /sys/devices/virtual/dmi/id/bios_vendor
Xen

however result.json has the info we need.

/run/cloud-init/result.json
{
 "v1": {
  "datasource": "DataSourceEc2Local",
  "errors": []
 }
}

Issue-Id: RELENG-3163

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Iece691579622608616a6abc8d27b9ca7653ad9ae

4 years agoNeed to export SET_JDK_VERSION 12/65512/1
Aric Gardner [Thu, 17 Sep 2020 16:01:38 +0000 (12:01 -0400)]
Need to export SET_JDK_VERSION

This var is needed by the exteral script
must export so that it is inherited.

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I627e6ecb369a5f0dab1f047663d24ac8a07b6330

4 years agoUse java11 for sonar cloud upload 83/65483/6
Aric Gardner [Mon, 14 Sep 2020 20:47:42 +0000 (16:47 -0400)]
Use java11 for sonar cloud upload

default to openjdk11
before calling sonarcloud upload.

sonar cloud upload will stop working
with java8 on october 1st

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I8c321b053a86e1f15149552af25b775353051444

4 years agoMerge "Remove tox from python-tools-install." v0.57.3
Anil Belur [Thu, 10 Sep 2020 04:09:02 +0000 (04:09 +0000)]
Merge "Remove tox from python-tools-install."

4 years agoMerge "The openstack-cron job email report."
Andrew Grimberg [Wed, 9 Sep 2020 20:36:48 +0000 (20:36 +0000)]
Merge "The openstack-cron job email report."

4 years agoRemove tox from python-tools-install. 00/65400/2
Aric Gardner [Wed, 9 Sep 2020 16:02:15 +0000 (12:02 -0400)]
Remove tox from python-tools-install.

If users need a system pip install of tox
they can include tox-install.
However, unless they need pyenv
I would suggest they use lf-env.

Order of system pip install of tox matters,
whichever one is installed most recently
is the one that is going to be called.
removing tox from python tools install
allows the install of tox in tox-install
to not be skipped, and in this way
we can ensure that the py3 version
is the once most recently installed

Issue-Id: RELENG-3189
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Ie904f42821f6598a389854e8876448237cfac3dd

4 years agoMerge "Improve python-tools-install.sh alpine compat"
Andrew Grimberg [Thu, 3 Sep 2020 17:16:43 +0000 (17:16 +0000)]
Merge "Improve python-tools-install.sh alpine compat"

4 years agoworkaround for bad tox on centos builders 95/65295/3 v0.57.2
Aric Gardner [Thu, 3 Sep 2020 00:02:57 +0000 (20:02 -0400)]
workaround for bad tox on centos builders

system tox is py2.7 needs updated pip and setuptools.
this needs to be fixed in packer. tox is py3 on my machine

tox --version on builders shows
imported from /usr/lib/python2.7
should be
imported from /usr/lib/python3.6

also as tox  is called from python2 and needs the zipp package

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I1f7230a8483ae3c33a574056492fb283d3f8eac2

4 years agoImprove python-tools-install.sh alpine compat 25/65325/1
Eric Ball [Thu, 3 Sep 2020 00:29:32 +0000 (17:29 -0700)]
Improve python-tools-install.sh alpine compat

Improve alpine compatibility by changing the mktemp call in
python-tools-install.sh to be compatible with both GNU mktemp as well
as BusyBox mktemp (which is used in alpine).

Change-Id: I7123d8928415c163ec26540c42fa0a96108b84e4
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
4 years agoCheck if ${NOMAD_DC} is undefined or unset 74/65274/5 v0.57.1
Anil Belur [Wed, 2 Sep 2020 04:27:35 +0000 (14:27 +1000)]
Check if ${NOMAD_DC} is undefined or unset

The condition in the commit 31a4b7fc00fe90494b6efb512ad0f1db02481ff3
should instead check when the environment variable $NOMAD_DC is not
defined or unset, this avoids the script from exiting without capturing
instance metadata.

Issue: RELENG-3142
Change-Id: I80240d77abfacb59c81e41e63629468350e24b7b
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoTox is called from system python2 85/65285/3
Aric Gardner [Wed, 2 Sep 2020 17:01:19 +0000 (13:01 -0400)]
Tox is called from system python2

which tox
/usr/bin/tox
tox --version
3.14.5 imported from
/usr/lib/python2.7/site-packages/tox/__init__.pyc

if $PYTHON=python3 we are not updaing tox
to the lastest version.

for some reason running this old tox
broke things. this appears to fix.

Also revert the lfenv stuff that we
don't want in tox-run

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I2f964ec5123bd4a3b5fcc75f3d1ba38c0810cd8a

4 years agoFix Python env issues - Part2 49/65249/4
Jessica Wagantall [Mon, 31 Aug 2020 21:13:22 +0000 (14:13 -0700)]
Fix Python env issues - Part2

This is a follow up for commit
f4f795bda67897b1b1e527e99e0b5f627ca1a4db
The release-job.sh script needed the same fix

Issue-ID: IT-20528
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I88fbad702fcd329c062a1f86a415ba99325882ab

4 years agoThe openstack-cron job email report. 91/65191/11
Aric Gardner [Tue, 25 Aug 2020 18:28:47 +0000 (14:28 -0400)]
The openstack-cron job email report.

The openstack-cron job should report to LF staff when it fails

Upgrade notes:
Will require all projects to add:
failure-notification: "releng+PROJECTCODE@linuxfoundation.org"
failure-notification-prefix: "[releng]"
to jjb/defaults.yaml

Issue-ID: IT-20409
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I0f4f9ad5f6e810898b9815d7ff66784557e2fc71

4 years agoFix pypi build and upload. 39/65239/1
Aric Gardner [Mon, 31 Aug 2020 17:39:04 +0000 (13:39 -0400)]
Fix pypi build and upload.

No need for job to create its own
venv like this.
virtualenv -p python3 /tmp/pypi

this does not update virtualenv first
also it was failing, I think due to
needing a readline package update.

python3 -m venv is preferred.
however we have lf-env.sh, which does things properly.
se we are going to use it.

Issue-ID: IT-20528
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: If2b8369296256368659c41e9e77231d4442f55bd

4 years agoRun Sonar scans using JDK11. 01/65201/2 v0.57.0
Jessica Wagantall [Tue, 25 Aug 2020 22:15:42 +0000 (15:15 -0700)]
Run Sonar scans using JDK11.

java-version will only set java for the maven build
part of the job, the sonar scan will use
sonarcloud-java-version which is set to openjdk11.

Projects not compatible with JDK11 will be able to
run their build with java-version set to their JDK
preference.
https://sonarcloud.io/documentation/user-guide/move-analysis-java-11/

Issue: RELENG-3176
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: Ida96043317da7671034416df6a438350041f4731

4 years agoOptionally set docker-tag to stream 63/65163/1
Jessica Wagantall [Fri, 21 Aug 2020 16:45:15 +0000 (09:45 -0700)]
Optionally set docker-tag to stream

If container-tag-method: stream, docker jobs will
use the value of the variable stream as docker tag.

Issue: IT-20480
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: Ib595b6bf9439a45985cc41233f588cb0e56b7920

4 years agoMerge "Run packer against a specific builder"
Andrew Grimberg [Thu, 20 Aug 2020 20:23:33 +0000 (20:23 +0000)]
Merge "Run packer against a specific builder"

4 years agoMerge "Fix job cost amazon exception"
Andrew Grimberg [Thu, 20 Aug 2020 19:57:49 +0000 (19:57 +0000)]
Merge "Fix job cost amazon exception"

4 years agoFix job cost amazon exception 26/65126/3
Aric Gardner [Mon, 17 Aug 2020 20:03:58 +0000 (16:03 -0400)]
Fix job cost amazon exception

bios_vendor file now says Xen, no amazon string
to match on any longer.

cat /sys/devices/virtual/dmi/id/bios_vendor
Xen

however result.json has the info we need.

/run/cloud-init/result.json
{
 "v1": {
  "datasource": "DataSourceEc2Local",
  "errors": []
 }
}

Issue-Id: RELENG-3163
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I57e27fe5bb2135e738b33c0f6de78f1759bcb93d

4 years agoRun packer against a specific builder 89/65089/3
Thanh Ha [Fri, 14 Aug 2020 14:09:35 +0000 (10:09 -0400)]
Run packer against a specific builder

As preparation to enable the ability to configure multiple builders
in a single packer template, we need to enable the ability in the
job to select which builder to run.

This patch sets up a new variable "packer-builder" which is used to
select the builder definition in the templates file.

Defaulting to "openstack" as that's the most common use case at the
moment.

Issue: RELENG-1052
Change-Id: Ib33a00f38a0005af69d52e7697322a2e84f6c674
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
4 years agoFix email notifications for lfdocs-conf releases 81/65081/5
Jessica Wagantall [Tue, 18 Aug 2020 17:55:54 +0000 (10:55 -0700)]
Fix email notifications for lfdocs-conf releases

Unfortunatelly the name of the repo does not match the
configuration in the code and we need to make sure the
email notification announces the corrected name of the
project which should be lfdocs-conf.
Add a new PROJECT_SLUG variable to use in case the name
of the repo is different from its configuration.

Issue: IT-20404
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I0dc7b0088851ce8abf8238bc63e8e5bdbfc2ea9b

4 years agoFix parallel tox mode when using bool jjb cfg 09/65109/1
Thanh Ha [Sat, 15 Aug 2020 01:08:45 +0000 (21:08 -0400)]
Fix parallel tox mode when using bool jjb cfg

JJB takes in the word "true" unquoted as a boolean and converts it to
the Python "True" boolean causing the bash script to ultimately receive
a string that starts with an uppercase letter. Since we are comparing
against a lowercase "true" we should ensure the string is lowercased
before comparison.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: If85c4411c73c18c931a339592a0dd352f574131a

4 years agoRevert "Fix formatting in script" 33/65033/1 v0.56.2
Thanh Ha [Thu, 13 Aug 2020 01:22:20 +0000 (21:22 -0400)]
Revert "Fix formatting in script"

This reverts commit 8ac864b735534ac4c8646aa9141dac1c51147912.

Change-Id: Ibd0d79eb81d486f17d27de78c9d5c64162e8c55e
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
4 years agoFix formatting in script 32/65032/1 v0.56.1
Anil Belur [Thu, 13 Aug 2020 00:18:40 +0000 (10:18 +1000)]
Fix formatting in script

The revert 0c66e6ab1d80770ca7060237b436190f9fa274a0
did not take care of the formatting.

Change-Id: I8efe5aed70d902a959c6653b56429dc9774276b6
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoRevert "Minor cleanup of openstack-protect script" 25/65025/5
Thanh Ha [Wed, 12 Aug 2020 19:07:32 +0000 (15:07 -0400)]
Revert "Minor cleanup of openstack-protect script"

The "Minor cleanup" ended up breaking the script entirely as the
image list array is not correctly created causing invalid image
name to be passed to the openstack command.

Also adds some code to strip double quotes from the prefix and suffix
if there are any.

This reverts commit b03a1792362654ad8fcd15092a64ed00f8509209.

Change-Id: Id4f734113d6dede7e7603f85784b46d164b91392
Co-authored-by: Aric Gardner <agardner@linuxfoundation.org>
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
4 years agoCorrect section type in release notes 81/64981/3
Anil Belur [Tue, 11 Aug 2020 04:21:42 +0000 (14:21 +1000)]
Correct section type in release notes

The CR 762400ee2c77 was submitted with a wrong
section type, the correct one should be fixes.

Ref: https://docs.openstack.org/reno/latest/user/usage.html#editing-a-release-note
Change-Id: I47e2b4532d5cf1ea94dffb5cf2b9fb26d3d0a250
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoFix image update script to handle $image_type 73/64973/4 v0.56.0
Anil Belur [Sun, 9 Aug 2020 03:06:12 +0000 (13:06 +1000)]
Fix image update script to handle $image_type

Issue: RELENG-3153
Change-Id: I82bab26605733ea3b65179f1107724f5b0a1ae06
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoMerge "Migrate write-good hook to pre-commit"
Anil Belur [Sun, 9 Aug 2020 04:01:32 +0000 (04:01 +0000)]
Merge "Migrate write-good hook to pre-commit"

4 years agoMerge "docs: Move release notes out of "Guides" section"
Anil Belur [Sun, 9 Aug 2020 04:00:53 +0000 (04:00 +0000)]
Merge "docs: Move release notes out of "Guides" section"

4 years agoMigrate write-good hook to pre-commit 67/64967/8
Thanh Ha [Sat, 8 Aug 2020 13:07:57 +0000 (09:07 -0400)]
Migrate write-good hook to pre-commit

This is a final patch to remove our dependency on the Coala platform
and use only pre-commit moving forward.

This patch also resolves write-good lint issues that were not caught
by the Coala version of write-good due to file ignore rules that
excluded several files from being scanned. With pre-commit we can
create separate rules to exclude more specific scans while no excluding
the file from scans entirely.

Issue: RELENG-2642
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Ic9a48b2964ff5c689a01f3fd611e69ef9525e055

4 years agodocs: Move release notes out of "Guides" section 66/64966/1
Eric Ball [Fri, 7 Aug 2020 12:43:41 +0000 (05:43 -0700)]
docs: Move release notes out of "Guides" section

The text in the "Release Notes" section of the docs index was
outdated (we have not used the relnotes directory in quite some
time), and the actual Release Notes links were under the "Guides"
section.
There is also a correction to put paths into monospace rather than
italics.

Change-Id: I3096bdce7ec666b762ab4be5811ae4c7ec2ffb55
Signed-off-by: Eric Ball <eball@linuxfoundation.org>