releng/global-jjb.git
2 years agoFeat: Upgrade NexusIQ Client 1.140.0-01. 62/70362/1 v0.78.0
Anil Belur [Tue, 5 Jul 2022 22:52:41 +0000 (08:52 +1000)]
Feat: Upgrade NexusIQ Client 1.140.0-01.

Issue-ID: RELENG-4306
Change-Id: Ic48cdfc7303e20a1070bfd35e26f22649520b6f0
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
2 years agoFix: Remove hosts file creation in sigul-install 68/70268/1 v0.77.4
Eric Ball [Wed, 8 Jun 2022 13:38:28 +0000 (06:38 -0700)]
Fix: Remove hosts file creation in sigul-install

Our Sigul bridges now have publicly accessible DNS names, so it is no
longer necessary to create an entry in the hosts file.

Issue: RELENG-4269
Change-Id: I8417747b598d4fad3bfef192ccf1056899ffdf0a
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
2 years agoFix: Update script and Dockerfile 01/70201/4 v0.77.3
Anil Belur [Tue, 31 May 2022 23:54:05 +0000 (09:54 +1000)]
Fix: Update script and Dockerfile

Fix URL path indent, add a default ARG to the Dockerfile to
remove WARNING. Set the .asc files permissions to jenkins after
the sigul has signed the files.

Issue-ID: IT-23826
Change-Id: Idfa32e797320e7a580516a4b137202faf5bd37b0
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
2 years agoMerge "Fix: Sign artifacts on CentOS Stream 8/9" v0.77.2
Andrew Grimberg [Tue, 31 May 2022 16:00:35 +0000 (16:00 +0000)]
Merge "Fix: Sign artifacts on CentOS Stream 8/9"

2 years agoFix: Sign artifacts on CentOS Stream 8/9 91/70191/4
Anil Belur [Thu, 26 May 2022 23:58:32 +0000 (09:58 +1000)]
Fix: Sign artifacts on CentOS Stream 8/9

Update the sigul-sign-dir.sh to sign artifacts using docker.
The docker image is built on CentOS Streams 8/9. The newer version
of sigul 1.1.1 available for CentOS 8 is not backwords compatible
with the version of sigul on CentOS 7.

As a temporary workaround build a CentOS7 docker image with
sigul installed and use it for signing artificats on platforms
where sigul is not readly available.

Issue-ID: IT-23826
Change-Id: Ie22e23240f7fe388219c0afc4d4c229f390efa9c
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
2 years agoFix: bashate E006 warnings for lines > 80 chars 92/70192/1
Anil Belur [Fri, 27 May 2022 03:30:16 +0000 (13:30 +1000)]
Fix: bashate E006 warnings for lines > 80 chars

Change-Id: Id0ddfbe03e2853019a53b3a76af7f63256c05582
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
2 years agoFix: Pin openstacksdk<0.99 85/70185/2 v0.77.1
Anil Belur [Tue, 24 May 2022 21:29:25 +0000 (07:29 +1000)]
Fix: Pin openstacksdk<0.99

This fixes the builder-openstack-cron job.

With the recent updates on the pypi dependencies,
some of attributes of various resources may have been
named differently to follow SDK attribute naming convention.

Issue-ID: RELENG-4243
Change-Id: Iadcfe9ddc9645ad6743bde9498ec85599f6fcf06
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
2 years agoChore: Update pre-commit hooks 64/70164/1 v0.77.0
Andrew Grimberg [Thu, 19 May 2022 16:55:43 +0000 (09:55 -0700)]
Chore: Update pre-commit hooks

github.com/pre-commit/pre-commit-hooks: v4.0.1 -> v4.2.0
github.com/pre-commit/mirrors-prettier: v2.4.1 -> v2.6.2
github.com/jorisroovers/gitlint: v0.16.0 -> v0.17.0
github.com/jumanjihouse/pre-commit-hooks: 2.1.5 -> 2.1.6
github.com/btford/write-good: v1.0.4 -> v1.0.8

Change-Id: I26bff86b5b0f9c6c624b9b2ba0400b6dadb59c2c
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
2 years agoChore: Upgrade base version of JJB to 4.1.0 61/70161/1
Andrew Grimberg [Wed, 18 May 2022 17:58:24 +0000 (10:58 -0700)]
Chore: Upgrade base version of JJB to 4.1.0

JJB has been pinned to v2.8.0 for nearly 2 years. It's time move on,
particularly since we've been hard at work dropping support for Python
2.7. This also brings in needed fixes to support recent changes in how
many plugins for Jenkins are getting versioned.

Change-Id: I1ee78f1b7b5a8fa019cb6fe5a322d02ff8d114e2
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
2 years agoFix: Add maven pre and post build scripts 95/70095/1 v0.76.4
Jessica Wagantall [Wed, 27 Apr 2022 19:09:12 +0000 (12:09 -0700)]
Fix: Add maven pre and post build scripts

Optionally run a script before and/or after maven goals.
This will help add dependencies and post process builds with more
flexibility to the project's needs.

Issue: IT-23957
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I7075972c796a2cac17d9ad7ec9e99be1699354c3

2 years agoMerge "Fix: Isolate SBOM PATH flag"
Andrew Grimberg [Wed, 27 Apr 2022 18:17:11 +0000 (18:17 +0000)]
Merge "Fix: Isolate SBOM PATH flag"

2 years agoFix: Isolate SBOM PATH flag 88/70088/1
Jessica Wagantall [Wed, 27 Apr 2022 02:30:50 +0000 (19:30 -0700)]
Fix: Isolate SBOM PATH flag

SBOM's path flag does not work as expected. We need to introduce a new
flag called SBOM_PATH to isolate the path where SBOM is going to be
extracted to and executed from. By default this is set to $WORKSPACE but
some projects need to execute the sbom from a different location in their
code. See https://github.com/opensbom-generator/spdx-sbom-generator/issues/227

Issue: RELENG-4213
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: Ibaaf4f89035f8aa9a773e199cf0cacb6c1e14947

2 years agoFeat: Choose best python version to run venv 66/70066/2 v0.76.3
Eric Ball [Thu, 21 Apr 2022 23:43:59 +0000 (16:43 -0700)]
Feat: Choose best python version to run venv

We've seen problems with capture-instance-metadata.sh choosing the
best python path, so this should help get the best version on each
builder.

If the $PYTHON variable is set, this will be used. If not, we check to
see if python3 is available, as this should point to the latest
version. If this is also not available, we run with the basic python
command.

Change-Id: I9950cc286c72fd17eac7a3c678ef8ca04ccd8921
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
2 years agoRevert "Fix: Remove "--python" flag from venv act" 57/70057/1 v0.76.2
Anil Belur [Wed, 20 Apr 2022 23:56:25 +0000 (09:56 +1000)]
Revert "Fix: Remove "--python" flag from venv act"

This reverts commit 18d90a9e0b74515b6aa2ac8fc4a17b9d98a7680e.

The issue does not happen on C7/C8 but mostly on Ubuntu and
needs to be addressed elsewhere.

Change-Id: I54abbea2550ed6d80e3c7d75ab86d246c0ec98d2
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
2 years agoFix: Remove "--python" flag from venv activation 40/70040/3 v0.76.1
Eric Ball [Tue, 19 Apr 2022 15:55:59 +0000 (08:55 -0700)]
Fix: Remove "--python" flag from venv activation

Remove "--python" flag from lf-activate-venv call in
capture-instance-metadata. This forces the system to use Python 2,
which is not available on newer systems, and is causing failures.

Change-Id: I456a1463885b6269b555edbd587c9cfc039c78da
Issue: RELENG-4192
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
2 years agoCI: Disable pre-commit in CI 55/70055/1
Jessica Wagantall [Wed, 20 Apr 2022 19:24:37 +0000 (12:24 -0700)]
CI: Disable pre-commit in CI

Packer has been having issues creating new images for a bit and
pre-commit is no longer working in CI because of something upstream
because of this.

Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I64d9e9b2d3edf3ff395ef0886d4e2fe0a5a2dad4

2 years agoMerge "Feat: Append build result to cost file" v0.76.0
Andrew Grimberg [Wed, 13 Apr 2022 23:07:30 +0000 (23:07 +0000)]
Merge "Feat: Append build result to cost file"

2 years agoFix: Activate the virtual environment 17/69917/5 v0.75.1
Anil Belur [Fri, 25 Mar 2022 00:43:03 +0000 (10:43 +1000)]
Fix: Activate the virtual environment

Install Python to make sure its available on all images.

Issue-ID: RELENG-4182
Change-Id: Ic8900474fdfd4ce1bdaeecc1e80fef74ca6538f7
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
2 years agoFeat: Append build result to cost file 71/69871/3
Kevin Sandi [Thu, 17 Mar 2022 23:04:29 +0000 (17:04 -0600)]
Feat: Append build result to cost file

Change-Id: I0f08f364432d4f102655dfe80e98d5202a6b0349
Signed-off-by: Kevin Sandi <ksandi@contractor.linuxfoundation.org>
2 years agoFeat: Add SBOM Generator conditional step 87/69687/14 v0.75.0
Jessica Wagantall [Mon, 7 Feb 2022 23:35:09 +0000 (15:35 -0800)]
Feat: Add SBOM Generator conditional step

This is a conditional step which calls a specific
version of SPDX SBOM generator, runs a scan and generates a
report of software bill of materials in a specific repo.

Issue: RELENG-4104
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I3433a93efc4141b5e5e1949d7260f7686a015506

2 years agoDocs: Update JJB documentation pointers 18/69818/1
Andrew Grimberg [Mon, 7 Mar 2022 21:43:22 +0000 (13:43 -0800)]
Docs: Update JJB documentation pointers

JJB changed where they are hosting documentation and it's causing issues
with Sphinx link checking

Change-Id: Iedd85caa71cd45a706a278a020b7dab6b4612438
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
2 years agoRefactor: future-proof Java version selector 08/69808/3
Robert Varga [Mon, 7 Mar 2022 07:01:55 +0000 (08:01 +0100)]
Refactor: future-proof Java version selector

In terms of Java versioning we have version epoch transition:
- Java 7 and Java 8 use 1.[7,8].0 as their version
- Java 9 and later use plain {9,10,...} as their version

Adjust the version matching machinery to cover not only Java 11/12, but
all future versions -- assuming versioning schema does not change.

Change-Id: I9b12223c39780353bd8921af4ce8ba1349fbcf13
Signed-off-by: Robert Varga <nite@hq.sk>
2 years agoFeat: Process orphaned coe clusters for K8S jobs 95/69595/6 v0.74.0
Anil Belur [Mon, 31 Jan 2022 00:48:51 +0000 (10:48 +1000)]
Feat: Process orphaned coe clusters for K8S jobs

K8s jobs by default creates stacks names that does not match
JOB_NAME, therefore ignore them while processing orphaned stacks
and handle them separatly when cleaning up the orphaned clusters.

The stack naming scheme is limited to take first 20 chars from the
JOB_NAME while the rest is randomly generated for uniqueness which
breaks the openstack cron jobs.

Ref: https://github.com/openstack/magnum/blob/master/magnum/
drivers/heat/driver.py#L202-L212

Issue-ID: RELENG-4106
Change-Id: Id3d9b74c3e6e2a0abbddb771b7fc7d5ba2b59ca5
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
2 years agoFeat: Add support for OpenJDK17 44/69544/1 v0.73.0
Anil Belur [Thu, 20 Jan 2022 06:45:06 +0000 (16:45 +1000)]
Feat: Add support for OpenJDK17

Change-Id: Ic6369e120cc256d081ebdd66c5d0d154c86fc73b
Issue-ID: RELENG-4097
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoFeat: Add new job gerrit-go-verify 56/69356/13 v0.72.0
Kevin Sandi [Wed, 8 Dec 2021 22:27:32 +0000 (16:27 -0600)]
Feat: Add new job gerrit-go-verify

This job adds support for running unit tests on Go projects

Issue-ID: RELENG-4055
Change-Id: I6c4d8bfbaf131837ba91aa1bc1cdf5e2f3f0a790
Signed-off-by: Kevin Sandi <ksandi@contractor.linuxfoundation.org>
3 years agoFeat: Add wait flag for SonarCloud quality gates 52/69152/9 v0.71.0
Kevin Sandi [Mon, 22 Nov 2021 23:15:39 +0000 (17:15 -0600)]
Feat: Add wait flag for SonarCloud quality gates

Add support for wait flag on SonarCloud quality gates, this way jobs
won't finish until the quality gate reports back the result during
the analysis step, which will fail anytime the quality gate fails.

Issue-ID: RELENG-4011
Signed-off-by: Kevin Sandi <ksandi@contractor.linuxfoundation.org>
Change-Id: Idd75faab59e55363d65fc930790336f370ce60d4

3 years agoMerge "Chore: Update pre-commit dependencies" v0.70.1
Anil Belur [Thu, 11 Nov 2021 22:39:58 +0000 (22:39 +0000)]
Merge "Chore: Update pre-commit dependencies"

3 years agoFix: sonary-verify choosing strategy to "gerrit" 05/69105/2 v0.70.0
Eric Ball [Thu, 11 Nov 2021 22:10:54 +0000 (14:10 -0800)]
Fix: sonary-verify choosing strategy to "gerrit"

Change-Id: Id5d1f3b2a832c2206788bb0a06a1173cb9018666
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoChore: Update pre-commit dependencies 04/69104/1
Anil Belur [Thu, 11 Nov 2021 22:03:43 +0000 (08:03 +1000)]
Chore: Update pre-commit dependencies

Change-Id: I15f1f1b59ca8d77e6b4052682af2a4b9e8236308
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoFix: bug in shell script maven-sonar.sh 97/69097/1 v0.69.1
Kevin Sandi [Tue, 9 Nov 2021 23:23:04 +0000 (17:23 -0600)]
Fix: bug in shell script maven-sonar.sh

In order to fix a bug in maven-sonar.sh we should treat parameter
scan-dev-branch as string instead of boolean

Issue-ID: RELENG-4011
Signed-off-by: Kevin Sandi <ksandi@contractor.linuxfoundation.org>
Change-Id: I6aa16ef65ded5c35174003439f7d183cfa8d0fcf

3 years agoFeat!: Add builder macro to set ansible.cfg file 95/69095/3
Anil Belur [Tue, 9 Nov 2021 00:25:03 +0000 (10:25 +1000)]
Feat!: Add builder macro to set ansible.cfg file

Add packer builder macros to create a 'ansible.cfg' file.
This is required by packer build jobs to set ansible host
configuration. The job or image specific configuration can
be created using JCasC custom files in the ci-man repository.

Issue-ID: RELENG-4032
Change-Id: Ia9fc4d26341228ba8009de6d2ec3c46e31bfc45b
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoChore!: Rename 'whitelist' to 'allowlist' 83/69083/5
Anil Belur [Thu, 4 Nov 2021 01:25:59 +0000 (11:25 +1000)]
Chore!: Rename 'whitelist' to 'allowlist'

Improve global-jjb code and documentation to minimize
non-inclusivity.

Change-Id: I3c70ad4ad2c4d34510410b0baab439ab8681954d
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoMerge "Feat: Add Maven SonarCloud verify job" v0.69.0
Kevin Sandi [Mon, 25 Oct 2021 23:39:41 +0000 (23:39 +0000)]
Merge "Feat: Add Maven SonarCloud verify job"

3 years agoFix: Pin pyparsing<3.0.0 required by httplib2 11/69011/1 v0.68.1
Anil Belur [Mon, 25 Oct 2021 10:40:08 +0000 (20:40 +1000)]
Fix: Pin pyparsing<3.0.0 required by httplib2

Pin pyparsing<3.0.0 required by httplib2 0.20.1. A new version
of pip 21.3.1 is out that has removed this dependency
(pyparsing<3,>=2.4.2) as required by httplib2.

Issue-ID: RELENG-4022
Change-Id: Ifc00a6c82f82b57768330b491828a159be561679
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoFeat: Add Maven SonarCloud verify job 76/68976/11
Kevin Sandi [Thu, 14 Oct 2021 22:55:42 +0000 (16:55 -0600)]
Feat: Add Maven SonarCloud verify job

Add new Maven SonarCloud verify job that will execute SonarCloud
scans before a change gets merged.

Issue-ID: RELENG-4011
Signed-off-by: Kevin Sandi <ksandi@contractor.linuxfoundation.org>
Change-Id: I6045b186bfde76e19d77f50ef14c98107e2cb0c5

3 years agoFix: Correct boot source options for OpenStack 90/68990/2
Andrew Grimberg [Mon, 18 Oct 2021 23:51:39 +0000 (16:51 -0700)]
Fix: Correct boot source options for OpenStack

If a VOLUME_SIZE option is not defined in the base cloud configuration
then the default OpenStack boot selection should be 'image' not
'volumeFromImage' otherwise we are unable to properly take advantage of
all options provided by an OpenStack cloud.

Change-Id: I76ae042d598c9f7fa1868d4ea6a3566f8c6b8115
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
3 years agoFix: Correct bad conversion of OpenStack config 79/68979/1 v0.68.0
Andrew Grimberg [Fri, 15 Oct 2021 00:05:27 +0000 (17:05 -0700)]
Fix: Correct bad conversion of OpenStack config

While attempting to get the OpenDaylight Jenkins Sandbox configured to
start working with JCasC cloud management it was discovered that the
numExecutors flag was being mistranslated.

Change-Id: I5598f9936a2a3190e0ff326dad8ef4503aaceaa2
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
3 years agoFeat: Update default version of JDK to openjdk11 32/68932/3
Anil Belur [Tue, 5 Oct 2021 01:47:15 +0000 (11:47 +1000)]
Feat: Update default version of JDK to openjdk11

JDK8 is soon going to be EOL by Mar 31, 2022. with most of the
LF projects already on JDK11 (LTS), upgrade the default
version to JDK11 (LTS).

https://www.oracle.com/java/technologies/java-se-support-roadmap.html

Issue-ID: RELENG-3978
Change-Id: I56359e72ef58602208242b076f35b19a1b7f5a1d
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoFeat: Allow use of newer JDKs for Sonar scans 93/68893/2 v0.67.1
Eric Ball [Fri, 24 Sep 2021 01:54:54 +0000 (18:54 -0700)]
Feat: Allow use of newer JDKs for Sonar scans

Many builders still default to openjdk8 for java, but Sonarcloud
only supports openjdk11+.

Issue: RELENG-3964
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Change-Id: Ida96f7f242a397a60e40e26f910510d665bcf577

3 years agoFix: Update sphinx requirements 92/68892/1
Eric Ball [Fri, 24 Sep 2021 00:08:24 +0000 (17:08 -0700)]
Fix: Update sphinx requirements

Our older sphnix requirements are out of date, and are trying to pull
in packages that are no longer supported.

Change-Id: I69d336d324af2a512b8d756f0e8f83c244642029
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoFeat: Add "Unmaintained" lifecycle_state for INFO 14/68814/1 v0.67.0
Eric Ball [Wed, 15 Sep 2021 22:54:21 +0000 (15:54 -0700)]
Feat: Add "Unmaintained" lifecycle_state for INFO

This adds "Unmaintained" as a valid lifecycle_state to be used in
INFO.yaml files, per request from ONAP.

Issue: RELENG-3955
Change-Id: I8dfb947b8a0936c552c087dda6c115563b69c74f
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoFix: Add missing v3-standard Vexxhost flavors 59/68759/1 v0.66.1
Eric Ball [Thu, 9 Sep 2021 21:34:38 +0000 (14:34 -0700)]
Fix: Add missing v3-standard Vexxhost flavors

Issue: RELENG-3935
Change-Id: Ib3f317bc918a9c742ee339808c84ec432b0af8ee
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoMerge "Feat: Allow scans in short lived branches" v0.66.0
Eric Ball [Thu, 9 Sep 2021 19:12:00 +0000 (19:12 +0000)]
Merge "Feat: Allow scans in short lived branches"

3 years agoFeat: Allow scans in short lived branches 51/68151/16
Jessica Wagantall [Fri, 25 Jun 2021 19:32:25 +0000 (12:32 -0700)]
Feat: Allow scans in short lived branches

Add functionality to allow scans of short lived
branches in SonarCloud.
Developers will be able to turn this feature on
by setting "scan-dev-branch: true"

Issue: RELENG-3937
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I6b32771ba00e86927ba179596c467b37c831bd2d

3 years agoFeat: Add v3-starter flavors to cloud lookup 58/68758/2
Eric Ball [Thu, 9 Sep 2021 18:29:14 +0000 (11:29 -0700)]
Feat: Add v3-starter flavors to cloud lookup

Issue: RELENG-3935
Change-Id: I114ca41c020c3d408d2afdf0934f57c166312ad8
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoChore: Upgrade pre-commit hooks 78/68678/1
Andrew Grimberg [Wed, 1 Sep 2021 18:04:48 +0000 (11:04 -0700)]
Chore: Upgrade pre-commit hooks

* github.com/pre-commit/pre-commit-hooks: v3.4.0 -> v4.0.1
* github.com/pre-commit/mirrors-prettier: v2.2.1 -> v2.3.2
* github.com/adrienverge/yamllint: v1.26.1 -> v1.26.3

Change-Id: I0ec2f3e17da3c5215a824ebe501468796f0d6316
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
3 years agoFix: Labels should always include config name 88/68588/1 v0.65.3
Eric Ball [Thu, 19 Aug 2021 21:58:20 +0000 (14:58 -0700)]
Fix: Labels should always include config name

Openstack labels need to include the config name, in addition to any
labels explicitly defined. This also changes the builder name to
match the config name, rather than using the labels (which can be
only one label, but is technically a space-separated list).

Change-Id: I29ccc9987aa02d1e96930d165908396b46223fa8
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoFix: If no volume set, use "image" type 85/68585/2
Eric Ball [Thu, 19 Aug 2021 19:02:44 +0000 (12:02 -0700)]
Fix: If no volume set, use "image" type

If no volume_size is defined, the default behavior was to set one to
10GB. However, the proper way to handle this is to use an "Image"
boot source rather than "Volume From Image".

Change-Id: Ib4a33f224ea1044b076d965faa164686da071cc4
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoFix: Use cloud agent name as default node label 07/68507/2 v0.65.2
Eric Ball [Tue, 10 Aug 2021 21:50:42 +0000 (14:50 -0700)]
Fix: Use cloud agent name as default node label

Rather than quitting if there are no labels defined, we can instead
use the agent name (e.g. "centos7-2c-1g") as the default label. This
recreates the functionality of the groovy scripts previously used.

Issue: RELENG-3863
Change-Id: I3817cca81358fa151fdeb1bbc8516df983327965
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoRefactor: Simplify and improve readability 06/68506/1
Eric Ball [Tue, 10 Aug 2021 21:44:05 +0000 (14:44 -0700)]
Refactor: Simplify and improve readability

This includes improvements to PEP8 formatting, grammar, and variable
naming, plus several changes to make code more efficient or readable.

Issue: RELENG-3863
Change-Id: I823db03175f96db3b99ee5fcc715345462a47139
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoFix: Install lftools before other packages 00/68300/1 v0.65.1
Eric Ball [Thu, 15 Jul 2021 18:41:22 +0000 (11:41 -0700)]
Fix: Install lftools before other packages

lftools has an incompatibility with python-docker >4.2, and this
requirement seems to be causing a conflict with other packages that
python-tools-install is installing. This causes an older version of
lftools to be installed (0.31.1, the last version when docker was not
pinned).

By installing lftools first, its requirements will take
precedence over other packages.

Issue: RELENG-3792
Change-Id: I0d3ca88dbe51c97ce495f6db0eb66829a61ad3bc
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoFeat: allow more tox parallel mode configurations 44/68044/7 v0.65.0
Guillaume Lambert [Thu, 17 Jun 2021 12:43:00 +0000 (14:43 +0200)]
Feat: allow more tox parallel mode configurations

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I8a812ed25f3836c100704914f08491a61642679f

3 years agoRefactor: how to pass options to tox in tox-run.sh 43/68043/3
Guillaume Lambert [Thu, 17 Jun 2021 12:15:24 +0000 (14:15 +0200)]
Refactor: how to pass options to tox in tox-run.sh

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I3cae2a28f8cdf10b4b547f729d436cb204322247

3 years agoDocs: remove reference to detox 41/68041/3
Guillaume Lambert [Thu, 17 Jun 2021 09:36:01 +0000 (11:36 +0200)]
Docs: remove reference to detox

Detox globaljjb support was removed in a previous commit.
1873622e1b5bf98150db6850901d190dd6c35422
Parallel jobs are now natively supported by tox since version 3.7.0
thanks to the option "-p" / "--parallel".
This new option offers more possibilities than detox and other options
have also been introduced to tune tox behavior in parallel mode.
The configuration choices made in globaljjb must be clarified.

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I82c88ea96ee1332b95ab13f0b7dbf2a04e00f37a

3 years agoFix: Skip vote for jjb-deploy also when successful 10/67910/2
Vratko Polak [Tue, 8 Jun 2021 15:16:48 +0000 (17:16 +0200)]
Fix: Skip vote for jjb-deploy also when successful

The previous code used "success" instead of "successful".

Fixes: bfe91be03d516a56adb74abd1e16c50b86c27c6e

Change-Id: If73f2f2756c10d0a7230ad1214e8ea3c86b1b114
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
3 years agoFeat: Add support for Ubuntu 20.04 and Docker 14/67714/6 v0.63.2 v0.64.0
Vanessa Rene Valderrama [Thu, 13 May 2021 17:22:28 +0000 (12:22 -0500)]
Feat: Add support for Ubuntu 20.04 and Docker

Updating the sysstat script to support Ubuntu 20.04 and Docker systems

Signed-off-by: Vanessa Rene Valderrama <vvalderrama@linuxfoundation.org>
Change-Id: I12e6daa8a04228337c99b004e67338eb8f6a8f1a

3 years agoCI: bashate warns lines>80 and forbids >120 74/67674/7
Guillaume Lambert [Tue, 11 May 2021 16:30:51 +0000 (18:30 +0200)]
CI: bashate warns lines>80 and forbids >120

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: Ib981e33c31c704ef3e8fb434f8766b0e217b2b1d

3 years agoFix: bashate E006 warnings for lines > 120 chars 76/67576/14
Guillaume Lambert [Tue, 4 May 2021 11:46:48 +0000 (13:46 +0200)]
Fix: bashate E006 warnings for lines > 120 chars

Lines too long can not only cause problems when reading/writing code,
but also often indicates a bad smell,
e.g. too many levels of indentation due to overly complex functions
which requires refactoring into smaller chunks.

Note that the bashate current profile reports every line > 80 chars.
After this commit, 211 warnings of this category will still remain.

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: Ibbb171353fd960ea93145f7913e62e327bd6a86b

3 years agoCI: enforce bashate tox profile 84/67584/16
Guillaume Lambert [Tue, 4 May 2021 15:21:00 +0000 (17:21 +0200)]
CI: enforce bashate tox profile

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I920542ec62e67ee37aebd554076bd79d6863212a

3 years agoFix: bashate E003 warnings 78/67578/16 v0.63.1
Guillaume Lambert [Tue, 4 May 2021 12:13:45 +0000 (14:13 +0200)]
Fix: bashate E003 warnings

Indentations must be aligned to multiple of 4 columns.

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I0bdb8b2fafd1811e55fbbacaa4c4cb6a2b59f822

3 years agoRefactor: use POSIX shell function declaration 75/67575/13
Guillaume Lambert [Tue, 4 May 2021 09:27:12 +0000 (11:27 +0200)]
Refactor: use POSIX shell function declaration

POSIX function style is already used in most of shell scripts.
By the way, this style fixes bashate E020 warnings.

Non POSIX function declarations in a format different from
^function name {$ are warned by bashate.
There are several equivalent ways to define functions in Bash.
This is for consistency.

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I5f02e7052101617314157ec408532dd024e80a72

3 years agoFix: bashate E002 warnings about tabs indents 74/67574/12
Guillaume Lambert [Tue, 4 May 2021 08:31:14 +0000 (10:31 +0200)]
Fix: bashate E002 warnings about tabs indents

Tabs must be avoided and replaced by 4 whitespaces according to the
standard usage.

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: Ic21fe1646b60b6630d54fd16b8b2e1fa6d4c49fb

3 years agoFix: bashate E011 warnings 73/67573/12
Guillaume Lambert [Tue, 4 May 2021 09:01:41 +0000 (11:01 +0200)]
Fix: bashate E011 warnings

E011 rule ensures consistency of "then" directive being on the same line
as it's condition.
This is similar to E010 in the previous commit
but to ensure the consistency of if/elif statements.

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I6d3ff5b468fa47948b0123c017980e00a94cf82f

3 years agoFix: bashate E010 warnings 72/67572/11
Guillaume Lambert [Tue, 4 May 2021 08:25:50 +0000 (10:25 +0200)]
Fix: bashate E010 warnings

E010 rule ensures consistency of "do" directive being on the same line
as it's command.

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: Id7717658728968da017107c4b496cdfbe0c5b6b0

3 years agoFix: bashate E043 warnings 71/67571/9
Guillaume Lambert [Tue, 4 May 2021 08:14:24 +0000 (10:14 +0200)]
Fix: bashate E043 warnings

Arithmetic compound has inconsistent return semantics

((expr)) has a trap that if expr is 0, the return value is 1.
This will trigger a failure with "set -e" and can be very confusing.
It is good defensive programming to avoid this with explicit assignment.

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I412bd8872560c90c28642ed5433eab9b2a29a5e8

3 years agoCI: Add bashate tox profile 70/67570/7
Guillaume Lambert [Mon, 3 May 2021 18:52:24 +0000 (20:52 +0200)]
CI: Add bashate tox profile

Bashate is a shell linter inspired from PEP8.
It helps improving shell scripts.

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I351a205cbfdc160660958069a8060badb68aa28a

3 years agoMerge "Fix: Set S3 URL in the target framename"
Andrew Grimberg [Mon, 10 May 2021 13:38:54 +0000 (13:38 +0000)]
Merge "Fix: Set S3 URL in the target framename"

3 years agoFix: Set S3 URL in the target framename 40/67640/2
Anil Belur [Sat, 8 May 2021 07:21:02 +0000 (17:21 +1000)]
Fix: Set S3 URL in the target framename

Set S3 URL in the framename of the target attribute.
The description-setter plugin does not read the URL
when the framename is unset.

Issue: RELENG-3269
Change-Id: I47c488b586cf68164a0273a14134e490c8c12cec
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoCI: Properly run gitlint in CI 41/67641/4
Andrew Grimberg [Sat, 8 May 2021 14:40:45 +0000 (07:40 -0700)]
CI: Properly run gitlint in CI

When running pre-commit in a CI system the COMMIT_EDITMSG does not
(normally) get created as that is an artifact of editing the commit
message. If the file doesn't exist then gitlint will skip which makes it
possible for pre-commit checks that should fail, to pass.

Since we want tox to run in a consistent manner both locally and in CI
we need to play around a little with how we are checking the commit
message.

Change-Id: I3990aa9846dc8479cadaad5025e45f863623bf86
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
3 years agoCI: Configure gitlint for Conventional Commits 14/67614/4
Andrew Grimberg [Wed, 5 May 2021 16:52:33 +0000 (09:52 -0700)]
CI: Configure gitlint for Conventional Commits

Turn on Conventional Commit message subject validation. This will affect
CI specifically, but developers can avoid having CI fail by assuring
that they have installed pre-commit and that they have also run

pre-commit install --hook-type commit-msg

The above is needed because pre-commit does not install commit-msg hooks
by default

Change-Id: If45c7a256f7d769eb092d867217468b9d6c53384
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
3 years agoCI: Enforce yamllint in pre-commit 13/67613/2
Andrew Grimberg [Wed, 5 May 2021 16:45:36 +0000 (09:45 -0700)]
CI: Enforce yamllint in pre-commit

Turn on yamllint enforcement via pre-commit. This affects both the CI
system as well as developers that pre-commit configured in their
environment (like they should)

Change-Id: If1ad8619a181ea916bcadc9eaddc58207fbef720
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
3 years agoStyle: Cleanup yamllint errors 12/67612/2
Andrew Grimberg [Wed, 5 May 2021 16:42:25 +0000 (09:42 -0700)]
Style: Cleanup yamllint errors

Cleanup yamllint errors before we start enforcing via pre-commit

Change-Id: Ie476be877d2896e1bc66ed4238bac0f2a1a33239
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
3 years agoCI: Rename yamllint config file 11/67611/3
Andrew Grimberg [Wed, 5 May 2021 16:32:23 +0000 (09:32 -0700)]
CI: Rename yamllint config file

Yamllint does not look for a yamllint.conf file and we're calling it
(that I can find) anywhere with an explicit config file definition.
Since .yamllint is the default file used, we are fixing the filename

Additionally, we are adding a section on comments to protect against the
prettier pre-commit hook that enforces a single space instead of
yamllints normal double space for comments after content

Change-Id: I0df4b467ea57b1558dbb3c71d74775f84a88b037
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
3 years agoCI: Update pre-commit checks to latest versions 10/67610/3
Andrew Grimberg [Wed, 5 May 2021 15:58:48 +0000 (08:58 -0700)]
CI: Update pre-commit checks to latest versions

This updates the pre-commit checks to the latest versions of the checks
we're using and fixes a linting issue in our yaml config

Change-Id: Ib9dc085ab6bf77fca33b884eb2005094b53f7d40
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
3 years agoAdd protection in python-tools-install.sh 50/67550/1
Guillaume Lambert [Mon, 3 May 2021 13:22:38 +0000 (15:22 +0200)]
Add protection in python-tools-install.sh

This script may result in unexpected behavior on distributions
other than Ubuntu with a matching release.
DISTRIB_ID must be tested to avoid such problems.

Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I2ba02b613a4aaadfeb09bd66ddfde8f36a9510b4

3 years agoFix: Correct choosing strategy use on merge 38/67538/2 v0.63.0
Andrew Grimberg [Thu, 29 Apr 2021 16:40:01 +0000 (09:40 -0700)]
Fix: Correct choosing strategy use on merge

The choosing strategy being used by the docker-merge-{stream} jobs for
gerrit is set to gerrit and not default. Since this is a merge job it's
supposed to always pick up the tip / head of the repository.

Issue: RELENG-3657
Change-Id: I3b97bf01966aa85d8066ebea301d3205398e7e63
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
3 years agoMerge "Remove all refs to python2"
Eric Ball [Tue, 30 Mar 2021 21:14:51 +0000 (21:14 +0000)]
Merge "Remove all refs to python2"

3 years agoProvision settings for WS scan 72/67272/2
Jessica Wagantall [Tue, 30 Mar 2021 17:04:40 +0000 (10:04 -0700)]
Provision settings for WS scan

Provision global-settings to replace the
default used by the Unified Agent.

This will allow dependencies to be downloaded the
same way the maven build does.

Issue: RELENG-3430
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I2b751c7d7d966b0b9b07a90405ec61b2254bed6d

3 years agoRemove all refs to python2 63/67263/1
Anil Belur [Tue, 30 Mar 2021 04:13:59 +0000 (14:13 +1000)]
Remove all refs to python2

Python 2.7 is EOL and no longer supported. Some of the
lf-macros and docs still refer to py2 which could
mislead users.

Change-Id: I9f39948cec30fb573f25ee7ecb3bed8bf3600ab9
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoMerge "Update prettier pre-commit to ignore stage error"
Andrew Grimberg [Fri, 26 Mar 2021 14:59:24 +0000 (14:59 +0000)]
Merge "Update prettier pre-commit to ignore stage error"

3 years agoAdd sonar-prescan-script jobs for maven 18/67218/2 v0.62.0
Eric Ball [Thu, 25 Mar 2021 19:58:09 +0000 (12:58 -0700)]
Add sonar-prescan-script jobs for maven

We added these for the more generic Sonar jobs in lf-ci-jobs, but had
not added them for maven. We now have a need for this job type (it can
be used to ensure that a Sonarcloud-compatible version of nodejs is on
the system), so they are being added now.

Issue: IT-21602
Change-Id: Ic32f418a042839ab277dc1b85777d0eef475d620
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoMerge "Pin git review to 1.78"
Andrew Grimberg [Tue, 23 Mar 2021 21:58:05 +0000 (21:58 +0000)]
Merge "Pin git review to 1.78"

3 years agoUpdate WS Unified Agent version 86/67086/1
Jessica Wagantall [Fri, 12 Mar 2021 19:53:19 +0000 (11:53 -0800)]
Update WS Unified Agent version

Update WhiteSource Unified Agent to
version 21.2.1

Issue: RELENG-3430
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I5c4ae6d49b1822a04fad1ce9af955bf9e06d3851

3 years agoUpdate prettier pre-commit to ignore stage error 25/67025/1
Anil Belur [Wed, 10 Mar 2021 07:20:54 +0000 (17:20 +1000)]
Update prettier pre-commit to ignore stage error

Set stages to 'commit' to overide the default_stages.

This fixes the issue seen while using git-commmit:
[error] No matching files. Patterns: .git/COMMIT_EDITMSG

Change-Id: I959744b89e6cb22e84fca9119a1405e354734cbb
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoPin git review to 1.78 14/67014/1
Anil Belur [Tue, 9 Mar 2021 04:34:53 +0000 (14:34 +1000)]
Pin git review to 1.78

The latest version of module trys to look for git hook recursively within
the submodules.

Error:
    Entering 'global-jjb'
    cannot stat '.git/hooks/commit-msg': Not a directory
    fatal: run_command returned non-zero status for global-jjb

Remove workaround that has been resolved in v1.28 and use lf-activate-venv
to install git-review

Issue: RELENG-3435
Change-Id: I67b3730dcdf5a410020c63312897bb67b42017bd
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoMerge "Update the create script with V3 flavors."
Bengt Thuree [Fri, 5 Mar 2021 03:29:41 +0000 (03:29 +0000)]
Merge "Update the create script with V3 flavors."

3 years agoReorder release-job script 37/66937/2
Jessica Wagantall [Tue, 2 Mar 2021 19:25:28 +0000 (11:25 -0800)]
Reorder release-job script

Reorder functions and add function labels to make
release-job.sh easier to read.
Fix "Double quote to prevent globbing" line 34

Issue: RELENG-3376
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: Ic8349f692a50a645a99e884c852013f75332cccd

3 years agoAdd support for artifact releases 47/66647/6
Jessica Wagantall [Wed, 20 Jan 2021 21:10:01 +0000 (13:10 -0800)]
Add support for artifact releases

Currently Dent is the only customer for artifact
releases where they need a particular binary to be
pulled from Snapshots and pushed manually into Releases.

These are artifacts built using make.

Issue: RELENG-3376
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I5a8ec8e2906757b1700505ad10dd2ba8ac844f9a

3 years agoPin cryptography to 3.3.2 35/66835/7 v0.61.1
Anil Belur [Sat, 20 Feb 2021 01:00:50 +0000 (11:00 +1000)]
Pin cryptography to 3.3.2

The latest version of module breaks compatibility with the
latest version of pip.

Error:

 Traceback (most recent call last):
   File "<string>", line 1, in <module>
   File "/tmp/pip-build-vqk6fya9/cryptography/setup.py", line 14, i
   in <module> from setuptools_rust import RustExtension
 ModuleNotFoundError: No module named 'setuptools_rust'

Issue-Id: IT-21538 IT-21509
Change-Id: I83db995ae8321377b861cabcf6ac53485778e471
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoFix reno lint error in sention names 36/66836/1
Anil Belur [Mon, 22 Feb 2021 08:39:51 +0000 (18:39 +1000)]
Fix reno lint error in sention names

Error:
unrecognized section name

Update the version of reno as per the requirements.txt

Change-Id: Iadc07de41934c936fc82147bc8ebcd4f6135764e
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoUpdate pre-commit hook shellcheck to 2.1.5 10/66810/1 v0.61.0
Anil Belur [Thu, 18 Feb 2021 02:06:33 +0000 (12:06 +1000)]
Update pre-commit hook shellcheck to 2.1.5

The workaround with 4d088120467 is no longer
required with the latest 2.1.5 release.

Change-Id: Idf059bb7e184beb7c5ca9c5c5c97f5244ecbcec5
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoMerge "Add workaround for pre-commit 2.10.0"
Anil Belur [Wed, 17 Feb 2021 21:43:16 +0000 (21:43 +0000)]
Merge "Add workaround for pre-commit 2.10.0"

3 years agoAdd workaround for pre-commit 2.10.0 95/66795/1
Eric Ball [Wed, 17 Feb 2021 00:40:46 +0000 (16:40 -0800)]
Add workaround for pre-commit 2.10.0

The jumanjihouse shellcheck hook errors without
additional_dependencies being defined. While there is a PR open to
add this fix, it has not yet been merged and released. Until it is,
this workaround will enable using the latest versions of pre-commit
and the jumanjihouse hooks.
Github PR: https://github.com/jumanjihouse/pre-commit-hooks/pull/80

Change-Id: I0ec16952009128e25cccb6833698bf1bd864af2d
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoMerge "Fix rtd-verify submodule update command"
Andrew Grimberg [Tue, 16 Feb 2021 22:35:58 +0000 (22:35 +0000)]
Merge "Fix rtd-verify submodule update command"

3 years agoFix rtd-verify submodule update command 42/66742/3
Eric Ball [Fri, 5 Feb 2021 00:11:09 +0000 (16:11 -0800)]
Fix rtd-verify submodule update command

The old version is missing the "--init" flag, which causes it to not
add new submodules. This did not show up as an issue until a new
submodule was added in a production environment.
Additionally, the "--recursive" flag has been included to ensure
proper loading of recursive submodules.

Issue: IT-21490
Change-Id: Ie882980d3721ae0e1fea9fa1d3321b86d18be484
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoMerge "Add schema for artifact releases"
Jessica Wagantall [Wed, 10 Feb 2021 22:29:25 +0000 (22:29 +0000)]
Merge "Add schema for artifact releases"

3 years agoRemove python2 tox installation 97/66697/2
Aric Gardner [Fri, 29 Jan 2021 16:33:45 +0000 (11:33 -0500)]
Remove python2 tox installation

Update path to ensure that tox from tox-install.sh is run

Should still work for python2 tox
as pyenv will continue to provide a py2 env

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I568ebf679bbce20b76a0a686699702f81e7e7777

3 years agoSkip Metadata Capture on unsupported Clouds 84/66684/1
Trevor Bramwell [Thu, 28 Jan 2021 17:53:35 +0000 (09:53 -0800)]
Skip Metadata Capture on unsupported Clouds

Updates the 'capture-instance-metadata.sh' script to skip attempting to
capture instance metadata needed for job-cost.sh if the build is being
run on an unsupported cloud or platform.

Change-Id: I8bfea1416d11de1caeed2d414b90513f63de9b6f
Signed-off-by: Trevor Bramwell <tbramwell@linuxfoundation.org>
3 years agoUpdate the create script with V3 flavors. 31/66431/3
Anil Belur [Fri, 8 Jan 2021 23:15:36 +0000 (09:15 +1000)]
Update the create script with V3 flavors.

Issue: RELENG-3329
Change-Id: I46f8b87a60a6f4538f5b76168a51ec242b5963c2
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoMerge "Upgrade packer version 1.6.6"
Anil Belur [Thu, 28 Jan 2021 12:28:09 +0000 (12:28 +0000)]
Merge "Upgrade packer version 1.6.6"