Anil Belur [Thu, 11 Nov 2021 22:03:43 +0000 (08:03 +1000)]
Chore: Update pre-commit dependencies
Change-Id: I15f1f1b59ca8d77e6b4052682af2a4b9e8236308
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Kevin Sandi [Tue, 9 Nov 2021 23:23:04 +0000 (17:23 -0600)]
Fix: bug in shell script maven-sonar.sh
In order to fix a bug in maven-sonar.sh we should treat parameter
scan-dev-branch as string instead of boolean
Issue-ID: RELENG-4011
Signed-off-by: Kevin Sandi <ksandi@contractor.linuxfoundation.org>
Change-Id: I6aa16ef65ded5c35174003439f7d183cfa8d0fcf
Anil Belur [Tue, 9 Nov 2021 00:25:03 +0000 (10:25 +1000)]
Feat!: Add builder macro to set ansible.cfg file
Add packer builder macros to create a 'ansible.cfg' file.
This is required by packer build jobs to set ansible host
configuration. The job or image specific configuration can
be created using JCasC custom files in the ci-man repository.
Issue-ID: RELENG-4032
Change-Id: Ia9fc4d26341228ba8009de6d2ec3c46e31bfc45b
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Anil Belur [Thu, 4 Nov 2021 01:25:59 +0000 (11:25 +1000)]
Chore!: Rename 'whitelist' to 'allowlist'
Improve global-jjb code and documentation to minimize
non-inclusivity.
Change-Id: I3c70ad4ad2c4d34510410b0baab439ab8681954d
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Kevin Sandi [Mon, 25 Oct 2021 23:39:41 +0000 (23:39 +0000)]
Merge "Feat: Add Maven SonarCloud verify job"
Anil Belur [Mon, 25 Oct 2021 10:40:08 +0000 (20:40 +1000)]
Fix: Pin pyparsing<3.0.0 required by httplib2
Pin pyparsing<3.0.0 required by httplib2 0.20.1. A new version
of pip 21.3.1 is out that has removed this dependency
(pyparsing<3,>=2.4.2) as required by httplib2.
Issue-ID: RELENG-4022
Change-Id: Ifc00a6c82f82b57768330b491828a159be561679
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Kevin Sandi [Thu, 14 Oct 2021 22:55:42 +0000 (16:55 -0600)]
Feat: Add Maven SonarCloud verify job
Add new Maven SonarCloud verify job that will execute SonarCloud
scans before a change gets merged.
Issue-ID: RELENG-4011
Signed-off-by: Kevin Sandi <ksandi@contractor.linuxfoundation.org>
Change-Id: I6045b186bfde76e19d77f50ef14c98107e2cb0c5
Andrew Grimberg [Mon, 18 Oct 2021 23:51:39 +0000 (16:51 -0700)]
Fix: Correct boot source options for OpenStack
If a VOLUME_SIZE option is not defined in the base cloud configuration
then the default OpenStack boot selection should be 'image' not
'volumeFromImage' otherwise we are unable to properly take advantage of
all options provided by an OpenStack cloud.
Change-Id: I76ae042d598c9f7fa1868d4ea6a3566f8c6b8115
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
Andrew Grimberg [Fri, 15 Oct 2021 00:05:27 +0000 (17:05 -0700)]
Fix: Correct bad conversion of OpenStack config
While attempting to get the OpenDaylight Jenkins Sandbox configured to
start working with JCasC cloud management it was discovered that the
numExecutors flag was being mistranslated.
Change-Id: I5598f9936a2a3190e0ff326dad8ef4503aaceaa2
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
Anil Belur [Tue, 5 Oct 2021 01:47:15 +0000 (11:47 +1000)]
Feat: Update default version of JDK to openjdk11
JDK8 is soon going to be EOL by Mar 31, 2022. with most of the
LF projects already on JDK11 (LTS), upgrade the default
version to JDK11 (LTS).
https://www.oracle.com/java/technologies/java-se-support-roadmap.html
Issue-ID: RELENG-3978
Change-Id: I56359e72ef58602208242b076f35b19a1b7f5a1d
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Eric Ball [Fri, 24 Sep 2021 01:54:54 +0000 (18:54 -0700)]
Feat: Allow use of newer JDKs for Sonar scans
Many builders still default to openjdk8 for java, but Sonarcloud
only supports openjdk11+.
Issue: RELENG-3964
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Change-Id: Ida96f7f242a397a60e40e26f910510d665bcf577
Eric Ball [Fri, 24 Sep 2021 00:08:24 +0000 (17:08 -0700)]
Fix: Update sphinx requirements
Our older sphnix requirements are out of date, and are trying to pull
in packages that are no longer supported.
Change-Id: I69d336d324af2a512b8d756f0e8f83c244642029
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Eric Ball [Wed, 15 Sep 2021 22:54:21 +0000 (15:54 -0700)]
Feat: Add "Unmaintained" lifecycle_state for INFO
This adds "Unmaintained" as a valid lifecycle_state to be used in
INFO.yaml files, per request from ONAP.
Issue: RELENG-3955
Change-Id: I8dfb947b8a0936c552c087dda6c115563b69c74f
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Eric Ball [Thu, 9 Sep 2021 21:34:38 +0000 (14:34 -0700)]
Fix: Add missing v3-standard Vexxhost flavors
Issue: RELENG-3935
Change-Id: Ib3f317bc918a9c742ee339808c84ec432b0af8ee
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Eric Ball [Thu, 9 Sep 2021 19:12:00 +0000 (19:12 +0000)]
Merge "Feat: Allow scans in short lived branches"
Jessica Wagantall [Fri, 25 Jun 2021 19:32:25 +0000 (12:32 -0700)]
Feat: Allow scans in short lived branches
Add functionality to allow scans of short lived
branches in SonarCloud.
Developers will be able to turn this feature on
by setting "scan-dev-branch: true"
Issue: RELENG-3937
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I6b32771ba00e86927ba179596c467b37c831bd2d
Eric Ball [Thu, 9 Sep 2021 18:29:14 +0000 (11:29 -0700)]
Feat: Add v3-starter flavors to cloud lookup
Issue: RELENG-3935
Change-Id: I114ca41c020c3d408d2afdf0934f57c166312ad8
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Andrew Grimberg [Wed, 1 Sep 2021 18:04:48 +0000 (11:04 -0700)]
Chore: Upgrade pre-commit hooks
* github.com/pre-commit/pre-commit-hooks: v3.4.0 -> v4.0.1
* github.com/pre-commit/mirrors-prettier: v2.2.1 -> v2.3.2
* github.com/adrienverge/yamllint: v1.26.1 -> v1.26.3
Change-Id: I0ec2f3e17da3c5215a824ebe501468796f0d6316
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
Eric Ball [Thu, 19 Aug 2021 21:58:20 +0000 (14:58 -0700)]
Fix: Labels should always include config name
Openstack labels need to include the config name, in addition to any
labels explicitly defined. This also changes the builder name to
match the config name, rather than using the labels (which can be
only one label, but is technically a space-separated list).
Change-Id: I29ccc9987aa02d1e96930d165908396b46223fa8
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Eric Ball [Thu, 19 Aug 2021 19:02:44 +0000 (12:02 -0700)]
Fix: If no volume set, use "image" type
If no volume_size is defined, the default behavior was to set one to
10GB. However, the proper way to handle this is to use an "Image"
boot source rather than "Volume From Image".
Change-Id: Ib4a33f224ea1044b076d965faa164686da071cc4
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Eric Ball [Tue, 10 Aug 2021 21:50:42 +0000 (14:50 -0700)]
Fix: Use cloud agent name as default node label
Rather than quitting if there are no labels defined, we can instead
use the agent name (e.g. "centos7-2c-1g") as the default label. This
recreates the functionality of the groovy scripts previously used.
Issue: RELENG-3863
Change-Id: I3817cca81358fa151fdeb1bbc8516df983327965
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Eric Ball [Tue, 10 Aug 2021 21:44:05 +0000 (14:44 -0700)]
Refactor: Simplify and improve readability
This includes improvements to PEP8 formatting, grammar, and variable
naming, plus several changes to make code more efficient or readable.
Issue: RELENG-3863
Change-Id: I823db03175f96db3b99ee5fcc715345462a47139
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Eric Ball [Thu, 15 Jul 2021 18:41:22 +0000 (11:41 -0700)]
Fix: Install lftools before other packages
lftools has an incompatibility with python-docker >4.2, and this
requirement seems to be causing a conflict with other packages that
python-tools-install is installing. This causes an older version of
lftools to be installed (0.31.1, the last version when docker was not
pinned).
By installing lftools first, its requirements will take
precedence over other packages.
Issue: RELENG-3792
Change-Id: I0d3ca88dbe51c97ce495f6db0eb66829a61ad3bc
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Guillaume Lambert [Thu, 17 Jun 2021 12:43:00 +0000 (14:43 +0200)]
Feat: allow more tox parallel mode configurations
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I8a812ed25f3836c100704914f08491a61642679f
Guillaume Lambert [Thu, 17 Jun 2021 12:15:24 +0000 (14:15 +0200)]
Refactor: how to pass options to tox in tox-run.sh
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I3cae2a28f8cdf10b4b547f729d436cb204322247
Guillaume Lambert [Thu, 17 Jun 2021 09:36:01 +0000 (11:36 +0200)]
Docs: remove reference to detox
Detox globaljjb support was removed in a previous commit.
1873622e1b5bf98150db6850901d190dd6c35422
Parallel jobs are now natively supported by tox since version 3.7.0
thanks to the option "-p" / "--parallel".
This new option offers more possibilities than detox and other options
have also been introduced to tune tox behavior in parallel mode.
The configuration choices made in globaljjb must be clarified.
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I82c88ea96ee1332b95ab13f0b7dbf2a04e00f37a
Vratko Polak [Tue, 8 Jun 2021 15:16:48 +0000 (17:16 +0200)]
Fix: Skip vote for jjb-deploy also when successful
The previous code used "success" instead of "successful".
Fixes:
bfe91be03d516a56adb74abd1e16c50b86c27c6e
Change-Id: If73f2f2756c10d0a7230ad1214e8ea3c86b1b114
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
Vanessa Rene Valderrama [Thu, 13 May 2021 17:22:28 +0000 (12:22 -0500)]
Feat: Add support for Ubuntu 20.04 and Docker
Updating the sysstat script to support Ubuntu 20.04 and Docker systems
Signed-off-by: Vanessa Rene Valderrama <vvalderrama@linuxfoundation.org>
Change-Id: I12e6daa8a04228337c99b004e67338eb8f6a8f1a
Guillaume Lambert [Tue, 11 May 2021 16:30:51 +0000 (18:30 +0200)]
CI: bashate warns lines>80 and forbids >120
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: Ib981e33c31c704ef3e8fb434f8766b0e217b2b1d
Guillaume Lambert [Tue, 4 May 2021 11:46:48 +0000 (13:46 +0200)]
Fix: bashate E006 warnings for lines > 120 chars
Lines too long can not only cause problems when reading/writing code,
but also often indicates a bad smell,
e.g. too many levels of indentation due to overly complex functions
which requires refactoring into smaller chunks.
Note that the bashate current profile reports every line > 80 chars.
After this commit, 211 warnings of this category will still remain.
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: Ibbb171353fd960ea93145f7913e62e327bd6a86b
Guillaume Lambert [Tue, 4 May 2021 15:21:00 +0000 (17:21 +0200)]
CI: enforce bashate tox profile
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I920542ec62e67ee37aebd554076bd79d6863212a
Guillaume Lambert [Tue, 4 May 2021 12:13:45 +0000 (14:13 +0200)]
Fix: bashate E003 warnings
Indentations must be aligned to multiple of 4 columns.
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I0bdb8b2fafd1811e55fbbacaa4c4cb6a2b59f822
Guillaume Lambert [Tue, 4 May 2021 09:27:12 +0000 (11:27 +0200)]
Refactor: use POSIX shell function declaration
POSIX function style is already used in most of shell scripts.
By the way, this style fixes bashate E020 warnings.
Non POSIX function declarations in a format different from
^function name {$ are warned by bashate.
There are several equivalent ways to define functions in Bash.
This is for consistency.
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I5f02e7052101617314157ec408532dd024e80a72
Guillaume Lambert [Tue, 4 May 2021 08:31:14 +0000 (10:31 +0200)]
Fix: bashate E002 warnings about tabs indents
Tabs must be avoided and replaced by 4 whitespaces according to the
standard usage.
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: Ic21fe1646b60b6630d54fd16b8b2e1fa6d4c49fb
Guillaume Lambert [Tue, 4 May 2021 09:01:41 +0000 (11:01 +0200)]
Fix: bashate E011 warnings
E011 rule ensures consistency of "then" directive being on the same line
as it's condition.
This is similar to E010 in the previous commit
but to ensure the consistency of if/elif statements.
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I6d3ff5b468fa47948b0123c017980e00a94cf82f
Guillaume Lambert [Tue, 4 May 2021 08:25:50 +0000 (10:25 +0200)]
Fix: bashate E010 warnings
E010 rule ensures consistency of "do" directive being on the same line
as it's command.
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: Id7717658728968da017107c4b496cdfbe0c5b6b0
Guillaume Lambert [Tue, 4 May 2021 08:14:24 +0000 (10:14 +0200)]
Fix: bashate E043 warnings
Arithmetic compound has inconsistent return semantics
((expr)) has a trap that if expr is 0, the return value is 1.
This will trigger a failure with "set -e" and can be very confusing.
It is good defensive programming to avoid this with explicit assignment.
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I412bd8872560c90c28642ed5433eab9b2a29a5e8
Guillaume Lambert [Mon, 3 May 2021 18:52:24 +0000 (20:52 +0200)]
CI: Add bashate tox profile
Bashate is a shell linter inspired from PEP8.
It helps improving shell scripts.
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I351a205cbfdc160660958069a8060badb68aa28a
Andrew Grimberg [Mon, 10 May 2021 13:38:54 +0000 (13:38 +0000)]
Merge "Fix: Set S3 URL in the target framename"
Anil Belur [Sat, 8 May 2021 07:21:02 +0000 (17:21 +1000)]
Fix: Set S3 URL in the target framename
Set S3 URL in the framename of the target attribute.
The description-setter plugin does not read the URL
when the framename is unset.
Issue: RELENG-3269
Change-Id: I47c488b586cf68164a0273a14134e490c8c12cec
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Andrew Grimberg [Sat, 8 May 2021 14:40:45 +0000 (07:40 -0700)]
CI: Properly run gitlint in CI
When running pre-commit in a CI system the COMMIT_EDITMSG does not
(normally) get created as that is an artifact of editing the commit
message. If the file doesn't exist then gitlint will skip which makes it
possible for pre-commit checks that should fail, to pass.
Since we want tox to run in a consistent manner both locally and in CI
we need to play around a little with how we are checking the commit
message.
Change-Id: I3990aa9846dc8479cadaad5025e45f863623bf86
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
Andrew Grimberg [Wed, 5 May 2021 16:52:33 +0000 (09:52 -0700)]
CI: Configure gitlint for Conventional Commits
Turn on Conventional Commit message subject validation. This will affect
CI specifically, but developers can avoid having CI fail by assuring
that they have installed pre-commit and that they have also run
pre-commit install --hook-type commit-msg
The above is needed because pre-commit does not install commit-msg hooks
by default
Change-Id: If45c7a256f7d769eb092d867217468b9d6c53384
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
Andrew Grimberg [Wed, 5 May 2021 16:45:36 +0000 (09:45 -0700)]
CI: Enforce yamllint in pre-commit
Turn on yamllint enforcement via pre-commit. This affects both the CI
system as well as developers that pre-commit configured in their
environment (like they should)
Change-Id: If1ad8619a181ea916bcadc9eaddc58207fbef720
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
Andrew Grimberg [Wed, 5 May 2021 16:42:25 +0000 (09:42 -0700)]
Style: Cleanup yamllint errors
Cleanup yamllint errors before we start enforcing via pre-commit
Change-Id: Ie476be877d2896e1bc66ed4238bac0f2a1a33239
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
Andrew Grimberg [Wed, 5 May 2021 16:32:23 +0000 (09:32 -0700)]
CI: Rename yamllint config file
Yamllint does not look for a yamllint.conf file and we're calling it
(that I can find) anywhere with an explicit config file definition.
Since .yamllint is the default file used, we are fixing the filename
Additionally, we are adding a section on comments to protect against the
prettier pre-commit hook that enforces a single space instead of
yamllints normal double space for comments after content
Change-Id: I0df4b467ea57b1558dbb3c71d74775f84a88b037
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
Andrew Grimberg [Wed, 5 May 2021 15:58:48 +0000 (08:58 -0700)]
CI: Update pre-commit checks to latest versions
This updates the pre-commit checks to the latest versions of the checks
we're using and fixes a linting issue in our yaml config
Change-Id: Ib9dc085ab6bf77fca33b884eb2005094b53f7d40
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
Guillaume Lambert [Mon, 3 May 2021 13:22:38 +0000 (15:22 +0200)]
Add protection in python-tools-install.sh
This script may result in unexpected behavior on distributions
other than Ubuntu with a matching release.
DISTRIB_ID must be tested to avoid such problems.
Signed-off-by: Guillaume Lambert <guillaume.lambert@orange.com>
Change-Id: I2ba02b613a4aaadfeb09bd66ddfde8f36a9510b4
Andrew Grimberg [Thu, 29 Apr 2021 16:40:01 +0000 (09:40 -0700)]
Fix: Correct choosing strategy use on merge
The choosing strategy being used by the docker-merge-{stream} jobs for
gerrit is set to gerrit and not default. Since this is a merge job it's
supposed to always pick up the tip / head of the repository.
Issue: RELENG-3657
Change-Id: I3b97bf01966aa85d8066ebea301d3205398e7e63
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
Eric Ball [Tue, 30 Mar 2021 21:14:51 +0000 (21:14 +0000)]
Merge "Remove all refs to python2"
Jessica Wagantall [Tue, 30 Mar 2021 17:04:40 +0000 (10:04 -0700)]
Provision settings for WS scan
Provision global-settings to replace the
default used by the Unified Agent.
This will allow dependencies to be downloaded the
same way the maven build does.
Issue: RELENG-3430
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I2b751c7d7d966b0b9b07a90405ec61b2254bed6d
Anil Belur [Tue, 30 Mar 2021 04:13:59 +0000 (14:13 +1000)]
Remove all refs to python2
Python 2.7 is EOL and no longer supported. Some of the
lf-macros and docs still refer to py2 which could
mislead users.
Change-Id: I9f39948cec30fb573f25ee7ecb3bed8bf3600ab9
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Andrew Grimberg [Fri, 26 Mar 2021 14:59:24 +0000 (14:59 +0000)]
Merge "Update prettier pre-commit to ignore stage error"
Eric Ball [Thu, 25 Mar 2021 19:58:09 +0000 (12:58 -0700)]
Add sonar-prescan-script jobs for maven
We added these for the more generic Sonar jobs in lf-ci-jobs, but had
not added them for maven. We now have a need for this job type (it can
be used to ensure that a Sonarcloud-compatible version of nodejs is on
the system), so they are being added now.
Issue: IT-21602
Change-Id: Ic32f418a042839ab277dc1b85777d0eef475d620
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Andrew Grimberg [Tue, 23 Mar 2021 21:58:05 +0000 (21:58 +0000)]
Merge "Pin git review to 1.78"
Jessica Wagantall [Fri, 12 Mar 2021 19:53:19 +0000 (11:53 -0800)]
Update WS Unified Agent version
Update WhiteSource Unified Agent to
version 21.2.1
Issue: RELENG-3430
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I5c4ae6d49b1822a04fad1ce9af955bf9e06d3851
Anil Belur [Wed, 10 Mar 2021 07:20:54 +0000 (17:20 +1000)]
Update prettier pre-commit to ignore stage error
Set stages to 'commit' to overide the default_stages.
This fixes the issue seen while using git-commmit:
[error] No matching files. Patterns: .git/COMMIT_EDITMSG
Change-Id: I959744b89e6cb22e84fca9119a1405e354734cbb
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Anil Belur [Tue, 9 Mar 2021 04:34:53 +0000 (14:34 +1000)]
Pin git review to 1.78
The latest version of module trys to look for git hook recursively within
the submodules.
Error:
Entering 'global-jjb'
cannot stat '.git/hooks/commit-msg': Not a directory
fatal: run_command returned non-zero status for global-jjb
Remove workaround that has been resolved in v1.28 and use lf-activate-venv
to install git-review
Issue: RELENG-3435
Change-Id: I67b3730dcdf5a410020c63312897bb67b42017bd
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Bengt Thuree [Fri, 5 Mar 2021 03:29:41 +0000 (03:29 +0000)]
Merge "Update the create script with V3 flavors."
Jessica Wagantall [Tue, 2 Mar 2021 19:25:28 +0000 (11:25 -0800)]
Reorder release-job script
Reorder functions and add function labels to make
release-job.sh easier to read.
Fix "Double quote to prevent globbing" line 34
Issue: RELENG-3376
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: Ic8349f692a50a645a99e884c852013f75332cccd
Jessica Wagantall [Wed, 20 Jan 2021 21:10:01 +0000 (13:10 -0800)]
Add support for artifact releases
Currently Dent is the only customer for artifact
releases where they need a particular binary to be
pulled from Snapshots and pushed manually into Releases.
These are artifacts built using make.
Issue: RELENG-3376
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I5a8ec8e2906757b1700505ad10dd2ba8ac844f9a
Anil Belur [Sat, 20 Feb 2021 01:00:50 +0000 (11:00 +1000)]
Pin cryptography to 3.3.2
The latest version of module breaks compatibility with the
latest version of pip.
Error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-vqk6fya9/cryptography/setup.py", line 14, i
in <module> from setuptools_rust import RustExtension
ModuleNotFoundError: No module named 'setuptools_rust'
Issue-Id: IT-21538 IT-21509
Change-Id: I83db995ae8321377b861cabcf6ac53485778e471
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Anil Belur [Mon, 22 Feb 2021 08:39:51 +0000 (18:39 +1000)]
Fix reno lint error in sention names
Error:
unrecognized section name
Update the version of reno as per the requirements.txt
Change-Id: Iadc07de41934c936fc82147bc8ebcd4f6135764e
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Anil Belur [Thu, 18 Feb 2021 02:06:33 +0000 (12:06 +1000)]
Update pre-commit hook shellcheck to 2.1.5
The workaround with
4d088120467 is no longer
required with the latest 2.1.5 release.
Change-Id: Idf059bb7e184beb7c5ca9c5c5c97f5244ecbcec5
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Anil Belur [Wed, 17 Feb 2021 21:43:16 +0000 (21:43 +0000)]
Merge "Add workaround for pre-commit 2.10.0"
Eric Ball [Wed, 17 Feb 2021 00:40:46 +0000 (16:40 -0800)]
Add workaround for pre-commit 2.10.0
The jumanjihouse shellcheck hook errors without
additional_dependencies being defined. While there is a PR open to
add this fix, it has not yet been merged and released. Until it is,
this workaround will enable using the latest versions of pre-commit
and the jumanjihouse hooks.
Github PR: https://github.com/jumanjihouse/pre-commit-hooks/pull/80
Change-Id: I0ec16952009128e25cccb6833698bf1bd864af2d
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Andrew Grimberg [Tue, 16 Feb 2021 22:35:58 +0000 (22:35 +0000)]
Merge "Fix rtd-verify submodule update command"
Eric Ball [Fri, 5 Feb 2021 00:11:09 +0000 (16:11 -0800)]
Fix rtd-verify submodule update command
The old version is missing the "--init" flag, which causes it to not
add new submodules. This did not show up as an issue until a new
submodule was added in a production environment.
Additionally, the "--recursive" flag has been included to ensure
proper loading of recursive submodules.
Issue: IT-21490
Change-Id: Ie882980d3721ae0e1fea9fa1d3321b86d18be484
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Jessica Wagantall [Wed, 10 Feb 2021 22:29:25 +0000 (22:29 +0000)]
Merge "Add schema for artifact releases"
Aric Gardner [Fri, 29 Jan 2021 16:33:45 +0000 (11:33 -0500)]
Remove python2 tox installation
Update path to ensure that tox from tox-install.sh is run
Should still work for python2 tox
as pyenv will continue to provide a py2 env
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I568ebf679bbce20b76a0a686699702f81e7e7777
Trevor Bramwell [Thu, 28 Jan 2021 17:53:35 +0000 (09:53 -0800)]
Skip Metadata Capture on unsupported Clouds
Updates the 'capture-instance-metadata.sh' script to skip attempting to
capture instance metadata needed for job-cost.sh if the build is being
run on an unsupported cloud or platform.
Change-Id: I8bfea1416d11de1caeed2d414b90513f63de9b6f
Signed-off-by: Trevor Bramwell <tbramwell@linuxfoundation.org>
Anil Belur [Fri, 8 Jan 2021 23:15:36 +0000 (09:15 +1000)]
Update the create script with V3 flavors.
Issue: RELENG-3329
Change-Id: I46f8b87a60a6f4538f5b76168a51ec242b5963c2
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Anil Belur [Thu, 28 Jan 2021 12:28:09 +0000 (12:28 +0000)]
Merge "Upgrade packer version 1.6.6"
Trevor Bramwell [Wed, 27 Jan 2021 22:37:51 +0000 (14:37 -0800)]
Fix Sudo Log Ownership for non-'jenkins' users
If builds are run with a user other than 'jenkins', chown'ing of the
sudoers log will fail. Switching the command to use the current
user (and user's default login group) fixes this.
Note: See chown(1) DESCRIPTION for an explanation of using a colon after
the username while omitting the group
Change-Id: Ia9b96e93a250fd22eb36c94471a06c2e211dc9e3
Signed-off-by: Trevor Bramwell <tbramwell@linuxfoundation.org>
Jessica Wagantall [Wed, 27 Jan 2021 19:24:19 +0000 (11:24 -0800)]
Add schema for artifact releases
Add initial schema for artifact self
releases.
Issue: RELENG-3376
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I78fe9b602b34b9f062599a6ef823f177975f26e4
Eric Ball [Wed, 27 Jan 2021 23:08:25 +0000 (15:08 -0800)]
Pin pre-commit to v2.9.3 in tox.ini
A change in v2.10.0 of pre-commit introduces an incompatibility with
jumanjihouse/pre-commit-hooks, particularly the shellcheck hook. It
isn't clear whether this is intentional or not, but until either
pre-commit or the jumanjihouse hooks are updated, we need to pin to
the working version of pre-commit.
Change-Id: I2aa74f9310f55603d2e57d83f79c6f1a51726994
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Trevor Bramwell [Wed, 27 Jan 2021 20:05:06 +0000 (12:05 -0800)]
Skip Job Cost Calulations on Non-Cloud Platforms
When 'job-cost.sh' is ran on a VM not initialized with cloud-init, or
running in OpenStack, attempts to read '/run/cloud-init/result.json'
will fail causing the job to become unsable.
This change adds a check for any relevant files to the beginning of
'job-cost.sh' in order to skip the calculation on VMs not running on
either AWS or OpenStack.
Change-Id: I878adee1ba3ceeffa574a1b1c7ae103811344123
Signed-off-by: Trevor Bramwell <tbramwell@linuxfoundation.org>
Jessica Wagantall [Tue, 26 Jan 2021 21:17:49 +0000 (13:17 -0800)]
Add release examples GitHub
Add release-job examples for Github
projects.
Issue: RELENG-3346
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I41ff49f1d36f724fae060f243eed0521a1fc8859
Jessica Wagantall [Wed, 20 Jan 2021 21:55:55 +0000 (13:55 -0800)]
Add self release jobs for GitHub
Add self release support for GitHub based
repos.
Issue: RELENG-3366
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I84e56c8f63459bf0ecd69f6521a580383df1d6fd
Eric Ball [Mon, 25 Jan 2021 20:21:52 +0000 (20:21 +0000)]
Merge "Add regex for common-packer var and templates"
Andrew Grimberg [Mon, 25 Jan 2021 18:13:50 +0000 (18:13 +0000)]
Merge "Fix release job script"
Anil Belur [Mon, 25 Jan 2021 04:00:59 +0000 (14:00 +1000)]
Remove python 2.7 support
As per the deprecation notice python 2.7 is not long supported.
This causing job failures since the dependencies install
are not maintained.
DEPRECATION: Python 2.7 reached the end of its life on January 1st,
2020. Please upgrade your Python as Python 2.7 is no longer
maintained. pip 21.0 will drop support for Python 2.7 in January
2021. More details about Python 2 support in pip can be found at
https://pip.pypa.io/en/latest/development/release-process/#python-2-support
pip 21.0 will remove support for this functionality.
Issue: RELENG-3375
Change-Id: I48cc4b91b9916918d222794534db5ade9b58dda0
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Anil Belur [Sat, 23 Jan 2021 09:55:11 +0000 (19:55 +1000)]
Add regex for common-packer var and templates
Add regex to trigger packer jobs when common-packer templates
are updated.
Change-Id: Ie9d79e0c2e4e525cd19c3ab10066b5df5ae65742
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Anil Belur [Sat, 23 Jan 2021 07:43:52 +0000 (17:43 +1000)]
Upgrade packer version 1.6.6
Change-Id: I3d5c36e6e9452f09784883e86eab4eb37dccf13f
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Anil Belur [Sat, 23 Jan 2021 07:33:06 +0000 (17:33 +1000)]
Fix release job script
Fix the release job script to handle LOG_DIR unbound variable
and condition to check if the LOGS_SERVER or CDN_URL is being used.
Issue: IT-21396
Change-Id: I3960a49aa12858af2ed4f47a1ec714ff793b70fd
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Anil Belur [Sat, 23 Jan 2021 02:24:04 +0000 (02:24 +0000)]
Merge "Revert "Remove the -only CLI option for packer build""
Anil Belur [Sat, 23 Jan 2021 02:23:15 +0000 (02:23 +0000)]
Merge "Revert "Pass the builder type to verify jobs""
Anil Belur [Sat, 23 Jan 2021 01:41:25 +0000 (11:41 +1000)]
Revert "Remove the -only CLI option for packer build"
This reverts commit
c6d9571b076faf5d02305bd93faeda7bf23fe766.
Based on the input provided on the Gihub packer issue
hard-coded for our parser to be able to decode the rest of the config
properly."
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Change-Id: I58debd1a961b92c3f7705a3f65a8bc883722596a
Anil Belur [Sat, 23 Jan 2021 01:37:14 +0000 (11:37 +1000)]
Revert "Pass the builder type to verify jobs"
This reverts commit
e0a9d10d3d6b488ed3a3b00050b34d9faee4a6f8.
Based on the input provided on the Gihub packer issue
hard-coded for our parser to be able to decode the rest of the config
properly."
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Change-Id: If7162ced328d58b24ab9339f7e81d7dff76c90bb
Jessica Wagantall [Fri, 22 Jan 2021 20:10:50 +0000 (20:10 +0000)]
Merge "Add repo tag support for GitHub"
Anil Belur [Fri, 22 Jan 2021 02:04:03 +0000 (12:04 +1000)]
Pass the builder type to verify jobs
The default builder type must be passed to packer verify jobs. For
packer jobs build on aws, set the `packer-builder: aws` in downstream
CI-management.
Change-Id: I599d167a957189584ef5b796c049d82e5a85ec7a
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Jessica Wagantall [Wed, 20 Jan 2021 22:54:19 +0000 (14:54 -0800)]
Add repo tag support for GitHub
Rename "tag-gerrit-repo" to "tag-git-repo" in
preparation to support self releases for Github
based projects.
Issue: RELENG-3368
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I6ceb6675a2f37cbfd1d008046aecb734665db6d3
Anil Belur [Sat, 16 Jan 2021 05:21:59 +0000 (15:21 +1000)]
Remove the -only CLI option for packer build
Remove the -only CLI option for packer build
The #CR
90dd675df482a8 submitted with common-packer v0.7.0
fails to build packer jobs because of a bug in the packer build CLI.
The `-only` option packer build CLI does not set the cloud provider
name/type as per the documentation. The alternative is to use an environment
variable that is passed through the packer job and can be reference in the
packer templates.
Issue: JIRA IT-21300
Github Issue: https://github.com/hashicorp/packer/issues/10495
Ref: https://gerrit.linuxfoundation.org/infra/c/releng/common-packer/+/66528
Change-Id: I97bc479ddac9a8f1f0dc83201735b0a084a8b3e4
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Andrew Grimberg [Wed, 30 Dec 2020 23:07:43 +0000 (23:07 +0000)]
Merge "Handle S3_BUCKET & CDN_URL in release jobs"
Anil Belur [Sun, 22 Nov 2020 22:26:30 +0000 (08:26 +1000)]
Handle S3_BUCKET & CDN_URL in release jobs
Fix release merge jobs to work with AWS S3_BUCKET when CDN_URL
is set.
Log shipping is being migrated from Nexus2 log server to AWS S3
Buckets. This requires the release job scripts to handle CDN_URL
and set the correct value for LOG_SERVER.
Issue: RELENG-3307
Change-Id: I508dd15508d9c9dd3741edf220d64526abed644f
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Andrew Grimberg [Mon, 28 Dec 2020 19:56:04 +0000 (19:56 +0000)]
Merge "Add pipeline-verify jobs to lint and check agents"
Eric Ball [Wed, 9 Dec 2020 16:19:51 +0000 (08:19 -0800)]
Add pipeline-verify jobs to lint and check agents
This adds a simple pipeline verification job that will lint any
pipelines found, and check to make sure that they are not set up to
run on master.
Issue: RELENG-2185
Change-Id: I76d400d145857887a6d28d7c2a9869ec4acf414d
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Andrew Grimberg [Mon, 14 Dec 2020 17:55:38 +0000 (17:55 +0000)]
Merge "Use write-good release tag v1.0.4"
James E. Blair [Wed, 2 Dec 2020 22:24:39 +0000 (14:24 -0800)]
Create the archives directory in deploy-logs
The 'lftools deploy archives' command expects $WORKSPACE/archives
to already exist. Normally it would be created by lf-infra-sysstat,
but to support the use of the deploy-logs macro in jobs which do not
run sysstat (and generally to make this macro entirely self-contained),
ensure the archives directory exists at the start of the script.
Signed-off-by: James E. Blair <jeblair@redhat.com>
Change-Id: Id90c77c3f2a6a9590b9b17f9223279d21066a3cf
Jessica Wagantall [Wed, 9 Dec 2020 19:08:39 +0000 (11:08 -0800)]
Activate virtual env for logs-deploy
Activate lftools env before processing the
logs-deploy script.
This will allow us to fetch the latest version of
pyOpenSSL and cryptography to resolve the following
issue:
https://github.com/pyca/pyopenssl/issues/973
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: Ib8872cad64630573cf6209ce1012dd5d5eb7cf69
Anil Belur [Tue, 8 Dec 2020 09:35:13 +0000 (19:35 +1000)]
Use write-good release tag v1.0.4
Issue: RELENG-3257
Change-Id: Ibf51a3f144ea9ce4c235659e7cd51661deddfb8f
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>