releng/global-jjb.git
4 years agoMerge "Add packer-verify-build jobs" v0.53.0
Anil Belur [Tue, 21 Apr 2020 01:44:31 +0000 (01:44 +0000)]
Merge "Add packer-verify-build jobs"

4 years agoAdd packer-verify-build jobs 37/63637/7
Eric Ball [Thu, 9 Apr 2020 01:53:58 +0000 (18:53 -0700)]
Add packer-verify-build jobs

This job is made to be manually triggered in order to build the
packer image, so that the full build process can be validated before
merging. If this is done on the final patch that is merged, the
merge job will not run another build (thereby avoiding building two
identical images).

The verify-build job piggybacks on the merge template, since it is
essentially the same job. However, it only triggers when given the
keyword "verify-build" or "packer-build". Due to the similarities,
some documentation improvements have also been shifted over to the
Packer Merge docs, as well as some small changes to Packer Verify.

Issue: RELENG-2671
Change-Id: Id4fb33a88447bbb2f09ac0967ac303a8f6d62326
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
4 years agoMerge "Fix lftools install in rtd verify job"
Andrew Grimberg [Fri, 17 Apr 2020 17:24:09 +0000 (17:24 +0000)]
Merge "Fix lftools install in rtd verify job"

4 years agoFix lftools install in rtd verify job 81/63681/4
Anil Belur [Thu, 16 Apr 2020 04:06:56 +0000 (14:06 +1000)]
Fix lftools install in rtd verify job

Issue: RELENG-2846
Change-Id: I5c3cd6bf8f4af887ba00aa2e1a4d42775caac80a
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoUse yq instead of niet 79/63679/2
Jessica Wagantall [Wed, 15 Apr 2020 23:10:24 +0000 (16:10 -0700)]
Use yq instead of niet

Standardize yaml parsing tools used across global-jjb.
Switch from using niet package to yq for parsing yaml files
in release-job.sh.

Issue: RELENG-2844
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: Ia57884a7a8efffa85417becd8c914ab8d23721a9

4 years agoBool is not a valid schema type 78/63678/3
Jessica Wagantall [Wed, 15 Apr 2020 22:51:08 +0000 (15:51 -0700)]
Bool is not a valid schema type

Using boolean rather than bool.

Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: Ied68c7f1a2fd4474e9fee65b2820e6c25f662b5f

4 years agoOptionally skip tag during release process 25/63625/12
Jessica Wagantall [Tue, 7 Apr 2020 19:34:14 +0000 (12:34 -0700)]
Optionally skip tag during release process

Make tag process optional. True by default.

Example scenario:
Some ONAP projects make both maven and docker releases
for a single repo. Docker releases depend on maven releases.
This means that the team needs to make a maven release first,
then make a docker release and lastly, tag the repo.

This change will allow them to hold the tagging part during
maven releases to allow them to tag after the docker release is made.

Issue: RELENG-2835
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: Ibd206269b80e7fb788682a81e038aa74d9eaf97c

4 years agoMerge "Add "logs" prefix to s3_path"
Houa Yang [Thu, 9 Apr 2020 21:24:11 +0000 (21:24 +0000)]
Merge "Add "logs" prefix to s3_path"

4 years agoAdd "logs" prefix to s3_path 46/63646/3
Houa Yang [Thu, 9 Apr 2020 19:25:05 +0000 (14:25 -0500)]
Add "logs" prefix to s3_path

We are not allowed to have an index.html file at the root level of the
S3 bucket. Adding the additional prefix allows us to mirror the directory
structure of Nexus where you are able to see both production and sandbox
logs when browsing to https://logs.<project>.org/

Issue-Id: RELENG-2379, RELENG-2380
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I7b6bd340170f9390c670f2b6ba507f2de1cfbc0e

4 years agoAllow rtdv3 jobs to run in parallel 32/63632/1
Jessica Wagantall [Wed, 8 Apr 2020 17:23:23 +0000 (10:23 -0700)]
Allow rtdv3 jobs to run in parallel

These jobs are submitted against different projects.
Allow them to run in parallel to save time.

Change-Id: I199b07c4b580fdf7be82e5503b8dc8a1e3915e89
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
4 years agoAdd doc for jjb-verify build-node label check 21/63621/1
Lott, Christopher (cl778h) [Tue, 7 Apr 2020 13:11:26 +0000 (09:11 -0400)]
Add doc for jjb-verify build-node label check

Document the jjb-verify feature that checks build-node labels named
in JJB YAML files against nodes in cloud config files.
Change the feature's JJB configuration variables to share a prefix.
Improve the shell script to be robust to errors such as a suffix of
".cfg" or an external label of "" (just two double quotes).

Change-Id: I8beb5669531fe2b7f59a9bcf3706b27ede3965b9
Issue-ID: RELENG-2828
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoMerge "Add CMake templates to build/publish DEB/RPM pkgs"
Anil Belur [Tue, 7 Apr 2020 01:39:32 +0000 (01:39 +0000)]
Merge "Add CMake templates to build/publish DEB/RPM pkgs"

4 years agoAdd CMake templates to build/publish DEB/RPM pkgs 93/63593/5
Lott, Christopher (cl778h) [Fri, 3 Apr 2020 14:53:43 +0000 (10:53 -0400)]
Add CMake templates to build/publish DEB/RPM pkgs

New templates gerrit-cmake-packagecloud-stage and
github-cmake-packagecloud-stage for building DEB/RPM package
files and publishing them to a PackageCloud.io repository on
posted comment.  Adds builder macros lf-packagecloud-file-provider
and lf-packagecloud-push.  Adds script packagecloud-push.sh to
call Ruby gem package_cloud.
The new templates are lifted from the ORAN project.

Change-Id: Ia39f5b43c2ef22485166a5f54fcf31d53eab84f8
Issue-ID: RELENG-2832
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoFix aws credential file provider conditional 94/63594/3
Houa Yang [Fri, 3 Apr 2020 17:04:16 +0000 (12:04 -0500)]
Fix aws credential file provider conditional

Add "on-evaluation-failure: dont-run" to conditional, so that if the
global env variable is not set, the job will not fail.

Issue-Id: RELENG-2749
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I3a7d992bfd93853ffe910713c00380852868f20e

4 years agoMerge "Verify build nodes in YAML against config files"
Thanh Ha (zxiiro) [Fri, 3 Apr 2020 13:34:19 +0000 (13:34 +0000)]
Merge "Verify build nodes in YAML against config files"

4 years agoMerge "Add AWS credentials config file"
Houa Yang [Wed, 1 Apr 2020 18:33:59 +0000 (18:33 +0000)]
Merge "Add AWS credentials config file"

4 years agoVerify build nodes in YAML against config files 34/63534/18
Lott, Christopher (cl778h) [Sun, 29 Mar 2020 22:53:05 +0000 (18:53 -0400)]
Verify build nodes in YAML against config files

Extend lf-jjb-verify anchor with boolean configuration variable
check-build-node-labels that guards a conditional build step.
If true, run script jjb-verify-build-nodes.sh to check build-node
labels in YAML files within the jjb subdirectory against labels
defined by config files in the jenkins cloud configuration
directory.  Disabled by default.  Projects should enable and
configure the job; e.g., for external build node labels.

Issue-ID: RELENG-2828
Change-Id: Iaf315b4a68c3488b5ead204a57361a8bcbca91d7
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoMerge "Add PyPI stage templates"
Jessica Wagantall [Tue, 31 Mar 2020 23:44:43 +0000 (23:44 +0000)]
Merge "Add PyPI stage templates"

4 years agoRepair tox-run for case of no tox logs to archive 20/63520/1
Lott, Christopher (cl778h) [Thu, 26 Mar 2020 21:32:23 +0000 (17:32 -0400)]
Repair tox-run for case of no tox logs to archive

Revise tox-run.sh to guard against glob matching no tox log files.
In that case the pattern is passed to the cp command, which fails.
Detect the cp failure ('cp: cannot stat ..') and exit the loop.
I introduced this new and undesired behavior by adding -e in
change 76a0761, so the script stops when the cp command fails.

Change-Id: I1d60bb69e270c4aea413f4f14cf2d4f8f672853b
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoAdd AWS credentials config file 68/63168/11
Houa Yang [Fri, 21 Feb 2020 21:59:59 +0000 (15:59 -0600)]
Add AWS credentials config file

Add new jenkins-s3-log-ship config file provider utilizing a conditional
step. This is needed to be able to use new lftools deploy s3 api call to
upload logs to AWS S3.

Issue-Id: RELENG-2749
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I2171c4f5a8a18259feca265b628cb7288a18c54f

4 years agoAdd PyPI stage templates 09/63509/4
Lott, Christopher (cl778h) [Wed, 25 Mar 2020 22:15:17 +0000 (18:15 -0400)]
Add PyPI stage templates

Add templates gerrit-pypi-stage/github-pypi-stage to allow projects
flexibility to control when Jenkins publishes a redistributable
Python package, either on merge (with the merge template) or on
demand (with the stage template).

Change-Id: I49d8101b9da8c58d10518cb3ad522cd71b1019dc
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoFix update to logs-deploy.sh 01/63501/4
Houa Yang [Wed, 25 Mar 2020 16:05:47 +0000 (11:05 -0500)]
Fix update to logs-deploy.sh

Fix script to remove outputting $ARCHIVE_ARTIFACTS. Outputting required
that $ARCHIVE_ARTIFACTS be set for every job. Instead it will output
${pattern_opts:-}.

Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I5f75a12745d1358b89dc7af18a35736514143072

4 years agoMerge "Run docker-merge job weekly by default"
Andrew Grimberg [Tue, 24 Mar 2020 20:42:06 +0000 (20:42 +0000)]
Merge "Run docker-merge job weekly by default"

4 years agoMerge "Update logs-deploy.sh"
Andrew Grimberg [Tue, 24 Mar 2020 20:33:25 +0000 (20:33 +0000)]
Merge "Update logs-deploy.sh"

4 years agoRun docker-merge job weekly by default 88/63488/2
Lott, Christopher (cl778h) [Tue, 24 Mar 2020 09:53:23 +0000 (05:53 -0400)]
Run docker-merge job weekly by default

Use @weekly cron default value in docker_merge_common anchor, no longer
empty. The merge job for docker images is like the stage job for java
artifacts, every docker image is a release candidate.  Run the merge job
regularly to check dependencies like base images and to push updated
images to the Nexus3 staging registry.

Change-Id: Ic26f534953c77047515d2678fa654e21ac9d6336
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoUpdate logs-deploy.sh 07/63307/5
Houa Yang [Thu, 5 Mar 2020 00:53:41 +0000 (18:53 -0600)]
Update logs-deploy.sh

Update script with get_pattern_opts function. Function refactors code to
be reusable.

Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: Ic5366f2177c401ce3bedda67b03fc5dee288cb8f

4 years agoStop on error in nexus-iq-cli.sh script 59/63459/1 v0.52.0
Lott, Christopher (cl778h) [Thu, 19 Mar 2020 20:32:38 +0000 (16:32 -0400)]
Stop on error in nexus-iq-cli.sh script

Revise script nexus-iq-cli.sh to stop on error or unbound variable.
This should fail the build if the scanner returns a non-zero code,
for example if credentials are missing or wrong.

Change-Id: I39f7ae9d6ba552ff3e3f3dd13df5bf998372e20f
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoUse lf-env for info-file-validate 00/63400/3
Aric Gardner [Thu, 12 Mar 2020 20:26:27 +0000 (16:26 -0400)]
Use lf-env for info-file-validate

Job was failing due to using python2
use lf-env to manage the environment.

ISSUE-ID: IT-19236
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I388ce7d8471e62efdae8c6131b4924cd89bfcc53

4 years agoMerge "Python 3.5 and python2 tox in Ubuntu 16.04"
Aric Gardner [Thu, 12 Mar 2020 15:21:49 +0000 (15:21 +0000)]
Merge "Python 3.5 and python2 tox in Ubuntu 16.04"

4 years agoMerge "Refactor lf-c-cpp-jobs for common anchor"
Eric Ball [Thu, 12 Mar 2020 15:08:40 +0000 (15:08 +0000)]
Merge "Refactor lf-c-cpp-jobs for common anchor"

4 years agoFix jq query testing if we got valid json 72/63372/2
Andrew Grimberg [Wed, 11 Mar 2020 16:37:00 +0000 (09:37 -0700)]
Fix jq query testing if we got valid json

While looking over cost data from OpenDaylight I noticed that we started
getting data that had unknown resources with 0 cost associated with it.
After digging in I found out that the job-cost.sh had been updated to
verify that the data received back was valid json.

Testing this out locally everything worked just fine, so I tested the
work flow on an instance in the build environment and found that the
version of jq on the systems throws an error if you try to redirect the
output without having a query, which is exactly what was being done for
the test for if we received a valid json block.

A little further testing showed me that changing the call to include a
query of '.' would just parse everything (as I expected) and that it
doesn't fail the test.

For further information this appears to be a bug in jq v1.5 which is
installed on the Centos7 systems but jq v1.6 which I have locally works
just fine.

Change-Id: Id1ffc4ce71903f62d56e62e6569975710294c630
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
4 years agoRefactor lf-c-cpp-jobs for common anchor 70/63370/6
Lott, Christopher (cl778h) [Thu, 5 Mar 2020 19:22:47 +0000 (14:22 -0500)]
Refactor lf-c-cpp-jobs for common anchor

Refactor templates in lf-c-cpp-jobs.yaml with common anchor/alias
lf_cmake_common to reduce redundant configuration.  This includes
gerrit-cmake-sonar, github-cmake-sonar, gerrit-cmake-sonarqube,
github-cmake-sonarqube, gerrit-cmake-stage, github-cmake-stage,
gerrit-cmake-verify and github-cmake-verify. No functional change.

Document maven settings parameters in g*t-cmake-stage templates.

Change-Id: I6477ad41da2b46adac2f76658dbf334db949b313
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoPython 3.5 and python2 tox in Ubuntu 16.04 90/63290/8
Aric Gardner [Wed, 4 Mar 2020 20:46:18 +0000 (15:46 -0500)]
Python 3.5 and python2 tox in Ubuntu 16.04

Ubuntu 16.04 calls a python2 version of tox.
so we must upgrade it even if we are testing with python3

ubuntu 16.04 has python 3.5 we must pinn the zipp package

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I4d2513c15baf4f00313885b35f8f0a297874b2b3

4 years agoPatch new Cmake/Tox + SonarQube templates 03/63303/2
Lott, Christopher (cl778h) [Thu, 5 Mar 2020 16:31:42 +0000 (11:31 -0500)]
Patch new Cmake/Tox + SonarQube templates

Revise templates gerrit-cmake-sonarqube and gerrit-tox-sonarqube
to move the triggering comment string into a parameter that can be
overridden in a job definition.  Github comment remains hardcoded.
This makes these new templates consistent with existing templates.
No functional change, the default is still "run-sonar".

Rename gerrit-cmake-sonarqube tempplate configuration parameter
from sonar-prescan-script to pre-build to be consistent with
existing CMake stage and verify templates. This is a breaking
change for any early adopters of this new template.

Correct release note file name prefix from cmake to tox

Change-Id: Ie98a58d2bf8cbffee110571df5d759c101cc3fc7
Issue-ID: RELENG-2760
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoMerge "Add templates Tox + SonarQube Jenkins plugin"
Eric Ball [Wed, 4 Mar 2020 18:46:51 +0000 (18:46 +0000)]
Merge "Add templates Tox + SonarQube Jenkins plugin"

4 years agoMerge "Add templates Cmake + SonarQube Jenkins plugin"
Eric Ball [Wed, 4 Mar 2020 17:33:20 +0000 (17:33 +0000)]
Merge "Add templates Cmake + SonarQube Jenkins plugin"

4 years agoAdd templates Cmake + SonarQube Jenkins plugin 20/63220/14
Lott, Christopher (cl778h) [Thu, 27 Feb 2020 19:45:39 +0000 (14:45 -0500)]
Add templates Cmake + SonarQube Jenkins plugin

New templates gerrit-cmake-sonarqube and github-cmake-sonarqube
use the SonarQube Jenkins plug-in to analyze CXX code and publish
the results.  Modeled after the generic gerrit/github-sonar templates.
The new templates lift the limitations of the existing templates
gerrit/github-cmake-sonar which download and install a Sonar scanner,
and cannot report unit-test code coverage statistics.

Change-Id: I2457ab0a40e226b8b5b2bbe18d2b9f5cde3281e1
Issue-ID: RELENG-2760
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoAdd templates Tox + SonarQube Jenkins plugin 67/63267/3
Lott, Christopher (cl778h) [Tue, 3 Mar 2020 18:20:42 +0000 (13:20 -0500)]
Add templates Tox + SonarQube Jenkins plugin

New templates gerrit-tox-sonarqube and github-tox-sonarqube
use the SonarQube Jenkins plug-in to analyze Python code and
publish the results.  Modeled after the generic gerrit/github-sonar
templates. The new templates eliminate the need for mvn-settings in
the job configuration and the need for a pom.xml file in the repo.

Change-Id: I6d5bc5200937a0230c5d8eb0c522941d2affcee4
Issue-ID: RELENG-2767
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoRevise self-serve release docs to match jobs 57/63257/3
Lott, Christopher (cl778h) [Mon, 2 Mar 2020 19:30:13 +0000 (14:30 -0500)]
Revise self-serve release docs to match jobs

Adjust formatting of the trigger file path default value
for container, maven and PyPI release jobs.  Drop the
parameter from release job docs to match the template,
and add a sentence with the regular expression.

Change-Id: Ibff31ac5c82fc1ed4f025fb4c16d9c907456f3c8
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoMerge "Show details and detect errors in tox-run.sh"
Chris Lott [Tue, 3 Mar 2020 15:44:40 +0000 (15:44 +0000)]
Merge "Show details and detect errors in tox-run.sh"

4 years agoShow details and detect errors in tox-run.sh 30/63230/2
Lott, Christopher (cl778h) [Fri, 28 Feb 2020 18:34:22 +0000 (13:34 -0500)]
Show details and detect errors in tox-run.sh

Revise tox-run.sh to guard against unbound variable TOX_ENVS;
stop on error or unbound variable (options -eu); and print
commands before executing (option -x). Add echo command at end.
Main effect is to show TOX_ENVS parameter value in the job log.

Change-Id: Iad78123f32f5c32792ba5ec8584921066ff9af70
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoAdjust cmake-sonar template and script 16/63216/5
Lott, Christopher (cl778h) [Thu, 27 Feb 2020 14:22:56 +0000 (09:22 -0500)]
Adjust cmake-sonar template and script

Add missing config property "stream: master" to lf_cmake_sonar macro
in lf-c-cpp-jobs.yaml to match existing "branch: master" property.
This makes the template consistent with other Sonar templates.

Remove double-quote chars that were added recently around $make_opts
from the invocation of the build wrapper in cmake-sonar.sh.  This
makes the invocation consistent with the invocation of cmake in the
same file.

Update documentation to reflect actual property names and defaults.

Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Change-Id: Ia1eb0573213cb06720ede8d9e47e1387b5b0303c

4 years agoSquelch progress messages in cmake-sonar.sh 99/63199/2
Lott, Christopher (cl778h) [Wed, 26 Feb 2020 19:46:20 +0000 (14:46 -0500)]
Squelch progress messages in cmake-sonar.sh

Reduce volume of output from wget and unzip by adding the -q flag to
the invocations in cmake-sonar.sh  There's little need to see wget
progress messages or archive contents in the build log.

Change-Id: Iaaaa4071e17b6ef42c85409e8df3653df1f42252
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoMerge "Add conditional block for s3"
Andrew Grimberg [Wed, 26 Feb 2020 17:18:39 +0000 (17:18 +0000)]
Merge "Add conditional block for s3"

4 years agoPin python idna library to v2.8 61/63161/1 v0.51.1
DW Talton [Thu, 20 Feb 2020 17:24:55 +0000 (10:24 -0700)]
Pin python idna library to v2.8

The idna library updated and broke twine/requests, which breaks
our *-pypi-release-merge jobs. Pinning the versions fixes this.

Issue-ID: IT-19046
Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: I62806d07600b975992f49de0f9b0e5d6401c18cc

4 years agoUpdate Sphinx and bootstrap theme versions 53/63153/1
Aric Gardner [Wed, 19 Feb 2020 21:08:51 +0000 (16:08 -0500)]
Update Sphinx and bootstrap theme versions

lfdocs conf has been updated, this should fix the
read the docs builds

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I5ff350989edafe14cdd6b54d069dfb6aaf23326b

4 years agoPin python zipp library to python3.5-supported version 35/63135/1
DW Talton [Mon, 17 Feb 2020 23:45:04 +0000 (16:45 -0700)]
Pin python zipp library to python3.5-supported version

Zipp just added a hard check for py36. This pins zipp to a version
that still works with py35.

Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: I44100186ce14abe068a9c919fec2277cc94f0113

4 years agoMerge "Add support for Sonar pre-scan shell script" v0.51.0
Eric Ball [Wed, 12 Feb 2020 20:04:39 +0000 (20:04 +0000)]
Merge "Add support for Sonar pre-scan shell script"

4 years agoAdd support for Sonar pre-scan shell script 79/63079/4
Lott, Christopher (cl778h) [Mon, 10 Feb 2020 20:58:11 +0000 (15:58 -0500)]
Add support for Sonar pre-scan shell script

New templates gerrit-sonar-prescan-script, github-sonar-prescan-script
accept an arbitrary shell-script body that can install prerequisites,
build and test to generate a code-coverage report for the Sonar Scanner
to find and upload.  This lifts limitations in existing templates
gerrit-sonar-prescan, github-sonar-prescan.

Change-Id: Ie61a5bed99a21fd62cb74f89d3e153ab0c3a4108
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Issue-ID: RELENG-2712

4 years agoRemove ref-update trigger for rtd-merge template 68/63068/2
Thanh Ha [Fri, 7 Feb 2020 20:07:52 +0000 (15:07 -0500)]
Remove ref-update trigger for rtd-merge template

This was originally added to help docs projects trigger a docs
update when a tag is deployed. Unfortunately ref-update appears to
trigger even for comments in Gerrit now. Not sure why but likely due
to Gerrit NoteDB. The Jenkins Gerrit Trigger configuration for
ref-update unfortunately does not provide any configuration for
specifying specific refs to trigger on.

Until there's an alternative it's better to disable this so that
Jenkins does not get flooded with queued jobs for Gerrit code
reviews that have active comments.

Issue: IT-18792
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I417e53d698e1454deb28b30d87e97147ce8d3038

4 years agoBad Floating Point Format in Cost File 58/63058/3
Tim Johnson [Thu, 6 Feb 2020 17:25:09 +0000 (09:25 -0800)]
Bad Floating Point Format in Cost File

Update format for uptime, cost & stack_cost. Changed default value for
resource (Instance Type) to 'unknown'. Created script (format-csv) to
fix the formats of the existing cost files.

ISSUE: RELENG-2715
Change-Id: Ibd8464faa1ba3983bf1e8ef5cd484d2f24c26e31
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoAdd configurable doc-dir 60/63060/8
Aric Gardner [Thu, 6 Feb 2020 18:22:07 +0000 (13:22 -0500)]
Add configurable doc-dir

defaults to "docs/_build/html" needed for relative path
modifications if you change the tox-dir

tox-run also needs to handle these relative paths.

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I6b18dc27e605dc28cb4420a5591a911e956df337

4 years agoAdd conditional block for s3 90/61890/7
Houa Yang [Tue, 1 Oct 2019 20:06:03 +0000 (15:06 -0500)]
Add conditional block for s3

Add conditional block for log shipping via AWS s3.
This will need some sort of environment variable, so
the build will know to also ship logs to s3.

Issue-Id: RELENG-2379, RELENG-2380
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: Iff922d22e3ffafb068ce7053194f8b7d289111b8

4 years agoMerge "Do not clone submodules in rtdv3"
Jessica Wagantall [Wed, 5 Feb 2020 21:30:24 +0000 (21:30 +0000)]
Merge "Do not clone submodules in rtdv3"

4 years agoDo not clone submodules in rtdv3 21/63021/2
Aric Gardner [Mon, 3 Feb 2020 20:29:59 +0000 (15:29 -0500)]
Do not clone submodules in rtdv3

The verify job should not be cloning
submodules.

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Id4abf1d513a87303078e9cd950a765e1f1fa95dd

4 years agoRedirect error to /dev/null 35/63035/1
Tim Johnson [Tue, 4 Feb 2020 22:37:32 +0000 (14:37 -0800)]
Redirect error to /dev/null

Change-Id: I5e02875c033b8bda4c293b9b3842f26f3ca6133a
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoClean up gerrit push change script 34/62734/16
Tim Johnson [Mon, 6 Jan 2020 23:09:05 +0000 (15:09 -0800)]
Clean up gerrit push change script

Removed broken code that removed leading/trailing whitespace from
variables. Use lf-acitvate-venv() to instll openstack. Enabled 'set -euf
pipefail' and updated code to handle errors. Updated out of date
comments.

ISSUE: RELENG-2633
Change-Id: I76d2bf9ed9f7f26303f56dd98ebbd4b458d0d54f
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoDelete deprecated files 04/63004/2
Tim Johnson [Fri, 31 Jan 2020 17:33:39 +0000 (09:33 -0800)]
Delete deprecated files

Change-Id: I41e86ee5e11f94464e77416e10232bf54416a9da
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoAdd Script to scrape cost data 72/62572/13
Tim Johnson [Mon, 9 Dec 2019 16:13:09 +0000 (08:13 -0800)]
Add Script to scrape cost data

Issue: RELENG-2555
Change-Id: I46e719f88721fd8cf0060181297d2655995e47f7
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoMerge "Fix release file detection"
Andrew Grimberg [Fri, 31 Jan 2020 21:21:55 +0000 (21:21 +0000)]
Merge "Fix release file detection"

4 years agoMerge "Add GitLint which replaces GitCommitBear"
Andrew Grimberg [Fri, 31 Jan 2020 20:34:47 +0000 (20:34 +0000)]
Merge "Add GitLint which replaces GitCommitBear"

4 years agoMerge "Update docs with missing cloud config params"
Andrew Grimberg [Fri, 31 Jan 2020 20:25:53 +0000 (20:25 +0000)]
Merge "Update docs with missing cloud config params"

4 years agoFix release file detection 58/62958/2
Aric Gardner [Mon, 27 Jan 2020 19:49:59 +0000 (14:49 -0500)]
Fix release file detection

commits with multiple parents were not being detected.
break long shell command across two lines

ISSUE-ID: RELENG-2685
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I660fabfba0497ab4c6825853c0fa117bacc93581
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoDont set jvm options to null 05/63005/1
Aric Gardner [Fri, 31 Jan 2020 18:45:24 +0000 (13:45 -0500)]
Dont set jvm options to null

should be empty

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I6a37b3922e9c40541d0e824178b33aee4c934703

4 years agoMerge "Python merge use git choosing strategy default"
Aric Gardner [Thu, 30 Jan 2020 19:30:51 +0000 (19:30 +0000)]
Merge "Python merge use git choosing strategy default"

4 years agoMerge "Branch discovery and build polling"
Aric Gardner [Thu, 30 Jan 2020 19:13:59 +0000 (19:13 +0000)]
Merge "Branch discovery and build polling"

4 years agoBranch discovery and build polling 22/62922/13
Aric Gardner [Thu, 23 Jan 2020 20:23:33 +0000 (15:23 -0500)]
Branch discovery and build polling

If a Branch has not been seen by rtd
we trigger a build and poll till that build
is complete.

we can then enable the branch and trigger a build
on it, again polling all builds untill they are
complete

ISSUE-ID: RELENG-2682

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Id59d4200dd5a883fc1fdc2f898c589bd58658312

4 years agoUpdate docs with missing cloud config params 92/62992/1
Anil Belur [Thu, 30 Jan 2020 02:19:56 +0000 (12:19 +1000)]
Update docs with missing cloud config params

Also correct the default value as defined in the
jenkins-configure-clouds.sh script.

Change-Id: I56cae3723339e8f8ec2630db2dd7744d95b8ec49
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoFix FLOATING_IP_POOL default value 91/62991/2
Eric Ball [Thu, 30 Jan 2020 00:42:19 +0000 (16:42 -0800)]
Fix FLOATING_IP_POOL default value

A recent change has made the "null" string a bad value for
FLOATING_IP_POOL. By making it an empty string, we recreate the old
functionality of having the default floating IP pool set to "No
value".

Change-Id: If6f2af7a8a6ccf4bb5079e8282962098f2f725b6
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
4 years agoShow error if distribution_type is missing 87/62987/1
Lott, Christopher (cl778h) [Wed, 29 Jan 2020 19:32:49 +0000 (14:32 -0500)]
Show error if distribution_type is missing

Extend release-job.sh to detect if distribution_type is missing
from the release yaml file and show a meaningful error.  The
shell option pipefile causes the script to halt silently if
niet fails to find that key, which utterly baffles users.

Change-Id: I926097a1e9d94bd867879ce6fdc549415a911dbe
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoPython merge use git choosing strategy default 86/62986/2
Lott, Christopher (cl778h) [Wed, 29 Jan 2020 18:22:50 +0000 (13:22 -0500)]
Python merge use git choosing strategy default

Use git choosing strategy default in tox and pypi merge jobs for
gerrit.  This makes those jobs consistent with maven and other
merge jobs for gerrit that always build from tip of the target
branch to create artifacts from the latest & greatest code.
Building from tip (not from Gerrit commit/merge point) avoids
confusion about content when changes are merged out of order.
For example, a fix is submitted and merged, but the merge job
fails. In the mean time, a different change that happened
earlier in commit history gets merged (or the merge job is
retriggered), causing a new artifact to be pushed.  But that
artifact does not have the expected fix.

Add comments to release merge jobs why their choosing strategy
is not default.

Document the git commit choosing strategy for the release merge jobs.

Change-Id: I2e32dfa946d678fa59a290ddb53fc09387d59baf
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoError handling for job cost script 23/62923/5
Zack Williams [Thu, 23 Jan 2020 18:42:17 +0000 (11:42 -0700)]
Error handling for job cost script

- Exit early if on AWS, where job cost calculation isn't supported
- Error handling for vexhost API - may return a 404 which doesn't parse
  properly in jq, which would fails an otherwise working build.
- Report correct name when script is run.

Change-Id: I375b4a16588b1ab12e4aa7da0e37b351ed52f64c
Signed-off-by: Zack Williams <zdw@opennetworking.org>
4 years agoAdd GitLint which replaces GitCommitBear 77/62977/1
Thanh Ha [Tue, 28 Jan 2020 15:55:55 +0000 (10:55 -0500)]
Add GitLint which replaces GitCommitBear

This replaces GitCommitBear with GitLint which does similar
linting for git commit messages but using pre-commit hooks.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I1622148cc36f072c7c412e88c1945b518b816793

4 years agoRelease jobs use Gerrit Trigger choosing strategy 70/62970/1
Lott, Christopher (cl778h) [Tue, 28 Jan 2020 15:03:14 +0000 (10:03 -0500)]
Release jobs use Gerrit Trigger choosing strategy

Use choosing strategy Gerrit Trigger in container/jar and
package cloud release merge jobs. This retains the current
behavior in the simple merge case, and ensures that a job
triggered by a "remerge" comment uses the release file at
that commit.  The previous choosing strategy, default, uses
the tip of the target branch.  That does not allow recovery
from merge job failure if the target branch has advanced past
the commit with the release file.

Change-Id: I6685e54da9f1510243f52506794cf52a28b9eb61
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoMerge "Call "lftools jenkins" after credentials are set"
Eric Ball [Mon, 27 Jan 2020 15:54:44 +0000 (15:54 +0000)]
Merge "Call "lftools jenkins" after credentials are set"

4 years agoCall "lftools jenkins" after credentials are set 51/62951/3
Eric Ball [Fri, 24 Jan 2020 23:25:38 +0000 (15:25 -0800)]
Call "lftools jenkins" after credentials are set

We have been seeing failures due to the "lftools jenkins plugins"
call being made without credentials being set first. The previous
method did not require credentials, so the failure was introduced
when we switched to using lftools. The os_plugin_version variable
is not needed before this part of the script, so by moving the
assignment to after JENKINS_USER and JENKINS_PASSWORD are set, the
script is once again successful.

Issue: IT-18593
Change-Id: Ib3e6099992abade5860eedd1bda07015b13f0f91
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
4 years agoUpgade WS Agent to 19.12.2 49/62949/1
Jessica Wagantall [Fri, 24 Jan 2020 21:34:33 +0000 (13:34 -0800)]
Upgade WS Agent to 19.12.2

Change-Id: I8a5ceeb55e8b4478667c516b8ba6c2ed0924972d
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
4 years agoShow python and pip versions in log 26/62926/2
Lott, Christopher (cl778h) [Fri, 24 Jan 2020 00:25:57 +0000 (19:25 -0500)]
Show python and pip versions in log

Extend shell scripts that invoke pip freeze to show
python and pip versions also.

Change-Id: I75fff4d379a7cbd27aa867c2cae4f2fd2dde2c31
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoMerge "Add pip freeze to python-tools-install" v0.50.0
Aric Gardner [Thu, 23 Jan 2020 20:14:43 +0000 (20:14 +0000)]
Merge "Add pip freeze to python-tools-install"

4 years agoAdd pip freeze to python-tools-install 19/62919/2
Lott, Christopher (cl778h) [Thu, 23 Jan 2020 19:07:42 +0000 (14:07 -0500)]
Add pip freeze to python-tools-install

Package version details support detection of changes in dependencies
like pip

Change-Id: Ic1b8a943c03e497aa1af85370d16e2bc5bc2c081
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoCatchup release notes for python-tools-install.sh 16/62916/4
Lott, Christopher (cl778h) [Thu, 23 Jan 2020 18:19:31 +0000 (13:19 -0500)]
Catchup release notes for python-tools-install.sh

Change-Id: I5964d216a89e629c86e56b2a0e6e49fa352368c2
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoDO NOT create a virtualenv in ~/.local 13/62913/2
Thanh Ha [Thu, 23 Jan 2020 17:31:29 +0000 (12:31 -0500)]
DO NOT create a virtualenv in ~/.local

~/.local is NOT a Python virtualenv and should not be created as such.
Using `--user` installs python modules into
~/.local/lib/PYTHON_VERSION/site-packages

Making ~/.local a virtualenv messes with the paths in site-packages.

This reverts one of the changes from
I4b2d778f3fd81565c5dd009d50c969696faba0d2

Ref: https://www.python.org/dev/peps/pep-0370
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Ie2635b691c8cdf5718fc2617d4de41a3d5a2c347

4 years agoRevert "remove --user" 02/62902/1
Aric Gardner [Thu, 23 Jan 2020 12:17:41 +0000 (07:17 -0500)]
Revert "remove --user"

Revert "remove --user now that pip is 20.0.x"
This reverts commit 8a2651316c6c2f121a32d1121dd3ff7dc0e8ac24.

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: If462b4d6ad672c044f0e8fb78f9d36211562e24d

4 years agoMerge "Upgrade pre-commit hook repos to latest"
Anil Belur [Thu, 23 Jan 2020 00:19:26 +0000 (00:19 +0000)]
Merge "Upgrade pre-commit hook repos to latest"

4 years agoMerge "Accept maven configuration in Python tox sonar"
Anil Belur [Thu, 23 Jan 2020 00:18:51 +0000 (00:18 +0000)]
Merge "Accept maven configuration in Python tox sonar"

4 years agoMerge "Typo jason -> json"
Anil Belur [Thu, 23 Jan 2020 00:17:46 +0000 (00:17 +0000)]
Merge "Typo jason -> json"

4 years agoremove --user now that pip is 20.0.x 74/62874/1
Aric Gardner [Wed, 22 Jan 2020 20:18:11 +0000 (15:18 -0500)]
remove --user now that pip is 20.0.x

I will find the release notes and add them here

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I321596b2d05bd3d13d9fde9cef598f95084777ea

4 years agoAccept maven configuration in Python tox sonar 71/62871/3
Lott, Christopher (cl778h) [Wed, 22 Jan 2020 15:06:18 +0000 (10:06 -0500)]
Accept maven configuration in Python tox sonar

Extend lf_tox_sonar with parameters macro lf-infra-maven-parameters
so job definitions can configure mvn-opts, mvn-params and mvn-version
values. Default values are defined so existing jobs are not affected.
This change makes the python-tox sonar template consistent with the
maven sonar template.

Extend the PyPI package structure recommendation with a way to share
the docs/ folder.

Change-Id: I2f785cabc9058238d36505c8b4af8f87114a685a
Issue-Id: RELENG-2677
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoPython2/3 problem in jjb-deploy job 88/62788/10
Tim Johnson [Mon, 13 Jan 2020 21:21:18 +0000 (13:21 -0800)]
Python2/3 problem in jjb-deploy job

Removed ref of macro: lf-pip-install from macro: lf_jjb_deploy_job. A
call to lf-activate-venv() has beed added to jjb-deploy-job.sh to
install jenkins-jobs command. The pip-install.sh is no longer needed.

ISSUE: IT-18720
Change-Id: Ice1e9793f131eace41a3df9bbbe598aac071c1df
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoMerge "Disable script location warnings"
Anil Belur [Tue, 21 Jan 2020 05:04:51 +0000 (05:04 +0000)]
Merge "Disable script location warnings"

4 years agoAdd quotes around variable to silence shellcheck 51/62851/1
Lott, Christopher (cl778h) [Mon, 20 Jan 2020 23:13:08 +0000 (18:13 -0500)]
Add quotes around variable to silence shellcheck

Modify maven-javadoc-generate.sh to put double quotes around use of
environment variable to silence shellcheck warning that I missed.
No functional change to behavior.

Change-Id: Icfd01adcff8ae03cf43fefd3751427ee2d0dc2b6
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoMerge "Revise javadoc script dash-f argument"
Eric Ball [Mon, 20 Jan 2020 21:00:09 +0000 (21:00 +0000)]
Merge "Revise javadoc script dash-f argument"

4 years agoTypo jason -> json 35/62835/1
Thanh Ha [Thu, 16 Jan 2020 17:44:51 +0000 (12:44 -0500)]
Typo jason -> json

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I7f48f94726e2c4efd8021dc6b0aede2f5f67da51

4 years agoUpgrade pre-commit hook repos to latest 27/62827/2
Thanh Ha [Thu, 16 Jan 2020 00:23:22 +0000 (19:23 -0500)]
Upgrade pre-commit hook repos to latest

Use the latest available hook repo tags for our pre-commit
hooks.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Id1a702dc410faf85ad6ab6f739fcc1bbddcc270e

4 years agoMigrate shellcheck hook to pre-commit 26/62826/3
Thanh Ha [Wed, 15 Jan 2020 23:21:56 +0000 (18:21 -0500)]
Migrate shellcheck hook to pre-commit

Use pre-commit to run the shellcheck hook rather than coala as
we want to eventually phase out coala.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Id8068a29379a5c71c6f016e00e99325939e1e693

4 years agoResolve shellcheck SC2001 25/62825/3
Thanh Ha [Thu, 16 Jan 2020 00:09:46 +0000 (19:09 -0500)]
Resolve shellcheck SC2001

SC2001: See if you can use ${variable//search/replace} instead.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Ib4cce69c29ade0b37c2bffcf9f566f9ab8821b29

4 years agoResolve shellcheck SC2086 double quote 24/62824/3
Thanh Ha [Wed, 15 Jan 2020 23:27:05 +0000 (18:27 -0500)]
Resolve shellcheck SC2086 double quote

SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Ib5fee4b949ec02e265b9d0e628a36ce63ce18784

4 years agoResolve shellcheck SC219[67] ?grep non-standard 23/62823/3
Thanh Ha [Thu, 16 Jan 2020 00:06:08 +0000 (19:06 -0500)]
Resolve shellcheck SC219[67] ?grep non-standard

SC2196: egrep is non-standard and deprecated. Use grep -E instead.
SC2197: fgrep is non-standard and deprecated. Use grep -F instead.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Ibbeb8d799c6c31b6706306c4760c371efde573c8

4 years agoResolve shellcheck SC2002 useless cat 22/62822/3
Thanh Ha [Wed, 15 Jan 2020 23:47:01 +0000 (18:47 -0500)]
Resolve shellcheck SC2002 useless cat

SC2002: Useless cat. Consider 'cmd < file | ..' or
        'cmd file | ..' instead.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Icdbeec332969ce5d8769aea1eb325a7fe35bcc77

4 years agoResolve shellcheck SC2116 useless echo 21/62821/3
Thanh Ha [Wed, 15 Jan 2020 23:44:15 +0000 (18:44 -0500)]
Resolve shellcheck SC2116 useless echo

SC2116: Useless echo? Instead of 'cmd $(echo foo)', just use 'cmd foo'

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: If656687c1968b901255bf4d4108f193bffc69cce