releng/global-jjb.git
4 years agoClean up gerrit push change script 34/62734/16
Tim Johnson [Mon, 6 Jan 2020 23:09:05 +0000 (15:09 -0800)]
Clean up gerrit push change script

Removed broken code that removed leading/trailing whitespace from
variables. Use lf-acitvate-venv() to instll openstack. Enabled 'set -euf
pipefail' and updated code to handle errors. Updated out of date
comments.

ISSUE: RELENG-2633
Change-Id: I76d2bf9ed9f7f26303f56dd98ebbd4b458d0d54f
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoDelete deprecated files 04/63004/2
Tim Johnson [Fri, 31 Jan 2020 17:33:39 +0000 (09:33 -0800)]
Delete deprecated files

Change-Id: I41e86ee5e11f94464e77416e10232bf54416a9da
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoAdd Script to scrape cost data 72/62572/13
Tim Johnson [Mon, 9 Dec 2019 16:13:09 +0000 (08:13 -0800)]
Add Script to scrape cost data

Issue: RELENG-2555
Change-Id: I46e719f88721fd8cf0060181297d2655995e47f7
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoMerge "Fix release file detection"
Andrew Grimberg [Fri, 31 Jan 2020 21:21:55 +0000 (21:21 +0000)]
Merge "Fix release file detection"

4 years agoMerge "Add GitLint which replaces GitCommitBear"
Andrew Grimberg [Fri, 31 Jan 2020 20:34:47 +0000 (20:34 +0000)]
Merge "Add GitLint which replaces GitCommitBear"

4 years agoMerge "Update docs with missing cloud config params"
Andrew Grimberg [Fri, 31 Jan 2020 20:25:53 +0000 (20:25 +0000)]
Merge "Update docs with missing cloud config params"

4 years agoFix release file detection 58/62958/2
Aric Gardner [Mon, 27 Jan 2020 19:49:59 +0000 (14:49 -0500)]
Fix release file detection

commits with multiple parents were not being detected.
break long shell command across two lines

ISSUE-ID: RELENG-2685
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I660fabfba0497ab4c6825853c0fa117bacc93581
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoDont set jvm options to null 05/63005/1
Aric Gardner [Fri, 31 Jan 2020 18:45:24 +0000 (13:45 -0500)]
Dont set jvm options to null

should be empty

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I6a37b3922e9c40541d0e824178b33aee4c934703

4 years agoMerge "Python merge use git choosing strategy default"
Aric Gardner [Thu, 30 Jan 2020 19:30:51 +0000 (19:30 +0000)]
Merge "Python merge use git choosing strategy default"

4 years agoMerge "Branch discovery and build polling"
Aric Gardner [Thu, 30 Jan 2020 19:13:59 +0000 (19:13 +0000)]
Merge "Branch discovery and build polling"

4 years agoBranch discovery and build polling 22/62922/13
Aric Gardner [Thu, 23 Jan 2020 20:23:33 +0000 (15:23 -0500)]
Branch discovery and build polling

If a Branch has not been seen by rtd
we trigger a build and poll till that build
is complete.

we can then enable the branch and trigger a build
on it, again polling all builds untill they are
complete

ISSUE-ID: RELENG-2682

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Id59d4200dd5a883fc1fdc2f898c589bd58658312

4 years agoUpdate docs with missing cloud config params 92/62992/1
Anil Belur [Thu, 30 Jan 2020 02:19:56 +0000 (12:19 +1000)]
Update docs with missing cloud config params

Also correct the default value as defined in the
jenkins-configure-clouds.sh script.

Change-Id: I56cae3723339e8f8ec2630db2dd7744d95b8ec49
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoFix FLOATING_IP_POOL default value 91/62991/2
Eric Ball [Thu, 30 Jan 2020 00:42:19 +0000 (16:42 -0800)]
Fix FLOATING_IP_POOL default value

A recent change has made the "null" string a bad value for
FLOATING_IP_POOL. By making it an empty string, we recreate the old
functionality of having the default floating IP pool set to "No
value".

Change-Id: If6f2af7a8a6ccf4bb5079e8282962098f2f725b6
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
4 years agoShow error if distribution_type is missing 87/62987/1
Lott, Christopher (cl778h) [Wed, 29 Jan 2020 19:32:49 +0000 (14:32 -0500)]
Show error if distribution_type is missing

Extend release-job.sh to detect if distribution_type is missing
from the release yaml file and show a meaningful error.  The
shell option pipefile causes the script to halt silently if
niet fails to find that key, which utterly baffles users.

Change-Id: I926097a1e9d94bd867879ce6fdc549415a911dbe
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoPython merge use git choosing strategy default 86/62986/2
Lott, Christopher (cl778h) [Wed, 29 Jan 2020 18:22:50 +0000 (13:22 -0500)]
Python merge use git choosing strategy default

Use git choosing strategy default in tox and pypi merge jobs for
gerrit.  This makes those jobs consistent with maven and other
merge jobs for gerrit that always build from tip of the target
branch to create artifacts from the latest & greatest code.
Building from tip (not from Gerrit commit/merge point) avoids
confusion about content when changes are merged out of order.
For example, a fix is submitted and merged, but the merge job
fails. In the mean time, a different change that happened
earlier in commit history gets merged (or the merge job is
retriggered), causing a new artifact to be pushed.  But that
artifact does not have the expected fix.

Add comments to release merge jobs why their choosing strategy
is not default.

Document the git commit choosing strategy for the release merge jobs.

Change-Id: I2e32dfa946d678fa59a290ddb53fc09387d59baf
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoError handling for job cost script 23/62923/5
Zack Williams [Thu, 23 Jan 2020 18:42:17 +0000 (11:42 -0700)]
Error handling for job cost script

- Exit early if on AWS, where job cost calculation isn't supported
- Error handling for vexhost API - may return a 404 which doesn't parse
  properly in jq, which would fails an otherwise working build.
- Report correct name when script is run.

Change-Id: I375b4a16588b1ab12e4aa7da0e37b351ed52f64c
Signed-off-by: Zack Williams <zdw@opennetworking.org>
4 years agoAdd GitLint which replaces GitCommitBear 77/62977/1
Thanh Ha [Tue, 28 Jan 2020 15:55:55 +0000 (10:55 -0500)]
Add GitLint which replaces GitCommitBear

This replaces GitCommitBear with GitLint which does similar
linting for git commit messages but using pre-commit hooks.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I1622148cc36f072c7c412e88c1945b518b816793

4 years agoRelease jobs use Gerrit Trigger choosing strategy 70/62970/1
Lott, Christopher (cl778h) [Tue, 28 Jan 2020 15:03:14 +0000 (10:03 -0500)]
Release jobs use Gerrit Trigger choosing strategy

Use choosing strategy Gerrit Trigger in container/jar and
package cloud release merge jobs. This retains the current
behavior in the simple merge case, and ensures that a job
triggered by a "remerge" comment uses the release file at
that commit.  The previous choosing strategy, default, uses
the tip of the target branch.  That does not allow recovery
from merge job failure if the target branch has advanced past
the commit with the release file.

Change-Id: I6685e54da9f1510243f52506794cf52a28b9eb61
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoMerge "Call "lftools jenkins" after credentials are set"
Eric Ball [Mon, 27 Jan 2020 15:54:44 +0000 (15:54 +0000)]
Merge "Call "lftools jenkins" after credentials are set"

4 years agoCall "lftools jenkins" after credentials are set 51/62951/3
Eric Ball [Fri, 24 Jan 2020 23:25:38 +0000 (15:25 -0800)]
Call "lftools jenkins" after credentials are set

We have been seeing failures due to the "lftools jenkins plugins"
call being made without credentials being set first. The previous
method did not require credentials, so the failure was introduced
when we switched to using lftools. The os_plugin_version variable
is not needed before this part of the script, so by moving the
assignment to after JENKINS_USER and JENKINS_PASSWORD are set, the
script is once again successful.

Issue: IT-18593
Change-Id: Ib3e6099992abade5860eedd1bda07015b13f0f91
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
4 years agoUpgade WS Agent to 19.12.2 49/62949/1
Jessica Wagantall [Fri, 24 Jan 2020 21:34:33 +0000 (13:34 -0800)]
Upgade WS Agent to 19.12.2

Change-Id: I8a5ceeb55e8b4478667c516b8ba6c2ed0924972d
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
4 years agoShow python and pip versions in log 26/62926/2
Lott, Christopher (cl778h) [Fri, 24 Jan 2020 00:25:57 +0000 (19:25 -0500)]
Show python and pip versions in log

Extend shell scripts that invoke pip freeze to show
python and pip versions also.

Change-Id: I75fff4d379a7cbd27aa867c2cae4f2fd2dde2c31
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoMerge "Add pip freeze to python-tools-install" v0.50.0
Aric Gardner [Thu, 23 Jan 2020 20:14:43 +0000 (20:14 +0000)]
Merge "Add pip freeze to python-tools-install"

4 years agoAdd pip freeze to python-tools-install 19/62919/2
Lott, Christopher (cl778h) [Thu, 23 Jan 2020 19:07:42 +0000 (14:07 -0500)]
Add pip freeze to python-tools-install

Package version details support detection of changes in dependencies
like pip

Change-Id: Ic1b8a943c03e497aa1af85370d16e2bc5bc2c081
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoCatchup release notes for python-tools-install.sh 16/62916/4
Lott, Christopher (cl778h) [Thu, 23 Jan 2020 18:19:31 +0000 (13:19 -0500)]
Catchup release notes for python-tools-install.sh

Change-Id: I5964d216a89e629c86e56b2a0e6e49fa352368c2
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoDO NOT create a virtualenv in ~/.local 13/62913/2
Thanh Ha [Thu, 23 Jan 2020 17:31:29 +0000 (12:31 -0500)]
DO NOT create a virtualenv in ~/.local

~/.local is NOT a Python virtualenv and should not be created as such.
Using `--user` installs python modules into
~/.local/lib/PYTHON_VERSION/site-packages

Making ~/.local a virtualenv messes with the paths in site-packages.

This reverts one of the changes from
I4b2d778f3fd81565c5dd009d50c969696faba0d2

Ref: https://www.python.org/dev/peps/pep-0370
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Ie2635b691c8cdf5718fc2617d4de41a3d5a2c347

4 years agoRevert "remove --user" 02/62902/1
Aric Gardner [Thu, 23 Jan 2020 12:17:41 +0000 (07:17 -0500)]
Revert "remove --user"

Revert "remove --user now that pip is 20.0.x"
This reverts commit 8a2651316c6c2f121a32d1121dd3ff7dc0e8ac24.

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: If462b4d6ad672c044f0e8fb78f9d36211562e24d

4 years agoMerge "Upgrade pre-commit hook repos to latest"
Anil Belur [Thu, 23 Jan 2020 00:19:26 +0000 (00:19 +0000)]
Merge "Upgrade pre-commit hook repos to latest"

4 years agoMerge "Accept maven configuration in Python tox sonar"
Anil Belur [Thu, 23 Jan 2020 00:18:51 +0000 (00:18 +0000)]
Merge "Accept maven configuration in Python tox sonar"

4 years agoMerge "Typo jason -> json"
Anil Belur [Thu, 23 Jan 2020 00:17:46 +0000 (00:17 +0000)]
Merge "Typo jason -> json"

4 years agoremove --user now that pip is 20.0.x 74/62874/1
Aric Gardner [Wed, 22 Jan 2020 20:18:11 +0000 (15:18 -0500)]
remove --user now that pip is 20.0.x

I will find the release notes and add them here

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I321596b2d05bd3d13d9fde9cef598f95084777ea

4 years agoAccept maven configuration in Python tox sonar 71/62871/3
Lott, Christopher (cl778h) [Wed, 22 Jan 2020 15:06:18 +0000 (10:06 -0500)]
Accept maven configuration in Python tox sonar

Extend lf_tox_sonar with parameters macro lf-infra-maven-parameters
so job definitions can configure mvn-opts, mvn-params and mvn-version
values. Default values are defined so existing jobs are not affected.
This change makes the python-tox sonar template consistent with the
maven sonar template.

Extend the PyPI package structure recommendation with a way to share
the docs/ folder.

Change-Id: I2f785cabc9058238d36505c8b4af8f87114a685a
Issue-Id: RELENG-2677
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoPython2/3 problem in jjb-deploy job 88/62788/10
Tim Johnson [Mon, 13 Jan 2020 21:21:18 +0000 (13:21 -0800)]
Python2/3 problem in jjb-deploy job

Removed ref of macro: lf-pip-install from macro: lf_jjb_deploy_job. A
call to lf-activate-venv() has beed added to jjb-deploy-job.sh to
install jenkins-jobs command. The pip-install.sh is no longer needed.

ISSUE: IT-18720
Change-Id: Ice1e9793f131eace41a3df9bbbe598aac071c1df
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoMerge "Disable script location warnings"
Anil Belur [Tue, 21 Jan 2020 05:04:51 +0000 (05:04 +0000)]
Merge "Disable script location warnings"

4 years agoAdd quotes around variable to silence shellcheck 51/62851/1
Lott, Christopher (cl778h) [Mon, 20 Jan 2020 23:13:08 +0000 (18:13 -0500)]
Add quotes around variable to silence shellcheck

Modify maven-javadoc-generate.sh to put double quotes around use of
environment variable to silence shellcheck warning that I missed.
No functional change to behavior.

Change-Id: Icfd01adcff8ae03cf43fefd3751427ee2d0dc2b6
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoMerge "Revise javadoc script dash-f argument"
Eric Ball [Mon, 20 Jan 2020 21:00:09 +0000 (21:00 +0000)]
Merge "Revise javadoc script dash-f argument"

4 years agoTypo jason -> json 35/62835/1
Thanh Ha [Thu, 16 Jan 2020 17:44:51 +0000 (12:44 -0500)]
Typo jason -> json

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I7f48f94726e2c4efd8021dc6b0aede2f5f67da51

4 years agoUpgrade pre-commit hook repos to latest 27/62827/2
Thanh Ha [Thu, 16 Jan 2020 00:23:22 +0000 (19:23 -0500)]
Upgrade pre-commit hook repos to latest

Use the latest available hook repo tags for our pre-commit
hooks.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Id1a702dc410faf85ad6ab6f739fcc1bbddcc270e

4 years agoMigrate shellcheck hook to pre-commit 26/62826/3
Thanh Ha [Wed, 15 Jan 2020 23:21:56 +0000 (18:21 -0500)]
Migrate shellcheck hook to pre-commit

Use pre-commit to run the shellcheck hook rather than coala as
we want to eventually phase out coala.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Id8068a29379a5c71c6f016e00e99325939e1e693

4 years agoResolve shellcheck SC2001 25/62825/3
Thanh Ha [Thu, 16 Jan 2020 00:09:46 +0000 (19:09 -0500)]
Resolve shellcheck SC2001

SC2001: See if you can use ${variable//search/replace} instead.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Ib4cce69c29ade0b37c2bffcf9f566f9ab8821b29

4 years agoResolve shellcheck SC2086 double quote 24/62824/3
Thanh Ha [Wed, 15 Jan 2020 23:27:05 +0000 (18:27 -0500)]
Resolve shellcheck SC2086 double quote

SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Ib5fee4b949ec02e265b9d0e628a36ce63ce18784

4 years agoResolve shellcheck SC219[67] ?grep non-standard 23/62823/3
Thanh Ha [Thu, 16 Jan 2020 00:06:08 +0000 (19:06 -0500)]
Resolve shellcheck SC219[67] ?grep non-standard

SC2196: egrep is non-standard and deprecated. Use grep -E instead.
SC2197: fgrep is non-standard and deprecated. Use grep -F instead.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Ibbeb8d799c6c31b6706306c4760c371efde573c8

4 years agoResolve shellcheck SC2002 useless cat 22/62822/3
Thanh Ha [Wed, 15 Jan 2020 23:47:01 +0000 (18:47 -0500)]
Resolve shellcheck SC2002 useless cat

SC2002: Useless cat. Consider 'cmd < file | ..' or
        'cmd file | ..' instead.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Icdbeec332969ce5d8769aea1eb325a7fe35bcc77

4 years agoResolve shellcheck SC2116 useless echo 21/62821/3
Thanh Ha [Wed, 15 Jan 2020 23:44:15 +0000 (18:44 -0500)]
Resolve shellcheck SC2116 useless echo

SC2116: Useless echo? Instead of 'cmd $(echo foo)', just use 'cmd foo'

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: If656687c1968b901255bf4d4108f193bffc69cce

4 years agoResolve shellcheck SC2219 use (()) instead of let 20/62820/3
Thanh Ha [Thu, 16 Jan 2020 00:20:54 +0000 (19:20 -0500)]
Resolve shellcheck SC2219 use (()) instead of let

SC2219: Instead of 'let expr', prefer (( expr )) .

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Ibb7b4c66915c1d4c95f3475a9a9535115a00463e

4 years agoIgnore shellcheck SC2012 use find instead of ls 19/62819/3
Thanh Ha [Thu, 16 Jan 2020 00:18:18 +0000 (19:18 -0500)]
Ignore shellcheck SC2012 use find instead of ls

It's reasonable to ignore this here as we're just piping ls to wc
to get a file count. The list of filenames is not being parsed in
anyway in this case.

SC2012: Use find instead of ls to better handle non-alphanumeric
        filenames.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I0687b24816c2b4920893162297cd318f813d7109

4 years agoResolve shellcheck SC2129 use {...} >> file 18/62818/2
Thanh Ha [Thu, 16 Jan 2020 02:26:37 +0000 (12:26 +1000)]
Resolve shellcheck SC2129 use {...} >> file

SC2129: Consider using { cmd1; cmd2; } >> file instead of
individual redirects.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Ib7327c9f5867379654ada6b89d415321fe9fc89a
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoRevise javadoc script dash-f argument 14/62814/7
Lott, Christopher (cl778h) [Wed, 15 Jan 2020 20:17:03 +0000 (15:17 -0500)]
Revise javadoc script dash-f argument

Revise shell script maven-javadoc-generate.sh to supply -f
argument to maven with absolute path determined by readlink.
This works around javadoc:aggregate behavior of silently
generating no files in parent project if invoked as "-f ."

Change-Id: I6856756539dd8aee80d8841dfbfe9efb76b8cb93
Issue-Id: RELENG-2672
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoResolve shellcheck SC1001 \= will be a regular = 17/62817/1
Thanh Ha [Wed, 15 Jan 2020 23:59:09 +0000 (18:59 -0500)]
Resolve shellcheck SC1001 \= will be a regular =

SC1001: This \= will be a regular '=' in this context.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I93f57b15802c9f07a789e9c3aa1a60cba40ab00b

4 years agoResolve SC1008 shebang was unrecognized 16/62816/1
Thanh Ha [Wed, 15 Jan 2020 23:55:46 +0000 (18:55 -0500)]
Resolve SC1008 shebang was unrecognized

SC1008: This shebang was unrecognized. ShellCheck only supports
sh/bash/dash/ksh. Add a 'shell' directive to specify.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I63aef5c1634c26860c2bf63a9a930c1584515551

4 years agoDisable script location warnings 00/62800/3
Anil Belur [Wed, 15 Jan 2020 01:44:27 +0000 (11:44 +1000)]
Disable script location warnings

Disable these pip install warning that are not useful in the console
logs.

WARNING: The scripts easy_install and easy_install-3.6 are installed
in '/home/jenkins/.local/bin' which is not on PATH. Consider adding
this directory to PATH or, if you prefer to suppress this warning use
--no-warn-script-location.

Change-Id: Iddf89e31cc18324ec936799cce378beb3d5809de
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoFix packagecloud match_release function and docs. 96/62796/8
Houa Yang [Tue, 14 Jan 2020 17:57:08 +0000 (11:57 -0600)]
Fix packagecloud match_release function and docs.

Fix search for string that matches the output used by
packagecloud gem when pushing packages.
Correct docs to reflect not having the version in the package name.

Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I1a696c77be94f3057c30764ae6ba710beed8bb84

4 years agoAdd info output in maven-javadoc-generate.sh 82/62782/3
Lott, Christopher (cl778h) [Mon, 13 Jan 2020 15:36:56 +0000 (10:36 -0500)]
Add info output in maven-javadoc-generate.sh

Revise shell script maven-javadoc-generate.sh to
drop unnecessary -l for login shell;
restore shell -x behavior to maven command for debug details in log;
add info output when creating tarball in verify path.

Change-Id: Ifdd5693dee73ffd7a75a97f3a572f3a31aafd7c4
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Issue-ID: RELENG-1872

4 years agoGenerate javadoc from a project in a subdirectory 95/62695/9
Lott, Christopher (cl778h) [Mon, 23 Dec 2019 20:32:48 +0000 (15:32 -0500)]
Generate javadoc from a project in a subdirectory

Lift the assumption that all files are in the git repository root.
Extend maven-javadoc publish and javadoc-verify macros with mvn-dir
configuration parameter (like tox-dir), defaults to '.' to preserve
existing behavior.
Extend maven-javadoc-generate.sh script to invoke mvn with the -f
option and use the directory name when archiving the generated HTML;
drop -x setting to conform with other scripts.
Document the new configuration parameter.

Projects using macros lf-maven-javadoc-publish, lf-maven-javadoc-verify
(i.e., using job templates gerrit-maven-javadoc-publish,
github-maven-javadoc-publish, gerrit-maven-javadoc-verify or
github-maven-javadoc-verify) must ensure the maven -f option is not
used in config parameter mvn-params.

Change-Id: Ib02a12a3192b8be577a0367476628c46db333231
Issue-ID: RELENG-1872
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoUpdate update-cloud-images script to use python3. 64/62764/5
Tim Johnson [Thu, 9 Jan 2020 17:31:08 +0000 (09:31 -0800)]
Update update-cloud-images script to use python3.

Use lf-activate-venv() to install openstack based on python3. Also added
error handling (script was ignoring errors by default). Minor cleanup of
syntax that does not effect functionality. Some improvements made to
comments and improved logging.

Issue: RELENG-2647
Change-Id: I4ddc0036bf178ca7185a00454dfa6f26433a8bab
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoAllow projects to set their tox-dir 92/62692/7 v0.49.0
Aric Gardner [Fri, 20 Dec 2019 20:21:48 +0000 (15:21 -0500)]
Allow projects to set their tox-dir

default is "."
Projects that don't want tox dir to be "."
can set it to "docs/"

also call to shell must be a
include-raw-escape

ISSUE-ID: RELENG-2520
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I53296072928b1a112213c7d041059c4e0b185812

4 years agoFix broken pipe in jenkins-sandbox-cleanup script 80/62680/3
Tim Johnson [Thu, 19 Dec 2019 19:13:30 +0000 (11:13 -0800)]
Fix broken pipe in jenkins-sandbox-cleanup script

Issue: RELENG-2640
Change-Id: I67b66146838e816b6da3e9846205ea68a7ce965c
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoUpdate lf-env Bash Library 30/62630/6
Tim Johnson [Sun, 15 Dec 2019 21:35:10 +0000 (13:35 -0800)]
Update lf-env Bash Library

lf-env is now a shell variable pointing to the head of the python
venv. Added --no-path & --system-site-packages flags to the
lf-activate-venv() function.

Issue: : RELENG-2566
Change-Id: Id5c5df6efb1da8c6912c346f56cb2d87f19cae7b
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoUpdate packagecloud jobs 12/62612/10
Houa Yang [Thu, 12 Dec 2019 17:34:31 +0000 (11:34 -0600)]
Update packagecloud jobs

Add sigul to lf-packagecloud-release-common. Add
set_variables_packagecloud. Allow tagging repo with ref or git_tag.
Add logs_dir to release-packagecloud-schema.yaml.

Issue-Id: RELENG-2579
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I23a8ef1e0aed261b0fbca6375c694b4ed268c082

4 years agoMerge "RTD Allow projects to change their default-version"
Aric Gardner [Mon, 16 Dec 2019 21:23:25 +0000 (21:23 +0000)]
Merge "RTD Allow projects to change their default-version"

4 years agoGet build cost estimates 04/62204/22
Tim Johnson [Fri, 8 Nov 2019 17:11:01 +0000 (09:11 -0800)]
Get build cost estimates

Generate and archive files containing cost, time & job info. If this is
a Openstack job call 'lftools openstack stack cost' to retrieve and
include costs from the stack members. Added two new scripts:
openstack-stack-delete.sh & job-cost.sh. The 'openstack-stack-delete
script' will retrieve the costs for the stack and delete the stack. The
'job-cost' will generate the job cost file which will include the costs
for the build agent and any Openstack members.

Issue: RELENG-1867
Change-Id: I658f5b42f90a28010e6f98f2f3c3ebb166192c30
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoRTD Allow projects to change their default-version 73/62373/25
Aric Gardner [Thu, 21 Nov 2019 21:32:18 +0000 (16:32 -0500)]
RTD Allow projects to change their default-version

Allow docs release via jjb config
Changes the default landing page from /latest/ to /stable/

by default we se to /latest/ (a no-op)
And on release change to stable via jjb config.

read the docs, by design creates /latest/ and /stable/ landing points
for the user.

If read the docs sees a tag or a new branch, it creates a /stable/
directory, which follows the latest tag or if there is no tag, the
latest branch.

Related issue where I ask for clarification as to their default
behavior in this regards.
https://github.com/readthedocs/readthedocs.org/issues/6415

ISSUE: RELENG-2549
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I33a8df14e68340ba268c953a471daa02ee6fe766

4 years agoMerge "Add JOB info to commit message"
Andrew Grimberg [Sat, 14 Dec 2019 16:31:16 +0000 (16:31 +0000)]
Merge "Add JOB info to commit message"

4 years agoAdd JOB info to commit message 15/62615/3
Jamo Luhrsen [Thu, 12 Dec 2019 20:25:36 +0000 (12:25 -0800)]
Add JOB info to commit message

Signed-off-by: Jamo Luhrsen <jluhrsen@gmail.com>
Change-Id: Icb6e7e40752a1c6f91b2c0f73c9cefc4ccfb91de
Signed-off-by: Jamo Luhrsen <jluhrsen@gmail.com>
4 years agoRemove Invalid function call from Bash library 11/62611/2
Tim Johnson [Thu, 12 Dec 2019 04:00:14 +0000 (20:00 -0800)]
Remove Invalid function call from Bash library

The function lf-echo-error() was renamed lf-echo-stderr().  Rename the
bad references. Some cleanup of log messages.

Issue: RELENG-2581
Change-Id: Ieaccc6049e3a7e70c3bb973f993849bc2a50579c
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoAdd explanation of ReadTheDocs v3 projects 10/62610/2
Lott, Christopher (cl778h) [Thu, 12 Dec 2019 14:43:45 +0000 (09:43 -0500)]
Add explanation of ReadTheDocs v3 projects

Change-Id: I2e25d97d1df9cc22ad3660e58764f256c0ee2e4c
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoMerge "Extend tag feature of release jobs"
Anil Belur [Thu, 12 Dec 2019 12:51:28 +0000 (12:51 +0000)]
Merge "Extend tag feature of release jobs"

4 years agoExtend tag feature of release jobs 03/62603/7
Lott, Christopher (cl778h) [Wed, 11 Dec 2019 17:08:07 +0000 (12:08 -0500)]
Extend tag feature of release jobs

Extend release-yaml schema files to allow git_tag entry.
Extend lf-build-with-parameters-maven-release macro with GIT_TAG.
Extend release-job.sh:
  * detect and use optional git_tag string.
  * detect and reject an existing lightweight tag that blocks
    push of a gpg-signed tag.
  * change GERRIT_HOST to GERRIT_URL in method for obtaining LF
    umbrella project to allow testing in the sandbox.
  * rename function from tag to tag-gerrit-repo.
  * simplify tests for presence of Jenkins parameter values
  * show more details about steps as INFO statements
Include schema file contents into documentation; remove the copies.

Change-Id: Iee2167b059c7f0ecf86ab6eb8dead36ec96be1b5
Issue-ID: RELENG-2568, RELENG-2586
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoRevert "Add step to verify stage repo is closed" 00/62600/1
Jessica Wagantall [Wed, 11 Dec 2019 17:04:57 +0000 (09:04 -0800)]
Revert "Add step to verify stage repo is closed"

This reverts commit 9c9feb37d817b1829d82e1531fa3a85499cbe7c4.

Change-Id: Ibd8060a990f2f39a20bb1e72cfd3b8a4e6e26c9c
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
4 years agoMerge "Revise docker-get-container-tag-script macro"
Eric Ball [Wed, 11 Dec 2019 02:16:16 +0000 (02:16 +0000)]
Merge "Revise docker-get-container-tag-script macro"

4 years agoFix code for generating Build Config Tarball 24/62424/10
Tim Johnson [Fri, 29 Nov 2019 19:18:11 +0000 (11:18 -0800)]
Fix code for generating Build Config Tarball

Also cleanup log messages (reduce and refine)

Issue: RELENG-2542
Change-Id: I9d7a9fa9e083afb54069a3f5166c8f05ce20fe16
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoMerge "Make gerrit-branch-lock work for all proj/branches"
Anil Belur [Tue, 10 Dec 2019 21:43:32 +0000 (21:43 +0000)]
Merge "Make gerrit-branch-lock work for all proj/branches"

4 years agoRevise docker-get-container-tag-script macro 90/62590/4
Lott, Christopher (cl778h) [Tue, 10 Dec 2019 19:04:06 +0000 (14:04 -0500)]
Revise docker-get-container-tag-script macro

Silence JJB warning by calling include-raw (not include-raw-escape)
when pulling in shell-script content; the result is the same.
Extend the documentation for the supported methods.
Add details on the contract of the shell script.

Change-Id: I6de0746653d3ceaf6df9fc63eb7aff5259f4b062
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoMake gerrit-branch-lock work for all proj/branches 47/62547/3
Eric Ball [Fri, 6 Dec 2019 04:04:19 +0000 (20:04 -0800)]
Make gerrit-branch-lock work for all proj/branches

Add gerrit-branch-lock to gerrit ci-jobs, and make it so that it can
be triggered from any change, for any branch.

Issue: RELENG-1236
Change-Id: I757f76c940bdbed10e3c1bc8a422ef94f1495dee
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
4 years agoExtend logs-deploy.sh with echo statements 53/62553/1
Lott, Christopher (cl778h) [Fri, 6 Dec 2019 15:10:24 +0000 (10:10 -0500)]
Extend logs-deploy.sh with echo statements

Show config/parameter values to document the script actions in logs.

Change-Id: I2e5e54d2f0c45bb4f3482e081ff8e17ad3b9e28a
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoAdd pypi prefix to example filename in layout 78/62478/2
Lott, Christopher (cl778h) [Wed, 4 Dec 2019 18:27:07 +0000 (13:27 -0500)]
Add pypi prefix to example filename in layout

A recent change to the PyPI release job trigger requires
"pypi" as the prefix on the release yaml file.
Also correct a one-word mistake in the release jobs doc,
should not say "verify" when discussing the merge template.

Change-Id: Ieeb7c96c225893ee52710e133afb06fa9639cb85
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
4 years agoMerge "Global JJB Disable generation of pip lists"
Andrew Grimberg [Tue, 3 Dec 2019 22:00:41 +0000 (22:00 +0000)]
Merge "Global JJB Disable generation of pip lists"

4 years agoFix lf-release-jobs 34/62434/10
Houa Yang [Mon, 2 Dec 2019 16:30:08 +0000 (10:30 -0600)]
Fix lf-release-jobs

Add -m to deal with merge commits.

-m
By default, git diff-tree --stdin does not show differences
for merge commits. With this flag, it shows differences
to that commit from all of its parents.

Refactor packagecloud release/merge parameters and add distribution_type
as environmental variable.
Remove distribution_type from pypi and packagecloud schema.
Add variable values for packagecloud in release-job.sh.

ISSUE-ID: RELENG-2567
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I4bf718874b754bd24c03e22a9184e0b1566075e9

4 years agoGlobal JJB Disable generation of pip lists 29/62429/1
Tim Johnson [Sun, 1 Dec 2019 16:31:04 +0000 (08:31 -0800)]
Global JJB Disable generation of pip lists

Issue: RELENG-2560
Change-Id: I2ec3cf5fa73e98334468067c76b605e1fbf4f4b9
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoMerge "Fix jenkins-verify-images.sh crash on grep"
Aric Gardner [Fri, 29 Nov 2019 17:54:15 +0000 (17:54 +0000)]
Merge "Fix jenkins-verify-images.sh crash on grep"

4 years agoRemove user venv from sandbox cleanup script 17/62417/3
Tim Johnson [Wed, 27 Nov 2019 16:37:38 +0000 (08:37 -0800)]
Remove user venv from sandbox cleanup script

No longer use jjb-install.sh & jjb-cleanup.sh

Issue: RELENG-2565
Change-Id: I08e0bab6fe31a48d933c8375b75f4c9264df7a08
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoFix jenkins-verify-images.sh crash on grep 16/62416/3
Bengt Thuree [Wed, 27 Nov 2019 13:24:10 +0000 (00:24 +1100)]
Fix jenkins-verify-images.sh crash on grep

This statement triggers the pipefail
    grep ^OS_CLOUD= jenkins-config/clouds/openstack/cattle/cloud.cfg
    $ echo $?
    1

ISSUE-ID: IT-18352

Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
Change-Id: Id71b3b664f5dca98bc4d784b86e8d68cec7fca9e

4 years agoMerge "Update lf-release-jobs gerrit trigger"
Houa Yang [Tue, 26 Nov 2019 21:04:11 +0000 (21:04 +0000)]
Merge "Update lf-release-jobs gerrit trigger"

4 years agoUpdate lf-release-jobs gerrit trigger 12/62412/3
Houa Yang [Tue, 26 Nov 2019 17:11:22 +0000 (11:11 -0600)]
Update lf-release-jobs gerrit trigger

Update release jobs regex to match based on file name and not on
releases/*.yaml.

Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I736a645a710f32228dc716d00662d5bbc322d4c4

4 years agoRemove use of user venv by builder-jjb-merge job 00/62400/2
Tim Johnson [Mon, 25 Nov 2019 21:35:15 +0000 (13:35 -0800)]
Remove use of user venv by builder-jjb-merge job

Call lf-activate-venv() to create venv to install jenkins-jobs. Remove
references to jjb-cleanup.sh & jjb-install.sh scripts. Some jobs still
depend on them, so they have to remain for now.

Issue: RELENG-2557
Change-Id: Ib6b8a6e65113a874994a447cb628ca08dc9604c0
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoMerge "Fix packagecloud wget schema url"
Houa Yang [Tue, 26 Nov 2019 17:32:13 +0000 (17:32 +0000)]
Merge "Fix packagecloud wget schema url"

4 years agoFix packagecloud wget schema url 13/62413/1
Houa Yang [Tue, 26 Nov 2019 17:14:10 +0000 (11:14 -0600)]
Fix packagecloud wget schema url

Fix breaking up the url for packagecloud schema.

Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I7799a2237173a714b1b98df39e2ac36ca34d6316

4 years agoAdd kubernetes cluster support 84/62184/17
DW Talton [Thu, 7 Nov 2019 00:00:01 +0000 (17:00 -0700)]
Add kubernetes cluster support

Add the ability for jobs to use kubernetes clusters hosted on Vexx.

Issue: RELENG-2528
Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: I730b0a57849bb0333cda6643c27d49148195bb7e

4 years agoMerge "Support OpenStack Cloud Jenkins plugin upgrade"
DW Talton [Mon, 25 Nov 2019 23:24:32 +0000 (23:24 +0000)]
Merge "Support OpenStack Cloud Jenkins plugin upgrade"

4 years agoSupport OpenStack Cloud Jenkins plugin upgrade 69/62369/5
DW Talton [Thu, 21 Nov 2019 00:11:42 +0000 (17:11 -0700)]
Support OpenStack Cloud Jenkins plugin upgrade

Version 2.47+ of the OpenStack Cloud plugin for Jenkins adds two new
SlaveOptions params, node_properties and config_drive. Failure to send
these params results in a failed request.

Replace method version_ge with testversion for plugin version
comparisons. Needed to enable deprecation and future plugin version
checking.

Refactor get_minion_options, replacing duplicated code blocks with if
checks.

Deprecate INSTANCE_MIN_CAPMAX config param, to be replaced with
INSTANCE_MIN, for configuring the minimum number of instances.

Issue: RELENG-2514
Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: Ic8773bd4ef44ea09b5635c4a84d6452e29ad777c

4 years agoMerge "Revert "Fix lf-release-jobs packagecloud jobname""
Houa Yang [Mon, 25 Nov 2019 18:55:21 +0000 (18:55 +0000)]
Merge "Revert "Fix lf-release-jobs packagecloud jobname""

4 years agoRevert "Fix lf-release-jobs packagecloud jobname" 97/62397/5
Houa Yang [Mon, 25 Nov 2019 17:35:02 +0000 (17:35 +0000)]
Revert "Fix lf-release-jobs packagecloud jobname"

This reverts commit 0fca5e8ba63946ad8dd568e545c9568c2c60f84f.
This also corrects the names to follow current standard.

Reason for revert: Revert to correct naming to comply with naming
standard already in use.

Change-Id: Ida9cb7b0a9d2909066c6f146fef3de39de3a1674
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
4 years agoMerge "Fix lf-release-jobs packagecloud jobname"
Houa Yang [Mon, 25 Nov 2019 17:24:27 +0000 (17:24 +0000)]
Merge "Fix lf-release-jobs packagecloud jobname"

4 years agoMerge "Update lf-docker-get-container-tag"
Andrew Grimberg [Fri, 22 Nov 2019 19:51:36 +0000 (19:51 +0000)]
Merge "Update lf-docker-get-container-tag"

4 years agoFix lf-release-jobs packagecloud jobname 80/62380/1
Houa Yang [Fri, 22 Nov 2019 16:30:04 +0000 (10:30 -0600)]
Fix lf-release-jobs packagecloud jobname

Fix name of the packagecloud-verify job name

Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: If09413b6aa531c117f09082d12dfbb9badb93bfb

4 years agoAdd packagecloud support global jjb 00/62300/13
Houa Yang [Sat, 16 Nov 2019 00:12:41 +0000 (18:12 -0600)]
Add packagecloud support global jjb

Add support for automating promotion of packages in packagecloud.

Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I97dfa3822462e0a1076e5546f97c216e18343795

4 years agoUpdate lf-docker-get-container-tag 70/62370/2
Jessica Wagantall [Thu, 21 Nov 2019 00:56:25 +0000 (16:56 -0800)]
Update lf-docker-get-container-tag

Allow shell script builder step to be customizable
in lf-docker-get-container-tag.
This will allow other teams like ONAP to call their
own script. Set to default to
"../shell/docker-get-container-tag.sh" which is what
teams were already using.

This feature is requested by ONAP's DCAE team to allow
them to handle tags differently via their own script.

Issue-ID: RELENG-2422
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I9afd52a0ad8916694db0e958e42eb80f98e4e28b

4 years agoRemove use of user venv by build-jjb-verify job 01/62301/5
Tim Johnson [Sat, 16 Nov 2019 00:23:02 +0000 (16:23 -0800)]
Remove use of user venv by build-jjb-verify job

Call lf-activate-venv() to create required venv. Not longer need
jjb-cleanup.sh & jjb-install.sh. Some project repos still depend on
them, so they have to remain for now.

Issue: RELENG-2538
Change-Id: I22f122d81a7f4977cb0be1e52f5c0317390f23cd
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoTrigger latest each time. 14/62314/4
Aric Gardner [Mon, 18 Nov 2019 14:59:28 +0000 (09:59 -0500)]
Trigger latest each time.

Trigger "latest" and "branch"

By default rtd serves documentation at /latest/

When a buid was triggered via curl, using a the old method.
"generic api integration"

curl -X POST -d "branches=${GERRIT_BRANCH}" -d "token=$RTD_TOKEN"
"$RTD_BUILD_URL

this returned for example:

{"build_triggered":true,"project":"o-ran-sc-doc","versions":
["latest","master"]}

So the previous behavior was to build latest and master from a call with
branches=master

when we post via the v3 api.
lftools rtd project-build-trigger "$rtdproject" "$STREAM"

It actually respects the $STREAM variable and only builds "master"
This is not what we want,
we want latest to be updated as well.
so we need to trigger both builds.

It is possible to, after project creation "enable master branch"
and then "delete default branch"
so that /latest/ ceases to exist.
but having /latest/ is the default behavior is seems best to
not fight against it, to simplify initial project setup.

The best solution I see right now is to just always trigger latest
as well as the branch we want to build.

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I6e2062aa30383ee79a1d64d8f5327ef9a05cece6

4 years agoFix OS_CLOUD export for image validation 18/62318/2 v0.48.1
Andrew Grimberg [Mon, 18 Nov 2019 17:47:44 +0000 (09:47 -0800)]
Fix OS_CLOUD export for image validation

Change fe2beb15bc9effb5b91ba17b520ab6c167dd6604 introduced a bug where
the CLOUD_CREDENTIAL_ID was being used for the OS_CLOUD export variable.
This was done since the test for multi-cloud image validation had the
credential IDs matching the OS_CLOUD variable. However, most of our
systems have the CLOUD_CREDENTIAL_ID set to the default of 'os-cloud'
and therefore after pulling updating to a global-jjb with that change
would no longer be able to validate images!

We now look for an OS_CLOUD variable in the cloud.cfg file and use it if
it is set, otherwise we default to an OS_CLOUD of 'vex' as we previously
did before change fe2beb15bc9effb5b91ba17b520ab6c167dd6604

Change-Id: I0af763495fa3a7ce2e8c080324abaf5d384e5188
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>