Lott, Christopher (cl778h) [Fri, 8 Nov 2019 10:36:38 +0000 (05:36 -0500)]
Report POST details in rtd-trigger-build.sh
Add echo to rtd-trigger-build.sh to report branch and URL used at
readthedocs.io, both of which may be modified in the script.
Change-Id: I301449b5f1c0359f3ee8fef823fc6dff13298b78
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Lott, Christopher (cl778h) [Wed, 6 Nov 2019 16:30:52 +0000 (11:30 -0500)]
Add .pypirc config file provider to release job
Add .pypirc config file provider to builders in release job.
Extend PyPI documentation with recommended directory layout.
Change-Id: I96814ad3c78187835ab26dd8adbb7532753315e9
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Lott, Christopher (cl778h) [Tue, 5 Nov 2019 14:42:33 +0000 (09:42 -0500)]
Document need for quotes around python version
In the python_version value in the release yaml file, a valid decimal
number like 3 or 3.7 must be quoted so the schema validator treats it
as string, not number.
Change-Id: I6ee1dda1b1d2133b5c9576bad00a344ddd58cea5
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Eric Ball [Mon, 4 Nov 2019 17:15:30 +0000 (17:15 +0000)]
Merge "Use "upgrade-strategy eager" for pip install"
Lott, Christopher (cl778h) [Mon, 4 Nov 2019 16:15:12 +0000 (11:15 -0500)]
Change additional to other to silence coala
Coala rejects use of 'additional' in documentation of Gerrit triggers,
but accepts the word 'other' which is equally readable.
Change-Id: Ibcc0235fc50c400c1b6eaeeac3ef8de542f7687c
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Lott, Christopher (cl778h) [Fri, 1 Nov 2019 18:02:09 +0000 (14:02 -0400)]
Use compare-type REG_EXP for Python job file paths
Change compare-type to REG_EXP in macros lf_python_clm_xc and
lf_pypi_common for config parameter gerrit_trigger_file_paths
with the regular expression .*
Previously was ANT, which didn't match anything with pattern .*
so recheck/reverify comment triggers did not work.
Clarify documentation of comment triggers for python jobs.
Change-Id: Id926d83e454fce2aa795b15d27c794fc41bbd1d3
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Anil Belur [Thu, 31 Oct 2019 13:57:20 +0000 (19:27 +0530)]
Update docker job templates gerrit trigger regexes
Update the gerrit trigger regexes for the docker job templates
to use shorter and more readable versions of the regexes
also used in other verify and merge job templates.
Change-Id: I8c189735c472ff22d9dd5093017234e74a2ac5c2
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Lott, Christopher (cl778h) [Thu, 31 Oct 2019 00:27:13 +0000 (20:27 -0400)]
Repair comment regexes on release templates
Change the comment-added-contains-event string for the release-verify
and release-merge templates to the same regexes used in all other verify
and merge templates, instead of custom versions that didn't work;
verify now uses '^Patch Set\s+\d+:\s+(recheck|reverify)\s*$'
merge now uses '^Patch Set\s+\d+:\s+remerge\s*$'
Change-Id: I4436568046ffbc15225115726e2664824bd6bea2
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Tim Johnson [Tue, 29 Oct 2019 19:00:48 +0000 (12:00 -0700)]
Update lf-venv-create function
Update the lf-venv-create() function to support a list of python
packages to install in the new venv.
Issue: RELENG-2502
Change-Id: Ic2e186e3d7db653f57b755974aeb1c63581df8f2
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
Eric Ball [Tue, 29 Oct 2019 19:43:10 +0000 (12:43 -0700)]
Use "upgrade-strategy eager" for pip install
Pip changed its default upgrade-strategy to "only-if-required", which
is not correctly upgrading requests (and potentially other packages
in the future) to meet other packages' requirements. This results in
errors in the build log. By using the upgrade-strategy "eager", pip
is able to properly install what is needed.
Issue: RELENG-2074
Change-Id: I0e37aac57553a352a4c0797d0ef8d0b597072440
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Lott, Christopher (cl778h) [Wed, 23 Oct 2019 01:43:27 +0000 (21:43 -0400)]
Revise PyPI release jobs to use a staging index
Rewrite PyPI release-verify and release-merge templates to download
distribution files from a PyPI staging index and upload the files to
a PyPI release index. Remove the tox etc. builders from the release
jobs. Install and upgrade python packages setuptools, twine etc.
Drop the branch/stream parameters for the pypi release templates;
only need one release-verify and one release-merge job per project.
Split the PyPI job groups because the verify & merge templates do not
accept the same arguments as release-verify & release-merge templates,
especially no stream/branch expected by release templates.
Extend release-job.sh with PyPI release features, drop the old file
pypi-tag-release.sh
Move the PyPI release features to the lf-release-jobs.(rst,yaml) files
from the lf-python-jobs.(rst,yaml) files.
Refactor the PyPI verify and merge macros and templates due to moving
out the release-verify and release-merge features.
Restore lf-python-jobs.rst to Coala processing by WriteGoodLintBear,
but turn off processing for sections with the word 'validate'.
Change-Id: Ie27071076f9c018ccf8765c9dd948a4f5c876ec9
Issue-ID: RELENG-2455, RELENG-2497
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Aric Gardner [Mon, 28 Oct 2019 21:16:09 +0000 (21:16 +0000)]
Merge "pip update virtualenv in tox-install"
Aric Gardner [Mon, 28 Oct 2019 18:03:20 +0000 (14:03 -0400)]
pip update virtualenv in tox-install
tox calls system virtualenv which breaks the coala linter.
as well as bringing with it a a very old version of pip from system.
updating virtualenv fixes this.
Also, envlist no longer has py35,py36,py37
so remove pins that were for supporting old python version
removed python2 workarounds.
-python3 -m nltk.downloader ect
Also removed
-passenv = TOX_WORK_DIR
-toxworkdir = {env:TOX_WORK_DIR:/tmp/v}
This was added in 11728
"By default tox environment will now be installed under /tmp/v/{envname}
but configurable through the CLI environment by specifying
TOX_WORK_DIR."
builder-jjb-merge and builder-jjb-verify to not call tox.
So it would not help with opnfv race conditions with multiple jjb
verify/deploys running on a static builder.
there are actually no tox jobs for opnfv/releng repo.
Pretty sure this does nothing and just adds a layer
of complexity that is not needed.
so removed.
ISSUE: RELENG-2366
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I07875a62341e28860a81ee7b2149393c148be84b
Thanh Ha [Mon, 28 Oct 2019 19:24:05 +0000 (15:24 -0400)]
Update script_dir to jjb dir
Configuration in global-jjb used to be in the root of global-jjb
but has since moved to a sub-dir to facilitate CI configuration files
that may be in yaml format.
Change-Id: I8a2ff9ccf2b41c941432b9ebe2d5702467c14967
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Tim Johnson [Mon, 21 Oct 2019 02:41:42 +0000 (19:41 -0700)]
Update Bash Library
Added lf-create-venv() and lf-venv-add().
Change-Id: If50c9c282adb8b11176bbe6223578ea7edad1839
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
Lott, Christopher (cl778h) [Mon, 28 Oct 2019 12:00:36 +0000 (08:00 -0400)]
Repair minor problems in self-release
Add missing underscore in two echo commands in release-job.sh.
Move comment for shellcheck disable onto separate line in lf-env.sh.
Extend documentation of the container self-release process.
Exclude lf-rtdv3.rst from WriteGoodLintBear coala processing.
Change-Id: Idc363f9e2b0a6189283b8dd0385926b870a2db27
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Bengt Thuree [Fri, 25 Oct 2019 00:35:00 +0000 (00:35 +0000)]
Merge "Use -o with grep to guarantee filename position"
Eric Ball [Thu, 24 Oct 2019 23:32:51 +0000 (16:32 -0700)]
Use -o with grep to guarantee filename position
The "-o" option makes grep only return the matching words. This way,
even if there are other params listed before -f, the filename will
always be in position 2 of the grep output.
Issue: RELENG-2428
Change-Id: Id52e3bf285de7f0e46d4a442ef4b0986d2e66cbc
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Eric Ball [Wed, 23 Oct 2019 15:33:27 +0000 (15:33 +0000)]
Merge "Extend PyPI script to call twine check"
Anil Belur [Tue, 22 Oct 2019 14:42:31 +0000 (14:42 +0000)]
Merge "Add step to verify stage repo is closed"
Anil Belur [Tue, 22 Oct 2019 12:07:36 +0000 (12:07 +0000)]
Merge "Verify maven release versions"
Lott, Christopher (cl778h) [Mon, 21 Oct 2019 19:52:46 +0000 (15:52 -0400)]
Extend PyPI script to call twine check
Extend pypi-dist-build.sh to call twine check, which tests
if the distribution's long description will render correctly.
Change-Id: I2229266f17bc4d1272219466f8fbf2e2614ca21f
Issue-Id: RELENG-2317
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Aric Gardner [Mon, 21 Oct 2019 18:05:41 +0000 (18:05 +0000)]
Merge "Fix pattern to trigger on any files in docs/"
Anil Belur [Sat, 19 Oct 2019 11:35:14 +0000 (17:05 +0530)]
Add step to verify stage repo is closed
Add step to verify Nexus staging repository is closed correctly or
fail the staging job, if the repository is still open. This allows
projects to address the validation failures earlier during
maven-stage phase, instead of waiting to catch them while running
the self-release job.
Issue: IT-17976
Change-Id: I88ca6cce67e2400ab1d24fd3cbac419bb8b3eb91
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Aric Gardner [Wed, 16 Oct 2019 15:03:39 +0000 (11:03 -0400)]
Fix pattern to trigger on any files in docs/
Also do not trigger when docs in global-jjb submodule
are updated.
add TOX_ENVs to rtdv3 macro so that it only runs
docs,docs-linkcheck
add conditional-step so that project must have a
.readthedocs.yaml for build to bother starting
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I7716ceae30e0b460da0bd427b0670a6e1db1688c
Jessica Wagantall [Tue, 15 Oct 2019 23:31:04 +0000 (16:31 -0700)]
Verify maven release versions
Add a verification step to maven self releases
to make sure the version being defined in the
releases file matches the actual version produced
by the maven-stage job that created the release
candidate.
This is to prevent releases being pushed in Nexus
with a version different from what the developer
intended in the releases file.
Issue: RELENG-2356
Change-Id: I0555ee782f608e04d10dda5112c283783bfc2a91
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Lott, Christopher (cl778h) [Wed, 16 Oct 2019 21:26:47 +0000 (17:26 -0400)]
Use release credential in pypi release merge
Add macro with lf-infra-wrappers block to set jenkins-ssh-credential
parameter to the value in parameter jenkins-ssh-release-credential,
which makes the PyPI release merge templates parallel to the release-job
merge template. Both need privileges to push a tag from Jenkins.
Document the revised configuration parameter. Silence yamllint issues.
Change-Id: I534c659ec4c5165184bc1de8bb7021dab116deab
Issue-Id: RELENG-2317
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Tim Johnson [Mon, 14 Oct 2019 16:27:23 +0000 (09:27 -0700)]
Change case of 'maven_options'
Shellcheck will check for 'used-before-set' only if variable is
lower-case.
Change-Id: I5c91d493a584d332cc587be37d53b3c98e5dbb3a
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
Anil Belur [Wed, 16 Oct 2019 13:49:44 +0000 (13:49 +0000)]
Merge "Support sonarcloud in python projects"
Aric Gardner [Wed, 16 Oct 2019 12:44:48 +0000 (12:44 +0000)]
Merge "Add comment trigger to PyPI release merge template"
Anil Belur [Wed, 16 Oct 2019 02:44:40 +0000 (02:44 +0000)]
Merge "Fix rtd merge job to handle new tag uploaded"
Lott, Christopher (cl778h) [Wed, 16 Oct 2019 01:43:40 +0000 (21:43 -0400)]
Add comment trigger to PyPI release merge template
Add missing trigger "remerge" to the PyPI release merge template.
Move trigger definitions into PyPI templates, instead of defining
four separate trigger definition blocks and using each exactly once.
Document the required comment text for the triggers.
Change-Id: I9c12b89a2e3ded80407e6399c62f6e9337bf5cd7
Issue-Id: RELENG-2317
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Aric Gardner [Tue, 15 Oct 2019 19:25:12 +0000 (15:25 -0400)]
Change project-compare-type to ANT
I missed this in the sandbox and jobs are triggered
by hand in the sandbox
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I3814c724e6f21e43dff5777753c8358a861a9280
Lott, Christopher (cl778h) [Tue, 15 Oct 2019 18:17:36 +0000 (14:17 -0400)]
Support sonarcloud in python projects
Extend lf_tox_sonar with sonarcloud, sonarcloud-project-key,
sonarcloud-project-organization, sonarcloud-api-token, tox-dir
and tox-envs properties; also with lf-infra-tox-parameters macro.
Use new sonarcloud property as guard for conditional builder step,
if true use lf-infra-maven-sonarcloud, else use lf-infra-maven-sonar.
Issue: RELENG-2418
Change-Id: Iafa2720da7f939a126374bc83191c328edcdc40b
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Aric Gardner [Tue, 15 Oct 2019 17:41:07 +0000 (17:41 +0000)]
Merge "Readthedocs v2 jobs"
Anil Belur [Tue, 15 Oct 2019 05:39:02 +0000 (11:09 +0530)]
Fix rtd merge job to handle new tag uploaded
Issue: RELENG-2445
Change-Id: I83bbb8b5b44ae6aee8e9cafd1711e8ac19b48811
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Aric Gardner [Mon, 9 Sep 2019 18:25:50 +0000 (14:25 -0400)]
Readthedocs v2 jobs
This change will need a new release of
lftools before It will work.
Global job that triggers on any docs changes.
Creates docs project in rtd if absent on merge
Creates subproject association if project does not
equal Jenkins global var MASTER_RTD_PROJECT
Triggers docs build on merge
fix tox job to push docs-dir to archives
so that we can see generated docs
ISSUE: RELENG-2362
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Id6d85068eb068301b91d4deb74490e76381a7e54
Lott, Christopher (cl778h) [Thu, 10 Oct 2019 18:43:55 +0000 (14:43 -0400)]
Repair pypi template tag and trigger behaviors
Add missing {branch} parameter to branch-pattern in gerrit trigger
blocks in PyPI release-verify and release-merge templates. Jobs
were starting on all defined branches, not limited to the target.
Change pypi-tag-release.sh script to continue (not declare error)
if the tag already exists.
Update docs for the tagging behavior.
Change-Id: I76f31520794cc673e7f398ea90cd17d5ef3851a8
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Eric Ball [Tue, 8 Oct 2019 21:19:59 +0000 (21:19 +0000)]
Merge "Add sonar-project-file parameter to ci sonar jobs"
Eric Ball [Mon, 7 Oct 2019 21:40:22 +0000 (14:40 -0700)]
Add sonar-project-file parameter to ci sonar jobs
By enabling the caller to override the default project file name with
an empty string, we enable the ability to provide project settings
directly in the sonar-properties field. This removes the requirement
for a "sonar-project.properties" file in the repo.
Change-Id: Ia1069417d3bb2a742a68d6113fbd1d8e0374cbdc
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Anil Belur [Sat, 5 Oct 2019 05:10:48 +0000 (05:10 +0000)]
Merge "Fix auto update image script to compare image type"
Thanh Ha (zxiiro) [Fri, 4 Oct 2019 15:19:08 +0000 (15:19 +0000)]
Revert "Update packer version to 1.4.4"
This reverts commit
0727a30524c2f708f81448c03dd37d1ebb272167.
Reason for revert: 1.4.4 still fails packer build vexxhost: Error setting image metadata: Bad request with: [POST https://block-storage-ca-ymq-1.vexxhost.net/v3/
12c36e260d8e4bb2913965203b1b491f/volumes/
d0bdc2ef-8827-441f-84ad-
3edf52616bdb/action], error message: {"badRequest": {"message": "Invalid volume: Updating volume metadata is not allowed for volumes in uploading status.", "code": 400}}
Change-Id: Ibc534b1a66f08e71e40780457eadba87ca7dda81
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Aric Gardner [Fri, 4 Oct 2019 05:09:42 +0000 (10:39 +0530)]
Update packer version to 1.4.4
Update packer version to 1.4.4, which fixes the issue
with the metadata update failure during image update.
https://github.com/hashicorp/packer/pull/8016
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Change-Id: Ibe2ee5e7b22008fd6ee7ec29361bbe110da5cbac
Aric Gardner [Thu, 3 Oct 2019 15:54:26 +0000 (15:54 +0000)]
Merge "Change PyPI merge repo default and fix doc URL"
Lott, Christopher (cl778h) [Thu, 3 Oct 2019 13:54:28 +0000 (09:54 -0400)]
Change PyPI merge repo default and fix doc URL
Correct doc .pypirc test URL to https://test.pypi.org/legacy/
Change doc .pypirc to use API tokens instead of username/password.
Use pypi-test as the default repository name in the PyPI merge template
(instead of "staging") to match established ONAP practice.
Change-Id: Id8a6f253f05a7c15790f45979d7dc47039aaf5cd
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Anil Belur [Thu, 3 Oct 2019 11:46:12 +0000 (17:16 +0530)]
Fix auto update image script to compare image type
Fix auto update image script to compare the image type before
updating an the image in the source repository. This fixes the
bug that updates images although they are the same flavour but
a diffirent type.
Issue: RELENG-2352
Change-Id: Ibaaeea41fb3d8a413e527c56b898a1fecadf5d22
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Tim Johnson [Mon, 30 Sep 2019 16:08:39 +0000 (09:08 -0700)]
Global JJB use python3 versions of lftools and jjb
Create LF bash 'library' and install in ~jenkins
Issue: RELENG-2411
Change-Id: I211c85b9823ca5582a263ec145797d4de9424e10
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
Lott, Christopher (cl778h) [Wed, 2 Oct 2019 17:35:10 +0000 (13:35 -0400)]
Add missing DRY_RUN parameter in PyPI merge
Add DRY_RUN parameter to the PyPI verify and merge template common
macro because it's required by the pypi_upload.sh script.
Also adjust verbosity of shell scripts - quiet down pip, add
repository name to output.
Change-Id: I947ee16fb6e42a6a3ec0f0de2722a6ecf7200ceb
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Lott, Christopher (cl778h) [Tue, 1 Oct 2019 10:37:56 +0000 (06:37 -0400)]
Adjust Jenkins triggers in PyPI templates
- In the PyPI merge template, add cron parameter to support daily
build and push to a staging repo, like the maven merge template.
- In PyPI release templates, change name of gerrit and github trigger
file patterns parameter. This avoids accidental overriding by jobs
that limit their actions to subdirectories. The release file patterns
are hardcoded in a shell script. Remove params from doc.
- In all PyPI templates, add disabled option and disable-job parameter
to be consistent with other python templates.
Change-Id: Ib2934728efbb4d957ad622dc84f1ee2a8383a27c
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Lott, Christopher (cl778h) [Fri, 27 Sep 2019 18:25:05 +0000 (14:25 -0400)]
Improve doc in maven job groups
Improve documentation about use of git, maven, docker.
Fold in maven-docker-job-groups file content and drop that file.
Move maven-docker test file into maven-jobs subdirectory.
No functional change.
Issue: RELENG-908
Change-Id: Ifca5364acfcd9afdb98b25afeb1d089d95311e34
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Aric Gardner [Fri, 27 Sep 2019 16:11:03 +0000 (12:11 -0400)]
Fix rtd verify for lftools
lftools docs build runs literal lftools commands
these commands are only avaliable if the current patchset
is installed.
this part is a bit of a hacke, suggestions welcome.
also, if anyone knows how this ever worked before,
id love to know.
Also, install tox so that path has python 3 version of tox
first in path.
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Ic3f9bd788db851bec815e86d57a1788e7eae945a
Aric Gardner [Thu, 26 Sep 2019 17:12:39 +0000 (13:12 -0400)]
RTD builds, python version should be python3
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Ib44e4291a88be05983d8df3bbf389403ad1a329e
Eric Ball [Wed, 25 Sep 2019 22:00:27 +0000 (15:00 -0700)]
Change Packer version back to 1.4.2
Packer 1.4.3 frequently has issues setting metadata after a build
completes.
There is a WIP change[0] to upgrade to 1.4.4, but per notes there,
it will not be available until October. Until then, the version
should be downgraded instead, as all Packer builds with 1.4.3 are
failing.
0: https://gerrit.linuxfoundation.org/infra/c/releng/global-jjb/+/61742
Change-Id: Ideb5aa607a9fccbd319e4638413414c5cb4c8a23
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Aric Gardner [Wed, 25 Sep 2019 18:35:57 +0000 (18:35 +0000)]
Merge "Revert "Fix rtd verify job choosing strategy""
Tim Johnson [Mon, 9 Sep 2019 17:51:54 +0000 (10:51 -0700)]
Capture sudo log on builders
Created new script sudo-logs.sh to archive the sudo logs
Issue: RELENG-1994
Change-Id: I86a44e6abf18e35c7dc0e1840797e0dde5c764fb
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
Chris Lott [Wed, 25 Sep 2019 16:12:51 +0000 (16:12 +0000)]
Revert "Fix rtd verify job choosing strategy"
This reverts commit
c02c7a2ac05e3a9b987edc15d555a5124a2ad976.
Reason for revert: Acumos RTD jobs are triggered by commits to ANY Acumos repo on a *.rst file. This change causes the doc job to checkout the commit ID from that other repo, which is never present in the doc repo.
Issue-ID: RELENG-911
Change-Id: Icab10bb8b0e3e3cc28458751a9de1edf88fdc3d7
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Jessica Wagantall [Wed, 25 Sep 2019 10:10:53 +0000 (10:10 +0000)]
Merge "Lift PyPI templates to Global-JJB from Acumos"
Aric Gardner [Mon, 23 Sep 2019 17:27:05 +0000 (13:27 -0400)]
Move python version to python3
Remove pin workarounds for python2
install tox via python3 pip
so that the version of tox is run from
/home/jenkins/.local/lib/python3.6/site-packages/tox/__init__.py
fix release-job to use venv
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I4b2d778f3fd81565c5dd009d50c969696faba0d2
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
Lott, Christopher (cl778h) [Wed, 21 Aug 2019 14:54:25 +0000 (10:54 -0400)]
Lift PyPI templates to Global-JJB from Acumos
New templates provide features to build and push Python
source and binary distributions to a PyPI repository.
Check version string and push a tag on release.
New job groups:
{project-name}-gerrit-pypi-jobs
{project-name}-github-pypi-jobs
New templates:
{project-name}-pypi-verify-{stream}
gerrit-pypi-verify
github-pypi-verify
{project-name}-pypi-merge-{stream}
gerrit-pypi-merge
github-pypi-merge
{project-name}-pypi-release-verify-{stream}
gerrit-pypi-release-verify
github-pypi-release-verify
{project-name}-pypi-release-merge-{stream}
gerrit-pypi-release-merge
github-pypi-release-merge
Change-Id: If74925623f2a5804de8684b9a9933573dfea29a1
Issue-Id: RELENG-2317
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Tim Johnson [Fri, 20 Sep 2019 22:21:31 +0000 (15:21 -0700)]
Global JJB Fatal Shellcheck Errors
Fix all tox warnings/errors
Issue: RELENG-1214
Change-Id: Ic4c1743828dbe4a230eaf2950c03d8c7fbfaf458
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
Bengt Thuree [Wed, 18 Sep 2019 08:56:15 +0000 (10:56 +0200)]
Change sonar trigger to run-sonar
It was stage-release, and with this change it
will be similar to other jobs.
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
Change-Id: I4c458979ec696c7d368bf410dabaa43ee7abb2f7
Thanh Ha [Thu, 19 Sep 2019 16:17:37 +0000 (12:17 -0400)]
Set DISTRIBUTION_TYPE to choice
As for as I can tell from the release_job.sh file there are only 3
valid options "None" as the default and choices are container or maven.
Set parameter to choices to help the user of the job select the right
option.
Change-Id: I587d63103011b9f42252a58ce3a79c44b46ceefd
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Jessica Wagantall [Tue, 17 Sep 2019 23:56:24 +0000 (23:56 +0000)]
Merge "Add sonar-prescan jobs"
Jessica Wagantall [Tue, 17 Sep 2019 17:02:58 +0000 (17:02 +0000)]
Merge "Allow java opts in WS scans"
Jessica Wagantall [Mon, 16 Sep 2019 23:47:54 +0000 (16:47 -0700)]
Allow java opts in WS scans
Allow java options for WhiteSource
Unified Agent calls
Issue: RELENG-2387
Change-Id: I4b0d717a38448e1a8ced1ba63416c1b0a92d26e2
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Aric Gardner [Mon, 16 Sep 2019 23:30:54 +0000 (23:30 +0000)]
Merge "Fix rtd verify job choosing strategy"
Houa Yang [Fri, 13 Sep 2019 16:03:35 +0000 (11:03 -0500)]
Fix rtd verify job choosing strategy
Change default setting of choosing strategy from default to gerrit
for rtd verify job. If the stratgey is not gerrit, you will need
to manually set the choosing strategy to gerrit in the jenkins job
before you can build using a specific parameter refspec.
Issue-Id:RELENG-911
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: If5f10f83048b907123bf423394ef0adf85607806
Anil Belur [Mon, 16 Sep 2019 17:18:02 +0000 (22:48 +0530)]
Add template to update OpenStack cloud images
This job finds and updates OpenStack cloud images on the
ci-management source repository.
The job is triggered in two ways:
1. When a packer merge job completes, the new image name created is
passed down to the job.
2. Manually trigger the job to update all images.
When the job is triggered through an upstream packer merge job, this
only generates a change request for the new image built.
When the job is triggered manually, this job finds the latest
images on OpenStack cloud and compares them with the images currently
used in the source ci-management source repository. If the compared
images have newer time stamps are **all** updated through a change
request.
This job requires a Jenkins configuration merge and verify job setup and
working on Jenkins.
Issue: RELENG-2352
Change-Id: I2c75407bbe9d6866ded492310b8d34b47c3a1341
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Aric Gardner [Mon, 16 Sep 2019 15:32:36 +0000 (11:32 -0400)]
pyenv pick latest installed version
Logic to pick latest version of pyev.
Builders may have diffrent versions installed.
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I2eb5cc75d815778bb35f166d1512711705b6ccd7
Jessica Wagantall [Fri, 13 Sep 2019 19:47:11 +0000 (12:47 -0700)]
Update WhiteSource scan timer
Run WhiteSource jobs weekly on Sunday.
Issue: RELENG-2386
Change-Id: I26d5b4a89e262ff2b046bdae8298e61597b71bc1
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Eric Ball [Fri, 13 Sep 2019 01:26:06 +0000 (18:26 -0700)]
Add sonar-prescan jobs
These jobs allow the caller to provide a builder that runs prior to
the Sonar scan.
Issue: RELENG-2314
Change-Id: I97ec30b27c83a7ce1ac7dc8a1736672dc8d15eee
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Andrew Grimberg [Fri, 13 Sep 2019 17:25:12 +0000 (17:25 +0000)]
Merge "Update release docs."
Aric Gardner [Mon, 9 Sep 2019 20:32:05 +0000 (16:32 -0400)]
Update release docs.
Add note about adding env variables to global-vars-releng.sh
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I3596434d7601829fdd92f27ce0646006452a4c89
Aric Gardner [Fri, 13 Sep 2019 15:31:00 +0000 (11:31 -0400)]
Pin more-itertools for python2
Pin more-itertools for python2 tox-install
tox-pyenv requires more-itertools.
Force to a compatible version if pip installed
via python2
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Ia5ade84925cdc6eac2f3407cb732eaf4946769e4
Anil Belur [Fri, 13 Sep 2019 07:28:33 +0000 (12:58 +0530)]
Pin python-cinderclient to 4.3.0.
A new version of python-cinderclient 5.0.0 is released which breaks
the openstack jobs.
ImportError: No module named v1.contrib
Issue: IT-17567
Change-Id: I2736c464ff82a3b08433031a5e924e1345c2c317
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Tim Johnson [Thu, 12 Sep 2019 22:58:33 +0000 (15:58 -0700)]
Cleanup tox warnings
Issue: RELENG-2365
Change-Id: I5a739a4446bfeabf73d020e6f72a8440b9d1c913
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
Jessica Wagantall [Wed, 11 Sep 2019 23:28:17 +0000 (16:28 -0700)]
Update release container info
- Update schema to require pull and push container
registries
- This will fix issues where for example ONAP pulls from
snapshots exclusively where other projects like O-RAN
use both snapshots or staging.
- Allows for future push to DockerHub
- Benefits projects like O-RAN and Acumos where they already
provide this information for other docker jobs.
- Pull and push registries need to be defined as Jenkins
global variables.
- Can be overwritten in releases files.
- Update some echos to allow users to find information faster.
Change-Id: Ibd0088c6419bacca16ded4ae86d70304c5bad45c
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Jessica Wagantall [Thu, 12 Sep 2019 18:22:42 +0000 (11:22 -0700)]
Fix WS CLI Agent calls
Remove quotes from optional variable
WSS_UNIFIED_AGENT_OPTIONS which, if empty, will
cause WS Unified Agent CLI issues.
Upgrade WS Unified Agent CLI to latest version 19.8.1
Change-Id: I3d21bad21bceee0a0c626b145c1ca74e0cd21ddc
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Jessica Wagantall [Wed, 11 Sep 2019 20:17:37 +0000 (13:17 -0700)]
Fix import key release jobs
Import GPG signing key before verifying Gerrit
tag details.
Change-Id: I132fb8dbba51de995b0e42765344bf218340415c
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Aric Gardner [Wed, 11 Sep 2019 15:22:00 +0000 (15:22 +0000)]
Merge "Add python templates to run tox on merge"
Lott, Christopher (cl778h) [Tue, 20 Aug 2019 15:46:31 +0000 (11:46 -0400)]
Add python templates to run tox on merge
Makes the Python job groups and templates parallel to
the maven job groups and templates. Although no build
is required by most Python projects, re-testing after
merge is a chance to reveal failures in the new tree.
Change-Id: I1fb793e08deb435a838d76d634dbbad730a3d7fa
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Aric Gardner [Wed, 11 Sep 2019 11:52:55 +0000 (11:52 +0000)]
Merge "Use configured python version to install tox-pyenv"
Lott, Christopher (cl778h) [Mon, 9 Sep 2019 15:08:37 +0000 (11:08 -0400)]
Use configured python version to install tox-pyenv
Revise the tox-install.sh script to use PYTHON env var to invoke pip,
instead of hardcoded 'python'.
Add python-version parameter to Read-The-Docs doc, macro and templates.
Change-Id: Ib0fe98dca62ce67f0ad002036b54f02faf27e888
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Issue: RELENG-2361
Jessica Wagantall [Mon, 9 Sep 2019 20:54:53 +0000 (20:54 +0000)]
Merge "Fix self release script indent"
Lott, Christopher (cl778h) [Sun, 8 Sep 2019 11:04:37 +0000 (07:04 -0400)]
Remove passive voice in doc to silence warnings
The write-good feature as invoked by tox coala WriteGoodLintBear
complains about usage of passive voice e.g., 'is applied'.
These changes silence all but two warnings about use of the word
'validate' which is a well-known Maven goal.
This makes no functional change to any templates.
Change-Id: I5a68a475997cbeeae5ce41a2e487d67cc3c2644f
Issue: RELENG-2365
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Anil Belur [Sun, 8 Sep 2019 03:56:42 +0000 (09:26 +0530)]
Fix self release script indent
Use 4 spaces indent. This makes the code easily readable
and consistant with rest of the code in global-jjb.
Change-Id: I62edca97517fe456d0fe688e2826b0d0fce8432a
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Lott, Christopher (cl778h) [Fri, 6 Sep 2019 00:38:46 +0000 (20:38 -0400)]
Repair tox-sonar behavior by restoring mvn goal
Use existing builder lf-infra-maven-sonar, drop the locally
defined builder lf-infra-tox-sonar, to get desired behavior
of generating code analysis then publishing to Sonar. This
broke when hardcoded goals were removed from maven-sonar.sh.
Change-Id: I862c4857d60361d77b390425d50f3c7c77e3919d
Issue: RELENG-2359
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Jessica Wagantall [Wed, 4 Sep 2019 00:25:52 +0000 (17:25 -0700)]
Update release vars functions
Release schema verification needs to happen
first before we attempt to assign values to
the variables.
Organize variable setup into functions.
Maven release files expects different variables
than container release files.
Rename "version" variable in container release
files to "container_release_tag" which is a
better user friendly name given the fact that
container versions are rather called tags.
Internally, we still process it as "version"
to allow reuse of the tag function.
Issue: RELENG-2353
Change-Id: Ie0456a1fa87ea62855cb5d1140f6ae9f32b3e566
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Aric Gardner [Wed, 4 Sep 2019 16:21:48 +0000 (16:21 +0000)]
Merge "Clarify limit of one release yaml file"
Lott, Christopher (cl778h) [Tue, 3 Sep 2019 19:06:03 +0000 (15:06 -0400)]
Clarify limit of one release yaml file
Specify that Centos + Sigul + Docker are required.
Rewrap all paragraphs to eliminate long lines.
Change-Id: I57f46405cb962aed31c578354147b2b414da811f
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Anil Belur [Wed, 4 Sep 2019 11:48:26 +0000 (17:18 +0530)]
Remove the unused MAVEN_CENTRAL_URL variable
Remove the unused MAVEN_CENTRAL_URL variable. The self-release job is
designed to work with any Nexus repository info published in
`staging-repo.txt.gz`, which makes the `MAVEN_CENTRAL_URL` redundant,
hence remove the unused variable.
Issue: RELENG-2129
Change-Id: Iabb772d908f8001a6626d728a9140ff14e3088f2
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Lott, Christopher (cl778h) [Tue, 3 Sep 2019 17:53:27 +0000 (13:53 -0400)]
Minor rewording and reformatting of release doc
Change-Id: I4d91d42bc2bd5cee4ae1e8dcebaba659bc168b6a
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Anil Belur [Sat, 31 Aug 2019 03:04:35 +0000 (03:04 +0000)]
Merge "INFO validate job improvements"
Houa Yang [Fri, 30 Aug 2019 14:23:08 +0000 (09:23 -0500)]
Add Sonar plug-in configurations to docs
Add documentation for configuring the Jenkins Sonarqube Scanner plug-in.
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I21b21efc6cefa3f4f643a0235d670c6e1ec4ac08
Aric Gardner [Thu, 29 Aug 2019 18:54:03 +0000 (14:54 -0400)]
INFO validate job improvements
removed +x trace as job is too verbose
verify now checks that repositories matches $PROJECT
Catches projects that replace / with - in their INFO file
Also ensures that repositories only has one entry.
We are not supporting multiple projects with a single INFO.yaml
file.
ISSUE: RELENG-2344
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Id847b7c10ef0515fd6f6a4b7d6809ebd37a9b51d
Anil Belur [Thu, 29 Aug 2019 14:49:46 +0000 (14:49 +0000)]
Merge "Remove detox in favor of tox --parallel"
Anil Belur [Wed, 28 Aug 2019 18:52:30 +0000 (00:22 +0530)]
Update lftools version to v0.26.2
Change-Id: I069dda979d203b1960821e4d112a5591843fd441
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Anil Belur [Wed, 28 Aug 2019 18:02:03 +0000 (23:32 +0530)]
Add DRY_RUN param to the release job
Add DRY_RUN build param to do a test run the job with
publishing artifacts.
Change-Id: I1b015db4be8ed3e8c39bec2cefe022819e31e59f
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Anil Belur [Wed, 28 Aug 2019 16:31:17 +0000 (16:31 +0000)]
Merge "Verify both repos before attempting release"
Aric Gardner [Wed, 28 Aug 2019 15:39:00 +0000 (11:39 -0400)]
Verify both repos before attempting release
We have run into a case where the repo on ODL
was good, and the repo on nexus was failed.
Cover this case by running the loop twice.
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I995a34918481db9e41d5486c06995f57100c1199