Anil Belur [Sun, 9 Aug 2020 04:01:32 +0000 (04:01 +0000)]
Merge "Migrate write-good hook to pre-commit"
Anil Belur [Sun, 9 Aug 2020 04:00:53 +0000 (04:00 +0000)]
Merge "docs: Move release notes out of "Guides" section"
Thanh Ha [Sat, 8 Aug 2020 13:07:57 +0000 (09:07 -0400)]
Migrate write-good hook to pre-commit
This is a final patch to remove our dependency on the Coala platform
and use only pre-commit moving forward.
This patch also resolves write-good lint issues that were not caught
by the Coala version of write-good due to file ignore rules that
excluded several files from being scanned. With pre-commit we can
create separate rules to exclude more specific scans while no excluding
the file from scans entirely.
Issue: RELENG-2642
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Ic9a48b2964ff5c689a01f3fd611e69ef9525e055
Eric Ball [Fri, 7 Aug 2020 12:43:41 +0000 (05:43 -0700)]
docs: Move release notes out of "Guides" section
The text in the "Release Notes" section of the docs index was
outdated (we have not used the relnotes directory in quite some
time), and the actual Release Notes links were under the "Guides"
section.
There is also a correction to put paths into monospace rather than
italics.
Change-Id: I3096bdce7ec666b762ab4be5811ae4c7ec2ffb55
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Anil Belur [Wed, 5 Aug 2020 09:21:09 +0000 (19:21 +1000)]
Capture instance metadata with job logs
Capture instance metadata for the VM's. This data is useful while
working with the cloud provider and debuging VM issues on the
infrastructure.
Issue: RELENG-3142
Change-Id: If54a6a02bf25f039c37fee2831f641ba662254a8
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Eric Ball [Tue, 4 Aug 2020 18:15:52 +0000 (11:15 -0700)]
Update sigul-install.sh to check for sigul
If sigul is already installed, we can skip the installation. This is
important due to kojipkgs being unreliable. We now have the sigul
binary baked into the base image creation, so newer images should
already have sigul on board. If they do and we still try to do this
manual installation, we could still run into connection issues with
kojipkgs.
Issue: RELENG-3115
Change-Id: I3580b92b8e4c1c72493069bf977f96e36aba6a85
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Eric Ball [Tue, 4 Aug 2020 19:25:53 +0000 (12:25 -0700)]
Fix shellcheck errors in scripts
Change-Id: I8ffc8ac98bc128f54aa515153615c8aa10560316
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Thanh Ha [Thu, 23 Jul 2020 19:27:17 +0000 (15:27 -0400)]
Explicity set Gerrit comment mode to PLAIN
The default setting for this config is BASE64 where it used to be
PLAIN. Explicity set what we expect the mode to be so that the
job does not fail to parse an unexpected base64 string.
Change-Id: Icf8ae74089f6f60cc03f41d4526d038c78e65606
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Anil Belur [Tue, 21 Jul 2020 23:22:25 +0000 (23:22 +0000)]
Merge "Fix port clean up race condition"
Jessica Wagantall [Mon, 20 Jul 2020 21:14:02 +0000 (21:14 +0000)]
Merge "lftools_activate should be deprecated"
Aric Gardner [Mon, 20 Jul 2020 17:11:41 +0000 (13:11 -0400)]
lftools_activate should be deprecated
We have lf-env.sh and lf-activate-venv
on all builders.
It is a more error resitant way to make sure a pip
package is installed
In tests lf-env passes in places that lftools_activate
fails.
Issue-ID: IT-20217
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Ib5610d6110e762f4510e62794215d17a086a1d20
Jessica Wagantall [Mon, 20 Jul 2020 17:12:06 +0000 (17:12 +0000)]
Merge "Clarifying read the docs documentation"
Anil Belur [Thu, 16 Jul 2020 23:31:33 +0000 (09:31 +1000)]
Fix jjb-deploy job to accept JJB_VERSION parameter
Fix the jjb-deploy job to accept a JJB_VERSION parameter in the
build parameters field so that projects can select which version
of JJB they want for the job.
Issue: RELENG-3073
Change-Id: I33653d50e11c79a00a5858c395bc05f20a03d283
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Anil Belur [Tue, 14 Jul 2020 04:10:43 +0000 (14:10 +1000)]
Fix port clean up race condition
Fix the race condition by checking the created_at timestamp and
clean up ports that were created atleast 30 minutes before.
Jira: RELENG-3062
Change-Id: I151d4d541e6858214fc49d3479cf932feb8b9acf
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Aric Gardner [Tue, 7 Jul 2020 14:58:06 +0000 (10:58 -0400)]
Clarifying read the docs documentation
This patchset hopes to elicit comments
from community members to ensure that this documentation
is understandable.
Please comment on any sections that are not clear.
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I359ac15094e90594eadc44a80885088e540cd766
Andrew Grimberg [Wed, 15 Jul 2020 16:58:02 +0000 (16:58 +0000)]
Merge "Add lf-pipelines-verify job for global pipelines"
Eric Ball [Fri, 10 Jul 2020 22:49:10 +0000 (15:49 -0700)]
Add lf-pipelines-verify job for global pipelines
This job will test the releng/pipelines repo. This also introduces a
framework for adding additonal pipeline jobs to global-jjb.
Issue: RELENG-2792
Change-Id: I5440b6d1a8dd4b56ebec2d810331cd52f8bbe4c4
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Anil Belur [Tue, 14 Jul 2020 00:58:13 +0000 (00:58 +0000)]
Merge "Use double bracket conditional compound command"
Aric Gardner [Thu, 9 Jul 2020 20:09:36 +0000 (16:09 -0400)]
Use double bracket conditional compound command
Odd failures in the read the docs jobs
casued by not quoting "$result"
Im going to quote it and switch to double bracket.
Fixed the issue in my testing.
Errors seen were:
jq: error (at <stdin>:1): Cannot index string with string "success"
./script: line 11: [: ==: unary operator expected
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Ic7554ddfa384b817ad3aac311972e6cfc52e0b49
Jessica Wagantall [Thu, 9 Jul 2020 16:25:35 +0000 (09:25 -0700)]
Update semver regex for release-job.sh
Update semver regex to the one recommended
by https://semver.org/
This regex was taken from
https://github.com/fsaintjacques/semver-tool
which follows the semver guidances and works in bash.
Issue: IT-20145
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I9d1fe49693279f71243df0280eebed266e55ff4e
Aric Gardner [Fri, 26 Jun 2020 18:51:07 +0000 (14:51 -0400)]
Fix ONAP exception.
Missed this one on my last go.
masterproject must equal rtdproject
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I1a184885dfee04f6abde6f29482d292bc7d85c58
Aric Gardner [Fri, 26 Jun 2020 17:20:23 +0000 (13:20 -0400)]
Need an exception for onap
We imported their docs for the migration
to not lose old versions.
this imported project does not follow the naming convention
and so I have added a hardcoded exception
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I8ac74bec953cf39b28cf8776c50fcb0e78869073
Aric Gardner [Thu, 18 Jun 2020 19:17:03 +0000 (15:17 -0400)]
Fix logic for discovering new branches
When a new branch exists on gerrit, but has never been seen by RTD
we need to trigger a job so that the branch is discovered
we can then mark it active in the following step.
API changed under out feet, where a 404 was returned before
we now get a 200 with the string null.
Change code to explicitly match the returned string of
"null"
Issue-Id: RELENG-3027
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I769b66384f5b53d87527faeb0f467350a847b65c
Thanh Ha [Thu, 11 Jun 2020 20:47:16 +0000 (16:47 -0400)]
Pin importlib-resources version to <2.0.0
The current virtualenv 20.0.21 is incompatible with importlib-resources
2.0.0. This patch pins the version of importlib-resources temporarily
until a virtualenv version that supports it is released. This resolves
the ignored error in tox-verify job builds:
ERROR: virtualenv 20.0.21 has requirement
importlib-resources<2,>=1.0; python_version < "3.7", but you'll
have importlib-resources 2.0.0 which is incompatible.
Issue: RELENG-2993
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I04cf74f4209104475c1548653f157821814970df
Andrew Grimberg [Wed, 10 Jun 2020 16:01:13 +0000 (09:01 -0700)]
Correctly evaluate UPDATE_CLOUD_IMAGE
While trying to get a clean packer build on an AWS based Jenkins I kept
running into the packer job failing. Digging into the code and turning
on debugging allowed me to spot that
if [[ ${UPDATE_CLOUD_IMAGE} ]]; then
Was translating to
[[ -n false ]]
During runs where $UPDATE_CLOUD_IMAGE wasn't set. This seemed strange to
me and then looking a bit further I realized that the boolean variable
in Jenkins is converting to a string ENV var of either 'true' or 'false'
depending the set. This means for the if statement to properly execute
it must evaluate as a string comparison as the variable is not actually
a boolean.
Change-Id: Ie98b51eb22412705c8c264b61be3c76b4e5ff985
Issue: RELENG-2985
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
Thanh Ha [Sat, 6 Jun 2020 13:32:36 +0000 (09:32 -0400)]
Add .editorconfig file
EditorConfig is a universal editor configuration file supported by many
IDEs and Editors (although some might require installing a plugin) to
share consistent editor configuration with developers using different
editors.
This also updates the README.md file to remove the single item bullet
list as that was causing prettier to reformat the bullet item to insert
extra spacing between the '-' and first word in the list.
Ref: https://editorconfig.org/
Change-Id: I22605008837c567d9f6a62abfc85a195acc708c6
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Andrew Grimberg [Sat, 6 Jun 2020 14:24:19 +0000 (14:24 +0000)]
Merge "Print git-diff when pre-commit fails"
Andrew Grimberg [Sat, 6 Jun 2020 14:24:11 +0000 (14:24 +0000)]
Merge "Update pre-commit hooks 2020-06"
Thanh Ha [Sat, 6 Jun 2020 13:43:00 +0000 (09:43 -0400)]
Print git-diff when pre-commit fails
When pre-commit fails and automatically fixes issues for us in Jenkins
we are not able to see what pre-commit automatically fixed. Adding
--show-diff-on-failure makes pre-commit perform a git-diff immediately
after failing to ensure more useful output is printed to inform the
contributor what changes are needed to pass the check.
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I0c9fc6afa8be214d03e342b2913205e946aa901e
Thanh Ha [Sat, 6 Jun 2020 13:37:13 +0000 (09:37 -0400)]
Update pre-commit hooks 2020-06
It's been awhile since we last bumped these so let's catch up with
the world.
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I3567df53ccbf7c78b8de8145bee4008fe785590a
Thanh Ha [Fri, 5 Jun 2020 21:57:37 +0000 (17:57 -0400)]
Fix gerrit-push-patch Job key not in trailers
Key-value pairs in Git should be appended to the Git trailers section
(AKA footer). This patch fixes the commit message produced by
gerrit-push-patch which currently creates a commit message where the
Job key appears in the commit message body rather than the trailer.
For example:
An example commit message
Job: builder-job/123
Change-Id:
1234567
Signed-off-by: Jenkins <jenkins@example.org>
This fixes it to:
An example commit with proper trailer
Job: builder-job/123
Change-Id: 123457
Signed-off-by: Jenkins <jenkins@example.org>
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I66c9b0813b36065c20adcf0e886eaa29354ca603
Eric Ball [Thu, 4 Jun 2020 20:37:28 +0000 (13:37 -0700)]
Improve upstream gjjb jobs, add Github version
ODL already has the upstream LF Gerrit in its Gerrit triggers, and it
is named "lf-releng". This should therefore be used as the standard
name for other projects.
Added a Github version of the upstream global-jjb verify, which just
needs SCM changes for pulling in its ci-man repo.
Documentation for preparing a Jenkins server to test upstream gjjb
changes is available here: https://confluence.linuxfoundation.org/display/RELEN/Add+upstream+global-jjb+testing+to+Jenkins
Issue: RELENG-1043
Change-Id: I0a2f757c0e58355bc59eaa1e56eaa396f60b0f32
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Aric Gardner [Thu, 28 May 2020 18:02:22 +0000 (14:02 -0400)]
Modify info-schema
make email and timezone optional.
Email if entered must be vaild, but is not required
Timezone if entered must be match regex, but is not required
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I1266c43edabed7fdb3882b8fc335bd8b3942699c
Eric Ball [Wed, 20 May 2020 00:38:18 +0000 (00:38 +0000)]
Merge "Add templates for CXX projects using autotools"
Lott, Christopher (cl778h) [Tue, 19 May 2020 12:32:15 +0000 (08:32 -0400)]
Echo error response from RTD without formatting
The rtd-trigger-build.sh starts a build at ReadTheDocs and parses
the response, a small JSON like this:
{"build_triggered":false,"project":"unicorn","versions":[]}
This change drops the invocation of jq that attempts to pretty-print
the JSON to the log when the build is not triggered, because that
call was failing for inexplicable reasons.
Change-Id: Ic4369b7e0b9172d4b4c9c3544224b6b5325813a5
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Lott, Christopher (cl778h) [Wed, 13 May 2020 14:26:41 +0000 (10:26 -0400)]
Add templates for CXX projects using autotools
New jobs for the "configure && make && make install" pattern used
by many open-source projects. Supports generation of the configure
script for projects that do not store generated files in version
control. Includes gerrit and github versions of templates
autotools-packagecloud-stage, autotools-sonarqube and
autotools-verify.
Change-Id: I4e35db15c73f99b38486f31f137b224e8d6acde0
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Lott, Christopher (cl778h) [Wed, 29 Apr 2020 17:58:30 +0000 (13:58 -0400)]
Report CLI version in scripts that invoke docker
Extend scripts that invoke the docker CLI to report the version
by invoking `docker --version`. Also add echo command to report
end of script where it was missing. Includes docker-build.sh,
docker-login.sh, docker-push.sh and release-job.sh. No functional
change to any script behavior, just a line of extra output.
Change-Id: Iac65366c7810d6f96fecb84267aa4c1dd9027dc1
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Anil Belur [Sun, 26 Apr 2020 00:55:31 +0000 (10:55 +1000)]
Update regexp in lf-infra-ship-logs macro
Missed updating the tested regex in
9e8bc48869a.
Test:
https://jenkins.opendaylight.org/releng/view/builder/job/builder-packer-verify/277
Issue: RELENG-2856
Change-Id: I28bf3410317a4e0962ceb4fd52b2857c41221a54
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Anil Belur [Fri, 24 Apr 2020 04:16:21 +0000 (14:16 +1000)]
Update regexp in lf-infra-ship-logs macro
Update regexp in lf-infra-ship-logs macro. This fixes a
regression from update to deploy-logs.sh script in
v0.53.0 causing missing logs links on Jenkins jobs.
https://gerrit.linuxfoundation.org/infra/c/releng/global-jjb/+/63501/4/shell/logs-deploy.sh
Test:
https://jenkins.opendaylight.org/releng/view/builder/job/builder-packer-verify/274
Issue: RELENG-2856
Change-Id: I2fb0bcf9660189172e2c7742a9b43d66e85493ce
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Eric Ball [Tue, 21 Apr 2020 20:37:13 +0000 (20:37 +0000)]
Merge "Drop removed CLM jobs from python job groups"
Andrew Grimberg [Tue, 21 Apr 2020 20:12:31 +0000 (20:12 +0000)]
Merge "Fix update cloud image list job to handle newlines"
Lott, Christopher (cl778h) [Tue, 21 Apr 2020 19:55:20 +0000 (15:55 -0400)]
Drop removed CLM jobs from python job groups
Drop jobs gerrit-python-xc-clm and github-python-xc-clm from
the two python job groups. Those templates were replaced by
gerrit-tox-nexus-iq-clm and github-tox-nexus-iq-clm. Not adding
those templates to the job groups because successful use requires
additional project configuration to report the package requirements.
Totally baffling that the tox test didn't catch this botch in the
change set that removed those templates.
Change-Id: Ibea19818ebf45cfc541cf910fe90cc3b1b3579ca
Issue-ID: RELENG-2821
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Aric Gardner [Tue, 21 Apr 2020 19:35:33 +0000 (19:35 +0000)]
Merge "Add Tox Nexus IQ (CLM) templates for Python"
Lott, Christopher (cl778h) [Mon, 23 Mar 2020 21:14:30 +0000 (17:14 -0400)]
Add Tox Nexus IQ (CLM) templates for Python
Add new templates gerrit-tox-nexus-iq-clm/github-tox-nexus-iq-clm
to analyze Python package requirements. Add script
nexus-iq-python-cli.sh with correct invocation and arguments for
the latest version of the Nexus IQ scanner.
Drop broken templates g*it-python-xc-clm, unused builder
lf-infra-clm-python and shell script nexus-iq-cli.sh.
Change-Id: I5c34ad02e3e37a0e04dfa7e64b1013ebc26119fc
Issue-ID: RELENG-2821
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Anil Belur [Mon, 20 Apr 2020 03:46:06 +0000 (13:46 +1000)]
Fix update cloud image list job to handle newlines
Fix update cloud image list job to handle newlines correctly and
update an existing gerrit change request.
A gerrit change submitted through git-review checks for an
existing change-id on Gerrit and either updates the patchset or
creates a new one. For this to work correctly the change-id
should go into the commit footer and not in the
GERRIT_COMMIT_MESSAGE. The embedded newlines are not processed
correctly in the script and fails to separate the footer and the
GERRIT_COMMIT_MESSAGE, instead it creates a new patches everytime
rather than updating an existing one.
The change is tested and now existing CR's are updated rather than
pushing a new CR everytime.
https://git.opendaylight.org/gerrit/c/releng/builder/+/89136
Issue: RELENG-2687
Change-Id: I8397d2d8a9402db07c99173c9f999152bc3c61ee
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Anil Belur [Tue, 21 Apr 2020 01:44:31 +0000 (01:44 +0000)]
Merge "Add packer-verify-build jobs"
Eric Ball [Thu, 9 Apr 2020 01:53:58 +0000 (18:53 -0700)]
Add packer-verify-build jobs
This job is made to be manually triggered in order to build the
packer image, so that the full build process can be validated before
merging. If this is done on the final patch that is merged, the
merge job will not run another build (thereby avoiding building two
identical images).
The verify-build job piggybacks on the merge template, since it is
essentially the same job. However, it only triggers when given the
keyword "verify-build" or "packer-build". Due to the similarities,
some documentation improvements have also been shifted over to the
Packer Merge docs, as well as some small changes to Packer Verify.
Issue: RELENG-2671
Change-Id: Id4fb33a88447bbb2f09ac0967ac303a8f6d62326
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Andrew Grimberg [Fri, 17 Apr 2020 17:24:09 +0000 (17:24 +0000)]
Merge "Fix lftools install in rtd verify job"
Anil Belur [Thu, 16 Apr 2020 04:06:56 +0000 (14:06 +1000)]
Fix lftools install in rtd verify job
Issue: RELENG-2846
Change-Id: I5c3cd6bf8f4af887ba00aa2e1a4d42775caac80a
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Jessica Wagantall [Wed, 15 Apr 2020 23:10:24 +0000 (16:10 -0700)]
Use yq instead of niet
Standardize yaml parsing tools used across global-jjb.
Switch from using niet package to yq for parsing yaml files
in release-job.sh.
Issue: RELENG-2844
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: Ia57884a7a8efffa85417becd8c914ab8d23721a9
Jessica Wagantall [Wed, 15 Apr 2020 22:51:08 +0000 (15:51 -0700)]
Bool is not a valid schema type
Using boolean rather than bool.
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: Ied68c7f1a2fd4474e9fee65b2820e6c25f662b5f
Jessica Wagantall [Tue, 7 Apr 2020 19:34:14 +0000 (12:34 -0700)]
Optionally skip tag during release process
Make tag process optional. True by default.
Example scenario:
Some ONAP projects make both maven and docker releases
for a single repo. Docker releases depend on maven releases.
This means that the team needs to make a maven release first,
then make a docker release and lastly, tag the repo.
This change will allow them to hold the tagging part during
maven releases to allow them to tag after the docker release is made.
Issue: RELENG-2835
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: Ibd206269b80e7fb788682a81e038aa74d9eaf97c
Houa Yang [Thu, 9 Apr 2020 21:24:11 +0000 (21:24 +0000)]
Merge "Add "logs" prefix to s3_path"
Houa Yang [Thu, 9 Apr 2020 19:25:05 +0000 (14:25 -0500)]
Add "logs" prefix to s3_path
We are not allowed to have an index.html file at the root level of the
S3 bucket. Adding the additional prefix allows us to mirror the directory
structure of Nexus where you are able to see both production and sandbox
logs when browsing to https://logs.<project>.org/
Issue-Id: RELENG-2379, RELENG-2380
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I7b6bd340170f9390c670f2b6ba507f2de1cfbc0e
Jessica Wagantall [Wed, 8 Apr 2020 17:23:23 +0000 (10:23 -0700)]
Allow rtdv3 jobs to run in parallel
These jobs are submitted against different projects.
Allow them to run in parallel to save time.
Change-Id: I199b07c4b580fdf7be82e5503b8dc8a1e3915e89
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Lott, Christopher (cl778h) [Tue, 7 Apr 2020 13:11:26 +0000 (09:11 -0400)]
Add doc for jjb-verify build-node label check
Document the jjb-verify feature that checks build-node labels named
in JJB YAML files against nodes in cloud config files.
Change the feature's JJB configuration variables to share a prefix.
Improve the shell script to be robust to errors such as a suffix of
".cfg" or an external label of "" (just two double quotes).
Change-Id: I8beb5669531fe2b7f59a9bcf3706b27ede3965b9
Issue-ID: RELENG-2828
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Anil Belur [Tue, 7 Apr 2020 01:39:32 +0000 (01:39 +0000)]
Merge "Add CMake templates to build/publish DEB/RPM pkgs"
Lott, Christopher (cl778h) [Fri, 3 Apr 2020 14:53:43 +0000 (10:53 -0400)]
Add CMake templates to build/publish DEB/RPM pkgs
New templates gerrit-cmake-packagecloud-stage and
github-cmake-packagecloud-stage for building DEB/RPM package
files and publishing them to a PackageCloud.io repository on
posted comment. Adds builder macros lf-packagecloud-file-provider
and lf-packagecloud-push. Adds script packagecloud-push.sh to
call Ruby gem package_cloud.
The new templates are lifted from the ORAN project.
Change-Id: Ia39f5b43c2ef22485166a5f54fcf31d53eab84f8
Issue-ID: RELENG-2832
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Houa Yang [Fri, 3 Apr 2020 17:04:16 +0000 (12:04 -0500)]
Fix aws credential file provider conditional
Add "on-evaluation-failure: dont-run" to conditional, so that if the
global env variable is not set, the job will not fail.
Issue-Id: RELENG-2749
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I3a7d992bfd93853ffe910713c00380852868f20e
Thanh Ha (zxiiro) [Fri, 3 Apr 2020 13:34:19 +0000 (13:34 +0000)]
Merge "Verify build nodes in YAML against config files"
Houa Yang [Wed, 1 Apr 2020 18:33:59 +0000 (18:33 +0000)]
Merge "Add AWS credentials config file"
Lott, Christopher (cl778h) [Sun, 29 Mar 2020 22:53:05 +0000 (18:53 -0400)]
Verify build nodes in YAML against config files
Extend lf-jjb-verify anchor with boolean configuration variable
check-build-node-labels that guards a conditional build step.
If true, run script jjb-verify-build-nodes.sh to check build-node
labels in YAML files within the jjb subdirectory against labels
defined by config files in the jenkins cloud configuration
directory. Disabled by default. Projects should enable and
configure the job; e.g., for external build node labels.
Issue-ID: RELENG-2828
Change-Id: Iaf315b4a68c3488b5ead204a57361a8bcbca91d7
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Jessica Wagantall [Tue, 31 Mar 2020 23:44:43 +0000 (23:44 +0000)]
Merge "Add PyPI stage templates"
Lott, Christopher (cl778h) [Thu, 26 Mar 2020 21:32:23 +0000 (17:32 -0400)]
Repair tox-run for case of no tox logs to archive
Revise tox-run.sh to guard against glob matching no tox log files.
In that case the pattern is passed to the cp command, which fails.
Detect the cp failure ('cp: cannot stat ..') and exit the loop.
I introduced this new and undesired behavior by adding -e in
change
76a0761, so the script stops when the cp command fails.
Change-Id: I1d60bb69e270c4aea413f4f14cf2d4f8f672853b
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Houa Yang [Fri, 21 Feb 2020 21:59:59 +0000 (15:59 -0600)]
Add AWS credentials config file
Add new jenkins-s3-log-ship config file provider utilizing a conditional
step. This is needed to be able to use new lftools deploy s3 api call to
upload logs to AWS S3.
Issue-Id: RELENG-2749
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I2171c4f5a8a18259feca265b628cb7288a18c54f
Lott, Christopher (cl778h) [Wed, 25 Mar 2020 22:15:17 +0000 (18:15 -0400)]
Add PyPI stage templates
Add templates gerrit-pypi-stage/github-pypi-stage to allow projects
flexibility to control when Jenkins publishes a redistributable
Python package, either on merge (with the merge template) or on
demand (with the stage template).
Change-Id: I49d8101b9da8c58d10518cb3ad522cd71b1019dc
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Houa Yang [Wed, 25 Mar 2020 16:05:47 +0000 (11:05 -0500)]
Fix update to logs-deploy.sh
Fix script to remove outputting $ARCHIVE_ARTIFACTS. Outputting required
that $ARCHIVE_ARTIFACTS be set for every job. Instead it will output
${pattern_opts:-}.
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I5f75a12745d1358b89dc7af18a35736514143072
Andrew Grimberg [Tue, 24 Mar 2020 20:42:06 +0000 (20:42 +0000)]
Merge "Run docker-merge job weekly by default"
Andrew Grimberg [Tue, 24 Mar 2020 20:33:25 +0000 (20:33 +0000)]
Merge "Update logs-deploy.sh"
Lott, Christopher (cl778h) [Tue, 24 Mar 2020 09:53:23 +0000 (05:53 -0400)]
Run docker-merge job weekly by default
Use @weekly cron default value in docker_merge_common anchor, no longer
empty. The merge job for docker images is like the stage job for java
artifacts, every docker image is a release candidate. Run the merge job
regularly to check dependencies like base images and to push updated
images to the Nexus3 staging registry.
Change-Id: Ic26f534953c77047515d2678fa654e21ac9d6336
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Houa Yang [Thu, 5 Mar 2020 00:53:41 +0000 (18:53 -0600)]
Update logs-deploy.sh
Update script with get_pattern_opts function. Function refactors code to
be reusable.
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: Ic5366f2177c401ce3bedda67b03fc5dee288cb8f
Lott, Christopher (cl778h) [Thu, 19 Mar 2020 20:32:38 +0000 (16:32 -0400)]
Stop on error in nexus-iq-cli.sh script
Revise script nexus-iq-cli.sh to stop on error or unbound variable.
This should fail the build if the scanner returns a non-zero code,
for example if credentials are missing or wrong.
Change-Id: I39f7ae9d6ba552ff3e3f3dd13df5bf998372e20f
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Aric Gardner [Thu, 12 Mar 2020 20:26:27 +0000 (16:26 -0400)]
Use lf-env for info-file-validate
Job was failing due to using python2
use lf-env to manage the environment.
ISSUE-ID: IT-19236
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I388ce7d8471e62efdae8c6131b4924cd89bfcc53
Aric Gardner [Thu, 12 Mar 2020 15:21:49 +0000 (15:21 +0000)]
Merge "Python 3.5 and python2 tox in Ubuntu 16.04"
Eric Ball [Thu, 12 Mar 2020 15:08:40 +0000 (15:08 +0000)]
Merge "Refactor lf-c-cpp-jobs for common anchor"
Andrew Grimberg [Wed, 11 Mar 2020 16:37:00 +0000 (09:37 -0700)]
Fix jq query testing if we got valid json
While looking over cost data from OpenDaylight I noticed that we started
getting data that had unknown resources with 0 cost associated with it.
After digging in I found out that the job-cost.sh had been updated to
verify that the data received back was valid json.
Testing this out locally everything worked just fine, so I tested the
work flow on an instance in the build environment and found that the
version of jq on the systems throws an error if you try to redirect the
output without having a query, which is exactly what was being done for
the test for if we received a valid json block.
A little further testing showed me that changing the call to include a
query of '.' would just parse everything (as I expected) and that it
doesn't fail the test.
For further information this appears to be a bug in jq v1.5 which is
installed on the Centos7 systems but jq v1.6 which I have locally works
just fine.
Change-Id: Id1ffc4ce71903f62d56e62e6569975710294c630
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
Lott, Christopher (cl778h) [Thu, 5 Mar 2020 19:22:47 +0000 (14:22 -0500)]
Refactor lf-c-cpp-jobs for common anchor
Refactor templates in lf-c-cpp-jobs.yaml with common anchor/alias
lf_cmake_common to reduce redundant configuration. This includes
gerrit-cmake-sonar, github-cmake-sonar, gerrit-cmake-sonarqube,
github-cmake-sonarqube, gerrit-cmake-stage, github-cmake-stage,
gerrit-cmake-verify and github-cmake-verify. No functional change.
Document maven settings parameters in g*t-cmake-stage templates.
Change-Id: I6477ad41da2b46adac2f76658dbf334db949b313
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Aric Gardner [Wed, 4 Mar 2020 20:46:18 +0000 (15:46 -0500)]
Python 3.5 and python2 tox in Ubuntu 16.04
Ubuntu 16.04 calls a python2 version of tox.
so we must upgrade it even if we are testing with python3
ubuntu 16.04 has python 3.5 we must pinn the zipp package
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I4d2513c15baf4f00313885b35f8f0a297874b2b3
Lott, Christopher (cl778h) [Thu, 5 Mar 2020 16:31:42 +0000 (11:31 -0500)]
Patch new Cmake/Tox + SonarQube templates
Revise templates gerrit-cmake-sonarqube and gerrit-tox-sonarqube
to move the triggering comment string into a parameter that can be
overridden in a job definition. Github comment remains hardcoded.
This makes these new templates consistent with existing templates.
No functional change, the default is still "run-sonar".
Rename gerrit-cmake-sonarqube tempplate configuration parameter
from sonar-prescan-script to pre-build to be consistent with
existing CMake stage and verify templates. This is a breaking
change for any early adopters of this new template.
Correct release note file name prefix from cmake to tox
Change-Id: Ie98a58d2bf8cbffee110571df5d759c101cc3fc7
Issue-ID: RELENG-2760
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Eric Ball [Wed, 4 Mar 2020 18:46:51 +0000 (18:46 +0000)]
Merge "Add templates Tox + SonarQube Jenkins plugin"
Eric Ball [Wed, 4 Mar 2020 17:33:20 +0000 (17:33 +0000)]
Merge "Add templates Cmake + SonarQube Jenkins plugin"
Lott, Christopher (cl778h) [Thu, 27 Feb 2020 19:45:39 +0000 (14:45 -0500)]
Add templates Cmake + SonarQube Jenkins plugin
New templates gerrit-cmake-sonarqube and github-cmake-sonarqube
use the SonarQube Jenkins plug-in to analyze CXX code and publish
the results. Modeled after the generic gerrit/github-sonar templates.
The new templates lift the limitations of the existing templates
gerrit/github-cmake-sonar which download and install a Sonar scanner,
and cannot report unit-test code coverage statistics.
Change-Id: I2457ab0a40e226b8b5b2bbe18d2b9f5cde3281e1
Issue-ID: RELENG-2760
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Lott, Christopher (cl778h) [Tue, 3 Mar 2020 18:20:42 +0000 (13:20 -0500)]
Add templates Tox + SonarQube Jenkins plugin
New templates gerrit-tox-sonarqube and github-tox-sonarqube
use the SonarQube Jenkins plug-in to analyze Python code and
publish the results. Modeled after the generic gerrit/github-sonar
templates. The new templates eliminate the need for mvn-settings in
the job configuration and the need for a pom.xml file in the repo.
Change-Id: I6d5bc5200937a0230c5d8eb0c522941d2affcee4
Issue-ID: RELENG-2767
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Lott, Christopher (cl778h) [Mon, 2 Mar 2020 19:30:13 +0000 (14:30 -0500)]
Revise self-serve release docs to match jobs
Adjust formatting of the trigger file path default value
for container, maven and PyPI release jobs. Drop the
parameter from release job docs to match the template,
and add a sentence with the regular expression.
Change-Id: Ibff31ac5c82fc1ed4f025fb4c16d9c907456f3c8
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Chris Lott [Tue, 3 Mar 2020 15:44:40 +0000 (15:44 +0000)]
Merge "Show details and detect errors in tox-run.sh"
Lott, Christopher (cl778h) [Fri, 28 Feb 2020 18:34:22 +0000 (13:34 -0500)]
Show details and detect errors in tox-run.sh
Revise tox-run.sh to guard against unbound variable TOX_ENVS;
stop on error or unbound variable (options -eu); and print
commands before executing (option -x). Add echo command at end.
Main effect is to show TOX_ENVS parameter value in the job log.
Change-Id: Iad78123f32f5c32792ba5ec8584921066ff9af70
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Lott, Christopher (cl778h) [Thu, 27 Feb 2020 14:22:56 +0000 (09:22 -0500)]
Adjust cmake-sonar template and script
Add missing config property "stream: master" to lf_cmake_sonar macro
in lf-c-cpp-jobs.yaml to match existing "branch: master" property.
This makes the template consistent with other Sonar templates.
Remove double-quote chars that were added recently around $make_opts
from the invocation of the build wrapper in cmake-sonar.sh. This
makes the invocation consistent with the invocation of cmake in the
same file.
Update documentation to reflect actual property names and defaults.
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Change-Id: Ia1eb0573213cb06720ede8d9e47e1387b5b0303c
Lott, Christopher (cl778h) [Wed, 26 Feb 2020 19:46:20 +0000 (14:46 -0500)]
Squelch progress messages in cmake-sonar.sh
Reduce volume of output from wget and unzip by adding the -q flag to
the invocations in cmake-sonar.sh There's little need to see wget
progress messages or archive contents in the build log.
Change-Id: Iaaaa4071e17b6ef42c85409e8df3653df1f42252
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Andrew Grimberg [Wed, 26 Feb 2020 17:18:39 +0000 (17:18 +0000)]
Merge "Add conditional block for s3"
DW Talton [Thu, 20 Feb 2020 17:24:55 +0000 (10:24 -0700)]
Pin python idna library to v2.8
The idna library updated and broke twine/requests, which breaks
our *-pypi-release-merge jobs. Pinning the versions fixes this.
Issue-ID: IT-19046
Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: I62806d07600b975992f49de0f9b0e5d6401c18cc
Aric Gardner [Wed, 19 Feb 2020 21:08:51 +0000 (16:08 -0500)]
Update Sphinx and bootstrap theme versions
lfdocs conf has been updated, this should fix the
read the docs builds
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I5ff350989edafe14cdd6b54d069dfb6aaf23326b
DW Talton [Mon, 17 Feb 2020 23:45:04 +0000 (16:45 -0700)]
Pin python zipp library to python3.5-supported version
Zipp just added a hard check for py36. This pins zipp to a version
that still works with py35.
Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: I44100186ce14abe068a9c919fec2277cc94f0113
Eric Ball [Wed, 12 Feb 2020 20:04:39 +0000 (20:04 +0000)]
Merge "Add support for Sonar pre-scan shell script"
Lott, Christopher (cl778h) [Mon, 10 Feb 2020 20:58:11 +0000 (15:58 -0500)]
Add support for Sonar pre-scan shell script
New templates gerrit-sonar-prescan-script, github-sonar-prescan-script
accept an arbitrary shell-script body that can install prerequisites,
build and test to generate a code-coverage report for the Sonar Scanner
to find and upload. This lifts limitations in existing templates
gerrit-sonar-prescan, github-sonar-prescan.
Change-Id: Ie61a5bed99a21fd62cb74f89d3e153ab0c3a4108
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
Issue-ID: RELENG-2712
Thanh Ha [Fri, 7 Feb 2020 20:07:52 +0000 (15:07 -0500)]
Remove ref-update trigger for rtd-merge template
This was originally added to help docs projects trigger a docs
update when a tag is deployed. Unfortunately ref-update appears to
trigger even for comments in Gerrit now. Not sure why but likely due
to Gerrit NoteDB. The Jenkins Gerrit Trigger configuration for
ref-update unfortunately does not provide any configuration for
specifying specific refs to trigger on.
Until there's an alternative it's better to disable this so that
Jenkins does not get flooded with queued jobs for Gerrit code
reviews that have active comments.
Issue: IT-18792
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I417e53d698e1454deb28b30d87e97147ce8d3038
Tim Johnson [Thu, 6 Feb 2020 17:25:09 +0000 (09:25 -0800)]
Bad Floating Point Format in Cost File
Update format for uptime, cost & stack_cost. Changed default value for
resource (Instance Type) to 'unknown'. Created script (format-csv) to
fix the formats of the existing cost files.
ISSUE: RELENG-2715
Change-Id: Ibd8464faa1ba3983bf1e8ef5cd484d2f24c26e31
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
Aric Gardner [Thu, 6 Feb 2020 18:22:07 +0000 (13:22 -0500)]
Add configurable doc-dir
defaults to "docs/_build/html" needed for relative path
modifications if you change the tox-dir
tox-run also needs to handle these relative paths.
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I6b18dc27e605dc28cb4420a5591a911e956df337
Houa Yang [Tue, 1 Oct 2019 20:06:03 +0000 (15:06 -0500)]
Add conditional block for s3
Add conditional block for log shipping via AWS s3.
This will need some sort of environment variable, so
the build will know to also ship logs to s3.
Issue-Id: RELENG-2379, RELENG-2380
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: Iff922d22e3ffafb068ce7053194f8b7d289111b8
Jessica Wagantall [Wed, 5 Feb 2020 21:30:24 +0000 (21:30 +0000)]
Merge "Do not clone submodules in rtdv3"
Aric Gardner [Mon, 3 Feb 2020 20:29:59 +0000 (15:29 -0500)]
Do not clone submodules in rtdv3
The verify job should not be cloning
submodules.
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Id4abf1d513a87303078e9cd950a765e1f1fa95dd