releng/lftools.git
3 years agoRefactor: Resolve W605 Invalid escape sequence 37/68937/2
Thanh Ha [Tue, 5 Oct 2021 16:27:09 +0000 (12:27 -0400)]
Refactor: Resolve W605 Invalid escape sequence

W605 is deprecated and will become a syntax error in Python 3.8
(https://www.flake8rules.com/rules/W605.html).

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Ic2a51f089ef9d6fc45a3db5e96a8f5196d59f02c

3 years agoMerge "Fix: Incorrect indent skips calling deploy_s3()" v0.35.10
Bengt Thuree [Thu, 2 Sep 2021 05:25:52 +0000 (05:25 +0000)]
Merge "Fix: Incorrect indent skips calling deploy_s3()"

3 years agoFix: Incorrect indent skips calling deploy_s3() 83/68683/1
Anil Belur [Thu, 2 Sep 2021 02:55:57 +0000 (12:55 +1000)]
Fix: Incorrect indent skips calling deploy_s3()

When Unix style glob patterns are passed through the CLI option
the deploy_s3() function skips the step to push logs to the S3 bucket.

Issue-Id: RELENG-3866
Change-Id: I58d6ef8dab41213fdbc431294ddd21854adc60dc
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoChore: Upgrade pre-commit hooks 79/68679/1
Andrew Grimberg [Wed, 1 Sep 2021 18:07:33 +0000 (11:07 -0700)]
Chore: Upgrade pre-commit hooks

* github.com/ambv/black: updating 21.7b0 -> 21.8b0

Change-Id: I4a2374c5e589c4adbfdc9e8e82cb7fc099acaa8c
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
3 years agoFix: typo in deploy.py 97/68597/1 v0.35.9
pmikus [Mon, 23 Aug 2021 10:37:58 +0000 (10:37 +0000)]
Fix: typo in deploy.py

Typo error in xml mimetype deploy to S3 storage. Thais patch remove the
extra sign to correctly map mimetype to content-type.

Signed-off-by: pmikus <peter.mikus@protonmail.ch>
Change-Id: Ia88ac75f483b31312ae72cc104021ab9efda6455

3 years agoMerge "Fix: Refactor image clean" v0.35.8
Andrew Grimberg [Mon, 16 Aug 2021 22:19:38 +0000 (22:19 +0000)]
Merge "Fix: Refactor image clean"

3 years agoFix: Refactor image clean 31/68531/3
Anil Belur [Fri, 13 Aug 2021 13:20:08 +0000 (23:20 +1000)]
Fix: Refactor image clean

Refactor image cleanup code to handle attribute types and
multi-clouds config correctly.

Error:
AttributeError: 'Connection' object has no attribute 'cloud_config'

Issue-ID: RELENG-3869
Change-Id: If6dd04507f6eb94786f6589299ce31f62a009b84
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
3 years agoMerge "Refactor: Fix flake8 F841"
Eric Ball [Sat, 14 Aug 2021 00:54:06 +0000 (00:54 +0000)]
Merge "Refactor: Fix flake8 F841"

3 years agoRefactor: Fix flake8 F841 28/68528/2
Thanh Ha [Fri, 13 Aug 2021 01:17:36 +0000 (21:17 -0400)]
Refactor: Fix flake8 F841

Resolves "F841 local variable ____ is assigned to but never used".

Change-Id: I4ab246dd96c96f2fdf33bae1d7d6b2d2c6fe6190
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
3 years agoFix: Move text/html mimetype check to after None 29/68529/2 v0.35.7
Eric Ball [Fri, 13 Aug 2021 01:40:17 +0000 (18:40 -0700)]
Fix: Move text/html mimetype check to after None

When checking mimetype during _upload_to_s3, the newly-added
text/html type was checking before it is checked for None, which is a
potential TypeError.

By moving it after the variable is checked for None, the feature is
safe without requiring any further modifications.

Change-Id: Ia945893adb8973608cb9527b4ea811a5610d6aba
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoFix: Add text/html mimetype to handle s3 logs 08/68508/1 v0.35.6
pmikus [Tue, 10 Aug 2021 11:31:16 +0000 (11:31 +0000)]
Fix: Add text/html mimetype to handle s3 logs

Currently html files in S3 bucket are not rendered properly
via Cloudfront as they are sent with text/plain content-type.
This patch is setting up content-tyepe for html for text/html.

Signed-off-by: pmikus <peter.mikus@protonmail.ch>
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
Change-Id: Id8a26f77446890320b212522236b68552d73c425

3 years agoMerge "Test: Improve unit test for nexus"
Andrew Grimberg [Tue, 3 Aug 2021 14:49:36 +0000 (14:49 +0000)]
Merge "Test: Improve unit test for nexus"

3 years agoTest: Improve unit test for nexus 02/68402/2
Eric Ball [Fri, 30 Jul 2021 22:45:05 +0000 (15:45 -0700)]
Test: Improve unit test for nexus

When waiting for a repo to complete releasing, a more realistic
"activity" return is to have the "release" status in place, but
without the "stopped" property (which is not added until the repo is
finished releasing). This improved unit test would have caught the
bug addressed by commit e083184d.

Change-Id: I0827cdaad07a8fb94d0f266dc8e53fd69c4aab2d
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoMerge "Refactor: Fix F401 module imported but unused"
Andrew Grimberg [Sat, 31 Jul 2021 14:19:10 +0000 (14:19 +0000)]
Merge "Refactor: Fix F401 module imported but unused"

3 years agoMerge "Chore: Remove botocore from requirements.txt"
Andrew Grimberg [Sat, 31 Jul 2021 14:18:04 +0000 (14:18 +0000)]
Merge "Chore: Remove botocore from requirements.txt"

3 years agoMerge "Refactor: Fix E713 form should be "not in""
Andrew Grimberg [Sat, 31 Jul 2021 14:17:48 +0000 (14:17 +0000)]
Merge "Refactor: Fix E713 form should be "not in""

3 years agoRefactor: Fix F401 module imported but unused 92/68392/1
Thanh Ha [Sat, 31 Jul 2021 12:01:18 +0000 (08:01 -0400)]
Refactor: Fix F401 module imported but unused

A module has been imported but is not used anywhere in the file.
The module should either be used or the import should be removed.

Ref: https://www.flake8rules.com/rules/F401.html
Change-Id: I3e73260628bdbeccab2a7dd78215aa83a2ee71ff
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
3 years agoChore: Remove botocore from requirements.txt 90/68390/1
Thanh Ha [Sat, 31 Jul 2021 11:44:23 +0000 (07:44 -0400)]
Chore: Remove botocore from requirements.txt

This is a transitive dependency of boto3, no need to explicitly
require it unless we are trying to pin the version.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I0ce2d72082707a969d8b4841957f6dd761c7c6a0

3 years agoRefactor: Fix E713 form should be "not in" 89/68389/1
Thanh Ha [Sat, 31 Jul 2021 03:53:40 +0000 (23:53 -0400)]
Refactor: Fix E713 form should be "not in"

Tests for membership should use the form x not in the_list
rather than not x in the_list. The former example is simply
more readable.

Ref: https://www.flake8rules.com/rules/E713.html
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I3ab1811c2a6b7e6135397e694b4ec4a1ab980c93

3 years agoRefactor: Fix Flake8 E711 87/68387/2
Thanh Ha [Fri, 30 Jul 2021 22:36:22 +0000 (18:36 -0400)]
Refactor: Fix Flake8 E711

Comparisons to the singleton objects, like True, False, and None,
should be done with identity, not equality. Use “is” or “is not”

Ref: https://www.flake8rules.com/rules/E711.html
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I33e80228341b93bdeb6cae1b587b8b19275c589d

3 years agoChore: Enable black for lftools 86/68386/1
Thanh Ha [Fri, 30 Jul 2021 22:31:07 +0000 (18:31 -0400)]
Chore: Enable black for lftools

Not sure why lftools was excluded but it effectively disabled black
in all of lftools defeating the purpose of having it in pre-commit
in the first place.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I60b3832cb180574e114b2dca78c0376a14c6d66e

3 years agoChore: Add isort to Python linters 81/68381/2
Thanh Ha [Fri, 30 Jul 2021 18:26:47 +0000 (14:26 -0400)]
Chore: Add isort to Python linters

ISort is an automatic import sorter for Python. More details at
the ISort GitHub page (https://github.com/PyCQA/isort).

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I2eaff523fab0966ea7196037669a1c603c68cd3e

3 years agoFix: Add check for "stopped" before reading text 52/68352/5 v0.35.5
Eric Ball [Wed, 28 Jul 2021 13:33:12 +0000 (06:33 -0700)]
Fix: Add check for "stopped" before reading text

nexus.release_staging_repo has an error condition if the staging repo
is not released when checked, as a non-existent "stopped" field will
return a NoneType, causing an error when .text is read. We avoid this
by checking for the existence of the "stopped" field before attempting
to read the text.

Issue: RELENG-3813
Change-Id: I0fae854a3ee52118806f70ea8b6faacc5fda430b
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoFix: Don't exit on error while collecting data for _sys-info.log 15/68315/2 v0.35.4
Dave Wallace [Thu, 22 Jul 2021 18:54:11 +0000 (14:54 -0400)]
Fix: Don't exit on error while collecting data for _sys-info.log

Change-Id: Ic3ed23b727b4663f01e52205641b23505518d3b9
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
3 years agoFix: Nexus release logic and naming, add unit test 06/68306/4 v0.35.3
Eric Ball [Wed, 21 Jul 2021 23:13:45 +0000 (16:13 -0700)]
Fix: Nexus release logic and naming, add unit test

nexus.release_staging_repos was failing with an error due to an
undeclared variable. Several other issues were identified and fixed
(see release note for details), and a unit test has been added to
help prevent similar issues in the future.

Issue: RELENG-3804
Change-Id: I8b7e4adf4561541acffa1dc81dbeba35d7783e51
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoChore: Unpin requirements 95/68295/2 v0.35.2
Eric Ball [Wed, 14 Jul 2021 21:21:39 +0000 (14:21 -0700)]
Chore: Unpin requirements

Having all requirements pinned was causing problems with dependency
resolution in pip. This worked fairly well in older versions, but
newer versions of pip are more strict in how they handle dependencies
of multiple packages. This was causing v0.31.1, the last version with
unpinned requirements, to be installed in cases where lftools was
installed alongside packages with conflicting requirements.

Issue: RELENG-3792
Change-Id: Ie8fb0dac45da5d3fba803ef66e216341516bc3ab
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoMerge "Refactor: Update setup.cfg due to deprecations"
Eric Ball [Fri, 9 Jul 2021 23:39:13 +0000 (23:39 +0000)]
Merge "Refactor: Update setup.cfg due to deprecations"

3 years agoRefactor: Update setup.cfg due to deprecations 75/68175/1
Andrew Grimberg [Tue, 29 Jun 2021 20:53:08 +0000 (13:53 -0700)]
Refactor: Update setup.cfg due to deprecations

While working on a few other updates it was noticed that tox was giving
deprecations warnings related to values in setup.cfg. This therefore
updates the keys with the new non-deprecated values.

Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
Change-Id: I04c526c5799a6a5da1ea6aae8a56ab4fc07d7465

3 years agoCI: Properly run gitlint in CI 62/68162/3
Andrew Grimberg [Mon, 28 Jun 2021 17:32:58 +0000 (10:32 -0700)]
CI: Properly run gitlint in CI

When running pre-commit in a CI system the COMMIT_EDITMSG does not
(normally) get created as that is an artifact of editing the commit
message. If the file doesn't exist then gitlint will skip which makes it
possible for pre-commit checks that should fail, to pass.

Since we want tox to run in a consistent manner both locally and in CI
we need to play around a little with how we are checking the commit
message.

Change-Id: I25c1815ea3195e47a67b529a06497ad3594b2c84
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
3 years agoCI: Configure gitlint for Conventional Commits 61/68161/3
Andrew Grimberg [Mon, 28 Jun 2021 17:30:05 +0000 (10:30 -0700)]
CI: Configure gitlint for Conventional Commits

Turn on Conventional Commit message subject validation. This will affect
CI specifically, but developers can avoid having CI fail by assuring
that they have installed pre-commit and that they have also run

pre-commit install --hook-type commit-msg

The above is needed because pre-commit does not install commit-msg hooks
by default

Change-Id: Ifa301f8f6b8fa5c5838934b8ffcfa47e746cd95d
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
3 years agoChore: Upgrade pre-commit hooks 60/68160/2
Andrew Grimberg [Mon, 28 Jun 2021 17:28:24 +0000 (10:28 -0700)]
Chore: Upgrade pre-commit hooks

This change also includes style updates enforced by black that have to
come in _with_ the upgrade.

Change-Id: I68eebed11c5aca19d530104ae1164c42371d4851
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
3 years agoRefactor: Migrate shade library to openstacksdk 29/67729/4
Chandra Dasari [Fri, 14 May 2021 13:54:55 +0000 (09:54 -0400)]
Refactor: Migrate shade library to openstacksdk

Replacing shade library with openstacksdk for openstack image commands.

Signed-off-by: Chandra Dasari <cdasari@contractor.linuxfoundation.org>
Change-Id: I23f901de37d502674edc617784ad3095dbc38288
Issue: RELENG-3392

3 years agoMerge "Modified printouts when releasing staging repo."
Andrew Grimberg [Mon, 12 Apr 2021 16:11:39 +0000 (16:11 +0000)]
Merge "Modified printouts when releasing staging repo."

3 years agoCheck dco_signoffs files for missing DCO hashes 31/66831/4
Eric Ball [Thu, 18 Feb 2021 23:48:42 +0000 (15:48 -0800)]
Check dco_signoffs files for missing DCO hashes

When checking for missing DCOs, any hashes found to be documented in
the dco_signoffs (or a user-specified alternative) directory will no
longer be included in the missing list.

Some formatting improvements have also been made to the relevant
files, particularly removing unnecessary #noqa and #nosec directives,
which were not affecting pylint scores.

Issue: RELENG-3396
Change-Id: I7cdac03fb406c6ce862b997c01fb0fab7cfecd76
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
3 years agoModified printouts when releasing staging repo. 96/66796/1
Bengt Thuree [Wed, 17 Feb 2021 05:30:22 +0000 (16:30 +1100)]
Modified printouts when releasing staging repo.

Modified printouts to be more human friendly.
Still waiting every 20 seconds.
Checking result every 40 seconds.

Issue-Id: RELENG-3398

Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
Change-Id: I7abc4b8fe612d14939ed0dd0c2d70f9a4ecf1774

3 years agoReplace print statements with log 91/66391/1
Anil Belur [Tue, 5 Jan 2021 04:37:55 +0000 (14:37 +1000)]
Replace print statements with log

Issue: RELENG-3244
Change-Id: I389658ee3e85f34a87d5799f58db07d1b1236f1d
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoMerge "Add user regexp for version tag validation"
Andrew Grimberg [Tue, 27 Oct 2020 18:46:55 +0000 (18:46 +0000)]
Merge "Add user regexp for version tag validation"

4 years agoCheck ownerid and visibility on image cleanup 91/65791/5 v0.35.1
Anil Belur [Sat, 17 Oct 2020 03:16:50 +0000 (13:16 +1000)]
Check ownerid and visibility on image cleanup

Recently update on the cloud provider caused the builder-cron-openstack
job to break, which exposed a bug in our cleanup scripts.

Check the visibility and ownerid of the image before attempting to
remove and image.

Issue: RELENG-3239
Change-Id: I38098c45fe73e872f3ca96960f704f168d9c4596
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoAdd create saml group to gerrit api 98/65698/11 v0.35.0
Aric Gardner [Tue, 6 Oct 2020 19:13:23 +0000 (15:13 -0400)]
Add create saml group to gerrit api

Project creation now needs an additional step
we must create a saml group.

Project creation now automatically
reformats your ldap group into a saml group

Also add error handling for unauthorized
when trying to see if a project exists

Issue-Id: RELENG-3231
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I72d8949331ecc66d320be5a1960f769ebfb2d962

4 years agoAdd user regexp for version tag validation 22/65222/6
Bengt Thuree [Fri, 28 Aug 2020 07:05:06 +0000 (17:05 +1000)]
Add user regexp for version tag validation

This patch adds the possibility to change the Version Schema.
This is done by the --version_regexp <regexp> parameter.
If this parameter is not used, then the default #.#.#
("^\d+.\d+.\d+$") regexp will be used.

The <regexp> could be an regexp expression or a filename pointing
to a file containing an regexp.

Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
Change-Id: I0099977f74c0da6bb87795111a48771fb3be4cd6

4 years agoAdd --repofile to releasedockerhub 15/63615/12
Bengt Thuree [Tue, 7 Apr 2020 01:39:46 +0000 (11:39 +1000)]
Add --repofile to  releasedockerhub

Enables providing a file with the repo names.

   -f, --repofile  Repo Name is a file name,
                   which contains one repo per row

ISSUE: RELENG-2553
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
Change-Id: I74a6eb5b95e1fef7d70d1e500c6df706b2eab8c9

4 years agoPreform check that disk-format maches 55/64955/5
Aric Gardner [Thu, 6 Aug 2020 20:18:10 +0000 (16:18 -0400)]
Preform check that disk-format maches

check that the disk format of the image to be uploaded
matches the disk-format specified by the user.
(default raw)

Issue-ID: RELENG-3149
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Ia989e53d2f4fc8ca541f68e17ea8d2663dd7bff4

4 years agoChange ci-managed filter default to false 54/64954/1
Aric Gardner [Thu, 6 Aug 2020 19:44:00 +0000 (15:44 -0400)]
Change ci-managed filter default to false

ODL does not tag images as ci-mangaged and its confusing
for the user to have nothing returned.

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Ib6e4fb15b0f211efde4b41ecb3d78327c1a8b4d9

4 years agoFix: openstack image upload defaults to qcow2 92/64792/3 v0.34.2
Aric Gardner [Tue, 28 Jul 2020 18:01:49 +0000 (14:01 -0400)]
Fix: openstack image upload defaults to qcow2

Openstack cmd and upload should default to raw not qcow.
builds were timing out. Internally ceph is converting from
qcow2 to raw. Which can cause the time outs.

Issue-ID: RELENG-3102
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I78b2fe7bcc6e6f631013cf1d58455cf17cbea291

4 years agoPin osc-lib to 2.2.0 to allow sharing images 03/64703/3
Anil Belur [Mon, 27 Jul 2020 01:37:17 +0000 (11:37 +1000)]
Pin osc-lib to 2.2.0 to allow sharing images

Using lftools openstack image share returns an error
"You are not authorized to find project with the name".
The issue is with osc_lib bug [1] resolved in osc_lib==2.2.0

[1] https://bugs.launchpad.net/ubuntu/+source/python-openstackclient/+bug/1864203
[2] https://opendev.org/openstack/osc-lib/commit/1ff3720daefd98a77557e5692fd7052b5930ae6c

Issue: RELENG-3099
Change-Id: I5e70443007d1ed6074802e502edb149008fcdced
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoRemove pinned distlib requirement 37/64537/3 v0.34.1
Eric Ball [Thu, 16 Jul 2020 00:47:52 +0000 (17:47 -0700)]
Remove pinned distlib requirement

Distlib is a common requirement for other libraries, and having it
pinned is causing failures in builds. It is not explicitly used in
lftools, so it does not need to be pinned.

Issue: RELENG-3071
Change-Id: I6a6a33585b1666b93656a43f16fb1db7e7b4f584
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
4 years agoRemove dependency on pre-commit 73/64273/2 v0.34.0
Thanh Ha [Sun, 14 Jun 2020 19:12:24 +0000 (15:12 -0400)]
Remove dependency on pre-commit

lftools has 0 dependencies on pre-commit and thus should not be
listed as one of its requirements. lftools simply uses pre-commit
as a linter which the dependency is already covered by the tox.ini
deps configuration.

Commit 54d1936f introduces what seems like an exceptionally large
list of dependencies which should be re-evaluated if lftools really
depends on any of those packages directly as some of those deps
are likely not really deps of lftools. This patch simply takes care
of pre-commit as I don't currently have time for a full audit.

Change-Id: I18acffc492ba6905e639cdb452dee0c2f5f2fcbb
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
4 years agoUpdate pre-commit hooks 2020-06 19/64219/2
Thanh Ha [Sat, 6 Jun 2020 13:52:04 +0000 (09:52 -0400)]
Update pre-commit hooks 2020-06

It's been awhile since we updated pre-commit hook versions so let's
catch up with the world.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I19d99ad86e8f60478af8daae2acfe19911b254ca

4 years agoSetup gitlint 18/64218/2
Thanh Ha [Sat, 6 Jun 2020 13:50:47 +0000 (09:50 -0400)]
Setup gitlint

Run gitlint to lint commit messages in the lftools repo.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I3daf64a3a234e191d035c583d6e752fa060accae

4 years agoPrint git-diff when pre-commit fails 17/64217/2
Thanh Ha [Sat, 6 Jun 2020 13:48:59 +0000 (09:48 -0400)]
Print git-diff when pre-commit fails

When pre-commit fails and automatically fixes issues for us in Jenkins
we are not able to see what pre-commit automatically fixed. Adding
--show-diff-on-failure makes pre-commit perform a git-diff immediately
after failing to ensure more useful output is printed to inform the
contributor what changes are needed to pass the check.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: Icd6484c8c4157d38885b1de460517d9b656a06d7

4 years agoUse black version 19.10b0 29/64229/1
Thanh Ha [Sat, 6 Jun 2020 14:13:38 +0000 (10:13 -0400)]
Use black version 19.10b0

The stable branch for some reason is failing to fetch in Jenkins build
now so let's switch to a tagged release.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I57c829e1cd831ae62cb5be1ca4414f144309005b

4 years agoFix string limit issue in _request_put_file 19/64119/1 v0.33.1
Eric Ball [Thu, 28 May 2020 15:22:42 +0000 (08:22 -0700)]
Fix string limit issue in _request_put_file

Requests can't handle a put call for very large data objects. However, it
can accept the data as a file-like object instead, and the size issue will
not show up. Documented here: https://github.com/psf/requests/issues/2717.

Issue: IT-19717
Change-Id: I826d3fa2eebbd3ba0389a0d7042701b4389e406d
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
4 years agoFix voting math 77/64077/4
Aric Gardner [Mon, 25 May 2020 19:00:58 +0000 (15:00 -0400)]
Fix voting math

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I450270710e67d08873f9bd40440417879f5f21c7

4 years agoMerge "Add ODL exception" v0.33.0
Anil Belur [Tue, 28 Apr 2020 20:51:57 +0000 (20:51 +0000)]
Merge "Add ODL exception"

4 years agoAdd ODL exception 12/63812/1
Aric Gardner [Tue, 28 Apr 2020 20:31:01 +0000 (16:31 -0400)]
Add ODL exception

umbrella name does not match ldap name
add exception

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I7edada167774a3078f86e1818bba70230bcc9c59

4 years agoAdd S3 log shipping subcommand 28/61828/98
Houa Yang [Tue, 24 Sep 2019 20:51:50 +0000 (15:51 -0500)]
Add S3 log shipping subcommand

Add new click command for logs-s3.

Issue-Id: RELENG-2379, RELENG-2380
Signed-off-by: Houa Yang <hyang@contractor.linuxfoundation.org>
Change-Id: I4295272cd903c9cc33962f2ee54667c28c3132e0

4 years agoAdd note to check for DCO exit status 14/63614/1
Anil Belur [Tue, 7 Apr 2020 01:25:44 +0000 (11:25 +1000)]
Add note to check for DCO exit status

Issue: RELENG-2764
Change-Id: I36f8cb328d4ecd42338a26e33a53b3a3ea7d3598
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
4 years agoMerge "Add utils command"
DW Talton [Mon, 6 Apr 2020 17:04:00 +0000 (17:04 +0000)]
Merge "Add utils command"

4 years agoAdd utils command 85/63585/9
DW Talton [Sun, 29 Mar 2020 21:15:01 +0000 (14:15 -0700)]
Add utils command

Add utils command to hold utilities like password generators
and whatnot.

Issue-ID: RELENG-2754
Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: I968c73f52a87c4436c355a1b9b66a49128a3bf56

4 years agoAdd password option to Nexus2 add user 96/63596/1
DW Talton [Fri, 3 Apr 2020 20:40:18 +0000 (13:40 -0700)]
Add password option to Nexus2 add user

Add the ability to optionally pass in password on user account
creation.

Issue-ID: RELENG-2755
Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: Iacda5eadb3f04503bf26ca551c990b6173d1ec0f

4 years agoChanges for nexus2 automation 58/63558/6 v0.32.0
Aric Gardner [Tue, 31 Mar 2020 18:56:58 +0000 (14:56 -0400)]
Changes for nexus2 automation

Allow more permissive regex match
project names like
integration-simulators-pnf-simulator
should allow . or - in their pom.xml
Enabled by strict_url_regex=False in config file

Now allows passing a url in the format https://
and sourcing nexus user and pass from lftools.ini

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I85d40f5964a2588093febd29ba9a1d25bc3e1046

4 years agoAdd Nexus3 user capabilities 60/63560/1
DW Talton [Tue, 31 Mar 2020 19:31:55 +0000 (12:31 -0700)]
Add Nexus3 user capabilities

Add Nexus3 CRUD user capabilities. Also includes password generator
helper.

Issue-ID: RELENG-2701
Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: I47fbf7939edb883a07b8aa421e776062317d6103

4 years agoUse development mode for tox testenv 44/13344/14
Thanh Ha [Tue, 6 Nov 2018 07:10:05 +0000 (15:10 +0800)]
Use development mode for tox testenv

Speeds up tests slighly as it skips the sdist install.

Change-Id: Ic4d4b860b373b1d1cd18ca2375971422f0034c4f
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
4 years agoMerge "Fix potential symlink loop in _compress_text()"
Andrew Grimberg [Tue, 24 Mar 2020 20:44:45 +0000 (20:44 +0000)]
Merge "Fix potential symlink loop in _compress_text()"

4 years agoFix potential symlink loop in _compress_text() 74/63474/1
Eric Ball [Fri, 20 Mar 2020 23:26:47 +0000 (16:26 -0700)]
Fix potential symlink loop in _compress_text()

By checking if os.path.exists can find the path, we stop the method
from attempting to traverse symlinked directories.
This also pins a more recent version of pre-commit, as the older
version was making a pre-commit file with errors.

Issue: RELENG-2576
Change-Id: I2265c9895444892877f684b807287ebe8319083a
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
4 years agoNot sure why lftools requires virtualenv 71/63471/1 v0.31.3
Aric Gardner [Fri, 20 Mar 2020 20:59:46 +0000 (16:59 -0400)]
Not sure why lftools requires virtualenv

In any case this breaks all tox runs.
as tox-install updates the virtualenv package.

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I78b07ee249c7201e99a3dbf2881910443dc94161

4 years agoMerge "Tweak initial project yaml file" v0.31.2
Aric Gardner [Mon, 16 Mar 2020 19:40:54 +0000 (19:40 +0000)]
Merge "Tweak initial project yaml file"

4 years agoTweak initial project yaml file 82/63382/6
Aric Gardner [Wed, 11 Mar 2020 21:58:58 +0000 (17:58 -0400)]
Tweak initial project yaml file

Adds a project view
drop branch (it is ignored)
dont append the file with -info
we will treat this as the projects first jjb file
projects can build off of it.

also change default builder if we are o-ran
also fix example text, we use fqdn now

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Ie7b56ad88769cd6ed6494162fc80ab8e8426392f

4 years agoFix initialization value of stack_cost 17/63417/1
Andrew Grimberg [Fri, 13 Mar 2020 20:51:42 +0000 (13:51 -0700)]
Fix initialization value of stack_cost

After having recently fixed the job-cost script in
1c99557ff1b60356f85193d00e23020bb9e53467 I realized that I should check
over the stack_cost generation. Much to my surprise I found that the
cost was being initialized to 0.7! This means that all stack costs that
we've generated so far have been increased by $0.70 in the reported
cost, job runs that have no stacks are still properly reporting $0 for
the cost.

This change fixes the initialization to be properly 0.0.

Issue: RELENG-2719
Change-Id: I8474604dbe0e2b32165b086754e29fac17f79785
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
4 years agoReplace python test harness 32/63232/11
DW Talton [Fri, 28 Feb 2020 20:30:59 +0000 (13:30 -0700)]
Replace python test harness

This commit is a comprehensive overhaul of the test harness used
in lftools. First, it removes coala. Black is added as a defacto
code formatter to ensure consistency across all commits. Black is
added as a pre-commit hook. Flake8 is added in conjunction with
black, and configured according to Black's specification.

Issue-ID: RELENG-2766
Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: Ib688b7cf9af8a22b7d067d3417a8e03f35f0bd64

4 years agoFormat lftools code with the Black code formatter 57/63357/1
DW Talton [Tue, 10 Mar 2020 16:41:45 +0000 (09:41 -0700)]
Format lftools code with the Black code formatter

This patch auto-formats most files contained in the lftools software.
Format changes are trivial, such as replacing a single quote with
double-quotes, and rearranging the length of certain lines. Lftools
will use Black for code formatting once this patch, and the test
harness overhaul patch, are merged.

Issue-ID: RELENG-2766
Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: I74318e0685aa44617af4575016c84caafa68b5c5

4 years agoRevert "Change the copy_archives method to use..." 66/63266/1 v0.31.1
Eric Ball [Tue, 3 Mar 2020 17:18:39 +0000 (09:18 -0800)]
Revert "Change the copy_archives method to use..."

This change is causing logs to not be copied correctly. See IT-19109,
IT-19131.
This reverts commit 3fad9329a6de8384446fb63215311d29e8aab363.

Change-Id: I7b134d2ac5f8c3b93808c1b863f3f01df0cc0a95
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
4 years agoRemove garbage var 96/63196/1 v0.31.0
Aric Gardner [Wed, 26 Feb 2020 17:29:22 +0000 (12:29 -0500)]
Remove garbage var

No memory of how this snuck in here, should be removed

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I8fa62db1ff6b481f865229643ac319413c8660f4

4 years agoMerge "Change the copy_archives method to use copy2"
Andrew Grimberg [Wed, 26 Feb 2020 17:16:13 +0000 (17:16 +0000)]
Merge "Change the copy_archives method to use copy2"

4 years agoChange the copy_archives method to use copy2 73/63173/3
Eric Ball [Sat, 22 Feb 2020 03:15:32 +0000 (19:15 -0800)]
Change the copy_archives method to use copy2

Uses copy2 with follow_symlinks set to False, which will copy
symlinks as symlinks, rather than following their path. Since glob
does not have an option to not follow symlinks, we instead just check
for file existence to ignore issues with globbing symlinked paths
that "open" fails to find.

Issue: RELENG-2576
Change-Id: Ie5a6ffba5c1cb79cff7dd9c691d9b9bd2eaeeba3
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
4 years agoTest released staging repo for closure 19/63119/4
DW Talton [Wed, 12 Feb 2020 20:59:53 +0000 (13:59 -0700)]
Test released staging repo for closure

Add long-polling to ensure that a released staging repo is fully
closed prior to exiting.

Issue-ID: RELENG-2695
Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: I032cccabe28ba7f5783b2ec82d73725d228ab797

4 years agoAdd nexus2 API endpoints 80/62780/13
DW Talton [Fri, 10 Jan 2020 21:03:59 +0000 (14:03 -0700)]
Add nexus2 API endpoints

Change-Id: I8ba68e819fbbcda8208e88d83007880cf543a194
Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
4 years agoMerge "Add GitLint which replaces GitCommitBear"
Andrew Grimberg [Tue, 4 Feb 2020 21:48:14 +0000 (21:48 +0000)]
Merge "Add GitLint which replaces GitCommitBear"

4 years agoRemove extra request error checking 90/62990/1 v0.30.1
DW Talton [Wed, 29 Jan 2020 21:59:59 +0000 (14:59 -0700)]
Remove extra request error checking

Remove additional requests error checking. Current lftools needs
the ability to continue despite a http session error.

Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: Icad82ce16f7df7466e43be7e9035abac3fe63033

4 years agoAdd GitLint which replaces GitCommitBear 78/62978/1
Thanh Ha [Tue, 28 Jan 2020 15:58:50 +0000 (10:58 -0500)]
Add GitLint which replaces GitCommitBear

This replaces GitCommitBear with GitLint which does similar
linting for git commit messages but using pre-commit hooks.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I3480ad81702fa6b5015523a7274e219ba2f240d0

4 years agoMake output parsable 21/62921/3 v0.30.0
Aric Gardner [Thu, 23 Jan 2020 20:11:11 +0000 (15:11 -0500)]
Make output parsable

Need valid JSON in the output for automation work

-thanks DW

ISSUE-ID: RELENG-2683

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Ie85adcec15577f432299846075068ed2a72783db

4 years agoEnable project_version_update 58/62858/11
Aric Gardner [Tue, 21 Jan 2020 18:37:26 +0000 (13:37 -0500)]
Enable project_version_update

Typo in the rtd docs api lead this to not be implemented.
corrected endpoint and implemented in CLI

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I09aee9e06e1b54c8f6c143762c501286627098d5

4 years agoImplement the Nexus 3 REST API in lftools 07/62607/16
DW Talton [Thu, 12 Dec 2019 00:26:02 +0000 (17:26 -0700)]
Implement the Nexus 3 REST API in lftools

Issue: RELENG-308
Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: I3dc1b1b7280fe6a2d69a75e2f8c7f5a9d64c5126

4 years agolftools github create-team no longer requires repo 70/62770/2
Aric Gardner [Thu, 9 Jan 2020 20:37:32 +0000 (15:37 -0500)]
lftools github create-team no longer requires repo

repo is now optional

ISSUE-ID: RELENG-2646
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I3cb31d17bb2fbe9e0ec7c419c99ad5cc2397d0f6

4 years agoFix reviewid logic 36/62636/2 v0.29.1
Aric Gardner [Mon, 16 Dec 2019 19:04:42 +0000 (14:04 -0500)]
Fix reviewid logic

When ammending a review we need to skip
create change not publish

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I3e2832ee3d228af12f337d63b49b66cb91357915

4 years agoMerge "Fix jenkins info job creation"
Aric Gardner [Thu, 12 Dec 2019 21:30:41 +0000 (21:30 +0000)]
Merge "Fix jenkins info job creation"

4 years agoMerge "Fix for empty INFO file creation"
Aric Gardner [Thu, 12 Dec 2019 01:50:10 +0000 (01:50 +0000)]
Merge "Fix for empty INFO file creation"

4 years agoFix jenkins info job creation 88/62588/1
Aric Gardner [Tue, 10 Dec 2019 17:36:55 +0000 (12:36 -0500)]
Fix jenkins info job creation

Fix jenkins info job creation.
Auto created info jobs need to go to the
"ci-managment" repo.

Also fix description on project creation.

ISSUE-ID: RELENG-2574
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Iab6860152c4623989997588b9a46804fc5693c2f

4 years agoGithub cli project specific tokens 86/62586/1
Aric Gardner [Tue, 10 Dec 2019 17:16:10 +0000 (12:16 -0500)]
Github cli project specific tokens

Github cli commands should first search for a
[github.organization] section.

Then fall back to a [github] section

ISSUE-ID: RELENG-2574
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I23957b43c50b4f6fda3483eec956f04afe9509e8

4 years agoFix for empty INFO file creation 17/62517/2
Aric Gardner [Thu, 5 Dec 2019 21:15:20 +0000 (16:15 -0500)]
Fix for empty INFO file creation

Do not require and lftools
[gerrit]
section.

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I118b4a945d6dd03231cdd7a2cbcf8cb82d35acee

4 years agoMerge "Create gerrit project via gerrit api" v0.29.0
Aric Gardner [Wed, 4 Dec 2019 16:58:36 +0000 (16:58 +0000)]
Merge "Create gerrit project via gerrit api"

4 years agoCreate gerrit project via gerrit api 17/62117/53
Aric Gardner [Wed, 30 Oct 2019 20:58:41 +0000 (16:58 -0400)]
Create gerrit project via gerrit api

internal jenkins implementation:
gerrit.linuxfoundation.org/infra/c/ci-management/+/62158

Extend lfidapi to exit with good message and code if
group does not exist. (check if group exists functionality)

Remove pythonsix compatability for config parser

Usage: lftools gerrit [OPTIONS] COMMAND [ARGS]...

  GERRIT TOOLS.

Options:
  --help  Show this message and exit.

Commands:
  abandonchanges              Abandon all OPEN changes for a gerrit
project.
  addfile                     Add an file for review to a Project.
  addgithubrights             Grant Github read for a project.
  addgitreview                Add git review to a project.
  addinfojob                  Add an INFO job for a new Project.
  createproject               Create a project via the gerrit API.
  list-project-inherits-from  List who a project inherits from.
  list-project-permissions    List Owners of a Project.

ISSUE: RELENG-2370
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I3a24751eeb2e739dee183074bbef6742df58a793

4 years agoSupport python 3.5.x 46/62446/1
Aric Gardner [Tue, 3 Dec 2019 21:57:21 +0000 (16:57 -0500)]
Support python 3.5.x

lftools must pin xdg to 3.x to support python 3.5
needed for onap ubuntu 16.04 builders

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: If97846bbb5a52410cfe23a66fb0f25c87b6bc5b4

4 years agoMerge "Add missing releasenote"
Bengt Thuree [Wed, 27 Nov 2019 02:07:01 +0000 (02:07 +0000)]
Merge "Add missing releasenote"

4 years agoAdd openstack cost command 82/62382/5
Tim Johnson [Fri, 22 Nov 2019 18:32:51 +0000 (10:32 -0800)]
Add openstack cost command

Issue: RELENG-2550
Change-Id: I9bee88cd5cb3d89f3f554470e86925d8f24dbb5a
Signed-off-by: Tim Johnson <tijohnson@linuxfoundation.org>
4 years agoAdd .venv to .gitignore 02/62402/1
DW Talton [Tue, 26 Nov 2019 03:32:04 +0000 (20:32 -0700)]
Add .venv to .gitignore

Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: I953896c94868ad1fbb39288be5a221c5541f9d26

4 years agoAdd HTTPBasicAuth support to REST client 68/62168/2
DW Talton [Tue, 5 Nov 2019 20:53:51 +0000 (13:53 -0700)]
Add HTTPBasicAuth support to REST client

Signed-off-by: DW Talton <dtalton@contractor.linuxfoundation.org>
Change-Id: I6467c8aeb2b2f24800ab1912ec6b485693a3378f

4 years agoAdd missing releasenote 88/62388/2
Bengt Thuree [Sun, 24 Nov 2019 22:39:07 +0000 (09:39 +1100)]
Add missing releasenote

Patch Id4efd6eaf870c9ce60d670094b6916b3c7b407cd did not have releasenote

Print rule failures for unclosed repos

Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
Change-Id: Ibc624e6e7c52716178e8e16995d7065982539271

4 years agoMerge "Fix help text for github create team"
Aric Gardner [Fri, 22 Nov 2019 19:47:30 +0000 (19:47 +0000)]
Merge "Fix help text for github create team"

4 years agoFix help text for github create team 84/62384/1
Aric Gardner [Fri, 22 Nov 2019 19:03:04 +0000 (14:03 -0500)]
Fix help text for github create team

let user know to set privacy to closed

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I9f2b583eb266cb6678a56ce6c4678f3f23e1af76