releng/lftools.git
5 years agoFix release notes typos 68/15868/2
Jessica Wagantall [Wed, 12 Jun 2019 00:16:26 +0000 (17:16 -0700)]
Fix release notes typos

Fix release notes typos for commit:
9d64c0644c4df403243eb89014d6257827d9d281

"Get jenkins config only with the key exists"

Change-Id: I78a567ffae6c5333021dd86e5b2aa8db8c3fc6be
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
5 years agoMerge "Get jenkins config only with the key exists"
Andrew Grimberg [Tue, 11 Jun 2019 21:43:00 +0000 (21:43 +0000)]
Merge "Get jenkins config only with the key exists"

5 years agoGet jenkins config only with the key exists 51/15851/5
Anil Belur [Tue, 11 Jun 2019 02:53:10 +0000 (12:53 +1000)]
Get jenkins config only with the key exists

Handle config parser correctly which defaults to "[jenkins]" section
when no server is pased. This fixes the issue with checking if the key
exists in the configuration read before reading the key-value.

The issue is reproduceable by running `lftools jenkins plugins --help`
or `tox -e docs`, with the ``jenkins.ini`` file having no "[jenkins]"
section.

Use SafeConfigParser which is a saner version for parsing config files.

Issue: RELENG-2120
Change-Id: I274ac46cf7a1d03cf7c3c6fdaec6094b5e02aa42
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
5 years agoMerge "Use six.text_type rather than str.encode('utf-8')"
Thanh Ha (zxiiro) [Tue, 11 Jun 2019 01:47:12 +0000 (01:47 +0000)]
Merge "Use six.text_type rather than str.encode('utf-8')"

5 years agoUse six.text_type rather than str.encode('utf-8') 40/15840/3
Eric Ball [Fri, 7 Jun 2019 13:59:20 +0000 (06:59 -0700)]
Use six.text_type rather than str.encode('utf-8')

Encoding the string in Python 3 results in a bytes object rather
than str. To avoid this, use six.text_type method, which is designed
to safely perform this function in Python 2 and 3.

Change-Id: Ice7f02f4c02fd71d115af627d5b15622cd04d35d
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
5 years agoMerge "Github api tools"
Andrew Grimberg [Mon, 10 Jun 2019 20:07:33 +0000 (20:07 +0000)]
Merge "Github api tools"

5 years agoGithub api tools 64/15264/30
Aric Gardner [Thu, 11 Apr 2019 20:35:20 +0000 (16:35 -0400)]
Github api tools

create-repo [OPTIONS] ORGANIZATION REPOSITORY DESCRIPTION
Create a Github repo within an Organization.
By default has_issues has_wiki and has_projects are set to false. See
--help to create a repo with these enabled.

modify-repo [OPTIONS] ORGANIZATION REPOSITORY
Modify a Github repo within an Organization.
By default has_issues has_wiki and has_projects are set to false. See
--help to use this command to enable these options

create-team [OPTIONS] ORGANIZATION NAME REPO PRIVACY
Create a Github team within an Organization.

list         List an Organization's GitHub repos.
  --audit          list members without 2fa
  --repos          list all repos
  --full           All members and their respective teams
  --teams          List avaliable teams
  --repofeatures   list enabled features for repos in an org
  --help           Show this message and exit.

user [OPTIONS] ORGANIZATION USER TEAM
Add and remove users
  --delete  remove user from a team within an org
  --admin   user is admin for org
  --help    Show this message and exit.

Github token is read from
~/.config/lftools/lftools.ini
In the form of:

[github]
token =

ISSUE: RELENG-1526
Change-Id: I558eb11897b555d26fac5abacb660d4bdb630a12
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
5 years agoMerge "lfidapi create group checks if group exists"
Andrew Grimberg [Fri, 7 Jun 2019 16:45:41 +0000 (16:45 +0000)]
Merge "lfidapi create group checks if group exists"

5 years agoMerge "Move numpy dependency into coala linter" v0.24.0
Anil Belur [Thu, 6 Jun 2019 21:56:56 +0000 (21:56 +0000)]
Merge "Move numpy dependency into coala linter"

5 years agoMove numpy dependency into coala linter 28/15828/1
Thanh Ha [Thu, 6 Jun 2019 21:25:37 +0000 (17:25 -0400)]
Move numpy dependency into coala linter

The NLTK upstream library started depending on numpy so pull it
in when calling Coala.

Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Change-Id: I50c4ee38eb84e570551ebc3dd8bd9d9d5cb4e473

5 years agoInclude *.{html,xml} file extensions to compress 93/15793/7
Anil Belur [Thu, 6 Jun 2019 14:05:51 +0000 (00:05 +1000)]
Include *.{html,xml} file extensions to compress

Add file extensions `*.{html,xml}` to ensure they are compressed.
`*.xml` files can be quite large on log server and this ensures
that log files are compressed on Nexus have a smaller foot print.

Issue: RELENG-2072
Change-Id: I64bcf7f75f45ba22d0fca50d5df55fa156193580
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
5 years agoadd numpy to requirements 27/15827/1
Aric Gardner [Thu, 6 Jun 2019 17:48:27 +0000 (13:48 -0400)]
add numpy to requirements

I dont understand why this error suddenly appeared.
Or what is pulling in NumPy

But this fixed it for a local run

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I5b9b69a91919135268e41a7ea6a7cc775a83e580

5 years agolfidapi create group checks if group exists 26/15826/2
Aric Gardner [Thu, 6 Jun 2019 17:25:14 +0000 (13:25 -0400)]
lfidapi create group checks if group exists

There is a bug in lfidapi where if a group exists
the reponse code does the right thing but some
shadow group is created.

Add a check so that we only create a group
if it does not yet exist.

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: Ia80400cf74431a8ffdea1ae06459969649d94501

5 years agoFix release notes format 22/15822/1
Anil Belur [Thu, 6 Jun 2019 00:22:54 +0000 (10:22 +1000)]
Fix release notes format

The change 6624bcb3191fdbe45bc7ae1c570a1b7873a5d193
breaks the verify jobs

Change-Id: I70778a7ed26c9c197b431a1f399eaeaae70a2baf
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
5 years agoMerge "Fix token generation."
Andrew Grimberg [Sun, 2 Jun 2019 19:08:42 +0000 (19:08 +0000)]
Merge "Fix token generation."

5 years agoMerge "List insecure plugins"
Andrew Grimberg [Sun, 2 Jun 2019 19:07:48 +0000 (19:07 +0000)]
Merge "List insecure plugins"

5 years agoPin detox 00/15800/1
Andrew Grimberg [Sun, 2 Jun 2019 00:24:27 +0000 (17:24 -0700)]
Pin detox

detox 0.19 is not compatible with the version of tox we are getting from
upstream

Change-Id: Icc0fd40057c08e5241cd4d06c6f4ee57f7c0e7ec
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
5 years agoList insecure plugins 12/15712/12
Aric Gardner [Fri, 17 May 2019 21:34:30 +0000 (17:34 -0400)]
List insecure plugins

List active plugins that have a known
vulnerability

Example output:

$ lftools jenkins -s 'build.opnfv.org/ci' plugins sec
pam-auth:1.5    pam-auth:1.5
https://jenkins.io/security/advisory/2019-05-21/#SECURITY-1316
credentials:2.1.18      credentials:2.1.18
https://jenkins.io/security/advisory/2019-05-21/#SECURITY-1322

jenkins_jobs.ini config
$ cat ~/.config/jenkins_jobs/jenkins_jobs.ini | grep build.opnfv.org/ci
[build.opnfv.org/ci]
user=
password=
url=https://build.opnfv.org/ci

ISSUE: RELENG-2046
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Signed-off-by: Jessica Wagantall <jwagantall@linuxfoundation.org>
Change-Id: I807a1e33d10c42c0d48f7fd179858eebc368c1e8

5 years agoFix token generation. 50/15750/4
Aric Gardner [Thu, 23 May 2019 16:36:25 +0000 (12:36 -0400)]
Fix token generation.

Remove legacy method of token generation.

username and password come from global section of lftools.ini.
[global]
username = lfid
password =

url comes from -s section of jenkins_jobs.ini
eg: [jenkins.acumos.org/sandbox]
url=jenkins.acumos.org/sandbox

example command
lftools jenkins -s 'jenkins.acumos.org/sandbox' token change

option --name set token name default is token-created-by-lftools

changeApiToken no longer exists, so we must allways generate a
new token.

NOTE: token print and token change now do the same thing.
That is generate a new token and return it to console.
I could remove one of them but Im wary of breaking any
legacy scripts that may exist.

NOTE: This will not work with SAML Plugin jenkins.
But I need it for automation before we switch all
jenkins to SAML

ISSUE: RELENG-2046
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I075c530f6e9fad7b477383867d52b83bba745662

5 years agoAdd documenation on lfidapi configuration 44/15344/6
Aric Gardner [Fri, 19 Apr 2019 18:20:43 +0000 (14:20 -0400)]
Add documenation on lfidapi configuration

Documentation on [lfid] for lftools was missing.

Change-Id: Ia8e4e41fc4412edc7b5153f6e5e51a822875d889
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
5 years agoNote in git-tag docs that tag must be annotated 42/15742/1
Eric Ball [Wed, 22 May 2019 21:12:27 +0000 (14:12 -0700)]
Note in git-tag docs that tag must be annotated

This caused a significant holdup in testing this function. Though
this is a very small change, I believe it can potentially save
future users a lot of frustration.

Change-Id: I1994391ab5cabdbec4aa7e751a5663f6de726f4d
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
5 years agoTitles wrong for gerrit infofile and lfid commands 42/15642/1
Aric Gardner [Mon, 13 May 2019 16:32:21 +0000 (12:32 -0400)]
Titles wrong for gerrit infofile and lfid commands

I noticed that I had done the titles incorrectly via some
copy paste lazyness in the docs.

Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
Change-Id: I3da9d6cfa7513b3aedcb68e8b6bed02823b31271

5 years agoRemove python 3.4 support 02/15302/1
Aric Gardner [Tue, 16 Apr 2019 20:38:08 +0000 (16:38 -0400)]
Remove python 3.4 support

Python 3.4 is EOL
And no longer receives security updates

Change-Id: Iefe7efae8614062b1fcf54b2555ffe40652502cd
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
5 years agoMerge "Update docs for git-tag and container signing"
Andrew Grimberg [Fri, 12 Apr 2019 20:57:10 +0000 (20:57 +0000)]
Merge "Update docs for git-tag and container signing"

5 years agoUpdate docs for git-tag and container signing 68/15268/1
Eric Ball [Fri, 12 Apr 2019 20:00:54 +0000 (13:00 -0700)]
Update docs for git-tag and container signing

This also reorganizes subcommands into alphabetical order in order
to match the "lftools sign --help" output.

Issue: RELENG-1851
Change-Id: I67a3a875d688ce47ab952ad82f39e39fb7093b1f
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
5 years agoMerge "Add git tag and Docker container sigul signing" v0.23.0 v0.23.1
Anil Belur [Sun, 7 Apr 2019 23:47:48 +0000 (23:47 +0000)]
Merge "Add git tag and Docker container sigul signing"

5 years agoAdd git tag and Docker container sigul signing 94/15194/4
Eric Ball [Thu, 4 Apr 2019 13:56:25 +0000 (06:56 -0700)]
Add git tag and Docker container sigul signing

Issue: RELENG-1851
Change-Id: I02ecde037ff603dafd5820f15b454f2a0f2571f1
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
5 years agoMerge "Fix OSError in lftools deploy archives" v0.22.2
Andrew Grimberg [Tue, 2 Apr 2019 23:42:45 +0000 (23:42 +0000)]
Merge "Fix OSError in lftools deploy archives"

5 years agoFix OSError in lftools deploy archives 66/15066/9
Bengt Thuree [Tue, 2 Apr 2019 00:04:03 +0000 (17:04 -0700)]
Fix OSError in lftools deploy archives

Adding a check if dir/file exists before executing for loop.

Change-Id: Id887b67ba3c2bea4e4978c6f5ce21bf499ed3d4f
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoFix help text 56/15156/2
Andrew Grimberg [Mon, 1 Apr 2019 20:37:59 +0000 (13:37 -0700)]
Fix help text

Cleanup the help text for 'openstack image upload'

Change-Id: I1e00b09332677b244c2d9d96c5d9ddcb5712e38d
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>
5 years agoMerge "Correct script exit condition."
Andrew Grimberg [Mon, 1 Apr 2019 17:12:57 +0000 (17:12 +0000)]
Merge "Correct script exit condition."

5 years agoFix AttributeError on HTTPError in deploy code 23/15123/2
Thanh Ha [Fri, 29 Mar 2019 15:21:23 +0000 (11:21 -0400)]
Fix AttributeError on HTTPError in deploy code

When a deploy nexus-zip fails it throws a stack trace with:

    AttributeError: 'HTTPError' object has no attribute 'value'

Which is a very unhelpful error message but it's caused by the
error handler code incorrectly handling the returned string value
for the exception.

Change-Id: I6d059dd73a5e774c15ab90a3b53728d6e051bf0c
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoCorrect script exit condition. 94/15094/3
Aric Gardner [Wed, 27 Mar 2019 20:30:28 +0000 (16:30 -0400)]
Correct script exit condition.

Logic error.
When no votes where registered script would exit 0

Change-Id: Ie7dad776c737dcf5e5ed7a0c39b53f362eff980d
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
5 years agoRevert "Refactor deploy_maven_file from shell ..." 28/14928/4 v0.22.1
Eric Ball [Fri, 15 Mar 2019 01:07:12 +0000 (18:07 -0700)]
Revert "Refactor deploy_maven_file from shell ..."

This reverts commit 5dfd489d3fe3e137f6845a046f3a69ed0fc24fbe.
This change is not fully support by Nexus. Nexus does not allow
using REST to push files to SNAPSHOT repos. Since this was causing
build failures, the feature is being removed entirely until it can
be implemented in a fully supported manner.

Change-Id: Iaf1faa82e19e2195a5564aa0f50627ae63072ec2
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
5 years agoMerge changes from topic 'release_docker_hub'
Thanh Ha (zxiiro) [Thu, 21 Mar 2019 23:33:19 +0000 (23:33 +0000)]
Merge changes from topic 'release_docker_hub'

* changes:
  Connect cli to release_docker_hub 7(7).
  Add multithread code to release_docker_hub 6(7).

5 years agoMerge "Add print functions to release_docker_hub 5(7)."
Thanh Ha (zxiiro) [Thu, 21 Mar 2019 23:31:12 +0000 (23:31 +0000)]
Merge "Add print functions to release_docker_hub 5(7)."

5 years agoAdd lftools infofile check-votes 62/14962/10 v0.22.0
Aric Gardner [Tue, 19 Mar 2019 19:00:50 +0000 (15:00 -0400)]
Add lftools infofile check-votes

Usage: lftools infofile check-votes [OPTIONS] INFO_FILE GERRIT_URL
                                    CHANGE_NUMBER
$ lftools infofile check-votes ~/lf/allrepos/pharos/INFO.yaml \
https://gerrit.opnfv.org/gerrit/ 67269
Number of Committers:
11
Committers have voted:
['mrichomme', 'trev', 'jose.lausuch', 'fdegir', 'Julien-zte',
'agardner', 'bramwelt', 'AlexandruAvadanii', 'gherrero']
9
Committers that have not voted:
['jmorgan1', 'wenjing']
2
majority vote reached
$ echo $?
0
(in this case exit 0 rather than 1)

Problems:
1) Someone will have to comment recheck when
majority is reached for the auto merge from
jenkins.

2) Non-Interactive Users will need submit.
(and possibly +2? (unless we expect one of the votes to be a +2)

ISSUE:RELENG-975

Change-Id: I83df2e20c3ae61681de3c0fbe5c90aaa1d873a8f
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
5 years agoConnect cli to release_docker_hub 7(7). 34/13734/10
Bengt Thuree [Thu, 29 Nov 2018 11:22:38 +0000 (22:22 +1100)]
Connect cli to release_docker_hub 7(7).

This is the seventh, and final part, which adds the
cli part.

Issue:RELENG-1549
Change-Id: I629d575730abd73425dacd5e694a30cadcd85657
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoAdd multithread code to release_docker_hub 6(7). 33/13733/11
Bengt Thuree [Thu, 29 Nov 2018 11:18:44 +0000 (22:18 +1100)]
Add multithread code to release_docker_hub 6(7).

This is the sixth part, with multithreading stuff,
fetching tags, copying to docker hub, and main script.

Issue:RELENG-1549
Change-Id: I03ec4adfe3225ec9ebbcc27092b578e4c0b36eae
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoAdd print functions to release_docker_hub 5(7). 32/13732/9
Bengt Thuree [Thu, 29 Nov 2018 11:08:20 +0000 (22:08 +1100)]
Add print functions to release_docker_hub 5(7).

This is the fifth part, which add various print functions
to the release_docker_hub function.

Issue:RELENG-1549
Change-Id: I23649c998f5a3b1e81edced6538741cbbd51a0d2
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoMerge "match-ldap-to-info matches ldap group to INFO file"
Aric Gardner [Fri, 8 Mar 2019 00:22:36 +0000 (00:22 +0000)]
Merge "match-ldap-to-info matches ldap group to INFO file"

5 years agoMerge "Add global extra roles for "nexus create repo""
Jessica Wagantall [Thu, 7 Mar 2019 22:24:50 +0000 (22:24 +0000)]
Merge "Add global extra roles for "nexus create repo""

5 years agomatch-ldap-to-info matches ldap group to INFO file 23/14823/6
Aric Gardner [Thu, 7 Mar 2019 18:48:22 +0000 (13:48 -0500)]
match-ldap-to-info matches ldap group to INFO file

I have abandoned the idea of using a diff on the INFO file as a driver
for making changes to ldap. It was messy for new info files, as there
was no previous diff. which meant if a user was removed,
we had no way of knowing. Now the script is idempotent.
Whatever is in the INFO.yaml will be reflected in the LDAP file.

example: you give it an info file and a group and it will
make the nessesary changes:

lftools lfidapi add-remove-committers path/to/INFO.yaml ldap-group

added --noop flag so you can see what would be changed

Change-Id: Ie1546f3e80e826742612b9f32f91dc6ce3360d0c
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
5 years agoExpose Python commands to shell signer mode param 99/14799/1 v0.21.0
Thanh Ha [Wed, 6 Mar 2019 16:56:13 +0000 (11:56 -0500)]
Expose Python commands to shell signer mode param

Change-Id: I3202f9b30d24ace82d583cf6b92c12fef0c3c85c
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoMerge "Allow requesting serial signing"
Thanh Ha (zxiiro) [Wed, 6 Mar 2019 16:56:49 +0000 (16:56 +0000)]
Merge "Allow requesting serial signing"

5 years agoFix typo in Jenkins key 98/14598/2 v0.20.0
Thanh Ha [Thu, 14 Feb 2019 22:38:30 +0000 (17:38 -0500)]
Fix typo in Jenkins key

Change-Id: I467f2b913061c98a65e63ef9c16f9fa895c7ece0
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agolfidapi for lftools 70/14670/23
Aric Gardner [Thu, 21 Feb 2019 21:06:50 +0000 (16:06 -0500)]
lfidapi for lftools

Note: as soon as by the end of the year
(whenever all clients have been migrated from LDAP to Auth0)
All groups will migrate to the "IAM"
product at which point this will be tooled to a different API.

All api methods implemented:

(search_members)
(user)
(invite)
(create_group)

uses oauth2_helper to retreive access_token from client_secret and
refresh_token

reads secrets from ~/.config/lftools/lftools.ini

[lfid]
clientid =
client_secret =
refresh_token =
token_uri = https://identitystg.linuxfoundation.org/oauth2/token
url = https://identitystg.linuxfoundation.org/rest/auth0/

Secrets are saved in LastPass under Release Engineering as
"Identity Groups REST API Client".

Change-Id: Ib7f575dda0c73a99ca65f0a7f87d2b62f85da03e
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
5 years agoAdd global extra roles for "nexus create repo" 52/14752/2
Eric Ball [Fri, 1 Mar 2019 23:43:29 +0000 (15:43 -0800)]
Add global extra roles for "nexus create repo"

This will allow users to specify common roles only once, rather than
having to add "extra_privs" to each repo. Documents in the docs repo
have also been updated, and are now linked to from the lftools docs.

Issue: RELENG-291
Change-Id: Ia35d92fa742ca8202f1d4ad7794bf8549e0a8a77
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
5 years agoAllow requesting serial signing 76/14676/2
Konstantin Ryabitsev [Fri, 22 Feb 2019 18:32:20 +0000 (13:32 -0500)]
Allow requesting serial signing

Sigul-signing directories containing a lot of large artifacts may not be
optimal, because whether serial or parallel, the same amount of
network IO is required and the job takes about the same amount of time
in the end. This allows passing "serial" as a third argument to sign_dir
to specifically request serial operation.

Change-Id: I6a6267090cee6cbed1f134b27eb6cec485fa1793
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
5 years agoPin more-itertools and update pytest 25/14625/1
Thanh Ha [Mon, 18 Feb 2019 21:52:29 +0000 (16:52 -0500)]
Pin more-itertools and update pytest

PyTest pulls in more-itertools but version 6.0.0 is Python 3.4
only. Pin more-itertools and update pytest to latest version.

Change-Id: Ia6ad6a5f1cf0707599884a892783d7aacc447e7a
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoAdd command to list openstack containers 75/14075/2
Thanh Ha [Fri, 4 Jan 2019 17:58:00 +0000 (12:58 -0500)]
Add command to list openstack containers

Change-Id: I5586888ba7c418f92ddc90727f349f695cd6d67d
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoMerge "Fetch nexus3 catalog to release_docker_hub 4(7)."
Thanh Ha (zxiiro) [Sun, 10 Feb 2019 17:05:12 +0000 (17:05 +0000)]
Merge "Fetch nexus3 catalog to release_docker_hub 4(7)."

5 years agoMerge "Add main project class to release_docker_hub 3(7)."
Thanh Ha (zxiiro) [Sun, 10 Feb 2019 17:04:48 +0000 (17:04 +0000)]
Merge "Add main project class to release_docker_hub 3(7)."

5 years agoMerge "Add tag classes to release_docker_hub 2(7)."
Thanh Ha (zxiiro) [Sun, 10 Feb 2019 17:04:29 +0000 (17:04 +0000)]
Merge "Add tag classes to release_docker_hub 2(7)."

5 years agoMerge "Add common functions to release_docker_hub 1(7)."
Thanh Ha (zxiiro) [Sun, 10 Feb 2019 17:03:28 +0000 (17:03 +0000)]
Merge "Add common functions to release_docker_hub 1(7)."

5 years agoAdd "schema validate" command 63/14363/8
Aric Gardner [Tue, 29 Jan 2019 20:25:44 +0000 (15:25 -0500)]
Add "schema validate" command

This command will validate a yaml file
based on a schema file

Issue: RELENG-1685
Change-Id: I59d427ff52ff61be3df9b1413828ff13b814653a
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
5 years agoFetch nexus3 catalog to release_docker_hub 4(7). 31/13731/7
Bengt Thuree [Thu, 29 Nov 2018 11:02:37 +0000 (22:02 +1100)]
Fetch nexus3 catalog to release_docker_hub 4(7).

This is the fourth part, which contains fetching nexus3
catalog.

Issue:RELENG-1549
Change-Id: Ie627ad04cc0d84304b585d126ed1e468c95661b7
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoAdd main project class to release_docker_hub 3(7). 30/13730/8
Bengt Thuree [Thu, 29 Nov 2018 10:58:12 +0000 (21:58 +1100)]
Add main project class to release_docker_hub 3(7).

This is the third part, which contains the Project class.
The project class initiates the various tag classes, as well
as contain the code to copy between nexus and docker.

Issue:RELENG-1549
Change-Id: I1b59a2452dd0f415cefe38a2d496f263bdf5d7b8
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoAdd tag classes to release_docker_hub 2(7). 29/13729/6
Bengt Thuree [Thu, 29 Nov 2018 10:44:27 +0000 (21:44 +1100)]
Add tag classes to release_docker_hub 2(7).

This is the second part, which contains the tag classes.
Generic tag class, which contains two functions:
  Validate version number.
  Add new tag.
The Nexus and Docker classes will fetch all tags upon init.

Issue:RELENG-1549
Change-Id: I305c98977a7d5828875d3cfd9bc59871b87d08f1
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoAdd common functions to release_docker_hub 1(7). 47/13047/17
Bengt Thuree [Thu, 29 Nov 2018 10:32:51 +0000 (21:32 +1100)]
Add common functions to release_docker_hub 1(7).

This is the first part, which contains the common stuff.

Issue:RELENG-1549
Change-Id: I6fd7658d85d696d39042f8210c7e2098d47f3bbc
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoMerge "lftools gerrit create for new projects"
Thanh Ha (zxiiro) [Fri, 25 Jan 2019 22:10:46 +0000 (22:10 +0000)]
Merge "lftools gerrit create for new projects"

5 years agolftools gerrit create for new projects 59/13659/26
Aric Gardner [Mon, 26 Nov 2018 20:48:48 +0000 (15:48 -0500)]
lftools gerrit create for new projects

Creates a project and sets up permissions

   calls gerrit create project:
   -creates the repo (Parent is configurable)
   -clones the repo
   -modifies groups file in /refs/meta/config to
     add GitHub Replication
   -adds a .gitreview and pushes it to
     HEAD:refs/heads/master
   If --enable is passed, it will
   -allow Github read (groups file is already in place)
   -replication start --wait

Issue: RELENG-1578

Change-Id: Id5bb3c52918c5fcad9e10b32ee14a90eb0b0d3ff
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
5 years agoFix corrupted zip/image upload 57/13957/6
Bengt Thuree [Fri, 18 Jan 2019 18:28:17 +0000 (05:28 +1100)]
Fix corrupted zip/image upload

Change the parameter to requests.posts from files=files,
where files contained the file reference, to
data=file.read().

This was done, since with the previous configuration binary files were
corrupted during upload.

Change-Id: I85c68adc32e031c7f1f101e9d8bfcb5a7acb0670
Issue-Id: RELENG-1591
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoRefactor deploy_maven_file from shell to python 52/13552/11
Eric Ball [Thu, 15 Nov 2018 08:00:24 +0000 (00:00 -0800)]
Refactor deploy_maven_file from shell to python

Refactoring the old shell script into pure python in order to
improve compatibility with Windows.

Issue: RELENG-1378
Change-Id: I4b9efbdd43fdc6605c5e152ca1a2a6b132770d1d
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
5 years agoAdd "nexus release" command 68/13868/7
Eric Ball [Fri, 7 Dec 2018 01:39:43 +0000 (17:39 -0800)]
Add "nexus release" command

This command will release one or more staging repositories in Nexus.
To ensure proper functionality, this commit also includes fixes for
lftools.nexus.cmd.get_credentials and lftools.cli.nexus.list_images.

Issue: RELENG-916
Change-Id: I5fffea04e160004485e09513834825f4c08b220d
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
5 years agopytest-responses requires pytest<4.1.0 96/14096/2
Eric Ball [Tue, 8 Jan 2019 19:36:52 +0000 (11:36 -0800)]
pytest-responses requires pytest<4.1.0

There is currently a PR open for pytest-responses to make it
compatible with 4.1.0, but until this upgrade is complete, the pytest
version needs to be limited.

Change-Id: I84c4ff8a20891d9a96532d1d773b61d0068cc3c6
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
5 years agoMerge "Make URL example more clear"
Bengt Thuree [Sun, 30 Dec 2018 23:21:16 +0000 (23:21 +0000)]
Merge "Make URL example more clear"

5 years agoOnly print docker image list if not empty 74/13974/4
Eric Ball [Thu, 13 Dec 2018 02:06:48 +0000 (18:06 -0800)]
Only print docker image list if not empty

Issue: RELENG-1595
Change-Id: I516f5e6cedbe1e1bbf25f687e473ca3534c0b683
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
5 years agoMake URL example more clear 51/13951/1
Thanh Ha [Wed, 12 Dec 2018 03:17:57 +0000 (11:17 +0800)]
Make URL example more clear

nexus_repo_url is expecting an exact repo URL and not just the
Nexus server address. Make that more clear.

Change-Id: If6f5f45fb85c719096a4691423c1e48f1c36081c
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoMerge "Change default ldap server"
Thanh Ha (zxiiro) [Thu, 6 Dec 2018 01:51:08 +0000 (01:51 +0000)]
Merge "Change default ldap server"

5 years agoMerge "Add filenames from zip file to error string"
Thanh Ha (zxiiro) [Thu, 6 Dec 2018 01:30:36 +0000 (01:30 +0000)]
Merge "Add filenames from zip file to error string"

5 years agoChange default ldap server 43/13843/1
Aric Gardner [Wed, 5 Dec 2018 19:09:37 +0000 (14:09 -0500)]
Change default ldap server

Default ldap server for jenkins.int is
ldaps://aws-us-west-2-lfit-openldap-2.linux-foundation.org

Change-Id: I2e1ac34eb8a124cbcc437e4d5e19addd53d5aee5
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
5 years agoAdd filenames from zip file to error string 27/13827/2
Bengt Thuree [Wed, 5 Dec 2018 06:40:12 +0000 (17:40 +1100)]
Add filenames from zip file to error string

When upload a zipfil failes, we embedd the actual file names
to the error string.

Change-Id: Idaa0386522414772cffe7b295d5f27330e70555d
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoFix UnicodeEncodeError when calling deploy logs 25/13825/1 v0.19.1
Thanh Ha [Wed, 5 Dec 2018 03:09:07 +0000 (11:09 +0800)]
Fix UnicodeEncodeError when calling deploy logs

Call .encode('utf-8') after splitting string as f.write() is expecting
an encoded string to write properly.

Issue: RELENG-1515
Change-Id: I88f70af17222fc7931a07e78eba23d9cc60eb9e7
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agofunction correct was replaced with sync-committers 12/13812/1
Aric Gardner [Tue, 4 Dec 2018 20:38:12 +0000 (15:38 -0500)]
function correct was replaced with sync-committers

This change was not reflected in the autocorrectinfofile
fuction

Change-Id: I7251508fce9afe5ebe7f14c941c278c186208258
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
5 years agoFix autocorrectinfofile 40/13740/1 v0.19.0
Aric Gardner [Thu, 29 Nov 2018 18:53:14 +0000 (13:53 -0500)]
Fix autocorrectinfofile

get_committers was renamed as get-committers

Change awk field seperator due to change in the output of
lftools infofile get-committers

Change-Id: Ic7bba44956907635434f97399f36da71a9207a1d
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
5 years agoOnly skip files start with maven-metadata.xml 54/13654/3
Thanh Ha [Mon, 26 Nov 2018 03:05:51 +0000 (11:05 +0800)]
Only skip files start with maven-metadata.xml

Ensure we are only skipping files that start with maven-metadata.xml*
Having the word "maven-metadata" in our artifact is fine as long as it
is not a Maven 2 repo file which is exactly 'maven-metadata.xml' the
trialing * is to ensure it does not pick up the checksum files too.

Also reword the function docs to be more clear about what it does.

Issue: RELENG-1512
Change-Id: Icc20e4581a74625506ac9a0ff5d5e17b45034ccc
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoPrint the time in seconds it took to upload 53/13653/2
Thanh Ha [Mon, 26 Nov 2018 02:58:08 +0000 (10:58 +0800)]
Print the time in seconds it took to upload

Issue: RELENG-1512
Change-Id: I53760367a57ab28f1afc29418cc45db60773e5b9
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoReword some of the upload output 52/13652/1
Thanh Ha [Mon, 26 Nov 2018 02:44:59 +0000 (10:44 +0800)]
Reword some of the upload output

Printing file and file upload URL on the same line makes the output a
little too busy and hard to read. We should print the upload URL once
before the upload and after only print the files because we can assume
the path will be the same from that point on.

This makes the output quite a bit easier to parse.

Issue: RELENG-1512
Change-Id: I3b8a0ac07b0cad72706a080a8abf3fd79e2dffed
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoOutput files being uploaded to Nexus 34/13634/1
Thanh Ha [Fri, 23 Nov 2018 04:09:44 +0000 (12:09 +0800)]
Output files being uploaded to Nexus

We need the files to be printed on info so that it is clear which
files were pushed to Nexus for a particular build run.

Issue: RELENG-1512
Change-Id: Id3d273b695bd3bf52e38fa6d5647ae59a48a2144
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoSkip archive copying when pattern is blank 33/13633/1
Thanh Ha [Fri, 23 Nov 2018 01:50:48 +0000 (09:50 +0800)]
Skip archive copying when pattern is blank

Blank patterns are invalid patterns to be copying and causes lftools
to incorrectly copy the entire current working directory. This patch
detects that pattern and skips the pattern as a pattern to archive.

Issue: RELENG-1512
Change-Id: Ia7b0ae4a5b1d6996919bfc2a10deea4c9db84f17
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoSave previous dir and restore it before rmtree 20/13620/2
Thanh Ha [Thu, 22 Nov 2018 05:18:24 +0000 (13:18 +0800)]
Save previous dir and restore it before rmtree

Make sure we restore the previous directory state before calling
rmtree.

Issue: RELENG-1512
Change-Id: Ie5ce6b3f4c19608c90a96c5cac63aa7c5ebd808d
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoRefactor deploy_nexus_stage to Python 88/13488/6
Bengt Thuree [Tue, 20 Nov 2018 13:02:20 +0000 (00:02 +1100)]
Refactor deploy_nexus_stage to Python

    This is part of the work to convert the existing SHELL
    scripts to Python scripts.

    * deploy nexus stage

Issue: RELENG-1380
Change-Id: I63e5546fb7455253d14effd638e65d45c36487cb
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoMerge "Refactor of Deploy_Nexus"
Thanh Ha (zxiiro) [Thu, 22 Nov 2018 04:42:08 +0000 (04:42 +0000)]
Merge "Refactor of Deploy_Nexus"

5 years agoRefactor of Deploy_Nexus 87/13487/18
Bengt Thuree [Tue, 13 Nov 2018 11:25:56 +0000 (22:25 +1100)]
Refactor of Deploy_Nexus

Refactoring deploy_nexus as part of the lftools-deploy-refactor from
shell to python.

Issue-ID: RELENG-1379
Issue-ID: RELENG-1374
Change-Id: Ibcf732643649600fa7049d92cdf1ac1072590371
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoAdd release note for upload_maven_file_to_nexus. 00/13600/2
Bengt Thuree [Tue, 20 Nov 2018 05:09:27 +0000 (16:09 +1100)]
Add release note for upload_maven_file_to_nexus.

This patch is just to add a release note to earlier provided and merged
patch.

Issue: RELENG-1373
Change-Id: I108b4253175d989c1d27686b5599d1df32b8e39e
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoRefactor upload_maven_file_to_nexus to Python 56/13556/8
Bengt Thuree [Fri, 16 Nov 2018 03:52:30 +0000 (14:52 +1100)]
Refactor upload_maven_file_to_nexus to Python

Issue: RELENG-1373
Change-Id: Ic896b85ee67a5138814b991d433fc27df9d5be54
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoUse new internal _request_post_file function 49/13549/3
Bengt Thuree [Thu, 15 Nov 2018 09:46:41 +0000 (20:46 +1100)]
Use new internal _request_post_file function

Refactor deploy_nexus_zip to use new internal function
_request_post_file

Change-Id: I42c1d7f1fae299680fb20483e0f14e62b85c5885
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoInternal func _request_post_file function 45/13545/7
Bengt Thuree [Thu, 15 Nov 2018 08:13:44 +0000 (19:13 +1100)]
Internal func _request_post_file function

Added one internal function
_request_post_file : requests.post call to upload a file.

Change-Id: Ie5a3ed3fc7c7c2b553d52b66eb8e7dcd9700eb43
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoAdd defusedxml and glob2 to known 3rdparty 77/13477/2
Thanh Ha [Tue, 13 Nov 2018 02:56:45 +0000 (10:56 +0800)]
Add defusedxml and glob2 to known 3rdparty

Change-Id: I2bca0139f1855340863200e82596a4db55fc7685
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoRefactor Nexus stage repo close|create cmds 36/13236/26
Bengt Thuree [Fri, 2 Nov 2018 00:51:38 +0000 (11:51 +1100)]
Refactor Nexus stage repo close|create cmds

This is part of the work to convert the existing SHELL
scripts to Python scripts.

Nexus stage repo close
Nexus stage repo create

Issue: RELENG-1371
Change-Id: If93fda483813e2acfc3b283b08cd2de90a467629
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoRefactor Deploy Nexus Zip command to Python 86/13286/7
Bengt Thuree [Thu, 1 Nov 2018 23:20:12 +0000 (19:20 -0400)]
Refactor Deploy Nexus Zip command to Python

Refactor the deploy nexus zip command for better portability with
Windows systems.

Issue: RELENG-1372
Change-Id: I58ea1d7703b626f791dcd74e63251c4f3261ca7d
Co-Authored-By: Thanh Ha <thanh.ha@linuxfoundation.org>
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoRefactor deploy-logs cmd to Python 13/13413/7
Thanh Ha [Thu, 8 Nov 2018 02:31:30 +0000 (10:31 +0800)]
Refactor deploy-logs cmd to Python

Issue: RELENG-1377
Change-Id: Ibf58a92fe8800ad77d8be32a123da603c6a36c56
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoRefactor deploy-archives to Python 72/13272/10
Thanh Ha [Thu, 1 Nov 2018 15:52:48 +0000 (11:52 -0400)]
Refactor deploy-archives to Python

CLI wise the function was reworked a bit to allow multiple
(-p pattern) calls for better usability.

Issue: RELENG-1376
Change-Id: I232a284c785b413129f8386fc7e799019da1f754
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoMerge "Add jenkins and xdg to known 3rdparty"
Thanh Ha (zxiiro) [Fri, 9 Nov 2018 17:24:37 +0000 (17:24 +0000)]
Merge "Add jenkins and xdg to known 3rdparty"

5 years agoInternal func _log_error_and_exit & _request_post 88/13288/11
Bengt Thuree [Fri, 2 Nov 2018 00:51:38 +0000 (11:51 +1100)]
Internal func _log_error_and_exit & _request_post

Adds two internal functions
_log_error_and_exist : prints out 0, 1 or 2 lines of text
                       and then exit with sys.exit(1)

_request_post        : This is a wrapper around request.post
                       Handles three exceptions
                         ConnectionError, MissingSchema, InvalidURL

Change-Id: If8a4d20054d958cbfb3ab651eae037a124420bd3
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>
5 years agoAdd jenkins and xdg to known 3rdparty 73/13373/1
Thanh Ha [Wed, 7 Nov 2018 03:49:45 +0000 (11:49 +0800)]
Add jenkins and xdg to known 3rdparty

Change-Id: Ica655586a045d6aaca226e667e1f3eb1edf1dad3
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
5 years agoMerge "Add docker list/delete subcommands for nexus"
Eric Ball [Tue, 6 Nov 2018 21:05:44 +0000 (21:05 +0000)]
Merge "Add docker list/delete subcommands for nexus"

5 years agoMerge changes from topic '13163'
Thanh Ha (zxiiro) [Mon, 5 Nov 2018 03:36:17 +0000 (03:36 +0000)]
Merge changes from topic '13163'

* changes:
  Allow additional credential input methods
  Refactor config sys error handler

5 years agoInternal func _format_url() to verify url fmt 87/13287/3
Bengt Thuree [Thu, 1 Nov 2018 23:48:58 +0000 (19:48 -0400)]
Internal func _format_url() to verify url fmt

Adds a internal function _format_url() to verify url formatting for
domain root.

Change-Id: I0020cf5fb6a848c0fdb2fbc3cd79a48f4c79df4c
Signed-off-by: Bengt Thuree <bthuree@linuxfoundation.org>