lf-rtd-verify
-------------
-ReadTheDocs verify script.
+ReadTheDocs verify script. Installs and runs tox.
+
+:Required parameters:
+
+ :doc-dir: Document directory.
+ :python-version: Python version.
check-info-votes
----------------
:python-version: Version of Python to install into the Tox virtualenv.
Eg. python2 / python3
-lf-infra-tox-sonar
-------------------
-
-Runs Sonar scanning against a Python project.
-
-:Required Parameters:
-
- :java-version: Version of Java to use to run Sonar.
- :mvn-version: Version of Maven to use to run Sonar.
-
lf-tox-install
--------------
:java-version: Version of Java to use for the build. (default: openjdk8)
:pre-build-script: Shell script to execute before the CLM builder.
For example, install prerequisites or move files to the repo root.
- (default: a string with only a comment)
+ (default: a string with a shell comment)
:stream: Keyword used to represent a release code-name.
Often the same as the branch. (default: master)
:submodule-recursive: Whether to checkout submodules recursively.
Python Sonar with Tox
---------------------
-Sonar scans for Python based repos. This job will perform a tox call which
-runs the coverage command to gather tests results. These test results get
-published back into Sonar after running the Sonar goals.
+Sonar scans for Python based repos. This job invokes tox to run tests and
+gather coverage statistics from the test results, then invokes Maven to
+publish the results to a Sonar server.
-To get the Sonar coverage results, tox.ini needs to exist and configured
-with coverage commands to run.
+To get the Sonar coverage results, file tox.ini must exist and contain coverage
+commands to run.
The coverage commands define the code that gets executed by the test suites.
-It does not guarantee that the code tests executed properly, but it will help
-pointing out the code that is not tested at all.
+Checking coverage does not guarantee that the tests execute properly, but it
+identifies code that is not executed by any test.
+
+This job reuses the Sonar builder used in Java/Maven projects which runs maven
+twice. The first invocation does nothing for Python projects, so the job uses
+the goal 'validate' by default. The second invocation publishes results using
+the goal 'sonar:sonar' by default.
For example:
:build-node: The node to run build on.
:jenkins-ssh-credential: Credential to use for SSH. (Generally should
get configured in defaults.yaml)
- :mvn-settings: The name of settings file containing credentials for the project.
+ :mvn-settings: The name of the settings file with credentials for the project.
:Optional parameters:
+ :branch: Git branch, should be master (default: master)
:build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
:build-timeout: Timeout in minutes before aborting build. (default: 60)
:cron: Cron schedule when to trigger the job. This parameter also
supports multiline input via YAML pipe | character in cases where
one may want to provide more than 1 cron timer. (default: H 11 * * *
to run once a day)
+ :disable-job: Whether to disable the job (default: false)
:git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+ :github-url: URL for Github. (default: https://github.com)
:java-version: Version of Java to use for the build. (default: openjdk8)
:mvn-global-settings: The name of the Maven global settings to use for
- Maven configuration. (default: global-settings)
+ :mvn-goals: The Maven goal to run first. (default: validate)
:mvn-version: Version of maven to use. (default: mvn35)
:pre-build-script: Shell script to execute before the Sonar builder.
For example, install prerequisites or move files to the repo root.
- (default: a string with only a comment)
+ (default: a string with a comment)
+ :python-version: Python version (default: python2)
+ :sonar-mvn-goal: The Maven goal to run the Sonar plugin. (default: sonar:sonar)
:stream: Keyword used to represent a release code-name.
Often the same as the branch. (default: master)
:submodule-recursive: Whether to checkout submodules recursively.
:git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
:pre-build-script: Shell script to execute before the Tox builder.
For example, install prerequisites or move files to the repo root.
- (default: a string with only a comment)
+ (default: a string with a shell comment)
:python-version: Version of Python to configure as a base in virtualenv.
(default: python3)
:stream: Keyword representing a release code-name.
.. _lf-global-jjb-release:
#######################
-Self Serve Release Jobs
+Self-Serve Release Jobs
#######################
-Self serve release jobs allow a project to create a releases/ or .releases/ directory and then place a release yaml file in it.
-Jenkins will pick this up and sign the ref extrapolated by log_dir and promote the artifact, whether maven or container.
+Self-serve release jobs allow a project team to direct Jenkins to
+promote a jar file or container image from a staging area to a release
+area. To trigger the action, create a releases/ or .releases/
+directory, add a release yaml file to it, and submit a change set with
+one release yaml file to Gerrit. Upon merge of the change, Jenkins will
+sign the reference extrapolated by log_dir and promote the artifact. The
+expected format of the release yaml file appears in schemas and examples
+below.
+
+The build node for maven and container release jobs must be CentOS,
+which supports the sigul client for accessing a signing server. The
+build node for container release jobs must have Docker installed.
+
+A Jenkins user can also trigger a release job via the "Build with
+parameters" action, removing the need for a release yaml file. The
+user must enter parameters in the same way as a release yaml file,
+except for the special USE_RELEASE_FILE and DRY_RUN check boxes. The
+user must uncheck the USE_RELEASE_FILE check box if the job should
+run with a release file, while passing the required information as
+build parameters. Similarly, the user must uncheck the DRY_RUN check
+box to test the job while skipping repository promotion to Nexus.
+
+The special parameters are as follows::
+
+ GERRIT_BRANCH = master
+ VERSION = 1.0.0
+ LOG_DIR = example-project-maven-stage-master/17/
+ DISTRIBUTION_TYPE = maven
+ USE_RELEASE_FILE = false
+ DRY_RUN = false
-Maven release jobs can also trigger via "Build with parameters" negating the need for a release file.
-The parameters will need to be filled out in the same was as a release file's would, excepting the special
-USE_RELEASE_FILE and DRY_RUN check boxes. The USE_RELEASE_FILE check box will need to be unchecked, if the job
-is expected to run with a release file, while passing the required information as build parameters.
-Similarly, the DRY_RUN check box will need to be unchecked, if the job needs to be tested while skipping
-repository promotion to Nexus.
+.. note::
-The Special Parameters are as follows:
+ The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml).
+ In words, the directory name can be ".releases" or "releases"; the file
+ name can be anything with suffix ".yaml".
-GERRIT_BRANCH = master
-VERSION = 1.0.0
-LOG_DIR = example-project-maven-stage-master/17/
-DISTRIBUTION_TYPE = maven
-USE_RELEASE_FILE = false
-DRY_RUN = false
+The JSON schema for a maven release job appears below.
-.. note::
+.. code-block:: none
- Example of a maven release file:
+ ---
+ $schema: "http://json-schema.org/schema#"
+ $id: "https://github.com/lfit/releng-global-jjb/blob/master/release-schema.yaml"
-.. note::
+ required:
+ - "distribution_type"
+ - "log_dir"
+ - "project"
+ - "version"
+
+ properties:
+ distribution_type:
+ type: "string"
+ log_dir:
+ type: "string"
+ project:
+ type: "string"
+ version:
+ type: "string"
- Release files regex: (releases\/.*\.yaml|\.releases\/.*\.yaml)
- directory can be .releases/ or releases/
- file can be ANYTHING.yaml
+Example of a maven release file:
.. code-block:: bash
- $ cat releases/maven-1.0.0.yaml
- ---
- distribution_type: 'maven'
- version: '1.0.0'
- project: 'example-project'
- log_dir: 'example-project-maven-stage-master/17/'
+ $ cat releases/1.0.0-maven.yaml
+ ---
+ distribution_type: 'maven'
+ version: '1.0.0'
+ project: 'example-project'
+ log_dir: 'example-project-maven-stage-master/17/'
+
+The JSON schema for a container release job appears below.
- Example of a container release file:
+.. code-block:: none
+
+ ---
+ $schema: "http://json-schema.org/schema#"
+ $id: "https://github.com/lfit/releng-global-jjb/blob/master/release-container-schema.yaml"
+
+ required:
+ - "containers"
+ - "distribution_type"
+ - "project"
+ - "container_release_tag"
+ - "ref"
+
+ properties:
+ containers:
+ type: "array"
+ properties:
+ name:
+ type: "string"
+ version:
+ type: "string"
+ additionalProperties: false
+ distribution_type:
+ type: "string"
+ project:
+ type: "string"
+ container_release_tag:
+ type: "string"
+ ref:
+ type: "string"
+
+
+An example of a container release file appears below. The job applies the
+container_release_tag string to all released containers. The job uses the
+per-container version strings to pull images from the container registry.
.. code-block:: bash
- $ cat releases/container-1.0.0.yaml
- ---
- distribution_type: 'container'
- version: '1.0.0'
- project: 'test'
- containers:
- - name: test-backend
- version: 1.0.0-20190806T184921Z
- - name: test-frontend
- version: 1.0.0-20190806T184921Z
+ $ cat releases/1.0.0-container.yaml
+ ---
+ distribution_type: 'container'
+ container_release_tag: '1.0.0'
+ project: 'test'
+ containers:
+ - name: test-backend
+ version: 1.0.0-20190806T184921Z
+ - name: test-frontend
+ version: 1.0.0-20190806T184921Z
.. note::
- Job should be appended under gerrit-maven-stage
- Example of a terse Jenkins job to call global-jjb macro:
+ Job should appear under gerrit-maven-stage
+
+Example of a terse Jenkins job to call the global-jjb macro:
.. code-block:: none
.. note::
- Release Engineers Please follow the setup guide before adding the job definition:
+ Release Engineers: please follow the setup guide below before adding the job definition.
-Setup for LFID Nexus Jenkins and Gerrit:
-========================================
+Setup for LFID, Nexus, Jenkins and Gerrit
+=========================================
LFID
====
Create an ``lfid`` and an ``ssh-key``
``YOUR_RELEASE_USERNAME`` for example: onap-release
+
``YOUR_RELEASE_EMAIL`` for example: collab-it+onap-release@linuxfoundation.org
ssh-key example:
Gerrit
======
-Log into your Gerrit with ``YOU_RELEASE_USERNAME``, upload the publick part of the ``ssh-key`` you created earlier.
-Log out of Gerrit and log in again with your normal account for the next steps.
+Log into your Gerrit with ``YOUR_RELEASE_USERNAME``, upload the public
+part of the ``ssh-key`` you created earlier. Log out of Gerrit and log
+in again with your normal account for the next steps.
-In Gerrit create a new group called ``self-serve-release`` and give it direct push rights via ``All-Projects``
-Add ``YOUR_RELEASE_USERNAME`` to group ``self-serve-release`` and group ``Non-Interactive Users``
+In Gerrit create a new group called ``self-serve-release`` and give it
+direct push rights via ``All-Projects`` Add ``YOUR_RELEASE_USERNAME``
+to group ``self-serve-release`` and group ``Non-Interactive Users``
In All project, grant group self-serve-release the following:
Jenkins
=======
-Add a global credential to Jenkins called ``jenkins-release`` and set the ID: ``'jenkins-release'``
-as its value insert the private half of the ``ssh-key`` that you created for your Gerrit user.
+Add a global credential to Jenkins called ``jenkins-release`` and set
+the ID: ``'jenkins-release'`` as its value insert the private half of
+the ``ssh-key`` that you created for your Gerrit user.
Add Global vars in Jenkins:
Jenkins configure -> Global properties -> Environment variables
-----BEGIN PGP PUBLIC KEY BLOCK-----
-Add or edit the managed file in Jenkins called ``lftoolsini``, appending a nexus section:
-Jenkins Settings -> Managed files -> Add (or edit) -> Custom file
+Add or edit the managed file in Jenkins called ``lftoolsini``,
+appending a nexus section: Jenkins Settings -> Managed files -> Add
+(or edit) -> Custom file
.. code-block:: none
Ci-management
=============
-Upgrade your projects global-jjb if needed
-add this to your global defaults file (eg: jjb/defaults.yaml).
+Upgrade your project's global-jjb if needed, then add the following to
+your global defaults file (e.g., jjb/defaults.yaml).
-.. code-block:: bash
+.. code-block:: none
jenkins-ssh-release-credential: 'jenkins-release'
lf-release
----------
-Release verify and merge jobs are the same except for their scm, trigger, and
-builders definition. This anchor is the common template.
+Release verify and merge jobs are the same except for their scm,
+trigger, and builders definition. This anchor is the common template.
Job Templates
=============
Release Merge
-------------
-:Template Name:
- - {project-name}-release-merge
+:Template Name: {project-name}-release-merge
:Comment Trigger: remerge
Release Verify
------------------
-:Template Names:
- - {project-name}-release-verify
+:Template Name: {project-name}-release-verify
:Comment Trigger: recheck|reverify
ReadTheDocs Verify
------------------
-Verify job which runs a tox build of the docs project
+Verify job which runs tox to test the docs project
:Template Names:
- {project-name}-rtd-verify-{stream}
:gerrit-skip-vote: Skip voting for this job. (default: false)
:git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
:project-pattern: Project to trigger build against. (default: \*\*)
+ :python-version: Python version (default: python2)
:stream: Keyword representing a release code-name.
Often the same as the branch. (default: master)
:submodule-recursive: Whether to checkout submodules recursively.
name: lf-rtd-verify
builders:
- inject:
- properties-content: DOC_DIR={doc-dir}
+ properties-content: |
+ DOC_DIR={doc-dir}
+ PYTHON={python-version}
- shell: !include-raw-escape: ../shell/tox-install.sh
- shell: !include-raw-escape: ../shell/rtd-verify.sh
name: lf-infra-tox-install
builders:
- inject:
- properties-content: "PYTHON_VERSION={python-version}"
+ properties-content: "PYTHON={python-version}"
- shell: !include-raw-escape: ../shell/tox-install.sh
- builder:
- "{branch}"
included-regions: "{obj:github_included_regions}"
-########################
+#########################
# Python Sonar with Tox #
-########################
+#########################
- lf_tox_sonar: &lf_tox_sonar
name: lf-tox_sonar
git-url: "$GIT_URL/$PROJECT"
github-url: "https://github.com"
java-version: openjdk8
- pre-build-script: "# pre-build script goes here"
- python-version: python2
mvn-global-settings: global-settings
+ mvn-goals: validate
mvn-settings: "{mvn-settings}"
mvn-version: mvn35
+ pre-build-script: "# pre-build script goes here"
+ python-version: python2
sonar-mvn-goal: "sonar:sonar"
stream: master
submodule-recursive: true
- lf-provide-maven-settings:
global-settings-file: "{mvn-global-settings}"
settings-file: "{mvn-settings}"
- - lf-infra-tox-sonar:
+ - lf-infra-maven-sonar:
java-version: "{java-version}"
+ mvn-goals: "{mvn-goals}"
mvn-settings: "{mvn-settings}"
mvn-version: "{mvn-version}"
publishers:
- lf-infra-publish
-- builder:
- name: lf-infra-tox-sonar
- # Run a Sonar build with Maven
- builders:
- - lf-maven-install:
- mvn-version: "{mvn-version}"
- - lf-update-java-alternatives:
- java-version: "{java-version}"
- - inject:
- # TODO: Switch this to the sonar wrapper when JJB 2.0 is available
- properties-content: SONAR_HOST_URL=$SONAR_URL
- - shell: !include-raw-escape:
- - ../shell/common-variables.sh
- - ../shell/maven-sonar.sh
- - lf-provide-maven-settings-cleanup
-
- job-template:
name: "{project-name}-tox-sonar"
id: gerrit-tox-sonar
git-url: "$GIT_URL/$PROJECT"
github-url: "https://github.com"
project-pattern: "**"
+ python-version: python2
stream: master
submodule-recursive: true
submodule-timeout: 10
- lf-infra-pre-build
- lf-rtd-verify:
doc-dir: "{doc-dir}"
+ python-version: "{python-version}"
- job-template:
name: "{project-name}-rtd-verify-{stream}"
--- /dev/null
+---
+fixes:
+ - |
+ Remove unused the MAVEN_CENTRAL_URL variable. The self-release job is
+ designed to work with any Nexus repository info published in
+ `staging-repo.txt.gz`, which makes the `MAVEN_CENTRAL_URL` redundant,
+ hence remove the unused variable.
--- /dev/null
+---
+fixes:
+ - |
+ Revise tox-install.sh script to invoke python using environment variable
+ PYTHON instead of hardcoding "python", and remove the pip --quiet flag.
+ Extend the RTD template to pass python-version, default python2.
--- /dev/null
+---
+fixes:
+ - |
+ Use existing builder lf-infra-maven sonar, drop incomplete builder
+ lf-tox-maven-sonar, to gain desired behavior of pushing code analysis
+ results to Sonar. Use trivial goal 'validate' by default.
+ The script maven-sonar.sh calls maven twice, first with build goals
+ and then with Sonar goals. The incomplete builder did not supply the
+ build goals.
--- /dev/null
+---
+fixes:
+ - |
+ Release schema verification needs to happen first before we attempt to assign
+ values to the variables. Validate version only after the schema validation has
+ passed and the variables are assigned.
+ - |
+ Organize variable setup into functions. Maven release files expects different
+ variables than container release files.
+ - |
+ Rename "version" variable in container release files to "container_release_tag"
+ which is a better user friendly name given the fact that container versions are
+ rather called tags. Internally, we still process it as "version" to allow reuse
+ of the tag function.
- "containers"
- "distribution_type"
- "project"
- - "version"
+ - "container_release_tag"
- "ref"
properties:
type: "string"
project:
type: "string"
- version:
+ container_release_tag:
type: "string"
ref:
type: "string"
pip install --user lftools[nexus] jsonschema niet yq
#Functions.
-verify_schema(){
- echo "---> INFO: Verifying $release_file schema."
- lftools schema verify "$release_file" "$RELEASE_SCHEMA"
- # Verify allowed versions
- # Allowed versions are "v#.#.#" or "#.#.#" aka SemVer
- allowed_version_regex="^((v?)([0-9]+)\.([0-9]+)\.([0-9]+))$"
- if [[ ! $VERSION =~ $allowed_version_regex ]]; then
- echo "The version $VERSION is not a semantic valid version"
- echo "Allowed versions are \"v#.#.#\" or \"#.#.#\" aka SemVer"
- echo "See https://semver.org/ for more details on SemVer"
- exit 1
- fi
-}
-tag(){
- if git tag -v "$VERSION"; then
- echo "---> OK: Repo already tagged $VERSION Continuting to release"
- else
-
- echo "---> INFO: Repo has not yet been tagged $VERSION"
- git tag -am "${PROJECT//\//-} $VERSION" "$VERSION"
- sigul --batch -c "$SIGUL_CONFIG" sign-git-tag "$SIGUL_KEY" "$VERSION" < "$SIGUL_PASSWORD"
- echo "Showing latest signature for $PROJECT:"
- gpg --import "$SIGNING_PUBKEY"
- echo "git tag -v $VERSION"
- git tag -v "$VERSION"
- ########## Merge Part ##############
- if [[ "$JOB_NAME" =~ "merge" ]] && [[ "$DRY_RUN" = false ]]; then
- echo "--> INFO: Running merge"
- gerrit_ssh=$(echo "$GERRIT_URL" | awk -F"/" '{print $3}')
- git remote set-url origin ssh://"$RELEASE_USERNAME"@"$gerrit_ssh":29418/"$PROJECT"
- git config user.name "$RELEASE_USERNAME"
- git config user.email "$RELEASE_EMAIL"
- echo -e "Host $gerrit_ssh\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config
- chmod 600 ~/.ssh/config
- git push origin "$VERSION"
+set_variables_common(){
+ echo "---> INFO: Setting all common variables"
+ LOGS_SERVER="${LOGS_SERVER:-None}"
+ if [ "${LOGS_SERVER}" == 'None' ]; then
+ echo "FAILED: log server not found"
+ exit 1
+ fi
+ NEXUS_PATH="${SILO}/${JENKINS_HOSTNAME}/"
+ # Verify if using release file or parameters
+ if $USE_RELEASE_FILE ; then
+ release_files=$(git diff-tree --no-commit-id -r "$GERRIT_PATCHSET_REVISION" --name-only -- "releases/" ".releases/")
+ if (( $(grep -c . <<<"$release_files") > 1 )); then
+ echo "---> INFO: RELEASE FILES ARE AS FOLLOWS: $release_files"
+ echo "---> ERROR: Committing multiple release files in the same commit OR rename/amend of existing files is not supported."
+ exit 1
+ else
+ release_file="$release_files"
+ echo "---> INFO: RELEASE FILE: $release_files"
+ fi
+ else
+ echo "This job is built with parameters, no release file needed. Continuing..."
+ release_file="None"
fi
- fi
-}
-nexus_release(){
- for staging_url in $(zcat "$PATCH_DIR"/staging-repo.txt.gz | awk -e '{print $2}'); do
- # extract the domain name from URL
- NEXUS_URL=$(echo "$staging_url" | sed -e 's|^[^/]*//||' -e 's|/.*$||')
- echo "---> INFO: NEXUS_URL: $NEXUS_URL"
- # extract the staging repo from URL
- STAGING_REPO=${staging_url#*repositories/}
- echo "Running Nexus Verify"
- lftools nexus release -v --server https://"$NEXUS_URL" "$STAGING_REPO"
- echo "Merge will run"
- echo "lftools nexus release --server https://$NEXUS_URL $STAGING_REPO"
- done
-
- #Run the loop twice, to catch errors on either nexus repo
- if [[ "$JOB_NAME" =~ "merge" ]] && [[ "$DRY_RUN" = false ]]; then
- for staging_url in $(zcat "$PATCH_DIR"/staging-repo.txt.gz | awk -e '{print $2}'); do
- NEXUS_URL=$(echo "$staging_url" | sed -e 's|^[^/]*//||' -e 's|/.*$||')
- STAGING_REPO=${staging_url#*repositories/}
- echo "Promoting $STAGING_REPO on $NEXUS_URL."
- lftools nexus release --server https://"$NEXUS_URL" "$STAGING_REPO"
- done
- fi
+ DISTRIBUTION_TYPE="${DISTRIBUTION_TYPE:-None}"
+ if [[ $DISTRIBUTION_TYPE == "None" ]]; then
+ DISTRIBUTION_TYPE="$(niet ".distribution_type" "$release_file")"
+ fi
+
+ PATCH_DIR="$(mktemp -d)"
+
+ # Displaying Release Information (Common variables)
+ echo "RELEASE ENVIRONMENT INFO:"
+ echo "RELEASE_FILE: $release_file"
+ echo "LOGS_SERVER: $LOGS_SERVER"
+ echo "NEXUS_PATH: $NEXUS_PATH"
+ echo "JENKINS_HOSTNAME: $JENKINS_HOSTNAME"
+ echo "SILO: $SILO"
+ echo "PROJECT: $PROJECT"
+ echo "PROJECT-DASHED: ${PROJECT//\//-}"
+ echo "DISTRIBUTION_TYPE: $DISTRIBUTION_TYPE"
}
+set_variables_maven(){
+ VERSION="${VERSION:-None}"
+ if [[ $VERSION == "None" ]]; then
+ VERSION="$(niet ".version" "$release_file")"
+ fi
+ LOG_DIR="${LOG_DIR:-None}"
+ if [[ $LOG_DIR == "None" ]]; then
+ LOG_DIR="$(niet ".log_dir" "$release_file")"
+ fi
+ LOGS_URL="${LOGS_SERVER}/${NEXUS_PATH}${LOG_DIR}"
+ LOGS_URL=${LOGS_URL%/} # strip any trailing '/'
-container_release_file(){
- echo "---> Processing container release"
- local lfn_umbrella
- lfn_umbrella="$(echo "$GERRIT_HOST" | awk -F"." '{print $2}')"
-
-
- for namequoted in $(cat $release_file | yq '.containers[].name'); do
- versionquoted=$(cat $release_file | yq ".containers[] |select(.name=="$namequoted") |.version")
-
- #Remove extra yaml quotes
- name="${namequoted#\"}"
- name="${name%\"}"
- version="${versionquoted#\"}"
- version="${version%\"}"
-
- echo "$name"
- echo "$version"
- echo "---> INFO: Merge will release $name $version as $VERSION"
- #Pull from public, to see if we have already tagged this.
- if docker pull "$DOCKER_REGISTRY":10002/"$lfn_umbrella"/"$name":"$VERSION"; then
- echo "---> OK: $VERSION is already released for image $name, Continuing..."
- else
- echo "---> OK: $VERSION not found in releases, release will be prepared. Continuing..."
- docker pull "$DOCKER_REGISTRY":10001/"$lfn_umbrella"/"$name":"$version"
- container_image_id="$(docker images | grep $name | grep $version | awk '{print $3}')"
- echo "---> INFO: Merge will run the following commands:"
- echo "docker tag $container_image_id $DOCKER_REGISTRY:10002/$lfn_umbrella/$name:$VERSION"
- echo "docker push $DOCKER_REGISTRY:10002/$lfn_umbrella/$name:$VERSION"
- if [[ "$JOB_NAME" =~ "merge" ]]; then
- docker tag "$container_image_id" "$DOCKER_REGISTRY":10002/"$lfn_umbrella"/"$name":"$VERSION"
- docker push "$DOCKER_REGISTRY":10002/"$lfn_umbrella"/"$name":"$VERSION"
- fi
- echo "#########################"
+ # Continuing displaying Release Information (Maven)
+ echo "RELEASE MAVEN INFO:"
+ echo "VERSION: $VERSION"
+ echo "LOG DIR: $LOG_DIR"
+ echo "LOGS URL: $LOGS_URL"
+}
+
+set_variables_container(){
+ VERSION="${VERSION:-None}"
+ if [[ $VERSION == "None" ]]; then
+ VERSION="$(niet ".container_release_tag" "$release_file")"
fi
- done
- ref="$(niet ".ref" "$release_file")"
- echo "---> INFO: Merge will tag ref: $ref"
- git checkout "$ref"
- tag
+ ref="$(niet ".ref" "$release_file")"
+
+ # Continuing displaying Release Information (Container)
+ echo "RELEASE CONTAINER INFO:"
+ echo "CONTAINER_RELEASE_TAG: $VERSION"
+ echo "GERRIT_REF_TO_TAG: $ref"
}
-maven_release_file(){
- echo "---> INFO: wget -P $PATCH_DIR ${LOGS_URL}/staging-repo.txt.gz"
- wget -P "$PATCH_DIR" "${LOGS_URL}/"staging-repo.txt.gz
- pushd "$PATCH_DIR"
- echo "---> INFO: wget ${LOGS_URL}/patches/{${PROJECT//\//-}.bundle,taglist.log.gz}"
- wget "${LOGS_URL}"/patches/{"${PROJECT//\//-}".bundle,taglist.log.gz}
- gunzip taglist.log.gz
- cat "$PATCH_DIR"/taglist.log
- popd
- git checkout "$(awk '{print $NF}' "$PATCH_DIR/taglist.log")"
- git fetch "$PATCH_DIR/${PROJECT//\//-}.bundle"
- git merge --ff-only FETCH_HEAD
- nexus_release
- tag
+verify_schema(){
+ echo "---> INFO: Verifying $release_file schema."
+ lftools schema verify "$release_file" "$RELEASE_SCHEMA"
}
-echo "########### Start Script release-job.sh ###################################"
-echo "---> INFO: Setting all VARS"
+verify_version(){
+ # Verify allowed versions
+ # Allowed versions are "v#.#.#" or "#.#.#" aka SemVer
+ allowed_version_regex="^((v?)([0-9]+)\.([0-9]+)\.([0-9]+))$"
+ if [[ ! $VERSION =~ $allowed_version_regex ]]; then
+ echo "The version $VERSION is not a semantic valid version"
+ echo "Allowed versions are \"v#.#.#\" or \"#.#.#\" aka SemVer"
+ echo "See https://semver.org/ for more details on SemVer"
+ exit 1
+ fi
+}
-LOGS_SERVER="${LOGS_SERVER:-None}"
-MAVEN_CENTRAL_URL="${MAVEN_CENTRAL_URL:-None}"
-if [ "${LOGS_SERVER}" == 'None' ]; then
- echo "FAILED: log server not found"
- exit 1
-fi
+tag(){
+ if git tag -v "$VERSION"; then
+ echo "---> OK: Repo already tagged $VERSION Continuting to release"
+ else
+ echo "---> INFO: Repo has not yet been tagged $VERSION"
+ git tag -am "${PROJECT//\//-} $VERSION" "$VERSION"
+ sigul --batch -c "$SIGUL_CONFIG" sign-git-tag "$SIGUL_KEY" "$VERSION" < "$SIGUL_PASSWORD"
+ echo "Showing latest signature for $PROJECT:"
+ gpg --import "$SIGNING_PUBKEY"
+ echo "git tag -v $VERSION"
+ git tag -v "$VERSION"
-if $USE_RELEASE_FILE ; then
+ ########## Merge Part ##############
+ if [[ "$JOB_NAME" =~ "merge" ]] && [[ "$DRY_RUN" = false ]]; then
+ echo "--> INFO: Running merge"
+ gerrit_ssh=$(echo "$GERRIT_URL" | awk -F"/" '{print $3}')
+ git remote set-url origin ssh://"$RELEASE_USERNAME"@"$gerrit_ssh":29418/"$PROJECT"
+ git config user.name "$RELEASE_USERNAME"
+ git config user.email "$RELEASE_EMAIL"
+ echo -e "Host $gerrit_ssh\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config
+ chmod 600 ~/.ssh/config
+ git push origin "$VERSION"
+ fi
+ fi
+}
- release_files=$(git diff-tree --no-commit-id -r "$GERRIT_PATCHSET_REVISION" --name-only -- "releases/" ".releases/")
- if (( $(grep -c . <<<"$release_files") > 1 )); then
- echo "---> INFO: RELEASE FILES ARE AS FOLLOWS: $release_files"
- echo "---> ERROR: Committing multiple release files in the same commit OR rename/amend of existing files is not supported."
- exit 1
- else
- release_file="$release_files"
- echo "---> INFO: RELEASE FILE: $release_files"
- fi
+nexus_release(){
+ for staging_url in $(zcat "$PATCH_DIR"/staging-repo.txt.gz | awk -e '{print $2}'); do
+ # extract the domain name from URL
+ NEXUS_URL=$(echo "$staging_url" | sed -e 's|^[^/]*//||' -e 's|/.*$||')
+ echo "---> INFO: NEXUS_URL: $NEXUS_URL"
+ # extract the staging repo from URL
+ STAGING_REPO=${staging_url#*repositories/}
+ echo "Running Nexus Verify"
+ lftools nexus release -v --server https://"$NEXUS_URL" "$STAGING_REPO"
+ echo "Merge will run"
+ echo "lftools nexus release --server https://$NEXUS_URL $STAGING_REPO"
+ done
-else
- echo "This job is built with parameters, no release file needed. Continuing..."
- release_file="None"
-fi
+ #Run the loop twice, to catch errors on either nexus repo
+ if [[ "$JOB_NAME" =~ "merge" ]] && [[ "$DRY_RUN" = false ]]; then
+ for staging_url in $(zcat "$PATCH_DIR"/staging-repo.txt.gz | awk -e '{print $2}'); do
+ NEXUS_URL=$(echo "$staging_url" | sed -e 's|^[^/]*//||' -e 's|/.*$||')
+ STAGING_REPO=${staging_url#*repositories/}
+ echo "Promoting $STAGING_REPO on $NEXUS_URL."
+ lftools nexus release --server https://"$NEXUS_URL" "$STAGING_REPO"
+ done
+ fi
+}
+container_release_file(){
+ echo "---> Processing container release"
+ local lfn_umbrella
+ lfn_umbrella="$(echo "$GERRIT_HOST" | awk -F"." '{print $2}')"
-NEXUS_PATH="${SILO}/${JENKINS_HOSTNAME}/"
+ for namequoted in $(cat $release_file | yq '.containers[].name'); do
+ versionquoted=$(cat $release_file | yq ".containers[] |select(.name=="$namequoted") |.version")
-VERSION="${VERSION:-None}"
-if [[ $VERSION == "None" ]]; then
- VERSION="$(niet ".version" "$release_file")"
-fi
+ #Remove extra yaml quotes
+ name="${namequoted#\"}"
+ name="${name%\"}"
+ version="${versionquoted#\"}"
+ version="${version%\"}"
-LOG_DIR="${LOG_DIR:-None}"
-if [[ $LOG_DIR == "None" ]]; then
- LOG_DIR="$(niet ".log_dir" "$release_file")"
-fi
+ echo "$name"
+ echo "$version"
+ echo "---> INFO: Merge will release $name $version as $VERSION"
+ #Pull from public, to see if we have already tagged this.
+ if docker pull "$DOCKER_REGISTRY":10002/"$lfn_umbrella"/"$name":"$VERSION"; then
+ echo "---> OK: $VERSION is already released for image $name, Continuing..."
+ else
+ echo "---> OK: $VERSION not found in releases, release will be prepared. Continuing..."
+ docker pull "$DOCKER_REGISTRY":10001/"$lfn_umbrella"/"$name":"$version"
+ container_image_id="$(docker images | grep $name | grep $version | awk '{print $3}')"
+ echo "---> INFO: Merge will run the following commands:"
+ echo "docker tag $container_image_id $DOCKER_REGISTRY:10002/$lfn_umbrella/$name:$VERSION"
+ echo "docker push $DOCKER_REGISTRY:10002/$lfn_umbrella/$name:$VERSION"
+ if [[ "$JOB_NAME" =~ "merge" ]]; then
+ docker tag "$container_image_id" "$DOCKER_REGISTRY":10002/"$lfn_umbrella"/"$name":"$VERSION"
+ docker push "$DOCKER_REGISTRY":10002/"$lfn_umbrella"/"$name":"$VERSION"
+ fi
+ echo "#########################"
+ fi
+ done
-DISTRIBUTION_TYPE="${DISTRIBUTION_TYPE:-None}"
-if [[ $DISTRIBUTION_TYPE == "None" ]]; then
- DISTRIBUTION_TYPE="$(niet ".distribution_type" "$release_file")"
-fi
+ echo "---> INFO: Merge will tag ref: $ref"
+ git checkout "$ref"
+ tag
+}
+
+maven_release_file(){
+ echo "---> INFO: wget -P $PATCH_DIR ${LOGS_URL}/staging-repo.txt.gz"
+ wget -P "$PATCH_DIR" "${LOGS_URL}/"staging-repo.txt.gz
+ pushd "$PATCH_DIR"
+ echo "---> INFO: wget ${LOGS_URL}/patches/{${PROJECT//\//-}.bundle,taglist.log.gz}"
+ wget "${LOGS_URL}"/patches/{"${PROJECT//\//-}".bundle,taglist.log.gz}
+ gunzip taglist.log.gz
+ cat "$PATCH_DIR"/taglist.log
+ popd
+ git checkout "$(awk '{print $NF}' "$PATCH_DIR/taglist.log")"
+ git fetch "$PATCH_DIR/${PROJECT//\//-}.bundle"
+ git merge --ff-only FETCH_HEAD
+ nexus_release
+ tag
+}
+
+echo "########### Start Script release-job.sh ###################################"
-####
-LOGS_URL="${LOGS_SERVER}/${NEXUS_PATH}${LOG_DIR}"
-LOGS_URL=${LOGS_URL%/} # strip any trailing '/'
-PATCH_DIR="$(mktemp -d)"
-#INFO
-echo "INFO:"
-echo "RELEASE_FILE: $release_file"
-echo "LOGS_SERVER: $LOGS_SERVER"
-echo "NEXUS_PATH: $NEXUS_PATH"
-echo "JENKINS_HOSTNAME: $JENKINS_HOSTNAME"
-echo "SILO: $SILO"
-echo "PROJECT: $PROJECT"
-echo "PROJECT-DASHED: ${PROJECT//\//-}"
-echo "VERSION: $VERSION"
-echo "LOG DIR: $LOG_DIR"
-echo "LOGS URL: $LOGS_URL"
-echo "DISTRIBUTION_TYPE: $DISTRIBUTION_TYPE"
-#Check if this is a container or maven release: release-container-schema.yaml vs release-schema.yaml
-
-#Logic to determine what we are releasing.
+# Check if this is a container or maven release: release-container-schema.yaml vs release-schema.yaml
+# Logic to determine what we are releasing.
##########################################
+
+# Set common environment variables
+set_variables_common
+
if [[ "$DISTRIBUTION_TYPE" == "maven" ]]; then
- wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/release-schema.yaml
- RELEASE_SCHEMA="release-schema.yaml"
- if $USE_RELEASE_FILE ; then
- verify_schema
- fi
- maven_release_file
+ wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/release-schema.yaml
+ RELEASE_SCHEMA="release-schema.yaml"
+ if $USE_RELEASE_FILE ; then
+ verify_schema
+ fi
+ set_variables_maven
+ verify_version
+ maven_release_file
elif [[ "$DISTRIBUTION_TYPE" == "container" ]]; then
- wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/release-container-schema.yaml
- RELEASE_SCHEMA="release-container-schema.yaml"
- verify_schema
- container_release_file
+ wget -q https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/release-container-schema.yaml
+ RELEASE_SCHEMA="release-container-schema.yaml"
+ verify_schema
+ set_variables_container
+ verify_version
+ container_release_file
else
- echo "---> ERROR: distribution_type: $DISTRIBUTION_TYPE not supported"
- echo "Must be maven or container"
- exit 1
+ echo "---> ERROR: distribution_type: $DISTRIBUTION_TYPE not supported"
+ echo "Must be maven or container"
+ exit 1
fi
##########################################
##############################################################################
echo "---> tox-install.sh"
-# Ensure we fail the job if any steps fail.
-# DO NOT set -u as virtualenv's activate script has unbound variables
-set -e -o pipefail
+# Ensure we fail the job if any steps fail or variables are missing.
+# Use -x to show value of $PYTHON in output
+set -eux -o pipefail
# Tox version is pulled in through detox to mitigate version conflict
-python -m pip install --user --quiet --upgrade tox-pyenv
+$PYTHON -m pip install --user --quiet --upgrade tox-pyenv
-echo "----> pip freeze"
-pip freeze
+$PYTHON -m pip freeze
Code Review / releng / global-jjb.git / commitdiff