Lift PyPI templates to Global-JJB from Acumos 32/16632/43
authorLott, Christopher (cl778h) <cl778h@att.com>
Wed, 21 Aug 2019 14:54:25 +0000 (10:54 -0400)
committerLott, Christopher (cl778h) <cl778h@att.com>
Mon, 23 Sep 2019 12:44:06 +0000 (08:44 -0400)
New templates provide features to build and push Python
source and binary distributions to a PyPI repository.
Check version string and push a tag on release.

New job groups:
    {project-name}-gerrit-pypi-jobs
    {project-name}-github-pypi-jobs

New templates:
    {project-name}-pypi-verify-{stream}
    gerrit-pypi-verify
    github-pypi-verify
    {project-name}-pypi-merge-{stream}
    gerrit-pypi-merge
    github-pypi-merge
    {project-name}-pypi-release-verify-{stream}
    gerrit-pypi-release-verify
    github-pypi-release-verify
    {project-name}-pypi-release-merge-{stream}
    gerrit-pypi-release-merge
    github-pypi-release-merge

Change-Id: If74925623f2a5804de8684b9a9933573dfea29a1
Issue-Id: RELENG-2317
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
.jjb-test/lf-python-jobs.yaml
docs/conf.py
docs/jjb/lf-python-jobs.rst
jjb/lf-python-job-groups.yaml
jjb/lf-python-jobs.yaml
releasenotes/notes/lf-pypi-merge-release-jobs-b729bc2331155364.yaml [new file with mode: 0644]
schema/release-pypi-schema.yaml [new file with mode: 0644]
shell/pypi-dist-build.sh [new file with mode: 0644]
shell/pypi-tag-release.sh [new file with mode: 0644]
shell/pypi-upload.sh [new file with mode: 0644]

index 6a10b8f..dd94c64 100644 (file)
@@ -4,6 +4,9 @@
     jobs:
       - "{project-name}-python-jobs"
       - gerrit-tox-sonar
+      - gerrit-pypi-merge
+      - gerrit-pypi-release-verify
+      - gerrit-pypi-release-merge
 
     project-name: gerrit-python
 
@@ -12,6 +15,9 @@
     jobs:
       - "{project-name}-github-python-jobs"
       - github-tox-sonar
+      - github-pypi-merge
+      - github-pypi-release-verify
+      - github-pypi-release-merge
 
     project-name: github-python
 
index 4457e12..ca63745 100644 (file)
@@ -74,7 +74,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = 'lf-releng-global-jjb'
-copyright = '2017, The Linux Foundation'
+copyright = '2019, The Linux Foundation'
 author = 'Linux Foundation Releng'
 
 # The version info for the project you're documenting, acts as replacement for
index 175f0e2..bea44fc 100644 (file)
@@ -7,7 +7,7 @@ Job Groups
 
 .. include:: ../job-groups.rst
 
-Below is a list of Maven job groups:
+Below is a list of Python job groups:
 
 .. literalinclude:: ../../jjb/lf-python-job-groups.yaml
    :language: yaml
@@ -19,31 +19,51 @@ Macros
 lf-infra-clm-python
 -------------------
 
-Run CLM scanning against a Python project.
+Runs CLM scanning against a Python project.
 
 :Required Parameters:
 
     :clm-project-name: Project name in Nexus IQ to send results to.
 
+lf-infra-pypi-tag-release
+-------------------------
+
+Checks the format of the release version string and checks the git
+repository for that tag. In a merge job, continues to tag the repository
+and push the tag to the git server. Also installs supporting tools
+including Sigul and lftools.  Sigul requires a CentOS build node.
+
+lf-infra-pypi-upload
+--------------------
+
+Uploads distribution files from subdirectory "dist" to a PyPI repository
+using a Python virtual enviroment to install required packages. The
+Jenkins server must have a configuration file ".pypirc".
+
+:Required Parameters:
+
+    :pypi-repo: PyPI repository key in .pypirc configuration file;
+        e.g., "staging" or "pypi".
+
 lf-infra-tox-install
 --------------------
 
-Install Tox into a virtualenv.
+Installs Tox into a virtualenv.
 
 :Required Parameters:
 
-    :python-version: Version of Python to install into the Tox virtualenv.
-        Eg. python2 / python3
+    :python-version: Version of Python to invoke the pip install of the tox-pyenv
+        package that creates a virtual environment, either "python2" or "python3".
 
-lf-tox-install
---------------
+lf-infra-tox-run
+----------------
 
-Runs a shell script that installs tox in a Python virtualenv.
+Creates a Tox virtual environment and invokes tox.
 
 :Required Parameters:
 
-    :python-version: Base Python version to use in the virtualenv. For example
-        python2 or python3.
+    :parallel: Boolean. If true use detox (distributed tox);
+        else use regular tox.
 
 
 Job Templates
@@ -55,9 +75,9 @@ Python XC CLM
 CLM scans for Python based repos. This job will call the Nexus IQ CLI
 directly to run the scans.
 
-A new credential named "nexus-iq-xc-clm" needs to exist in the Jenkins credentials.
-The credential should contain the username and password to access Nexus
-IQ Server.
+A new credential named "nexus-iq-xc-clm" needs to exist in the Jenkins
+credentials.  The credential should contain the username and password
+to access Nexus IQ Server.
 
 :Template Names:
 
@@ -97,30 +117,29 @@ IQ Server.
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
     :gerrit_clm_triggers: Override Gerrit Triggers.
-    :gerrit_trigger_file_paths: Override file paths which used to filter which
-        file modifications will trigger a build. Refer to JJB documentation for
-        "file-path" details.
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
         https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
 
 
 Python Sonar with Tox
 ---------------------
 
-Sonar scans for Python based repos. This job invokes tox to run tests and
-gather coverage statistics from the test results, then invokes Maven to
-publish the results to a Sonar server.
+Sonar scans for Python based repos. This job invokes tox to run tests
+and gather coverage statistics from the test results, then invokes
+Maven to publish the results to a Sonar server.
 
-To get the Sonar coverage results, file tox.ini must exist and contain coverage
-commands to run.
+To get the Sonar coverage results, file tox.ini must exist and contain
+coverage commands to run.
 
-The coverage commands define the code that gets executed by the test suites.
-Checking coverage does not guarantee that the tests execute properly, but it
-identifies code that is not executed by any test.
+The coverage commands define the code that gets executed by the test
+suites.  Checking coverage does not guarantee that the tests execute
+properly, but it identifies code that is not executed by any test.
 
-This job reuses the Sonar builder used in Java/Maven projects which runs maven
-twice. The first invocation does nothing for Python projects, so the job uses
-the goal 'validate' by default. The second invocation publishes results using
-the goal 'sonar:sonar' by default.
+This job reuses the Sonar builder used in Java/Maven projects which
+runs maven twice. The first invocation does nothing for Python
+projects, so the job uses the goal 'validate' by default. The second
+invocation publishes results using the goal 'sonar:sonar' by default.
 
 For example:
 
@@ -169,10 +188,13 @@ https://docs.sonarqube.org/display/PLUG/Python+Coverage+Results+Import
     :mvn-global-settings: The name of the Maven global settings to use for
     :mvn-goals: The Maven goal to run first. (default: validate)
     :mvn-version: Version of maven to use. (default: mvn35)
+    :parallel: Boolean indicator for tox to run tests in parallel or series.
+       (default: false, in series)
     :pre-build-script: Shell script to execute before the Sonar builder.
         For example, install prerequisites or move files to the repo root.
-        (default: a string with a comment)
-    :python-version: Python version (default: python2)
+        (default: a string with a shell comment)
+    :python-version: Python version to invoke pip install of tox-pyenv
+        (default: python2)
     :sonar-mvn-goal: The Maven goal to run the Sonar plugin. (default: sonar:sonar)
     :stream: Keyword used to represent a release code-name.
         Often the same as the branch. (default: master)
@@ -183,19 +205,19 @@ https://docs.sonarqube.org/display/PLUG/Python+Coverage+Results+Import
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
     :gerrit_sonar_triggers: Override Gerrit Triggers.
-    :gerrit_trigger_file_paths: Override file paths which used to filter which
-        file modifications will trigger a build. Refer to JJB documentation for
-        "file-path" details.
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
         https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
 
 
 Tox Verify
 ----------
 
-Tox runner to verify a project on creation of a patch set.
-This job is pyenv aware so if the image contains an installation of pyenv
-at /opt/pyenv it will pick it up and run Python tests with the appropriate
-Python versions. This job will set the following pyenv variables before running.
+Tox runner to verify a project on creation of a patch set.  This job
+is pyenv aware so if the image contains an installation of pyenv at
+/opt/pyenv it will pick it up and run Python tests with the
+appropriate Python versions. This job will set the following pyenv
+variables before running.
 
 .. code:: bash
 
@@ -225,8 +247,10 @@ Python versions. This job will set the following pyenv variables before running.
     :pre-build-script: Shell script to execute before the Tox builder.
         For example, install prerequisites or move files to the repo root.
         (default: a string with a shell comment)
-    :python-version: Version of Python to configure as a base in virtualenv.
-        (default: python3)
+    :parallel: Boolean indicator for tox to run tests in parallel or series.
+       (default: false, in series)
+    :python-version: Python version to invoke pip install of tox-pyenv
+        (default: python2)
     :stream: Keyword representing a release code-name.
         Often the same as the branch. (default: master)
     :submodule-recursive: Whether to checkout submodules recursively.
@@ -236,23 +260,23 @@ Python versions. This job will set the following pyenv variables before running.
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
     :tox-dir: Directory containing the project's tox.ini relative to
-        the workspace. Empty works if tox.ini is at project root.
-        (default: '')
+        the workspace. The default uses tox.ini at the project root.
+        (default: '.')
     :tox-envs: Tox environments to run. If blank run everything described
         in tox.ini. (default: '')
-    :gerrit_trigger_file_paths: Override file paths used to filter which
-        file modifications will trigger a build. Refer to JJB documentation for
-        "file-path" details.
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
         https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
 
 
 Tox Merge
 ---------
 
-Tox runner to verify a project after merge of a patch set.
-This job is pyenv aware so if the image contains an installation of pyenv
-at /opt/pyenv it will pick it up and run Python tests with the appropriate
-Python versions. This job will set the following pyenv variables before running.
+Tox runner to verify a project after merge of a patch set.  This job
+is pyenv aware so if the image contains an installation of pyenv at
+/opt/pyenv it will pick it up and run Python tests with the
+appropriate Python versions. This job will set the following pyenv
+variables before running.
 
 .. code:: bash
 
@@ -282,7 +306,352 @@ Python versions. This job will set the following pyenv variables before running.
     :pre-build-script: Shell script to execute before the CLM builder.
         For example, install prerequisites or move files to the repo root.
         (default: a string with only a comment)
-    :python-version: Version of Python to configure as a base in virtualenv.
+    :python-version: Python version to invoke pip install of tox-pyenv
+        (default: python2)
+    :stream: Keyword representing a release code-name.
+        Often the same as the branch. (default: master)
+    :submodule-recursive: Whether to checkout submodules recursively.
+        (default: true)
+    :submodule-timeout: Timeout (in minutes) for checkout operation.
+        (default: 10)
+    :submodule-disable: Disable submodule checkout operation.
+        (default: false)
+    :tox-dir: Directory containing the project's tox.ini relative to
+        the workspace. The default uses tox.ini at the project root.
+        (default: '.')
+    :tox-envs: Tox environments to run. If blank run everything described
+        in tox.ini. (default: '')
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
+        https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+
+
+PyPI Verify
+-----------
+
+Verifies a Python library project on creation of a patch set. Runs tox
+then builds a source distribution and (optionally) a binary
+distribution.  The project repository must have a setup.py file with
+configuration for packaging the component.
+
+The tox runner is pyenv aware so if the image contains an installation
+of pyenv at /opt/pyenv it will pick it up and run Python tests with
+the appropriate Python versions. The tox runner sets the following
+pyenv variables before running.
+
+.. code:: bash
+
+   export PYENV_ROOT="/opt/pyenv"
+   export PATH="$PYENV_ROOT/bin:$PATH"
+
+:Template Names:
+
+    - {project-name}-pypi-verify-{stream}
+    - gerrit-pypi-verify
+    - github-pypi-verify
+
+:Comment Trigger: recheck
+
+:Required Parameters:
+
+    :build-node: The node to run the build on.
+    :jenkins-ssh-credential: Credential to use for SSH. (Generally set
+        in defaults.yaml)
+
+:Optional Parameters:
+
+    :branch: The branch to build against. (default: master)
+    :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+    :build-timeout: Timeout in minutes before aborting build. (default: 15)
+    :dist-binary: Whether to build a binary wheel distribution. (default: true)
+    :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+    :parallel: Boolean indicator for tox to run tests in parallel or series.
+       (default: false, in series)
+    :pre-build-script: Shell script to execute before the tox builder. For
+        example, install system prerequisites. (default: a shell comment)
+    :python-version: Python version to invoke pip install of tox-pyenv
+        (default: python3)
+    :stream: Keyword representing a release code-name.
+        Often the same as the branch. (default: master)
+    :submodule-recursive: Whether to checkout submodules recursively.
+        (default: true)
+    :submodule-timeout: Timeout (in minutes) for checkout operation.
+        (default: 10)
+    :submodule-disable: Disable submodule checkout operation.
+        (default: false)
+    :tox-dir: Directory containing the project's tox.ini relative to
+        the workspace. The default uses tox.ini at the project root.
+        (default: '.')
+    :tox-envs: Tox environments to run. If blank run everything described
+        in tox.ini. (default: '')
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
+        https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+
+
+PyPI Merge
+----------
+
+Creates and uploads distribution files on merge of a patch set.  Runs
+tox, builds a source distribution and (optionally) a binary
+distribution, and uploads the distribution(s) to a PyPI repository.
+This job should be configured to use a staging PyPI repository like
+testpypi.python.org, not a public release area like the global PyPI
+repository. Like the verify job, this requires a setup.py file for
+packaging the component.
+
+The tox runner is pyenv aware so if the image contains an installation
+of pyenv at /opt/pyenv it will pick it up and run Python tests with
+the appropriate Python versions. The tox runner sets the following
+pyenv variables before running.
+
+.. code:: bash
+
+   export PYENV_ROOT="/opt/pyenv"
+   export PATH="$PYENV_ROOT/bin:$PATH"
+
+
+Requires a .pypirc configuration file in the Jenkins builder home
+directory, an example appears next.
+
+.. code-block:: bash
+
+    [distutils] # this tells distutils what package indexes you can push to
+    index-servers =
+    staging
+    pypi
+
+    [staging]
+    repository: https://testpypi.python.org/pypi
+    username: your_username
+    password: your_password
+
+    [pypi]
+    repository: https://pypi.python.org/pypi
+    username: your_username
+    password: your_password
+
+
+:Template Names:
+
+    - {project-name}-pypi-merge-{stream}
+    - gerrit-pypi-merge
+    - github-pypi-merge
+
+:Comment Trigger: pypi-remerge
+
+:Required Parameters:
+
+    :build-node: The node to run the build on.
+    :jenkins-ssh-credential: Credential to use for SSH. (Generally set
+        in defaults.yaml)
+
+:Optional Parameters:
+
+    :branch: The branch to build against. (default: master)
+    :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+    :build-timeout: Timeout in minutes before aborting build. (default: 15)
+    :dist-binary: Whether to build a binary wheel distribution. (default: true)
+    :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+    :parallel: Boolean indicator for tox to run tests in parallel or series.
+       (default: false, in series)
+    :pre-build-script: Shell script to execute before the tox builder. For
+        example, install system prerequisites. (default: a shell comment)
+    :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
+        Merge jobs should use a server like testpypi.python.org.  (default: staging)
+    :python-version: Python version to invoke pip install of tox-pyenv
+        (default: python3)
+    :stream: Keyword representing a release code-name.
+        Often the same as the branch. (default: master)
+    :submodule-recursive: Whether to checkout submodules recursively.
+        (default: true)
+    :submodule-timeout: Timeout (in minutes) for checkout operation.
+        (default: 10)
+    :submodule-disable: Disable submodule checkout operation.
+        (default: false)
+    :tox-dir: Directory containing the project's tox.ini relative to
+        the workspace. The default uses tox.ini at the project root.
+        (default: '.')
+    :tox-envs: Tox environments to run. If blank run everything described
+        in tox.ini. (default: '')
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
+        https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+
+
+PyPI Release Verify
+-------------------
+
+Verifies a Python library project on creation of a patch set with a
+release yaml file. Runs tox, builds source and (optionally) binary
+distributions, checks the format of the version string, checks that
+the distribution file names contain the release version string, and
+checks that no tag exists in the code repository for the release
+version.
+
+To initiate the release process, create a releases/ or .releases/
+directory at the root of the project repository, add one release yaml
+file to it, and submit a change set with that release yaml file. A
+schema and and an example for the release yaml file appear below.  The
+version in the release yaml file must be a valid Semantic Versioning
+(SemVer) string, matching either the pattern "v#.#.#" or "#.#.#" where
+"#" is one or more digits.
+
+This job is similar to the PyPI verify job, but is only triggered by a
+patch set with a release yaml file.
+
+The build node for PyPI release verify jobs must be CentOS, which
+supports the sigul client for accessing a signing server.
+
+.. note::
+
+   The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml).
+   In words, the directory name can be ".releases" or "releases"; the file
+   name can be anything with suffix ".yaml".
+
+The JSON schema for a pypi release file appears below.
+
+.. code-block:: none
+
+    ---
+    $schema: "http://json-schema.org/schema#"
+    $id: "https://github.com/lfit/releng-global-jjb/blob/master/release-pypi-schema.yaml"
+
+    required:
+      - "distribution_type"
+      - "project"
+      - "version"
+
+    properties:
+      distribution_type:
+        type: "string"
+      project:
+        type: "string"
+      version:
+        type: "string"
+
+
+An example of a pypi release file appears below.
+
+.. code-block:: none
+
+    $ cat releases/1.0.0-pypi.yaml
+    ---
+    distribution_type: pypi
+    version: 1.0.0
+    project: 'example-project'
+
+
+:Template Names:
+
+    - {project-name}-pypi-release-verify-{stream}
+    - gerrit-pypi-release-verify
+    - github-pypi-release-verify
+
+:Required Parameters:
+
+    :build-node: The node to run build on, which must be Centos.
+    :jenkins-ssh-credential: Credential to use for SSH. (Generally set
+        in defaults.yaml)
+
+:Optional Parameters:
+
+    :branch: The branch to build against. (default: master)
+    :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+    :build-timeout: Timeout in minutes before aborting build. (default: 15)
+    :dist-binary: Whether to build a binary wheel distribution. (default: true)
+    :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+    :parallel: Boolean indicator for tox to run tests in parallel or series.
+       (default: false, in series)
+    :pre-build-script: Shell script to execute before the tox builder.
+        For example, install prerequisites or move files to the repo root.
+        (default: a string with a shell comment)
+    :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
+        Release jobs should use a server like pypy.org.  (default: pypi)
+    :python-version: Python version to invoke pip install of tox-pyenv
+        (default: python3)
+    :stream: Keyword representing a release code-name.
+        Often the same as the branch. (default: master)
+    :submodule-recursive: Whether to checkout submodules recursively.
+        (default: true)
+    :submodule-timeout: Timeout (in minutes) for checkout operation.
+        (default: 10)
+    :submodule-disable: Disable submodule checkout operation.
+        (default: false)
+    :tox-dir: Directory containing the project's tox.ini relative to
+        the workspace. The default uses tox.ini at the project root.
+        (default: '.')
+    :tox-envs: Tox environments to run. If blank run everything described
+        in tox.ini. (default: '')
+    :use-release-file: Whether to use the release file. (default: true)
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
+        https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+
+
+PyPI Release Merge
+------------------
+
+Publishes a Python library on merge of a patch set with a release yaml
+file.  Runs tox, builds source and (optionally) binary distributions,
+checks the format of the version string, checks that the distribution
+file names contain the release version string, checks that no tag
+exists in the code repository for the release version, tags the code
+repository with the release version, pushes the tag to the git server,
+and uploads distributions to a PyPI repository.
+
+This job is similar to the PyPI merge job, but is only triggered by
+merge of a release yaml file and checks the version and tag before
+uploading to a public repository such as PyPI.
+
+See the PyPI Release Verify job above for documentation of the release
+yaml file format.
+
+The build node for PyPI release merge jobs must be CentOS, which
+supports the sigul client for accessing a signing server.
+
+A Jenkins user can also trigger this release job via the "Build with
+parameters" action, removing the need to merge a release yaml file.
+The user must enter parameters in the same way as a release yaml file,
+except for the special USE_RELEASE_FILE and DRY_RUN check boxes. The
+user must uncheck the USE_RELEASE_FILE check box if the job should run
+with a release file, while passing the required information as build
+parameters. Similarly, the user must uncheck the DRY_RUN check box to
+test the job while skipping upload of files to a repository.
+
+The special parameters are as follows::
+
+    VERSION = 1.0.0
+    USE_RELEASE_FILE = false
+    DRY_RUN = false
+
+:Template Names:
+
+    - {project-name}-pypi-release-merge-{stream}
+    - gerrit-pypi-release-merge
+    - github-pypi-release-merge
+
+:Required Parameters:
+
+    :build-node: The node to run build on, which must be Centos.
+    :jenkins-ssh-credential: Credential to use for SSH. (Generally set
+        in defaults.yaml)
+
+:Optional Parameters:
+
+    :branch: The branch to build against. (default: master)
+    :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+    :build-timeout: Timeout in minutes before aborting build. (default: 15)
+    :dist-binary: Whether to build a binary wheel distribution. (default: true)
+    :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+    :parallel: Boolean indicator for tox to run tests in parallel or series.
+       (default: false, in series)
+    :pre-build-script: Shell script to execute before the tox builder.
+        For example, install prerequisites or move files to the repo root.
+        (default: a string with a shell comment)
+    :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
+        Release jobs should use a server like pypy.org.  (default: pypi)
+    :python-version: Python version to invoke pip install of tox-pyenv
         (default: python3)
     :stream: Keyword representing a release code-name.
         Often the same as the branch. (default: master)
@@ -293,11 +662,11 @@ Python versions. This job will set the following pyenv variables before running.
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
     :tox-dir: Directory containing the project's tox.ini relative to
-        the workspace. Empty works if tox.ini is at project root.
-        (default: '')
+        the workspace. The default uses tox.ini at the project root.
+        (default: '.')
     :tox-envs: Tox environments to run. If blank run everything described
         in tox.ini. (default: '')
-    :gerrit_trigger_file_paths: Override file paths used to filter which
-        file modifications will trigger a build. Refer to JJB documentation for
-        "file-path" details.
+    :use-release-file: Whether to use the release file. (default: true)
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
         https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
index 59066f9..60ecc66 100644 (file)
@@ -3,7 +3,7 @@
     name: "{project-name}-python-jobs"
 
     # This job group contains all the recommended jobs that should be deployed
-    # for any project ci that is using Gerrit.
+    # for a Gerrit-based Python project to verify commits using tox.
 
     jobs:
       - gerrit-python-xc-clm
     name: "{project-name}-github-python-jobs"
 
     # This job group contains all the recommended jobs that should be deployed
-    # for any project ci that is using GitHub.
+    # for a Github-based Python project to verify commits using tox.
 
     jobs:
       - github-python-xc-clm
       - github-tox-verify
       - github-tox-merge
+
+- job-group:
+    name: "{project-name}-gerrit-pypi-jobs"
+
+    # This job group contains all the recommended jobs that should be deployed for
+    # a Gerrit-based Python project to test, build and deploy a library to PyPI.
+
+    jobs:
+      - gerrit-pypi-verify
+      - gerrit-pypi-merge
+      - gerrit-pypi-release-verify
+      - gerrit-pypi-release-merge
+
+- job-group:
+    name: "{project-name}-github-pypi-jobs"
+
+    # This job group contains all the recommended jobs that should be deployed for
+    # a Github-based Python project to test, build and deploy a library to PyPI.
+
+    jobs:
+      - github-pypi-verify
+      - github-pypi-merge
+      - github-pypi-release-verify
+      - github-pypi-release-merge
index 451eb63..de38fa1 100644 (file)
@@ -3,6 +3,41 @@
 # Macros #
 ##########
 
+- builder:
+    name: lf-infra-clm-python
+    builders:
+      - inject:
+          properties-content: "CLM_PROJECT_NAME={clm-project-name}"
+      - shell: !include-raw-escape: ../shell/nexus-iq-cli.sh
+
+- builder:
+    name: lf-infra-pypi-tag-release
+    builders:
+      - config-file-provider:
+          files:
+            - file-id: sigul-config
+              variable: SIGUL_CONFIG
+            - file-id: sigul-password
+              variable: SIGUL_PASSWORD
+            - file-id: sigul-pki
+              variable: SIGUL_PKI
+            - file-id: signing-pubkey
+              variable: SIGNING_PUBKEY
+      - shell: !include-raw: ../shell/sigul-configuration.sh
+      - shell: !include-raw: ../shell/sigul-install.sh
+      - shell: !include-raw: ../shell/pypi-tag-release.sh
+
+- builder:
+    name: lf-infra-pypi-upload
+    builders:
+      - config-file-provider:
+          files:
+            - file-id: pypirc
+              target: "$HOME/.pypirc"
+      - inject:
+          properties-content: "REPOSITORY={pypi-repo}"
+      - shell: !include-raw-escape: ../shell/pypi-upload.sh
+
 - builder:
     name: lf-infra-tox-install
     builders:
       - shell: !include-raw-escape: ../shell/tox-install.sh
 
 - builder:
-    name: lf-infra-clm-python
+    name: lf-infra-tox-run
     builders:
       - inject:
-          properties-content: "CLM_PROJECT_NAME={clm-project-name}"
-      - shell: !include-raw-escape:
-          - ../shell/nexus-iq-cli.sh
+          properties-content: "PARALLEL={parallel}"
+      - shell: !include-raw-escape: ../shell/tox-run.sh
 
 ####################
 # COMMON FUNCTIONS #
     mvn-goals: validate
     mvn-settings: "{mvn-settings}"
     mvn-version: mvn35
+    parallel: true
     pre-build-script: "# pre-build script goes here"
     python-version: python2
     sonar-mvn-goal: "sonar:sonar"
       - lf-infra-tox-install:
           python-version: "{python-version}"
       - shell: "{pre-build-script}"
-      - shell: !include-raw-escape: ../shell/tox-run.sh
+      - lf-infra-tox-run:
+          parallel: "{parallel}"
       - lf-provide-maven-settings:
           global-settings-file: "{mvn-global-settings}"
           settings-file: "{mvn-settings}"
     disable-job: false
     git-url: "$GIT_URL/$GERRIT_PROJECT"
     github-url: "https://github.com"
-    parallel: true
+    parallel: false
     pre-build-script: "# pre-build script goes here"
     python-version: python2
     stream: master
     submodule-recursive: true
     submodule-timeout: 10
     submodule-disable: false
-    tox-dir: ""
+    tox-dir: "."
     tox-envs: ""
 
     gerrit_trigger_file_paths:
       - lf-infra-tox-parameters:
           tox-dir: "{tox-dir}"
           tox-envs: "{tox-envs}"
-      - bool:
-          name: PARALLEL
-          default: "{parallel}"
-          description: Tox test type used to configure serial or parallel testing.
 
     wrappers:
       - lf-infra-wrappers:
       - lf-infra-tox-install:
           python-version: "{python-version}"
       - shell: "{pre-build-script}"
-      - shell: !include-raw-escape: ../shell/tox-run.sh
+      - lf-infra-tox-run:
+          parallel: "{parallel}"
 
     publishers:
       - lf-infra-publish
           white-list-target-branches:
             - "{branch}"
           included-regions: "{obj:github_included_regions}"
+
+########
+# PyPI #
+########
+
+- lf_pypi_common: &lf_pypi_common
+    name: lf-pypi-common
+
+    ######################
+    # Default parameters #
+    ######################
+
+    branch: master
+    build-days-to-keep: 7
+    build-timeout: 15
+    disable-job: false
+    dist-binary: true
+    git-url: "$GIT_URL/$GERRIT_PROJECT"
+    github-url: "https://github.com"
+    parallel: false
+    pre-build-script: "# pre-build script goes here"
+    python-version: python3
+    stream: master
+    submodule-recursive: true
+    submodule-timeout: 10
+    submodule-disable: false
+    tox-dir: "."
+    tox-envs: ""
+
+    gerrit_trigger_file_paths:
+      - compare-type: ANT
+        pattern: ".*"
+
+    # github_included_regions MUST match gerrit_trigger_file_paths
+    github_included_regions:
+      - ".*"
+
+    parameters:
+      - lf-infra-parameters:
+          project: "{project}"
+          branch: "{branch}"
+          stream: "{stream}"
+      - lf-infra-tox-parameters:
+          tox-dir: "{tox-dir}"
+          tox-envs: "{tox-envs}"
+      - bool:
+          name: BUILD_BDIST_WHEEL
+          default: "{dist-binary}"
+          description: "Set to True to build a wheel"
+
+- lf_pypi_verify_builders: &lf_pypi_verify_builders
+    name: lf-pypi-verify-builders
+
+    builders:
+      - lf-infra-pre-build
+      - lf-infra-tox-install:
+          python-version: "{python-version}"
+      - shell: "{pre-build-script}"
+      - lf-infra-tox-run:
+          parallel: "{parallel}"
+      - shell: !include-raw-escape: ../shell/pypi-dist-build.sh
+
+- lf_pypi_merge_builders: &lf_pypi_merge_builders
+    name: lf-pypi-merge-builders
+
+    builders:
+      - lf-infra-pre-build
+      - lf-infra-tox-install:
+          python-version: "{python-version}"
+      - shell: "{pre-build-script}"
+      - lf-infra-tox-run:
+          parallel: "{parallel}"
+      - shell: !include-raw-escape: ../shell/pypi-dist-build.sh
+      - lf-infra-pypi-upload:
+          pypi-repo: "{pypi-repo}"
+
+- lf_pypi_release_verify_builders: &lf_pypi_release_verify_builders
+    name: lf-pypi-release-verify-builders
+
+    builders:
+      - lf-infra-pre-build
+      - lf-infra-tox-install:
+          python-version: "{python-version}"
+      - shell: "{pre-build-script}"
+      - lf-infra-tox-run:
+          parallel: "{parallel}"
+      - shell: !include-raw-escape: ../shell/pypi-dist-build.sh
+      - lf-infra-pypi-tag-release
+
+- lf_pypi_release_merge_builders: &lf_pypi_release_merge_builders
+    name: lf-pypi-release-merge-builders
+
+    builders:
+      - lf-infra-pre-build
+      - lf-infra-tox-install:
+          python-version: "{python-version}"
+      - shell: "{pre-build-script}"
+      - lf-infra-tox-run:
+          parallel: "{parallel}"
+      - shell: !include-raw-escape: ../shell/pypi-dist-build.sh
+      - lf-infra-pypi-tag-release
+      - lf-infra-pypi-upload:
+          pypi-repo: "{pypi-repo}"
+
+- job-template:
+    name: "{project-name}-pypi-verify-{stream}"
+    id: gerrit-pypi-verify
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_verify_builders
+
+    gerrit_verify_triggers:
+      - patchset-created-event:
+          exclude-drafts: true
+          exclude-trivial-rebase: false
+          exclude-no-code-change: false
+      - draft-published-event
+      - comment-added-contains-event:
+          comment-contains-value: '^Patch Set\s+\d+:\s+(recheck|reverify)\s*$'
+
+    scm:
+      - lf-infra-gerrit-scm:
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+          git-url: "{git-url}"
+          refspec: "$GERRIT_REFSPEC"
+          branch: "$GERRIT_BRANCH"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: gerrit
+
+    triggers:
+      - gerrit:
+          server-name: "{gerrit-server-name}"
+          trigger-on: "{obj:gerrit_verify_triggers}"
+          projects:
+            - project-compare-type: ANT
+              project-pattern: "{project}"
+              branches:
+                - branch-compare-type: ANT
+                  branch-pattern: "**/{branch}"
+              file-paths: "{obj:gerrit_trigger_file_paths}"
+
+- job-template:
+    name: "{project-name}-pypi-verify-{stream}"
+    id: github-pypi-verify
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_verify_builders
+
+    properties:
+      - github:
+          url: "{github-url}/{github-org}/{project}"
+
+    scm:
+      - lf-infra-github-scm:
+          url: "{git-clone-url}{github-org}/{project}"
+          refspec: "+refs/pull/*:refs/remotes/origin/pr/*"
+          branch: "$sha1"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: default
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+
+    triggers:
+      - github-pull-request:
+          trigger-phrase: "^(recheck|reverify)$"
+          only-trigger-phrase: false
+          status-context: "PyPI Verify"
+          permit-all: true
+          github-hooks: true
+          white-list-target-branches:
+            - "{branch}"
+          included-regions: "{obj:github_included_regions}"
+
+- job-template:
+    name: "{project-name}-pypi-merge-{stream}"
+    id: gerrit-pypi-merge
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_merge_builders
+
+    pypi-repo: staging
+
+    gerrit_merge_triggers:
+      - change-merged-event
+      - comment-added-contains-event:
+          comment-contains-value: '^Patch Set\s+\d+:\s+remerge\s*$'
+
+    scm:
+      - lf-infra-gerrit-scm:
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+          git-url: "{git-url}"
+          refspec: "$GERRIT_REFSPEC"
+          branch: "$GERRIT_BRANCH"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: gerrit
+
+    triggers:
+      - gerrit:
+          server-name: "{gerrit-server-name}"
+          trigger-on: "{obj:gerrit_merge_triggers}"
+          projects:
+            - project-compare-type: ANT
+              project-pattern: "{project}"
+              branches:
+                - branch-compare-type: ANT
+                  branch-pattern: "**/{branch}"
+              file-paths: "{obj:gerrit_trigger_file_paths}"
+
+- job-template:
+    name: "{project-name}-pypi-merge-{stream}"
+    id: github-pypi-merge
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_merge_builders
+
+    pypi-repo: staging
+
+    properties:
+      - github:
+          url: "{github-url}/{github-org}/{project}"
+
+    scm:
+      - lf-infra-github-scm:
+          url: "{git-clone-url}{github-org}/{project}"
+          refspec: ""
+          branch: "refs/heads/{branch}"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: default
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+
+    triggers:
+      - github-pull-request:
+          trigger-phrase: "^remerge$"
+          only-trigger-phrase: false
+          status-context: "Merge"
+          permit-all: true
+          github-hooks: true
+          org-list:
+            - "{github-org}"
+          white-list: "{obj:github_pr_whitelist}"
+          admin-list: "{obj:github_pr_admin_list}"
+          white-list-target-branches:
+            - "{branch}"
+          included-regions: "{obj:github_included_regions}"
+
+- lf_pypi_release_common: &lf_pypi_release_common
+    name: lf-pypi-release-common
+
+    dist-binary: true
+    pypi-repo: pypi
+    use-release-file: true
+
+    gerrit_trigger_file_paths:
+      - compare-type: REG_EXP
+        pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)'
+
+    # github_included_regions MUST match gerrit_trigger_file_paths
+    github_included_regions:
+      - 'releases\/.*\.yaml'
+      - '.releases\/.*\.yaml'
+
+    parameters:
+      - lf-infra-parameters:
+          project: "{project}"
+          branch: "{branch}"
+          stream: "{stream}"
+      - lf-infra-tox-parameters:
+          tox-dir: "{tox-dir}"
+          tox-envs: "{tox-envs}"
+      - bool:
+          name: BUILD_BDIST_WHEEL
+          default: "{dist-binary}"
+          description: "Set to True to build a wheel"
+      - string:
+          name: VERSION
+          default: ""
+          description: "This is the version, example: 1.0.0"
+      - bool:
+          name: USE_RELEASE_FILE
+          default: "{use-release-file}"
+          description: "Set to False for job built with parameters"
+      - bool:
+          name: DRY_RUN
+          default: false
+          description: |
+            If DRY_RUN is enabled artifacts are not published.
+
+- job-template:
+    name: "{project-name}-pypi-release-verify-{stream}"
+    id: gerrit-pypi-release-verify
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_release_common
+    <<: *lf_pypi_release_verify_builders
+
+    gerrit_verify_triggers:
+      - patchset-created-event:
+          exclude-drafts: true
+          exclude-trivial-rebase: false
+          exclude-no-code-change: false
+      - draft-published-event
+      - comment-added-contains-event:
+          comment-contains-value: '^Patch Set\s+\d+:\s+(recheck|reverify)\s*$'
+
+    scm:
+      - lf-infra-gerrit-scm:
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+          git-url: "{git-url}"
+          refspec: "$GERRIT_REFSPEC"
+          branch: "$GERRIT_BRANCH"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: gerrit
+
+    triggers:
+      - gerrit:
+          server-name: "{gerrit-server-name}"
+          trigger-on: "{obj:gerrit_verify_triggers}"
+          projects:
+            - project-compare-type: "ANT"
+              project-pattern: "{project}"
+              branches:
+                - branch-compare-type: "ANT"
+                  branch-pattern: "**"
+              file-paths: "{obj:gerrit_trigger_file_paths}"
+
+- job-template:
+    name: "{project-name}-pypi-release-verify-{stream}"
+    id: github-pypi-release-verify
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_release_common
+    <<: *lf_pypi_release_verify_builders
+
+    properties:
+      - github:
+          url: "{github-url}/{github-org}/{project}"
+
+    scm:
+      - lf-infra-github-scm:
+          url: "{git-clone-url}{github-org}/{project}"
+          refspec: ""
+          branch: "refs/heads/{branch}"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: default
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+
+    triggers:
+      - github-pull-request:
+          trigger-phrase: "^(recheck|reverify)$"
+          only-trigger-phrase: false
+          status-context: "PyPI Release Verify"
+          permit-all: true
+          github-hooks: true
+          white-list-target-branches:
+            - "{branch}"
+          included-regions: "{obj:github_included_regions}"
+
+- job-template:
+    name: "{project-name}-pypi-release-merge-{stream}"
+    id: gerrit-pypi-release-merge
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_release_common
+    <<: *lf_pypi_release_merge_builders
+
+    gerrit_release_triggers:
+      - change-merged-event
+
+    scm:
+      - lf-infra-gerrit-scm:
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+          git-url: "{git-url}"
+          refspec: "$GERRIT_REFSPEC"
+          branch: "$GERRIT_BRANCH"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: gerrit
+
+    triggers:
+      - gerrit:
+          server-name: "{gerrit-server-name}"
+          trigger-on: "{obj:gerrit_release_triggers}"
+          projects:
+            - project-compare-type: "ANT"
+              project-pattern: "{project}"
+              branches:
+                - branch-compare-type: "ANT"
+                  branch-pattern: "**"
+              file-paths: "{obj:gerrit_trigger_file_paths}"
+
+- job-template:
+    name: "{project-name}-pypi-release-merge-{stream}"
+    id: github-pypi-release-merge
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_release_common
+    <<: *lf_pypi_release_merge_builders
+
+    properties:
+      - github:
+          url: "{github-url}/{github-org}/{project}"
+
+    scm:
+      - lf-infra-github-scm:
+          url: "{git-clone-url}{github-org}/{project}"
+          refspec: ""
+          branch: "refs/heads/{branch}"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: default
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+
+    triggers:
+      - github-pull-request:
+          trigger-phrase: "^(remerge)$"
+          only-trigger-phrase: false
+          status-context: "PyPI Release Merge"
+          permit-all: true
+          github-hooks: true
+          white-list-target-branches:
+            - "{branch}"
+          included-regions: "{obj:github_included_regions}"
diff --git a/releasenotes/notes/lf-pypi-merge-release-jobs-b729bc2331155364.yaml b/releasenotes/notes/lf-pypi-merge-release-jobs-b729bc2331155364.yaml
new file mode 100644 (file)
index 0000000..34947a2
--- /dev/null
@@ -0,0 +1,13 @@
+---
+features:
+  - |
+    New templates to build and push Python source and binary distributions
+    to a PyPI server. Includes:
+    ``{project-name}-pypi-verify-{stream}``, ``gerrit-pypi-verify``,
+    ``github-pypi-verify``,
+    ``{project-name}-pypi-merge-{stream}``, ``gerrit-pypi-merge``,
+    ``github-pypi-merge``,
+    ``{project-name}-pypi-release-verify-{stream}``,
+    ``gerrit-pypi-release-verify``, ``github-pypi-release-verify``,
+    ``{project-name}-pypi-release-merge-{stream}``,
+    ``gerrit-pypi-release-merge``, ``github-pypi-release-merge``,
diff --git a/schema/release-pypi-schema.yaml b/schema/release-pypi-schema.yaml
new file mode 100644 (file)
index 0000000..e56eed6
--- /dev/null
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: EPL-1.0
+##############################################################################
+# Copyright (c) 2019 The Linux Foundation and others.
+#
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Eclipse Public License v1.0
+# which accompanies this distribution, and is available at
+# http://www.eclipse.org/legal/epl-v10.html
+##############################################################################
+---
+$schema: "http://json-schema.org/schema#"
+$id: "https://github.com/lfit/releng-global-jjb/blob/master/release-pypi-schema.yaml"
+
+required:
+  - "distribution_type"
+  - "project"
+  - "version"
+
+properties:
+  distribution_type:
+    type: "string"
+  project:
+    type: "string"
+  version:
+    type: "string"
diff --git a/shell/pypi-dist-build.sh b/shell/pypi-dist-build.sh
new file mode 100644 (file)
index 0000000..eb10371
--- /dev/null
@@ -0,0 +1,33 @@
+#!/bin/bash
+# SPDX-License-Identifier: EPL-1.0
+##############################################################################
+# Copyright (c) 2019 The Linux Foundation and others.
+#
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Eclipse Public License v1.0
+# which accompanies this distribution, and is available at
+# http://www.eclipse.org/legal/epl-v10.html
+##############################################################################
+echo "---> pypi-dist-build.sh"
+
+# Script to create Python source and binary distributions
+# Requires project file "setup.py"
+
+# Ensure we fail the job if any steps fail.
+set -eu -o pipefail
+
+virtualenv -p python3 /tmp/pypi
+PATH=/tmp/pypi/bin:$PATH
+
+bdist=""
+if $BUILD_BDIST_WHEEL; then
+    echo "INFO: installing wheel to build binary distribution"
+    pip install wheel
+    bdist="bdist_wheel"
+fi
+
+echo "INFO: cd to tox-dir $TOX_DIR"
+cd "$WORKSPACE/$TOX_DIR"
+echo "INFO: creating distributions"
+python3 setup.py sdist $bdist
+echo "---> pypi-dist-build.sh ends"
diff --git a/shell/pypi-tag-release.sh b/shell/pypi-tag-release.sh
new file mode 100644 (file)
index 0000000..d920c54
--- /dev/null
@@ -0,0 +1,151 @@
+#!/bin/bash
+# SPDX-License-Identifier: EPL-1.0
+##############################################################################
+# Copyright (c) 2019 The Linux Foundation and others.
+#
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Eclipse Public License v1.0
+# which accompanies this distribution, and is available at
+# http://www.eclipse.org/legal/epl-v10.html
+##############################################################################
+echo "---> pypi-tag-release.sh"
+
+# Ensure we fail the job if any steps fail.
+set -eu -o pipefail
+
+# Functions.
+
+set_variables(){
+    echo "INFO: Setting variables"
+    # Verify if using release file or parameters
+    if $USE_RELEASE_FILE; then
+        echo "INFO: Checking number of release yaml files"
+        release_files=$(git diff-tree --no-commit-id -r "$GIT_COMMIT" --name-only -- "releases/" ".releases/")
+        if (( $(echo "$release_files" | wc -w) != 1 )); then
+          echo "ERROR: RELEASE FILES: $release_files"
+          echo "ERROR: Committing multiple release files in the same commit OR rename/amend of existing files is not supported."
+          exit 1
+        else
+          release_file="$release_files"
+          echo "INFO: RELEASE FILE: $release_file"
+        fi
+    else
+        echo "INFO: This job is built with parameters, no release file"
+        release_file="None"
+    fi
+
+    if [[ -z ${DISTRIBUTION_TYPE:-} ]]; then
+        echo "INFO: reading DISTRIBUTION_TYPE from file $release_file"
+        DISTRIBUTION_TYPE="$(niet ".distribution_type" "$release_file")"
+    fi
+    if [[ -z ${VERSION:-} ]]; then
+        echo "INFO: reading VERSION from file $release_file"
+        VERSION="$(niet ".version" "$release_file")"
+    fi
+
+    # Display Release Information
+    printf "\t%-30s\n" RELEASE_ENVIRONMENT_INFO:
+    printf "\t%-30s %s\n" RELEASE_FILE: $release_file
+    printf "\t%-30s %s\n" JENKINS_HOSTNAME: $JENKINS_HOSTNAME
+    printf "\t%-30s %s\n" SILO: $SILO
+    printf "\t%-30s %s\n" PROJECT: $PROJECT
+    printf "\t%-30s %s\n" PROJECT-DASHED: ${PROJECT//\//-}
+    printf "\t%-30s %s\n" DISTRIBUTION_TYPE: $DISTRIBUTION_TYPE
+    printf "\t%-30s %s\n" VERSION: $VERSION
+}
+
+# needs to run in the repository root
+verify_schema(){
+    echo "INFO: Fetching schema"
+    pypi_schema="release-pypi-schema.yaml"
+    wget https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/${pypi_schema}
+    echo "INFO: Verifying $release_file against schema $pypi_schema"
+    lftools schema verify "$release_file" "$pypi_schema"
+    echo "INFO: $release_file passed schema verification"
+}
+
+verify_version(){
+    # Verify allowed patterns "v#.#.#" or "#.#.#" aka SemVer
+    echo "INFO: Verifying version string $VERSION"
+    allowed_version_regex="^((v?)([0-9]+)\.([0-9]+)\.([0-9]+))$"
+    if [[ $VERSION =~ $allowed_version_regex ]]; then
+        echo "INFO: The version $VERSION is a valid semantic version"
+    else
+        echo "ERROR: The version $VERSION is not a valid semantic version"
+        echo "ERROR: Allowed versions are \"v#.#.#\" or \"#.#.#\" aka SemVer"
+        echo "ERROR: See https://semver.org/ for more details on SemVer"
+        exit 1
+    fi
+}
+
+verify_dist(){
+    # Verify all file names in dist folder have the expected version string
+    dir="$WORKSPACE/$TOX_DIR/dist"
+    echo "INFO: Listing files in $dir"
+    ls $dir
+    echo "INFO: Checking files in $dir for $VERSION"
+    if unex_files=$(find $dir | grep -v $VERSION | egrep -v "^$dir$"); then
+        echo "ERROR: found unexpected files: $unex_files"
+        exit 1
+    else
+        echo "INFO: All file names have expected string ${VERSION}"
+    fi
+}
+
+# TODO: how to tag Github?
+tag_gerrit(){
+    echo "INFO: Verifying tag $VERSION in repo"
+    # Import public signing key
+    gpg --import "$SIGNING_PUBKEY"
+    # Fail if tag exists
+    if git tag -v "$VERSION"; then
+        echo "ERROR: Repo already tagged"
+        exit 1
+    else
+        echo "INFO: Repo has not yet been tagged"
+    fi
+    echo "INFO: Tagging repo"
+    git tag -am "${PROJECT//\//-} $VERSION" "$VERSION"
+    echo "INFO: Signing tag"
+    sigul --batch -c "$SIGUL_CONFIG" sign-git-tag "$SIGUL_KEY" "$VERSION" < "$SIGUL_PASSWORD"
+    echo "INFO: Verifying tag"
+    # may fail due to missing public key
+    if ! git tag -v "$VERSION"; then
+        echo "WARN: failed to verify tag, continuing anyhow"
+    fi
+    # The verify job also calls this script
+    if [[ ! $JOB_NAME =~ "merge" ]] ; then
+        echo "INFO: job is not a merge, skipping push"
+    else
+        echo "INFO: configuring Gerrit remote"
+        gerrit_ssh=$(echo "$GERRIT_URL" | awk -F"/" '{print $3}')
+        git remote set-url origin "ssh://$RELEASE_USERNAME@$gerrit_ssh:29418/$PROJECT"
+        git config user.name "$RELEASE_USERNAME"
+        git config user.email "$RELEASE_EMAIL"
+        echo -e "Host $gerrit_ssh\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config
+        chmod 600 ~/.ssh/config
+        if $DRY_RUN; then
+            echo "INFO: dry run, skipping push"
+        else
+            echo "INFO: pushing tag"
+            git push origin "$VERSION"
+        fi
+    fi
+}
+
+# Main
+virtualenv -p python3 /tmp/pypi
+PATH=/tmp/pypi/bin:$PATH
+pip install lftools jsonschema niet
+set_variables
+if [[ $DISTRIBUTION_TYPE != "pypi" ]]; then
+    echo "ERROR: unexpected distribution type $DISTRIBUTION_TYPE"
+    exit 1
+fi
+if $USE_RELEASE_FILE; then
+    verify_schema
+fi
+verify_version
+verify_dist
+tag_gerrit
+echo "---> pypi-tag-release.sh ends"
diff --git a/shell/pypi-upload.sh b/shell/pypi-upload.sh
new file mode 100644 (file)
index 0000000..e1ecfdf
--- /dev/null
@@ -0,0 +1,34 @@
+#!/bin/bash
+# SPDX-License-Identifier: EPL-1.0
+##############################################################################
+# Copyright (c) 2019 The Linux Foundation and others.
+#
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Eclipse Public License v1.0
+# which accompanies this distribution, and is available at
+# http://www.eclipse.org/legal/epl-v10.html
+##############################################################################
+echo "---> pypi-upload.sh"
+
+# Script to publish Python distributions from a folder
+# to the PyPI repository in $REPOSITORY which must be a
+# key in the .pypirc file
+
+# Ensure we fail the job if any steps fail.
+set -eu -o pipefail
+
+virtualenv -p python3 /tmp/pypi
+PATH=/tmp/pypi/bin:$PATH
+
+pip install twine
+echo "INFO: cd to tox-dir $TOX_DIR"
+cd "$WORKSPACE/$TOX_DIR"
+cmd="twine upload -r $REPOSITORY dist/*"
+if $DRY_RUN; then
+    echo "INFO: dry-run is set, echoing command only"
+    echo $cmd
+else
+    echo "INFO: uploading distributions"
+    $cmd
+fi
+echo "---> pypi-upload.sh ends"