Lift PyPI templates to Global-JJB from Acumos 32/16632/43
authorLott, Christopher (cl778h) <cl778h@att.com>
Wed, 21 Aug 2019 14:54:25 +0000 (10:54 -0400)
committerLott, Christopher (cl778h) <cl778h@att.com>
Mon, 23 Sep 2019 12:44:06 +0000 (08:44 -0400)
New templates provide features to build and push Python
source and binary distributions to a PyPI repository.
Check version string and push a tag on release.

New job groups:
    {project-name}-gerrit-pypi-jobs
    {project-name}-github-pypi-jobs

New templates:
    {project-name}-pypi-verify-{stream}
    gerrit-pypi-verify
    github-pypi-verify
    {project-name}-pypi-merge-{stream}
    gerrit-pypi-merge
    github-pypi-merge
    {project-name}-pypi-release-verify-{stream}
    gerrit-pypi-release-verify
    github-pypi-release-verify
    {project-name}-pypi-release-merge-{stream}
    gerrit-pypi-release-merge
    github-pypi-release-merge

Change-Id: If74925623f2a5804de8684b9a9933573dfea29a1
Issue-Id: RELENG-2317
Signed-off-by: Lott, Christopher (cl778h) <cl778h@att.com>
.jjb-test/lf-python-jobs.yaml
docs/conf.py
docs/jjb/lf-python-jobs.rst
jjb/lf-python-job-groups.yaml
jjb/lf-python-jobs.yaml
releasenotes/notes/lf-pypi-merge-release-jobs-b729bc2331155364.yaml [new file with mode: 0644]
schema/release-pypi-schema.yaml [new file with mode: 0644]
shell/pypi-dist-build.sh [new file with mode: 0644]
shell/pypi-tag-release.sh [new file with mode: 0644]
shell/pypi-upload.sh [new file with mode: 0644]

index 6a10b8f..dd94c64 100644 (file)
@@ -4,6 +4,9 @@
     jobs:
       - "{project-name}-python-jobs"
       - gerrit-tox-sonar
     jobs:
       - "{project-name}-python-jobs"
       - gerrit-tox-sonar
+      - gerrit-pypi-merge
+      - gerrit-pypi-release-verify
+      - gerrit-pypi-release-merge
 
     project-name: gerrit-python
 
 
     project-name: gerrit-python
 
@@ -12,6 +15,9 @@
     jobs:
       - "{project-name}-github-python-jobs"
       - github-tox-sonar
     jobs:
       - "{project-name}-github-python-jobs"
       - github-tox-sonar
+      - github-pypi-merge
+      - github-pypi-release-verify
+      - github-pypi-release-merge
 
     project-name: github-python
 
 
     project-name: github-python
 
index 4457e12..ca63745 100644 (file)
@@ -74,7 +74,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = 'lf-releng-global-jjb'
 
 # General information about the project.
 project = 'lf-releng-global-jjb'
-copyright = '2017, The Linux Foundation'
+copyright = '2019, The Linux Foundation'
 author = 'Linux Foundation Releng'
 
 # The version info for the project you're documenting, acts as replacement for
 author = 'Linux Foundation Releng'
 
 # The version info for the project you're documenting, acts as replacement for
index 175f0e2..bea44fc 100644 (file)
@@ -7,7 +7,7 @@ Job Groups
 
 .. include:: ../job-groups.rst
 
 
 .. include:: ../job-groups.rst
 
-Below is a list of Maven job groups:
+Below is a list of Python job groups:
 
 .. literalinclude:: ../../jjb/lf-python-job-groups.yaml
    :language: yaml
 
 .. literalinclude:: ../../jjb/lf-python-job-groups.yaml
    :language: yaml
@@ -19,31 +19,51 @@ Macros
 lf-infra-clm-python
 -------------------
 
 lf-infra-clm-python
 -------------------
 
-Run CLM scanning against a Python project.
+Runs CLM scanning against a Python project.
 
 :Required Parameters:
 
     :clm-project-name: Project name in Nexus IQ to send results to.
 
 
 :Required Parameters:
 
     :clm-project-name: Project name in Nexus IQ to send results to.
 
+lf-infra-pypi-tag-release
+-------------------------
+
+Checks the format of the release version string and checks the git
+repository for that tag. In a merge job, continues to tag the repository
+and push the tag to the git server. Also installs supporting tools
+including Sigul and lftools.  Sigul requires a CentOS build node.
+
+lf-infra-pypi-upload
+--------------------
+
+Uploads distribution files from subdirectory "dist" to a PyPI repository
+using a Python virtual enviroment to install required packages. The
+Jenkins server must have a configuration file ".pypirc".
+
+:Required Parameters:
+
+    :pypi-repo: PyPI repository key in .pypirc configuration file;
+        e.g., "staging" or "pypi".
+
 lf-infra-tox-install
 --------------------
 
 lf-infra-tox-install
 --------------------
 
-Install Tox into a virtualenv.
+Installs Tox into a virtualenv.
 
 :Required Parameters:
 
 
 :Required Parameters:
 
-    :python-version: Version of Python to install into the Tox virtualenv.
-        Eg. python2 / python3
+    :python-version: Version of Python to invoke the pip install of the tox-pyenv
+        package that creates a virtual environment, either "python2" or "python3".
 
 
-lf-tox-install
---------------
+lf-infra-tox-run
+----------------
 
 
-Runs a shell script that installs tox in a Python virtualenv.
+Creates a Tox virtual environment and invokes tox.
 
 :Required Parameters:
 
 
 :Required Parameters:
 
-    :python-version: Base Python version to use in the virtualenv. For example
-        python2 or python3.
+    :parallel: Boolean. If true use detox (distributed tox);
+        else use regular tox.
 
 
 Job Templates
 
 
 Job Templates
@@ -55,9 +75,9 @@ Python XC CLM
 CLM scans for Python based repos. This job will call the Nexus IQ CLI
 directly to run the scans.
 
 CLM scans for Python based repos. This job will call the Nexus IQ CLI
 directly to run the scans.
 
-A new credential named "nexus-iq-xc-clm" needs to exist in the Jenkins credentials.
-The credential should contain the username and password to access Nexus
-IQ Server.
+A new credential named "nexus-iq-xc-clm" needs to exist in the Jenkins
+credentials.  The credential should contain the username and password
+to access Nexus IQ Server.
 
 :Template Names:
 
 
 :Template Names:
 
@@ -97,30 +117,29 @@ IQ Server.
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
     :gerrit_clm_triggers: Override Gerrit Triggers.
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
     :gerrit_clm_triggers: Override Gerrit Triggers.
-    :gerrit_trigger_file_paths: Override file paths which used to filter which
-        file modifications will trigger a build. Refer to JJB documentation for
-        "file-path" details.
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
         https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
 
 
 Python Sonar with Tox
 ---------------------
 
         https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
 
 
 Python Sonar with Tox
 ---------------------
 
-Sonar scans for Python based repos. This job invokes tox to run tests and
-gather coverage statistics from the test results, then invokes Maven to
-publish the results to a Sonar server.
+Sonar scans for Python based repos. This job invokes tox to run tests
+and gather coverage statistics from the test results, then invokes
+Maven to publish the results to a Sonar server.
 
 
-To get the Sonar coverage results, file tox.ini must exist and contain coverage
-commands to run.
+To get the Sonar coverage results, file tox.ini must exist and contain
+coverage commands to run.
 
 
-The coverage commands define the code that gets executed by the test suites.
-Checking coverage does not guarantee that the tests execute properly, but it
-identifies code that is not executed by any test.
+The coverage commands define the code that gets executed by the test
+suites.  Checking coverage does not guarantee that the tests execute
+properly, but it identifies code that is not executed by any test.
 
 
-This job reuses the Sonar builder used in Java/Maven projects which runs maven
-twice. The first invocation does nothing for Python projects, so the job uses
-the goal 'validate' by default. The second invocation publishes results using
-the goal 'sonar:sonar' by default.
+This job reuses the Sonar builder used in Java/Maven projects which
+runs maven twice. The first invocation does nothing for Python
+projects, so the job uses the goal 'validate' by default. The second
+invocation publishes results using the goal 'sonar:sonar' by default.
 
 For example:
 
 
 For example:
 
@@ -169,10 +188,13 @@ https://docs.sonarqube.org/display/PLUG/Python+Coverage+Results+Import
     :mvn-global-settings: The name of the Maven global settings to use for
     :mvn-goals: The Maven goal to run first. (default: validate)
     :mvn-version: Version of maven to use. (default: mvn35)
     :mvn-global-settings: The name of the Maven global settings to use for
     :mvn-goals: The Maven goal to run first. (default: validate)
     :mvn-version: Version of maven to use. (default: mvn35)
+    :parallel: Boolean indicator for tox to run tests in parallel or series.
+       (default: false, in series)
     :pre-build-script: Shell script to execute before the Sonar builder.
         For example, install prerequisites or move files to the repo root.
     :pre-build-script: Shell script to execute before the Sonar builder.
         For example, install prerequisites or move files to the repo root.
-        (default: a string with a comment)
-    :python-version: Python version (default: python2)
+        (default: a string with a shell comment)
+    :python-version: Python version to invoke pip install of tox-pyenv
+        (default: python2)
     :sonar-mvn-goal: The Maven goal to run the Sonar plugin. (default: sonar:sonar)
     :stream: Keyword used to represent a release code-name.
         Often the same as the branch. (default: master)
     :sonar-mvn-goal: The Maven goal to run the Sonar plugin. (default: sonar:sonar)
     :stream: Keyword used to represent a release code-name.
         Often the same as the branch. (default: master)
@@ -183,19 +205,19 @@ https://docs.sonarqube.org/display/PLUG/Python+Coverage+Results+Import
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
     :gerrit_sonar_triggers: Override Gerrit Triggers.
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
     :gerrit_sonar_triggers: Override Gerrit Triggers.
-    :gerrit_trigger_file_paths: Override file paths which used to filter which
-        file modifications will trigger a build. Refer to JJB documentation for
-        "file-path" details.
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
         https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
 
 
 Tox Verify
 ----------
 
         https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
 
 
 Tox Verify
 ----------
 
-Tox runner to verify a project on creation of a patch set.
-This job is pyenv aware so if the image contains an installation of pyenv
-at /opt/pyenv it will pick it up and run Python tests with the appropriate
-Python versions. This job will set the following pyenv variables before running.
+Tox runner to verify a project on creation of a patch set.  This job
+is pyenv aware so if the image contains an installation of pyenv at
+/opt/pyenv it will pick it up and run Python tests with the
+appropriate Python versions. This job will set the following pyenv
+variables before running.
 
 .. code:: bash
 
 
 .. code:: bash
 
@@ -225,8 +247,10 @@ Python versions. This job will set the following pyenv variables before running.
     :pre-build-script: Shell script to execute before the Tox builder.
         For example, install prerequisites or move files to the repo root.
         (default: a string with a shell comment)
     :pre-build-script: Shell script to execute before the Tox builder.
         For example, install prerequisites or move files to the repo root.
         (default: a string with a shell comment)
-    :python-version: Version of Python to configure as a base in virtualenv.
-        (default: python3)
+    :parallel: Boolean indicator for tox to run tests in parallel or series.
+       (default: false, in series)
+    :python-version: Python version to invoke pip install of tox-pyenv
+        (default: python2)
     :stream: Keyword representing a release code-name.
         Often the same as the branch. (default: master)
     :submodule-recursive: Whether to checkout submodules recursively.
     :stream: Keyword representing a release code-name.
         Often the same as the branch. (default: master)
     :submodule-recursive: Whether to checkout submodules recursively.
@@ -236,23 +260,23 @@ Python versions. This job will set the following pyenv variables before running.
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
     :tox-dir: Directory containing the project's tox.ini relative to
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
     :tox-dir: Directory containing the project's tox.ini relative to
-        the workspace. Empty works if tox.ini is at project root.
-        (default: '')
+        the workspace. The default uses tox.ini at the project root.
+        (default: '.')
     :tox-envs: Tox environments to run. If blank run everything described
         in tox.ini. (default: '')
     :tox-envs: Tox environments to run. If blank run everything described
         in tox.ini. (default: '')
-    :gerrit_trigger_file_paths: Override file paths used to filter which
-        file modifications will trigger a build. Refer to JJB documentation for
-        "file-path" details.
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
         https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
 
 
 Tox Merge
 ---------
 
         https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
 
 
 Tox Merge
 ---------
 
-Tox runner to verify a project after merge of a patch set.
-This job is pyenv aware so if the image contains an installation of pyenv
-at /opt/pyenv it will pick it up and run Python tests with the appropriate
-Python versions. This job will set the following pyenv variables before running.
+Tox runner to verify a project after merge of a patch set.  This job
+is pyenv aware so if the image contains an installation of pyenv at
+/opt/pyenv it will pick it up and run Python tests with the
+appropriate Python versions. This job will set the following pyenv
+variables before running.
 
 .. code:: bash
 
 
 .. code:: bash
 
@@ -282,7 +306,352 @@ Python versions. This job will set the following pyenv variables before running.
     :pre-build-script: Shell script to execute before the CLM builder.
         For example, install prerequisites or move files to the repo root.
         (default: a string with only a comment)
     :pre-build-script: Shell script to execute before the CLM builder.
         For example, install prerequisites or move files to the repo root.
         (default: a string with only a comment)
-    :python-version: Version of Python to configure as a base in virtualenv.
+    :python-version: Python version to invoke pip install of tox-pyenv
+        (default: python2)
+    :stream: Keyword representing a release code-name.
+        Often the same as the branch. (default: master)
+    :submodule-recursive: Whether to checkout submodules recursively.
+        (default: true)
+    :submodule-timeout: Timeout (in minutes) for checkout operation.
+        (default: 10)
+    :submodule-disable: Disable submodule checkout operation.
+        (default: false)
+    :tox-dir: Directory containing the project's tox.ini relative to
+        the workspace. The default uses tox.ini at the project root.
+        (default: '.')
+    :tox-envs: Tox environments to run. If blank run everything described
+        in tox.ini. (default: '')
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
+        https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+
+
+PyPI Verify
+-----------
+
+Verifies a Python library project on creation of a patch set. Runs tox
+then builds a source distribution and (optionally) a binary
+distribution.  The project repository must have a setup.py file with
+configuration for packaging the component.
+
+The tox runner is pyenv aware so if the image contains an installation
+of pyenv at /opt/pyenv it will pick it up and run Python tests with
+the appropriate Python versions. The tox runner sets the following
+pyenv variables before running.
+
+.. code:: bash
+
+   export PYENV_ROOT="/opt/pyenv"
+   export PATH="$PYENV_ROOT/bin:$PATH"
+
+:Template Names:
+
+    - {project-name}-pypi-verify-{stream}
+    - gerrit-pypi-verify
+    - github-pypi-verify
+
+:Comment Trigger: recheck
+
+:Required Parameters:
+
+    :build-node: The node to run the build on.
+    :jenkins-ssh-credential: Credential to use for SSH. (Generally set
+        in defaults.yaml)
+
+:Optional Parameters:
+
+    :branch: The branch to build against. (default: master)
+    :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+    :build-timeout: Timeout in minutes before aborting build. (default: 15)
+    :dist-binary: Whether to build a binary wheel distribution. (default: true)
+    :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+    :parallel: Boolean indicator for tox to run tests in parallel or series.
+       (default: false, in series)
+    :pre-build-script: Shell script to execute before the tox builder. For
+        example, install system prerequisites. (default: a shell comment)
+    :python-version: Python version to invoke pip install of tox-pyenv
+        (default: python3)
+    :stream: Keyword representing a release code-name.
+        Often the same as the branch. (default: master)
+    :submodule-recursive: Whether to checkout submodules recursively.
+        (default: true)
+    :submodule-timeout: Timeout (in minutes) for checkout operation.
+        (default: 10)
+    :submodule-disable: Disable submodule checkout operation.
+        (default: false)
+    :tox-dir: Directory containing the project's tox.ini relative to
+        the workspace. The default uses tox.ini at the project root.
+        (default: '.')
+    :tox-envs: Tox environments to run. If blank run everything described
+        in tox.ini. (default: '')
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
+        https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+
+
+PyPI Merge
+----------
+
+Creates and uploads distribution files on merge of a patch set.  Runs
+tox, builds a source distribution and (optionally) a binary
+distribution, and uploads the distribution(s) to a PyPI repository.
+This job should be configured to use a staging PyPI repository like
+testpypi.python.org, not a public release area like the global PyPI
+repository. Like the verify job, this requires a setup.py file for
+packaging the component.
+
+The tox runner is pyenv aware so if the image contains an installation
+of pyenv at /opt/pyenv it will pick it up and run Python tests with
+the appropriate Python versions. The tox runner sets the following
+pyenv variables before running.
+
+.. code:: bash
+
+   export PYENV_ROOT="/opt/pyenv"
+   export PATH="$PYENV_ROOT/bin:$PATH"
+
+
+Requires a .pypirc configuration file in the Jenkins builder home
+directory, an example appears next.
+
+.. code-block:: bash
+
+    [distutils] # this tells distutils what package indexes you can push to
+    index-servers =
+    staging
+    pypi
+
+    [staging]
+    repository: https://testpypi.python.org/pypi
+    username: your_username
+    password: your_password
+
+    [pypi]
+    repository: https://pypi.python.org/pypi
+    username: your_username
+    password: your_password
+
+
+:Template Names:
+
+    - {project-name}-pypi-merge-{stream}
+    - gerrit-pypi-merge
+    - github-pypi-merge
+
+:Comment Trigger: pypi-remerge
+
+:Required Parameters:
+
+    :build-node: The node to run the build on.
+    :jenkins-ssh-credential: Credential to use for SSH. (Generally set
+        in defaults.yaml)
+
+:Optional Parameters:
+
+    :branch: The branch to build against. (default: master)
+    :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+    :build-timeout: Timeout in minutes before aborting build. (default: 15)
+    :dist-binary: Whether to build a binary wheel distribution. (default: true)
+    :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+    :parallel: Boolean indicator for tox to run tests in parallel or series.
+       (default: false, in series)
+    :pre-build-script: Shell script to execute before the tox builder. For
+        example, install system prerequisites. (default: a shell comment)
+    :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
+        Merge jobs should use a server like testpypi.python.org.  (default: staging)
+    :python-version: Python version to invoke pip install of tox-pyenv
+        (default: python3)
+    :stream: Keyword representing a release code-name.
+        Often the same as the branch. (default: master)
+    :submodule-recursive: Whether to checkout submodules recursively.
+        (default: true)
+    :submodule-timeout: Timeout (in minutes) for checkout operation.
+        (default: 10)
+    :submodule-disable: Disable submodule checkout operation.
+        (default: false)
+    :tox-dir: Directory containing the project's tox.ini relative to
+        the workspace. The default uses tox.ini at the project root.
+        (default: '.')
+    :tox-envs: Tox environments to run. If blank run everything described
+        in tox.ini. (default: '')
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
+        https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+
+
+PyPI Release Verify
+-------------------
+
+Verifies a Python library project on creation of a patch set with a
+release yaml file. Runs tox, builds source and (optionally) binary
+distributions, checks the format of the version string, checks that
+the distribution file names contain the release version string, and
+checks that no tag exists in the code repository for the release
+version.
+
+To initiate the release process, create a releases/ or .releases/
+directory at the root of the project repository, add one release yaml
+file to it, and submit a change set with that release yaml file. A
+schema and and an example for the release yaml file appear below.  The
+version in the release yaml file must be a valid Semantic Versioning
+(SemVer) string, matching either the pattern "v#.#.#" or "#.#.#" where
+"#" is one or more digits.
+
+This job is similar to the PyPI verify job, but is only triggered by a
+patch set with a release yaml file.
+
+The build node for PyPI release verify jobs must be CentOS, which
+supports the sigul client for accessing a signing server.
+
+.. note::
+
+   The release file regex is: (releases\/.*\.yaml|\.releases\/.*\.yaml).
+   In words, the directory name can be ".releases" or "releases"; the file
+   name can be anything with suffix ".yaml".
+
+The JSON schema for a pypi release file appears below.
+
+.. code-block:: none
+
+    ---
+    $schema: "http://json-schema.org/schema#"
+    $id: "https://github.com/lfit/releng-global-jjb/blob/master/release-pypi-schema.yaml"
+
+    required:
+      - "distribution_type"
+      - "project"
+      - "version"
+
+    properties:
+      distribution_type:
+        type: "string"
+      project:
+        type: "string"
+      version:
+        type: "string"
+
+
+An example of a pypi release file appears below.
+
+.. code-block:: none
+
+    $ cat releases/1.0.0-pypi.yaml
+    ---
+    distribution_type: pypi
+    version: 1.0.0
+    project: 'example-project'
+
+
+:Template Names:
+
+    - {project-name}-pypi-release-verify-{stream}
+    - gerrit-pypi-release-verify
+    - github-pypi-release-verify
+
+:Required Parameters:
+
+    :build-node: The node to run build on, which must be Centos.
+    :jenkins-ssh-credential: Credential to use for SSH. (Generally set
+        in defaults.yaml)
+
+:Optional Parameters:
+
+    :branch: The branch to build against. (default: master)
+    :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+    :build-timeout: Timeout in minutes before aborting build. (default: 15)
+    :dist-binary: Whether to build a binary wheel distribution. (default: true)
+    :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+    :parallel: Boolean indicator for tox to run tests in parallel or series.
+       (default: false, in series)
+    :pre-build-script: Shell script to execute before the tox builder.
+        For example, install prerequisites or move files to the repo root.
+        (default: a string with a shell comment)
+    :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
+        Release jobs should use a server like pypy.org.  (default: pypi)
+    :python-version: Python version to invoke pip install of tox-pyenv
+        (default: python3)
+    :stream: Keyword representing a release code-name.
+        Often the same as the branch. (default: master)
+    :submodule-recursive: Whether to checkout submodules recursively.
+        (default: true)
+    :submodule-timeout: Timeout (in minutes) for checkout operation.
+        (default: 10)
+    :submodule-disable: Disable submodule checkout operation.
+        (default: false)
+    :tox-dir: Directory containing the project's tox.ini relative to
+        the workspace. The default uses tox.ini at the project root.
+        (default: '.')
+    :tox-envs: Tox environments to run. If blank run everything described
+        in tox.ini. (default: '')
+    :use-release-file: Whether to use the release file. (default: true)
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
+        https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
+
+
+PyPI Release Merge
+------------------
+
+Publishes a Python library on merge of a patch set with a release yaml
+file.  Runs tox, builds source and (optionally) binary distributions,
+checks the format of the version string, checks that the distribution
+file names contain the release version string, checks that no tag
+exists in the code repository for the release version, tags the code
+repository with the release version, pushes the tag to the git server,
+and uploads distributions to a PyPI repository.
+
+This job is similar to the PyPI merge job, but is only triggered by
+merge of a release yaml file and checks the version and tag before
+uploading to a public repository such as PyPI.
+
+See the PyPI Release Verify job above for documentation of the release
+yaml file format.
+
+The build node for PyPI release merge jobs must be CentOS, which
+supports the sigul client for accessing a signing server.
+
+A Jenkins user can also trigger this release job via the "Build with
+parameters" action, removing the need to merge a release yaml file.
+The user must enter parameters in the same way as a release yaml file,
+except for the special USE_RELEASE_FILE and DRY_RUN check boxes. The
+user must uncheck the USE_RELEASE_FILE check box if the job should run
+with a release file, while passing the required information as build
+parameters. Similarly, the user must uncheck the DRY_RUN check box to
+test the job while skipping upload of files to a repository.
+
+The special parameters are as follows::
+
+    VERSION = 1.0.0
+    USE_RELEASE_FILE = false
+    DRY_RUN = false
+
+:Template Names:
+
+    - {project-name}-pypi-release-merge-{stream}
+    - gerrit-pypi-release-merge
+    - github-pypi-release-merge
+
+:Required Parameters:
+
+    :build-node: The node to run build on, which must be Centos.
+    :jenkins-ssh-credential: Credential to use for SSH. (Generally set
+        in defaults.yaml)
+
+:Optional Parameters:
+
+    :branch: The branch to build against. (default: master)
+    :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7)
+    :build-timeout: Timeout in minutes before aborting build. (default: 15)
+    :dist-binary: Whether to build a binary wheel distribution. (default: true)
+    :git-url: URL clone project from. (default: $GIT_URL/$PROJECT)
+    :parallel: Boolean indicator for tox to run tests in parallel or series.
+       (default: false, in series)
+    :pre-build-script: Shell script to execute before the tox builder.
+        For example, install prerequisites or move files to the repo root.
+        (default: a string with a shell comment)
+    :pypi-repo: Key for PyPI repository parameters in the .pypirc file.
+        Release jobs should use a server like pypy.org.  (default: pypi)
+    :python-version: Python version to invoke pip install of tox-pyenv
         (default: python3)
     :stream: Keyword representing a release code-name.
         Often the same as the branch. (default: master)
         (default: python3)
     :stream: Keyword representing a release code-name.
         Often the same as the branch. (default: master)
@@ -293,11 +662,11 @@ Python versions. This job will set the following pyenv variables before running.
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
     :tox-dir: Directory containing the project's tox.ini relative to
     :submodule-disable: Disable submodule checkout operation.
         (default: false)
     :tox-dir: Directory containing the project's tox.ini relative to
-        the workspace. Empty works if tox.ini is at project root.
-        (default: '')
+        the workspace. The default uses tox.ini at the project root.
+        (default: '.')
     :tox-envs: Tox environments to run. If blank run everything described
         in tox.ini. (default: '')
     :tox-envs: Tox environments to run. If blank run everything described
         in tox.ini. (default: '')
-    :gerrit_trigger_file_paths: Override file paths used to filter which
-        file modifications will trigger a build. Refer to JJB documentation for
-        "file-path" details.
+    :use-release-file: Whether to use the release file. (default: true)
+    :gerrit_trigger_file_paths: Override file paths used to filter which file
+        modifications trigger a build. Refer to JJB documentation for "file-path" details.
         https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
         https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit
index 59066f9..60ecc66 100644 (file)
@@ -3,7 +3,7 @@
     name: "{project-name}-python-jobs"
 
     # This job group contains all the recommended jobs that should be deployed
     name: "{project-name}-python-jobs"
 
     # This job group contains all the recommended jobs that should be deployed
-    # for any project ci that is using Gerrit.
+    # for a Gerrit-based Python project to verify commits using tox.
 
     jobs:
       - gerrit-python-xc-clm
 
     jobs:
       - gerrit-python-xc-clm
     name: "{project-name}-github-python-jobs"
 
     # This job group contains all the recommended jobs that should be deployed
     name: "{project-name}-github-python-jobs"
 
     # This job group contains all the recommended jobs that should be deployed
-    # for any project ci that is using GitHub.
+    # for a Github-based Python project to verify commits using tox.
 
     jobs:
       - github-python-xc-clm
       - github-tox-verify
       - github-tox-merge
 
     jobs:
       - github-python-xc-clm
       - github-tox-verify
       - github-tox-merge
+
+- job-group:
+    name: "{project-name}-gerrit-pypi-jobs"
+
+    # This job group contains all the recommended jobs that should be deployed for
+    # a Gerrit-based Python project to test, build and deploy a library to PyPI.
+
+    jobs:
+      - gerrit-pypi-verify
+      - gerrit-pypi-merge
+      - gerrit-pypi-release-verify
+      - gerrit-pypi-release-merge
+
+- job-group:
+    name: "{project-name}-github-pypi-jobs"
+
+    # This job group contains all the recommended jobs that should be deployed for
+    # a Github-based Python project to test, build and deploy a library to PyPI.
+
+    jobs:
+      - github-pypi-verify
+      - github-pypi-merge
+      - github-pypi-release-verify
+      - github-pypi-release-merge
index 451eb63..de38fa1 100644 (file)
@@ -3,6 +3,41 @@
 # Macros #
 ##########
 
 # Macros #
 ##########
 
+- builder:
+    name: lf-infra-clm-python
+    builders:
+      - inject:
+          properties-content: "CLM_PROJECT_NAME={clm-project-name}"
+      - shell: !include-raw-escape: ../shell/nexus-iq-cli.sh
+
+- builder:
+    name: lf-infra-pypi-tag-release
+    builders:
+      - config-file-provider:
+          files:
+            - file-id: sigul-config
+              variable: SIGUL_CONFIG
+            - file-id: sigul-password
+              variable: SIGUL_PASSWORD
+            - file-id: sigul-pki
+              variable: SIGUL_PKI
+            - file-id: signing-pubkey
+              variable: SIGNING_PUBKEY
+      - shell: !include-raw: ../shell/sigul-configuration.sh
+      - shell: !include-raw: ../shell/sigul-install.sh
+      - shell: !include-raw: ../shell/pypi-tag-release.sh
+
+- builder:
+    name: lf-infra-pypi-upload
+    builders:
+      - config-file-provider:
+          files:
+            - file-id: pypirc
+              target: "$HOME/.pypirc"
+      - inject:
+          properties-content: "REPOSITORY={pypi-repo}"
+      - shell: !include-raw-escape: ../shell/pypi-upload.sh
+
 - builder:
     name: lf-infra-tox-install
     builders:
 - builder:
     name: lf-infra-tox-install
     builders:
       - shell: !include-raw-escape: ../shell/tox-install.sh
 
 - builder:
       - shell: !include-raw-escape: ../shell/tox-install.sh
 
 - builder:
-    name: lf-infra-clm-python
+    name: lf-infra-tox-run
     builders:
       - inject:
     builders:
       - inject:
-          properties-content: "CLM_PROJECT_NAME={clm-project-name}"
-      - shell: !include-raw-escape:
-          - ../shell/nexus-iq-cli.sh
+          properties-content: "PARALLEL={parallel}"
+      - shell: !include-raw-escape: ../shell/tox-run.sh
 
 ####################
 # COMMON FUNCTIONS #
 
 ####################
 # COMMON FUNCTIONS #
     mvn-goals: validate
     mvn-settings: "{mvn-settings}"
     mvn-version: mvn35
     mvn-goals: validate
     mvn-settings: "{mvn-settings}"
     mvn-version: mvn35
+    parallel: true
     pre-build-script: "# pre-build script goes here"
     python-version: python2
     sonar-mvn-goal: "sonar:sonar"
     pre-build-script: "# pre-build script goes here"
     python-version: python2
     sonar-mvn-goal: "sonar:sonar"
       - lf-infra-tox-install:
           python-version: "{python-version}"
       - shell: "{pre-build-script}"
       - lf-infra-tox-install:
           python-version: "{python-version}"
       - shell: "{pre-build-script}"
-      - shell: !include-raw-escape: ../shell/tox-run.sh
+      - lf-infra-tox-run:
+          parallel: "{parallel}"
       - lf-provide-maven-settings:
           global-settings-file: "{mvn-global-settings}"
           settings-file: "{mvn-settings}"
       - lf-provide-maven-settings:
           global-settings-file: "{mvn-global-settings}"
           settings-file: "{mvn-settings}"
     disable-job: false
     git-url: "$GIT_URL/$GERRIT_PROJECT"
     github-url: "https://github.com"
     disable-job: false
     git-url: "$GIT_URL/$GERRIT_PROJECT"
     github-url: "https://github.com"
-    parallel: true
+    parallel: false
     pre-build-script: "# pre-build script goes here"
     python-version: python2
     stream: master
     submodule-recursive: true
     submodule-timeout: 10
     submodule-disable: false
     pre-build-script: "# pre-build script goes here"
     python-version: python2
     stream: master
     submodule-recursive: true
     submodule-timeout: 10
     submodule-disable: false
-    tox-dir: ""
+    tox-dir: "."
     tox-envs: ""
 
     gerrit_trigger_file_paths:
     tox-envs: ""
 
     gerrit_trigger_file_paths:
       - lf-infra-tox-parameters:
           tox-dir: "{tox-dir}"
           tox-envs: "{tox-envs}"
       - lf-infra-tox-parameters:
           tox-dir: "{tox-dir}"
           tox-envs: "{tox-envs}"
-      - bool:
-          name: PARALLEL
-          default: "{parallel}"
-          description: Tox test type used to configure serial or parallel testing.
 
     wrappers:
       - lf-infra-wrappers:
 
     wrappers:
       - lf-infra-wrappers:
       - lf-infra-tox-install:
           python-version: "{python-version}"
       - shell: "{pre-build-script}"
       - lf-infra-tox-install:
           python-version: "{python-version}"
       - shell: "{pre-build-script}"
-      - shell: !include-raw-escape: ../shell/tox-run.sh
+      - lf-infra-tox-run:
+          parallel: "{parallel}"
 
     publishers:
       - lf-infra-publish
 
     publishers:
       - lf-infra-publish
           white-list-target-branches:
             - "{branch}"
           included-regions: "{obj:github_included_regions}"
           white-list-target-branches:
             - "{branch}"
           included-regions: "{obj:github_included_regions}"
+
+########
+# PyPI #
+########
+
+- lf_pypi_common: &lf_pypi_common
+    name: lf-pypi-common
+
+    ######################
+    # Default parameters #
+    ######################
+
+    branch: master
+    build-days-to-keep: 7
+    build-timeout: 15
+    disable-job: false
+    dist-binary: true
+    git-url: "$GIT_URL/$GERRIT_PROJECT"
+    github-url: "https://github.com"
+    parallel: false
+    pre-build-script: "# pre-build script goes here"
+    python-version: python3
+    stream: master
+    submodule-recursive: true
+    submodule-timeout: 10
+    submodule-disable: false
+    tox-dir: "."
+    tox-envs: ""
+
+    gerrit_trigger_file_paths:
+      - compare-type: ANT
+        pattern: ".*"
+
+    # github_included_regions MUST match gerrit_trigger_file_paths
+    github_included_regions:
+      - ".*"
+
+    parameters:
+      - lf-infra-parameters:
+          project: "{project}"
+          branch: "{branch}"
+          stream: "{stream}"
+      - lf-infra-tox-parameters:
+          tox-dir: "{tox-dir}"
+          tox-envs: "{tox-envs}"
+      - bool:
+          name: BUILD_BDIST_WHEEL
+          default: "{dist-binary}"
+          description: "Set to True to build a wheel"
+
+- lf_pypi_verify_builders: &lf_pypi_verify_builders
+    name: lf-pypi-verify-builders
+
+    builders:
+      - lf-infra-pre-build
+      - lf-infra-tox-install:
+          python-version: "{python-version}"
+      - shell: "{pre-build-script}"
+      - lf-infra-tox-run:
+          parallel: "{parallel}"
+      - shell: !include-raw-escape: ../shell/pypi-dist-build.sh
+
+- lf_pypi_merge_builders: &lf_pypi_merge_builders
+    name: lf-pypi-merge-builders
+
+    builders:
+      - lf-infra-pre-build
+      - lf-infra-tox-install:
+          python-version: "{python-version}"
+      - shell: "{pre-build-script}"
+      - lf-infra-tox-run:
+          parallel: "{parallel}"
+      - shell: !include-raw-escape: ../shell/pypi-dist-build.sh
+      - lf-infra-pypi-upload:
+          pypi-repo: "{pypi-repo}"
+
+- lf_pypi_release_verify_builders: &lf_pypi_release_verify_builders
+    name: lf-pypi-release-verify-builders
+
+    builders:
+      - lf-infra-pre-build
+      - lf-infra-tox-install:
+          python-version: "{python-version}"
+      - shell: "{pre-build-script}"
+      - lf-infra-tox-run:
+          parallel: "{parallel}"
+      - shell: !include-raw-escape: ../shell/pypi-dist-build.sh
+      - lf-infra-pypi-tag-release
+
+- lf_pypi_release_merge_builders: &lf_pypi_release_merge_builders
+    name: lf-pypi-release-merge-builders
+
+    builders:
+      - lf-infra-pre-build
+      - lf-infra-tox-install:
+          python-version: "{python-version}"
+      - shell: "{pre-build-script}"
+      - lf-infra-tox-run:
+          parallel: "{parallel}"
+      - shell: !include-raw-escape: ../shell/pypi-dist-build.sh
+      - lf-infra-pypi-tag-release
+      - lf-infra-pypi-upload:
+          pypi-repo: "{pypi-repo}"
+
+- job-template:
+    name: "{project-name}-pypi-verify-{stream}"
+    id: gerrit-pypi-verify
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_verify_builders
+
+    gerrit_verify_triggers:
+      - patchset-created-event:
+          exclude-drafts: true
+          exclude-trivial-rebase: false
+          exclude-no-code-change: false
+      - draft-published-event
+      - comment-added-contains-event:
+          comment-contains-value: '^Patch Set\s+\d+:\s+(recheck|reverify)\s*$'
+
+    scm:
+      - lf-infra-gerrit-scm:
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+          git-url: "{git-url}"
+          refspec: "$GERRIT_REFSPEC"
+          branch: "$GERRIT_BRANCH"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: gerrit
+
+    triggers:
+      - gerrit:
+          server-name: "{gerrit-server-name}"
+          trigger-on: "{obj:gerrit_verify_triggers}"
+          projects:
+            - project-compare-type: ANT
+              project-pattern: "{project}"
+              branches:
+                - branch-compare-type: ANT
+                  branch-pattern: "**/{branch}"
+              file-paths: "{obj:gerrit_trigger_file_paths}"
+
+- job-template:
+    name: "{project-name}-pypi-verify-{stream}"
+    id: github-pypi-verify
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_verify_builders
+
+    properties:
+      - github:
+          url: "{github-url}/{github-org}/{project}"
+
+    scm:
+      - lf-infra-github-scm:
+          url: "{git-clone-url}{github-org}/{project}"
+          refspec: "+refs/pull/*:refs/remotes/origin/pr/*"
+          branch: "$sha1"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: default
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+
+    triggers:
+      - github-pull-request:
+          trigger-phrase: "^(recheck|reverify)$"
+          only-trigger-phrase: false
+          status-context: "PyPI Verify"
+          permit-all: true
+          github-hooks: true
+          white-list-target-branches:
+            - "{branch}"
+          included-regions: "{obj:github_included_regions}"
+
+- job-template:
+    name: "{project-name}-pypi-merge-{stream}"
+    id: gerrit-pypi-merge
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_merge_builders
+
+    pypi-repo: staging
+
+    gerrit_merge_triggers:
+      - change-merged-event
+      - comment-added-contains-event:
+          comment-contains-value: '^Patch Set\s+\d+:\s+remerge\s*$'
+
+    scm:
+      - lf-infra-gerrit-scm:
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+          git-url: "{git-url}"
+          refspec: "$GERRIT_REFSPEC"
+          branch: "$GERRIT_BRANCH"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: gerrit
+
+    triggers:
+      - gerrit:
+          server-name: "{gerrit-server-name}"
+          trigger-on: "{obj:gerrit_merge_triggers}"
+          projects:
+            - project-compare-type: ANT
+              project-pattern: "{project}"
+              branches:
+                - branch-compare-type: ANT
+                  branch-pattern: "**/{branch}"
+              file-paths: "{obj:gerrit_trigger_file_paths}"
+
+- job-template:
+    name: "{project-name}-pypi-merge-{stream}"
+    id: github-pypi-merge
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_merge_builders
+
+    pypi-repo: staging
+
+    properties:
+      - github:
+          url: "{github-url}/{github-org}/{project}"
+
+    scm:
+      - lf-infra-github-scm:
+          url: "{git-clone-url}{github-org}/{project}"
+          refspec: ""
+          branch: "refs/heads/{branch}"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: default
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+
+    triggers:
+      - github-pull-request:
+          trigger-phrase: "^remerge$"
+          only-trigger-phrase: false
+          status-context: "Merge"
+          permit-all: true
+          github-hooks: true
+          org-list:
+            - "{github-org}"
+          white-list: "{obj:github_pr_whitelist}"
+          admin-list: "{obj:github_pr_admin_list}"
+          white-list-target-branches:
+            - "{branch}"
+          included-regions: "{obj:github_included_regions}"
+
+- lf_pypi_release_common: &lf_pypi_release_common
+    name: lf-pypi-release-common
+
+    dist-binary: true
+    pypi-repo: pypi
+    use-release-file: true
+
+    gerrit_trigger_file_paths:
+      - compare-type: REG_EXP
+        pattern: '(releases\/.*\.yaml|\.releases\/.*\.yaml)'
+
+    # github_included_regions MUST match gerrit_trigger_file_paths
+    github_included_regions:
+      - 'releases\/.*\.yaml'
+      - '.releases\/.*\.yaml'
+
+    parameters:
+      - lf-infra-parameters:
+          project: "{project}"
+          branch: "{branch}"
+          stream: "{stream}"
+      - lf-infra-tox-parameters:
+          tox-dir: "{tox-dir}"
+          tox-envs: "{tox-envs}"
+      - bool:
+          name: BUILD_BDIST_WHEEL
+          default: "{dist-binary}"
+          description: "Set to True to build a wheel"
+      - string:
+          name: VERSION
+          default: ""
+          description: "This is the version, example: 1.0.0"
+      - bool:
+          name: USE_RELEASE_FILE
+          default: "{use-release-file}"
+          description: "Set to False for job built with parameters"
+      - bool:
+          name: DRY_RUN
+          default: false
+          description: |
+            If DRY_RUN is enabled artifacts are not published.
+
+- job-template:
+    name: "{project-name}-pypi-release-verify-{stream}"
+    id: gerrit-pypi-release-verify
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_release_common
+    <<: *lf_pypi_release_verify_builders
+
+    gerrit_verify_triggers:
+      - patchset-created-event:
+          exclude-drafts: true
+          exclude-trivial-rebase: false
+          exclude-no-code-change: false
+      - draft-published-event
+      - comment-added-contains-event:
+          comment-contains-value: '^Patch Set\s+\d+:\s+(recheck|reverify)\s*$'
+
+    scm:
+      - lf-infra-gerrit-scm:
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+          git-url: "{git-url}"
+          refspec: "$GERRIT_REFSPEC"
+          branch: "$GERRIT_BRANCH"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: gerrit
+
+    triggers:
+      - gerrit:
+          server-name: "{gerrit-server-name}"
+          trigger-on: "{obj:gerrit_verify_triggers}"
+          projects:
+            - project-compare-type: "ANT"
+              project-pattern: "{project}"
+              branches:
+                - branch-compare-type: "ANT"
+                  branch-pattern: "**"
+              file-paths: "{obj:gerrit_trigger_file_paths}"
+
+- job-template:
+    name: "{project-name}-pypi-release-verify-{stream}"
+    id: github-pypi-release-verify
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_release_common
+    <<: *lf_pypi_release_verify_builders
+
+    properties:
+      - github:
+          url: "{github-url}/{github-org}/{project}"
+
+    scm:
+      - lf-infra-github-scm:
+          url: "{git-clone-url}{github-org}/{project}"
+          refspec: ""
+          branch: "refs/heads/{branch}"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: default
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+
+    triggers:
+      - github-pull-request:
+          trigger-phrase: "^(recheck|reverify)$"
+          only-trigger-phrase: false
+          status-context: "PyPI Release Verify"
+          permit-all: true
+          github-hooks: true
+          white-list-target-branches:
+            - "{branch}"
+          included-regions: "{obj:github_included_regions}"
+
+- job-template:
+    name: "{project-name}-pypi-release-merge-{stream}"
+    id: gerrit-pypi-release-merge
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_release_common
+    <<: *lf_pypi_release_merge_builders
+
+    gerrit_release_triggers:
+      - change-merged-event
+
+    scm:
+      - lf-infra-gerrit-scm:
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+          git-url: "{git-url}"
+          refspec: "$GERRIT_REFSPEC"
+          branch: "$GERRIT_BRANCH"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: gerrit
+
+    triggers:
+      - gerrit:
+          server-name: "{gerrit-server-name}"
+          trigger-on: "{obj:gerrit_release_triggers}"
+          projects:
+            - project-compare-type: "ANT"
+              project-pattern: "{project}"
+              branches:
+                - branch-compare-type: "ANT"
+                  branch-pattern: "**"
+              file-paths: "{obj:gerrit_trigger_file_paths}"
+
+- job-template:
+    name: "{project-name}-pypi-release-merge-{stream}"
+    id: github-pypi-release-merge
+    <<: *lf_python_common
+    # yamllint disable-line rule:key-duplicates
+    <<: *lf_pypi_common
+    <<: *lf_pypi_release_common
+    <<: *lf_pypi_release_merge_builders
+
+    properties:
+      - github:
+          url: "{github-url}/{github-org}/{project}"
+
+    scm:
+      - lf-infra-github-scm:
+          url: "{git-clone-url}{github-org}/{project}"
+          refspec: ""
+          branch: "refs/heads/{branch}"
+          submodule-recursive: "{submodule-recursive}"
+          submodule-timeout: "{submodule-timeout}"
+          submodule-disable: "{submodule-disable}"
+          choosing-strategy: default
+          jenkins-ssh-credential: "{jenkins-ssh-credential}"
+
+    triggers:
+      - github-pull-request:
+          trigger-phrase: "^(remerge)$"
+          only-trigger-phrase: false
+          status-context: "PyPI Release Merge"
+          permit-all: true
+          github-hooks: true
+          white-list-target-branches:
+            - "{branch}"
+          included-regions: "{obj:github_included_regions}"
diff --git a/releasenotes/notes/lf-pypi-merge-release-jobs-b729bc2331155364.yaml b/releasenotes/notes/lf-pypi-merge-release-jobs-b729bc2331155364.yaml
new file mode 100644 (file)
index 0000000..34947a2
--- /dev/null
@@ -0,0 +1,13 @@
+---
+features:
+  - |
+    New templates to build and push Python source and binary distributions
+    to a PyPI server. Includes:
+    ``{project-name}-pypi-verify-{stream}``, ``gerrit-pypi-verify``,
+    ``github-pypi-verify``,
+    ``{project-name}-pypi-merge-{stream}``, ``gerrit-pypi-merge``,
+    ``github-pypi-merge``,
+    ``{project-name}-pypi-release-verify-{stream}``,
+    ``gerrit-pypi-release-verify``, ``github-pypi-release-verify``,
+    ``{project-name}-pypi-release-merge-{stream}``,
+    ``gerrit-pypi-release-merge``, ``github-pypi-release-merge``,
diff --git a/schema/release-pypi-schema.yaml b/schema/release-pypi-schema.yaml
new file mode 100644 (file)
index 0000000..e56eed6
--- /dev/null
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: EPL-1.0
+##############################################################################
+# Copyright (c) 2019 The Linux Foundation and others.
+#
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Eclipse Public License v1.0
+# which accompanies this distribution, and is available at
+# http://www.eclipse.org/legal/epl-v10.html
+##############################################################################
+---
+$schema: "http://json-schema.org/schema#"
+$id: "https://github.com/lfit/releng-global-jjb/blob/master/release-pypi-schema.yaml"
+
+required:
+  - "distribution_type"
+  - "project"
+  - "version"
+
+properties:
+  distribution_type:
+    type: "string"
+  project:
+    type: "string"
+  version:
+    type: "string"
diff --git a/shell/pypi-dist-build.sh b/shell/pypi-dist-build.sh
new file mode 100644 (file)
index 0000000..eb10371
--- /dev/null
@@ -0,0 +1,33 @@
+#!/bin/bash
+# SPDX-License-Identifier: EPL-1.0
+##############################################################################
+# Copyright (c) 2019 The Linux Foundation and others.
+#
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Eclipse Public License v1.0
+# which accompanies this distribution, and is available at
+# http://www.eclipse.org/legal/epl-v10.html
+##############################################################################
+echo "---> pypi-dist-build.sh"
+
+# Script to create Python source and binary distributions
+# Requires project file "setup.py"
+
+# Ensure we fail the job if any steps fail.
+set -eu -o pipefail
+
+virtualenv -p python3 /tmp/pypi
+PATH=/tmp/pypi/bin:$PATH
+
+bdist=""
+if $BUILD_BDIST_WHEEL; then
+    echo "INFO: installing wheel to build binary distribution"
+    pip install wheel
+    bdist="bdist_wheel"
+fi
+
+echo "INFO: cd to tox-dir $TOX_DIR"
+cd "$WORKSPACE/$TOX_DIR"
+echo "INFO: creating distributions"
+python3 setup.py sdist $bdist
+echo "---> pypi-dist-build.sh ends"
diff --git a/shell/pypi-tag-release.sh b/shell/pypi-tag-release.sh
new file mode 100644 (file)
index 0000000..d920c54
--- /dev/null
@@ -0,0 +1,151 @@
+#!/bin/bash
+# SPDX-License-Identifier: EPL-1.0
+##############################################################################
+# Copyright (c) 2019 The Linux Foundation and others.
+#
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Eclipse Public License v1.0
+# which accompanies this distribution, and is available at
+# http://www.eclipse.org/legal/epl-v10.html
+##############################################################################
+echo "---> pypi-tag-release.sh"
+
+# Ensure we fail the job if any steps fail.
+set -eu -o pipefail
+
+# Functions.
+
+set_variables(){
+    echo "INFO: Setting variables"
+    # Verify if using release file or parameters
+    if $USE_RELEASE_FILE; then
+        echo "INFO: Checking number of release yaml files"
+        release_files=$(git diff-tree --no-commit-id -r "$GIT_COMMIT" --name-only -- "releases/" ".releases/")
+        if (( $(echo "$release_files" | wc -w) != 1 )); then
+          echo "ERROR: RELEASE FILES: $release_files"
+          echo "ERROR: Committing multiple release files in the same commit OR rename/amend of existing files is not supported."
+          exit 1
+        else
+          release_file="$release_files"
+          echo "INFO: RELEASE FILE: $release_file"
+        fi
+    else
+        echo "INFO: This job is built with parameters, no release file"
+        release_file="None"
+    fi
+
+    if [[ -z ${DISTRIBUTION_TYPE:-} ]]; then
+        echo "INFO: reading DISTRIBUTION_TYPE from file $release_file"
+        DISTRIBUTION_TYPE="$(niet ".distribution_type" "$release_file")"
+    fi
+    if [[ -z ${VERSION:-} ]]; then
+        echo "INFO: reading VERSION from file $release_file"
+        VERSION="$(niet ".version" "$release_file")"
+    fi
+
+    # Display Release Information
+    printf "\t%-30s\n" RELEASE_ENVIRONMENT_INFO:
+    printf "\t%-30s %s\n" RELEASE_FILE: $release_file
+    printf "\t%-30s %s\n" JENKINS_HOSTNAME: $JENKINS_HOSTNAME
+    printf "\t%-30s %s\n" SILO: $SILO
+    printf "\t%-30s %s\n" PROJECT: $PROJECT
+    printf "\t%-30s %s\n" PROJECT-DASHED: ${PROJECT//\//-}
+    printf "\t%-30s %s\n" DISTRIBUTION_TYPE: $DISTRIBUTION_TYPE
+    printf "\t%-30s %s\n" VERSION: $VERSION
+}
+
+# needs to run in the repository root
+verify_schema(){
+    echo "INFO: Fetching schema"
+    pypi_schema="release-pypi-schema.yaml"
+    wget https://raw.githubusercontent.com/lfit/releng-global-jjb/master/schema/${pypi_schema}
+    echo "INFO: Verifying $release_file against schema $pypi_schema"
+    lftools schema verify "$release_file" "$pypi_schema"
+    echo "INFO: $release_file passed schema verification"
+}
+
+verify_version(){
+    # Verify allowed patterns "v#.#.#" or "#.#.#" aka SemVer
+    echo "INFO: Verifying version string $VERSION"
+    allowed_version_regex="^((v?)([0-9]+)\.([0-9]+)\.([0-9]+))$"
+    if [[ $VERSION =~ $allowed_version_regex ]]; then
+        echo "INFO: The version $VERSION is a valid semantic version"
+    else
+        echo "ERROR: The version $VERSION is not a valid semantic version"
+        echo "ERROR: Allowed versions are \"v#.#.#\" or \"#.#.#\" aka SemVer"
+        echo "ERROR: See https://semver.org/ for more details on SemVer"
+        exit 1
+    fi
+}
+
+verify_dist(){
+    # Verify all file names in dist folder have the expected version string
+    dir="$WORKSPACE/$TOX_DIR/dist"
+    echo "INFO: Listing files in $dir"
+    ls $dir
+    echo "INFO: Checking files in $dir for $VERSION"
+    if unex_files=$(find $dir | grep -v $VERSION | egrep -v "^$dir$"); then
+        echo "ERROR: found unexpected files: $unex_files"
+        exit 1
+    else
+        echo "INFO: All file names have expected string ${VERSION}"
+    fi
+}
+
+# TODO: how to tag Github?
+tag_gerrit(){
+    echo "INFO: Verifying tag $VERSION in repo"
+    # Import public signing key
+    gpg --import "$SIGNING_PUBKEY"
+    # Fail if tag exists
+    if git tag -v "$VERSION"; then
+        echo "ERROR: Repo already tagged"
+        exit 1
+    else
+        echo "INFO: Repo has not yet been tagged"
+    fi
+    echo "INFO: Tagging repo"
+    git tag -am "${PROJECT//\//-} $VERSION" "$VERSION"
+    echo "INFO: Signing tag"
+    sigul --batch -c "$SIGUL_CONFIG" sign-git-tag "$SIGUL_KEY" "$VERSION" < "$SIGUL_PASSWORD"
+    echo "INFO: Verifying tag"
+    # may fail due to missing public key
+    if ! git tag -v "$VERSION"; then
+        echo "WARN: failed to verify tag, continuing anyhow"
+    fi
+    # The verify job also calls this script
+    if [[ ! $JOB_NAME =~ "merge" ]] ; then
+        echo "INFO: job is not a merge, skipping push"
+    else
+        echo "INFO: configuring Gerrit remote"
+        gerrit_ssh=$(echo "$GERRIT_URL" | awk -F"/" '{print $3}')
+        git remote set-url origin "ssh://$RELEASE_USERNAME@$gerrit_ssh:29418/$PROJECT"
+        git config user.name "$RELEASE_USERNAME"
+        git config user.email "$RELEASE_EMAIL"
+        echo -e "Host $gerrit_ssh\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config
+        chmod 600 ~/.ssh/config
+        if $DRY_RUN; then
+            echo "INFO: dry run, skipping push"
+        else
+            echo "INFO: pushing tag"
+            git push origin "$VERSION"
+        fi
+    fi
+}
+
+# Main
+virtualenv -p python3 /tmp/pypi
+PATH=/tmp/pypi/bin:$PATH
+pip install lftools jsonschema niet
+set_variables
+if [[ $DISTRIBUTION_TYPE != "pypi" ]]; then
+    echo "ERROR: unexpected distribution type $DISTRIBUTION_TYPE"
+    exit 1
+fi
+if $USE_RELEASE_FILE; then
+    verify_schema
+fi
+verify_version
+verify_dist
+tag_gerrit
+echo "---> pypi-tag-release.sh ends"
diff --git a/shell/pypi-upload.sh b/shell/pypi-upload.sh
new file mode 100644 (file)
index 0000000..e1ecfdf
--- /dev/null
@@ -0,0 +1,34 @@
+#!/bin/bash
+# SPDX-License-Identifier: EPL-1.0
+##############################################################################
+# Copyright (c) 2019 The Linux Foundation and others.
+#
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Eclipse Public License v1.0
+# which accompanies this distribution, and is available at
+# http://www.eclipse.org/legal/epl-v10.html
+##############################################################################
+echo "---> pypi-upload.sh"
+
+# Script to publish Python distributions from a folder
+# to the PyPI repository in $REPOSITORY which must be a
+# key in the .pypirc file
+
+# Ensure we fail the job if any steps fail.
+set -eu -o pipefail
+
+virtualenv -p python3 /tmp/pypi
+PATH=/tmp/pypi/bin:$PATH
+
+pip install twine
+echo "INFO: cd to tox-dir $TOX_DIR"
+cd "$WORKSPACE/$TOX_DIR"
+cmd="twine upload -r $REPOSITORY dist/*"
+if $DRY_RUN; then
+    echo "INFO: dry-run is set, echoing command only"
+    echo $cmd
+else
+    echo "INFO: uploading distributions"
+    $cmd
+fi
+echo "---> pypi-upload.sh ends"