From: Lott, Christopher (cl778h) Date: Wed, 25 Mar 2020 22:15:17 +0000 (-0400) Subject: Add PyPI stage templates X-Git-Tag: v0.53.0~12^2 X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=commitdiff_plain;h=refs%2Fchanges%2F09%2F63509%2F4;p=releng%2Fglobal-jjb.git Add PyPI stage templates Add templates gerrit-pypi-stage/github-pypi-stage to allow projects flexibility to control when Jenkins publishes a redistributable Python package, either on merge (with the merge template) or on demand (with the stage template). Change-Id: I49d8101b9da8c58d10518cb3ad522cd71b1019dc Signed-off-by: Lott, Christopher (cl778h) --- diff --git a/.jjb-test/lf-python-jobs.yaml b/.jjb-test/lf-python-jobs.yaml index 49f05ec0..657a1d80 100644 --- a/.jjb-test/lf-python-jobs.yaml +++ b/.jjb-test/lf-python-jobs.yaml @@ -6,9 +6,10 @@ - gerrit-pypi-merge - gerrit-pypi-release-merge - gerrit-pypi-release-verify + - gerrit-pypi-stage + - gerrit-pypi-verify - gerrit-tox-sonar - gerrit-tox-sonarqube - - gerrit-pypi-verify project-name: gerrit-python @@ -19,9 +20,10 @@ - github-pypi-merge - github-pypi-release-merge - github-pypi-release-verify + - github-pypi-stage + - github-pypi-verify - github-tox-sonar - github-tox-sonarqube - - github-pypi-verify project-name: github-python diff --git a/docs/jjb/lf-python-jobs.rst b/docs/jjb/lf-python-jobs.rst index eb110095..475bc2ab 100644 --- a/docs/jjb/lf-python-jobs.rst +++ b/docs/jjb/lf-python-jobs.rst @@ -341,6 +341,7 @@ uses configuration parameters in the umbrella project's defaults.yaml file. jobs: - gerrit-tox-sonarqube + Tox Verify ---------- @@ -464,12 +465,15 @@ variables before running. PyPI Merge ---------- -Creates and uploads distribution files on merge of a patch set. Runs -tox, builds a source distribution and (optionally) a binary +Creates and uploads package distribution files on merge of a patch set. +Runs tox, builds a source distribution and (optionally) a binary distribution, and uploads the distribution(s) to a PyPI repository. The project git repository must have a setup.py file with configuration for packaging the component. +Projects can choose **either** this template to publish on merge, +**or** the Stage template to publish on command. + This job should use a staging repository like testpypi.python.org, which sets up use of release jobs to promote the distributions later. This job can also use a public release area like the global PyPI @@ -489,50 +493,9 @@ pyenv variables before running. export PYENV_ROOT="/opt/pyenv" export PATH="$PYENV_ROOT/bin:$PATH" -Installable package projects should use the directory layout shown -below. All Python files are in a repo subdirectory separate from -non-Python files like documentation. This layout allows highly -specific build-job triggers in Jenkins using the subdirectory -paths. For example, a PyPI merge job should not run on a non-Python -file change such as documentation, because the job cannot upload the -same package twice. - -To make the document files available for building a Python package -long description in setup.py, add a symbolic link "docs" in the -package subdirectory pointing to the top-level docs directory. +See the recommended directory layout documented in the PyPI Verify job. -.. code-block:: bash - - git-repo-name/ - │ - ├── docs/ - │ ├── index.rst - │ └── release-notes.rst - │ - ├── helloworld-package/ - │ │ - │ └── helloworld/ - │ │ ├── __init__.py - │ │ ├── helloworld.py - │ │ └── helpers.py - │ │ - │ ├── tests/ - │ │ ├── helloworld_tests.py - │ │ └── helloworld_mocks.py - │ │ - │ ├── requirements.txt - │ └── setup.py - │ └── tox.ini - │ - ├── releases/ - │ └── pypi-helloworld.yaml - │ - ├── .gitignore - ├── LICENSE - └── README.md - - -Jobs built from the PyPI templates depend on a .pypirc configuration file +Jobs using this PyPI template depend on a .pypirc configuration file in the Jenkins builder home directory. An example appears next that uses API tokens. Note that in the [pypi] entry the repository key-value pair is optional, it defaults to pypi.org. @@ -576,9 +539,9 @@ is optional, it defaults to pypi.org. :branch: The branch to build against. (default: master) :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) :build-timeout: Timeout in minutes before aborting build. (default: 15) - :cron: Cron schedule when to trigger the job. Supports daily builds. - This parameter also supports multiline input via YAML pipe | character in - cases where one may want to provide more than 1 cron timer. (default: empty) + :cron: Cron schedule when to trigger the job. Supports regular builds. + Not useful when publishing to pypi.org because that rejects a package + if the version exists. (default: empty) :disable-job: Whether to disable the job (default: false) :dist-binary: Whether to build a binary wheel distribution. (default: true) :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) @@ -590,7 +553,7 @@ is optional, it defaults to pypi.org. :pre-build-script: Shell script to execute before the tox builder. For example, install system prerequisites. (default: a shell comment) :pypi-repo: Key for the PyPI target repository in the .pypirc file, - ideally a server like test.pypy.org. (default: pypi-test) + ideally a server like test.pypi.org. (default: pypi-test) :python-version: Python version to invoke pip install of tox-pyenv (default: python3) :stream: Keyword representing a release code-name. @@ -611,6 +574,117 @@ is optional, it defaults to pypi.org. https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit +PyPI Stage +---------- + +Creates and uploads package distribution files on receipt of a comment. +Runs tox, builds a source distribution and (optionally) a binary +distribution, and uploads the distribution(s) to a PyPI repository. +The project git repository must have a setup.py file with configuration +for packaging the component. + +Projects can choose **either** this template to publish on command, +**or** the Merge template to publish on merge. + +This job should use a staging repository like testpypi.python.org, +which sets up use of release jobs to promote the distributions later. +This job can also use a public release area like the global PyPI +repository if the release process is not needed. These PyPI +repositories allow upload of a package at a specific version once, +they do not allow overwrite of a package. This means that a job +will fail in the upload step if the package version already exists in +the target repository. + +The tox runner is pyenv aware so if the image contains an installation +of pyenv at /opt/pyenv it will pick it up and run Python tests with +the appropriate Python versions. The tox runner sets the following +pyenv variables before running. + +.. code:: bash + + export PYENV_ROOT="/opt/pyenv" + export PATH="$PYENV_ROOT/bin:$PATH" + +See the recommended directory layout documented in the PyPI Verify job. + +Jobs using this PyPI template depend on a .pypirc configuration file +in the Jenkins builder home directory. An example appears next that uses +API tokens. Note that in the [pypi] entry the repository key-value pair +is optional, it defaults to pypi.org. + +.. code-block:: bash + + [distutils] # this tells distutils what package indexes you can push to + index-servers = pypi-test pypi + + [pypi-test] + repository: https://test.pypi.org/legacy/ + username: __token__ + password: pypi-test-api-token-goes-here + + [pypi] + username: __token__ + password: pypi-api-token-goes-here + + +:Template Names: + + - {project-name}-pypi-stage-{stream} + - gerrit-pypi-stage + - github-pypi-stage + +:Comment Trigger: **stage-release** post a comment with the trigger to launch + this job manually. Do not include any other text or vote in the + same comment. + +:Required Parameters: + + :build-node: The node to run the build on. + :jenkins-ssh-credential: Credential to use for SSH. (Generally set + in defaults.yaml) + :mvn-settings: The settings file with credentials for the project + :project: Git repository name + :project-name: Jenkins job name prefix + +:Optional Parameters: + + :branch: The branch to build against. (default: master) + :build-days-to-keep: Days to keep build logs in Jenkins. (default: 7) + :build-timeout: Timeout in minutes before aborting build. (default: 15) + :cron: Cron schedule when to trigger the job. Supports regular builds. + Not useful when publishing to pypi.org because that rejects a package + if the version exists. (default: empty) + :disable-job: Whether to disable the job (default: false) + :dist-binary: Whether to build a binary wheel distribution. (default: true) + :git-url: URL clone project from. (default: $GIT_URL/$PROJECT) + :mvn-opts: Sets MAVEN_OPTS to start up the JVM running Maven. (default: '') + :mvn-params: Parameters to pass to the mvn CLI. (default: '') + :mvn-version: Version of maven to use. (default: mvn35) + :parallel: Boolean indicator for tox to run tests in parallel or series. + (default: false, in series) + :pre-build-script: Shell script to execute before the tox builder. For + example, install system prerequisites. (default: a shell comment) + :pypi-repo: Key for the PyPI target repository in the .pypirc file, + ideally a server like test.pypi.org. (default: pypi-test) + :python-version: Python version to invoke pip install of tox-pyenv + (default: python3) + :stream: Keyword representing a release code-name. + Often the same as the branch. (default: master) + :submodule-recursive: Whether to checkout submodules recursively. + (default: true) + :submodule-timeout: Timeout (in minutes) for checkout operation. + (default: 10) + :submodule-disable: Disable submodule checkout operation. + (default: false) + :tox-dir: Directory containing the project's tox.ini relative to + the workspace. The default uses tox.ini at the project root. + (default: '.') + :tox-envs: Tox environments to run. If blank run everything described + in tox.ini. (default: '') + :gerrit_trigger_file_paths: Override file paths used to filter which file + modifications trigger a build. Refer to JJB documentation for "file-path" details. + https://docs.openstack.org/infra/jenkins-job-builder/triggers.html#triggers.gerrit + PyPI Verify ----------- @@ -619,6 +693,49 @@ then builds a source distribution and (optionally) a binary distribution. The project repository must have a setup.py file with configuration for packaging the component. +Installable package projects should use the directory layout shown +below. All Python files are in a repo subdirectory separate from +non-Python files like documentation. This layout allows highly +specific build-job triggers in Jenkins using the subdirectory +paths. For example, a PyPI publisher job should not run on a non-Python +file change such as documentation, because the job cannot upload the +same package twice. + +To make the document files available for building a Python package +long description in setup.py, add a symbolic link "docs" in the +package subdirectory pointing to the top-level docs directory. + +.. code-block:: bash + + git-repo-name/ + │ + ├── docs/ + │ ├── index.rst + │ └── release-notes.rst + │ + ├── helloworld-package/ + │ │ + │ └── helloworld/ + │ │ ├── __init__.py + │ │ ├── helloworld.py + │ │ └── helpers.py + │ │ + │ ├── tests/ + │ │ ├── helloworld_tests.py + │ │ └── helloworld_mocks.py + │ │ + │ ├── requirements.txt + │ └── setup.py + │ └── tox.ini + │ + ├── releases/ + │ └── pypi-helloworld.yaml + │ + ├── .gitignore + ├── LICENSE + └── README.md + + The tox runner is pyenv aware so if the image contains an installation of pyenv at /opt/pyenv it will pick it up and run Python tests with the appropriate Python versions. The tox runner sets the following diff --git a/jjb/lf-python-jobs.yaml b/jjb/lf-python-jobs.yaml index 9f91a6f7..c3206e59 100644 --- a/jjb/lf-python-jobs.yaml +++ b/jjb/lf-python-jobs.yaml @@ -892,8 +892,8 @@ parallel: "{parallel}" - shell: !include-raw-escape: ../shell/pypi-dist-build.sh -- lf_pypi_merge_builders: &lf_pypi_merge_builders - name: lf-pypi-merge-builders +- lf_pypi_publish_builders: &lf_pypi_publish_builders + name: lf-pypi-publish-builders builders: - lf-infra-pre-build @@ -984,9 +984,9 @@ name: "{project-name}-pypi-merge-{stream}" id: gerrit-pypi-merge <<: *lf_pypi_common - <<: *lf_pypi_merge_builders + <<: *lf_pypi_publish_builders - cron: "" + cron: "" # avoid for pypi which rejects duplicates pypi-repo: pypi-test scm: @@ -1021,7 +1021,7 @@ name: "{project-name}-pypi-merge-{stream}" id: github-pypi-merge <<: *lf_pypi_common - <<: *lf_pypi_merge_builders + <<: *lf_pypi_publish_builders cron: "" pypi-repo: pypi-test @@ -1056,3 +1056,77 @@ white-list-target-branches: - "{branch}" included-regions: "{obj:github_included_regions}" + +- job-template: + name: "{project-name}-pypi-stage-{stream}" + id: gerrit-pypi-stage + <<: *lf_pypi_common + <<: *lf_pypi_publish_builders + + cron: "" + pypi-repo: pypi-test + + gerrit_stage_triggers: + - comment-added-contains-event: + comment-contains-value: '^Patch Set\s+\d+:\s+stage-release\s*$' + + scm: + - lf-infra-gerrit-scm: + jenkins-ssh-credential: "{jenkins-ssh-credential}" + git-url: "{git-url}" + refspec: "$GERRIT_REFSPEC" + branch: "$GERRIT_BRANCH" + submodule-recursive: "{submodule-recursive}" + submodule-timeout: "{submodule-timeout}" + submodule-disable: "{submodule-disable}" + # stage jobs always build from tip + choosing-strategy: default + + triggers: + - timed: "{obj:cron}" + - gerrit: + server-name: "{gerrit-server-name}" + trigger-on: "{obj:gerrit_stage_triggers}" + projects: + - project-compare-type: ANT + project-pattern: "{project}" + branches: + - branch-compare-type: ANT + branch-pattern: "**/{branch}" + file-paths: "{obj:gerrit_trigger_file_paths}" + +- job-template: + name: "{project-name}-pypi-stage-{stream}" + id: github-pypi-stage + <<: *lf_pypi_common + <<: *lf_pypi_publish_builders + + cron: "" + pypi-repo: pypi-test + + properties: + - github: + url: "{github-url}/{github-org}/{project}" + + scm: + - lf-infra-github-scm: + url: "{git-clone-url}{github-org}/{project}" + refspec: "" + branch: "refs/heads/{branch}" + submodule-recursive: "{submodule-recursive}" + submodule-timeout: "{submodule-timeout}" + submodule-disable: "{submodule-disable}" + choosing-strategy: default + jenkins-ssh-credential: "{jenkins-ssh-credential}" + + triggers: + - timed: "{obj:cron}" + - github-pull-request: + trigger-phrase: "^stage-release$" + only-trigger-phrase: true + status-context: "Release" + permit-all: true + github-hooks: true + white-list-target-branches: + - "{branch}" + included-regions: "{obj:github_included_regions}" diff --git a/releasenotes/notes/add-pypi-stage-templates-c33955d759789d53.yaml b/releasenotes/notes/add-pypi-stage-templates-c33955d759789d53.yaml new file mode 100644 index 00000000..d3d9d6fa --- /dev/null +++ b/releasenotes/notes/add-pypi-stage-templates-c33955d759789d53.yaml @@ -0,0 +1,7 @@ +--- +features: + - | + Add templates gerrit-pypi-stage/github-pypi-stage to allow projects + flexibility to control when Jenkins publishes a redistributable + Python package, either on merge (with the merge template) or on + command (with the stage template).