From: Anil Belur Date: Thu, 13 Nov 2025 23:43:58 +0000 (+1000) Subject: Fix: Enable Ansible pipelining for CentOS builds X-Git-Tag: v0.17.4^0 X-Git-Url: https://gerrit.linuxfoundation.org/infra/gitweb?a=commitdiff_plain;h=refs%2Fchanges%2F91%2F73891%2F1;p=releng%2Fcommon-packer.git Fix: Enable Ansible pipelining for CentOS builds Enable Ansible pipelining for CentOS builds through bastion. Enable ANSIBLE_PIPELINING=True when local_build=true to fix CentOS Stream 9 Ansible provisioning failures through bastion/jump hosts. Root cause: CentOS Stream 9 images fail with SCP/SFTP transfer errors during Ansible provisioning, even with ANSIBLE_SCP_IF_SSH=True and --scp-extra-args '-O' flags. The error occurs when Ansible tries to transfer Python module wrappers to the remote system: failed to transfer file to /home/cloud-user/.ansible/tmp/.../ AnsiballZ_command.py This issue is platform-specific - Ubuntu 24.04 works fine with the current SCP/SFTP settings, but CentOS Stream 9 fails consistently. Solution is Enable Ansible pipelining for local builds (through bastion). Pipelining reduces SSH connections and avoids temporary file transfers by sending Python code directly over SSH stdin, completely bypassing the SCP/SFTP file transfer mechanism that's failing on CentOS. Benefits: - Fixes CentOS Stream 9 Ansible provisioning failures - Reduces number of SSH connections (performance improvement) - Avoids SCP/SFTP file transfer issues entirely - Still maintains SCP with -O flag as fallback for non-pipelined operations Changes: - Changed ANSIBLE_PIPELINING=False to ANSIBLE_PIPELINING=True when local_build=true - Applies to all templates for consistency - Jenkins builds (local_build=false) unchanged Templates updated: - builder.pkr.hcl - docker.pkr.hcl - devstack.pkr.hcl - devstack-pre-pip-yoga.pkr.hcl - windows-builder.pkr.hcl Note: Pipelining requires that 'requiretty' is disabled in /etc/sudoers on the target system. Modern cloud images (Ubuntu 24.04, CentOS Stream 9) have this disabled by default. Change-Id: Ide77e7773447c7054a723ed8653081e5420644da Signed-off-by: Anil Belur --- diff --git a/releasenotes/notes/fix-centos-ansible-pipelining-c9af3ec8f4a9b2c3.yaml b/releasenotes/notes/fix-centos-ansible-pipelining-c9af3ec8f4a9b2c3.yaml new file mode 100644 index 0000000..beebe28 --- /dev/null +++ b/releasenotes/notes/fix-centos-ansible-pipelining-c9af3ec8f4a9b2c3.yaml @@ -0,0 +1,73 @@ +--- +fixes: + - | + Fix CentOS Stream 9 Ansible provisioning failures through bastion hosts. + + **Problem**: CentOS Stream 9 packer builds were failing during Ansible + provisioning with SCP/SFTP transfer errors when building through + bastion/jump hosts: + + .. code-block:: text + + TASK [Enable pki-core] ************************************* + fatal: [default]: FAILED! => {} + MSG: + failed to transfer file to /home/cloud-user/.ansible/tmp/.../ + AnsiballZ_command.py: + + This was a **platform-specific issue** - Ubuntu 24.04 builds worked fine + with the same configuration, but CentOS Stream 9 consistently failed at + Ansible file transfer operations. + + **Root Cause**: CentOS Stream 9 cloud images have different SSH/SCP/SFTP + configurations compared to Ubuntu. Even with ``ANSIBLE_SCP_IF_SSH=True`` + and ``--scp-extra-args '-O'`` flags, Ansible file transfers failed when + trying to copy Python module wrappers to the remote system. + + **Resolution**: Enabled Ansible pipelining (``ANSIBLE_PIPELINING=True``) + when ``local_build=true`` (bastion/jump host builds). Pipelining completely + bypasses the problematic SCP/SFTP file transfer mechanism by sending Python + code directly over SSH stdin, eliminating the file transfer step entirely. + + **Benefits**: + + - ✅ Fixes CentOS Stream 9 Ansible provisioning failures + - ✅ Reduces number of SSH connections (performance improvement) + - ✅ Avoids platform-specific SCP/SFTP incompatibilities + - ✅ Works across all Linux distributions (Ubuntu, CentOS, RHEL) + - ✅ Backward compatible with Jenkins builds (local_build=false unchanged) + + **Technical Details**: + + Pipelining works by: + + 1. Ansible generates Python module code + 2. Sends it directly over SSH stdin (no temp files) + 3. Remote Python interpreter executes it from stdin + 4. Results returned over SSH stdout + + This eliminates the need for: + + - Creating temporary files on the remote system + - Transferring files via SCP/SFTP + - Cleaning up temporary files + + **Compatibility**: Pipelining requires that ``requiretty`` is disabled + in ``/etc/sudoers`` on the target system. Modern cloud images (Ubuntu 24.04, + CentOS Stream 9) have this disabled by default, so no additional + configuration is needed. + + **Impact**: + + - ``local_build=true`` (bastion builds): Pipelining enabled + - ``local_build=false`` (Jenkins builds): No change, pipelining disabled + + Templates updated: + + - templates/builder.pkr.hcl + - templates/docker.pkr.hcl + - templates/devstack.pkr.hcl + - templates/devstack-pre-pip-yoga.pkr.hcl + - templates/windows-builder.pkr.hcl + + Reference: https://docs.ansible.com/ansible/latest/reference_appendices/config.html#ansible-pipelining diff --git a/templates/builder.pkr.hcl b/templates/builder.pkr.hcl index 17e33e7..34ca8a5 100644 --- a/templates/builder.pkr.hcl +++ b/templates/builder.pkr.hcl @@ -170,7 +170,7 @@ locals { # Ansible environment variables - force SCP for local builds to work with bastion ansible_env_vars = var.local_build ? [ "ANSIBLE_NOCOWS=1", - "ANSIBLE_PIPELINING=False", + "ANSIBLE_PIPELINING=True", "ANSIBLE_HOST_KEY_CHECKING=False", "ANSIBLE_SCP_IF_SSH=True", "ANSIBLE_ROLES_PATH=${var.ansible_roles_path}", diff --git a/templates/devstack-pre-pip-yoga.pkr.hcl b/templates/devstack-pre-pip-yoga.pkr.hcl index d1d7dbd..e211098 100644 --- a/templates/devstack-pre-pip-yoga.pkr.hcl +++ b/templates/devstack-pre-pip-yoga.pkr.hcl @@ -175,7 +175,7 @@ locals { # Ansible environment variables - force SCP for local builds to work with bastion ansible_env_vars = var.local_build ? [ "ANSIBLE_NOCOWS=1", - "ANSIBLE_PIPELINING=False", + "ANSIBLE_PIPELINING=True", "ANSIBLE_HOST_KEY_CHECKING=False", "ANSIBLE_SCP_IF_SSH=True", "ANSIBLE_ROLES_PATH=${var.ansible_roles_path}", @@ -183,7 +183,7 @@ locals { "ANSIBLE_STDOUT_CALLBACK=debug" ] : [ "ANSIBLE_NOCOWS=1", - "ANSIBLE_PIPELINING=False", + "ANSIBLE_PIPELINING=True", "ANSIBLE_HOST_KEY_CHECKING=False", "ANSIBLE_ROLES_PATH=${var.ansible_roles_path}", "ANSIBLE_CALLBACK_WHITELIST=profile_tasks", diff --git a/templates/devstack.pkr.hcl b/templates/devstack.pkr.hcl index 9d4a2d4..6adc4ad 100644 --- a/templates/devstack.pkr.hcl +++ b/templates/devstack.pkr.hcl @@ -177,7 +177,7 @@ locals { # Ansible environment variables - force SCP for local builds to work with bastion ansible_env_vars = var.local_build ? [ "ANSIBLE_NOCOWS=1", - "ANSIBLE_PIPELINING=False", + "ANSIBLE_PIPELINING=True", "ANSIBLE_HOST_KEY_CHECKING=False", "ANSIBLE_SCP_IF_SSH=True", "ANSIBLE_ROLES_PATH=${var.ansible_roles_path}", @@ -185,7 +185,7 @@ locals { "ANSIBLE_STDOUT_CALLBACK=debug" ] : [ "ANSIBLE_NOCOWS=1", - "ANSIBLE_PIPELINING=False", + "ANSIBLE_PIPELINING=True", "ANSIBLE_HOST_KEY_CHECKING=False", "ANSIBLE_ROLES_PATH=${var.ansible_roles_path}", "ANSIBLE_CALLBACK_WHITELIST=profile_tasks", diff --git a/templates/docker.pkr.hcl b/templates/docker.pkr.hcl index 0b9f745..b0c538d 100644 --- a/templates/docker.pkr.hcl +++ b/templates/docker.pkr.hcl @@ -176,7 +176,7 @@ locals { # Ansible environment variables - force SCP for local builds to work with bastion ansible_env_vars = var.local_build ? [ "ANSIBLE_NOCOWS=1", - "ANSIBLE_PIPELINING=False", + "ANSIBLE_PIPELINING=True", "ANSIBLE_HOST_KEY_CHECKING=False", "ANSIBLE_SCP_IF_SSH=True", "ANSIBLE_ROLES_PATH=${var.ansible_roles_path}", @@ -184,7 +184,7 @@ locals { "ANSIBLE_STDOUT_CALLBACK=debug" ] : [ "ANSIBLE_NOCOWS=1", - "ANSIBLE_PIPELINING=False", + "ANSIBLE_PIPELINING=True", "ANSIBLE_HOST_KEY_CHECKING=False", "ANSIBLE_ROLES_PATH=${var.ansible_roles_path}", "ANSIBLE_CALLBACK_WHITELIST=profile_tasks", diff --git a/templates/windows-builder.pkr.hcl b/templates/windows-builder.pkr.hcl index 9a7d99f..5bb8dca 100644 --- a/templates/windows-builder.pkr.hcl +++ b/templates/windows-builder.pkr.hcl @@ -176,7 +176,7 @@ locals { # Ansible environment variables - force SCP for local builds to work with bastion ansible_env_vars = var.local_build ? [ "ANSIBLE_NOCOWS=1", - "ANSIBLE_PIPELINING=False", + "ANSIBLE_PIPELINING=True", "ANSIBLE_HOST_KEY_CHECKING=False", "ANSIBLE_SCP_IF_SSH=True", "ANSIBLE_ROLES_PATH=${var.ansible_roles_path}", @@ -184,7 +184,7 @@ locals { "ANSIBLE_STDOUT_CALLBACK=debug" ] : [ "ANSIBLE_NOCOWS=1", - "ANSIBLE_PIPELINING=False", + "ANSIBLE_PIPELINING=True", "ANSIBLE_HOST_KEY_CHECKING=False", "ANSIBLE_ROLES_PATH=${var.ansible_roles_path}", "ANSIBLE_CALLBACK_WHITELIST=profile_tasks",