zuul_bubblewrap_copy_issue.md

Investiagate mystical tmp path when remote is Pod container

The Problem

When executing Ansible tasks targeting remote pod container, any of the Ansible bubble wrapped modules receive the invalid tmp directory which cannot be resolved.

The example of invalid path /~zuul/.ansible/tmp/ansible-tmp-1602657590.3517509-66-230912263536663/source where the ~ part is resovled to HOME directory and breaks everything.

Details

Nodepool config:

labels:
  - name: fedora30-ec2
  - name: fedora-eks
providers:
  - name: ec2
    driver: aws
    profile-name: {{ env }}
    region-name: eu-west-1
    cloud-images:
      - name: fedora30
        username: fedora
        image-filters:
          - name: name
            values:
             - fedora30-18
    pools:
      - name: main
        max-servers: 100
        subnet-id: "{{ zuul_subnet_id }}"
        security-group-id: "{{ zuul_security_group_id }}"
        public-ip-address: False
        host-key-checking: True
        labels:
          - name: fedora30-ec2
            cloud-image: fedora30
            instance-type: t3.medium
            key-name: nodepool

  - name: eks-cluster
    driver: kubernetes
    context: arn:aws:eks:eu-west-1:{{ amazon_account_id }}:cluster/eks-{{ env }}-eu-west-1
    pools:
      - name: main
        labels:
          - name: fedora-eks
            type: pod
            image: docker.example.com/runners/fedora-runner:47

In Zuul any base modules with potential security breach are bubble wrapped to validate only allowed parameters as well as provide tmp folder.

The executing task:

- name: Template out inventory file
  template:
    src: "../templates/infrastructure/inventory.j2"
    dest: "{{ infrastructure_path }}/inventory"

Success result when remote is VM (fedora30-ec2):

{
  "module_args": {
    "_original_basename":"inventory.j2"
    "attributes":NULL
    "backup":false
    "checksum":"11f73911b692a5901c3c53541f26337421c78734"
    "content":NULL
    "delimiter":NULL
    "dest":"/home/fedora/src/github.example.com/repo/app/inventory"
    "directory_mode":NULL
    "follow":false
    "force":true
    "group":NULL
    "local_follow":NULL
    "mode":NULL
    "owner":NULL
    "regexp":NULL
    "remote_src":NULL
    "selevel":NULL
    "serole":NULL
    "setype":NULL
    "seuser":NULL
    "src":"/home/fedora/.ansible/tmp/ansible-tmp-1602628588.9110763-33-249935670401163/source"
    "unsafe_writes":NULL
    "validate":NULL
  }
}

Failed result when remote is pod container (fedora-eks):

failed to transfer file /var/lib/zuul/builds/8e9fec346d584533a26f3afad58775f8/work/tmp/ansible-local-2gqkvcnrh/tmp4anqlvmb/inventory.j2 to '/~zuul/.ansible/tmp/ansible-tmp-1602657590.3517509-66-230912263536663/source':
b''
b"dd: failed to open '/~zuul/.ansible/tmp/ansible-tmp-1602657590.3517509-66-230912263536663/source': No such file or directory\ncommand terminated with exit code 1\n"

The difference is the tmp folder:

/home/fedora/.ansible/tmp/ansible-tmp-1602628588.9110763-33-249935670401163/source
and
/~zuul/.ansible/tmp/ansible-tmp-1602657590.3517509-66-230912263536663/source The /~zuul/ part is invalid and resolved incorectly.

When checking Ansible environment variable for fedora-eks:

HOME=/root

I cannot detect the source where the invalid /~zuul/ is being set. :(

As pointed in the #zuul IRC the issue could have something to do with how ansible copies files when kubectl is used: https://github.com/ansible/ansible/blob/stable-2.9/lib/ansible/plugins/connection/kubectl.py#L321
The ~ might be never expanded

zuul community helped to identify the issue, the problem is very trivial
https://github.com/ansible/ansible/blob/stable-2.9/lib/ansible/plugins/connection/kubectl.py#L296-L308
the kubectl connection plugin just cannot expand the ~ and they have a comment for possible usage of $HOME for that
atm it just prefixed with /