Minimal repro for failures seen with saltutil.sync_all in provisioning with Packer

README.md

What..

This is a minimal reproduction of an issue I am seeing when provisioning hosts with Saltstack and Packer. More specifically, running salt-call --local saltutil.sync_all in a script uploaded by packer does not function as expected (no files/modules are sync'd).

Note that saltutil.sync_all works fine on the new host when run manually from the shell directly.

How to use..

• Download and install packer to your $PATH from https://www.packer.io/downloads.html
• Create a new directory and copy the setup_salt.sh, base-host.json, and config.json files from this gist to that new directory
• This build uses AWS, so you'll need an existing VPC and subnet. Grab the VPC and Subnet ID, and put those in config.json.
• Update config.json with an Access Key ID and Secret Key you can use.
• Run packer build -var-file=config.json build.json
• If everything works, you'd see the foobar.py script sync with saltutil.sync_all, and you'll see salt run the foobar.test module with hello test printed to stdout. If it fails, you won't see foobar.py sync'd, and you'll see Module 'foobar' is not available as output.

build.json

{
  "variables": {
    "vpc_id": "",
    "subnet_id": "",
    "source_ami": "",
    "region": "us-west-1",
    "aws_access_key": "",
    "aws_secret_key": ""
  },
  "builders": [ {
      "type": "amazon-ebs",
      "source_ami": "{{user `source_ami`}}",
      "instance_type": "t2.small",
      "vpc_id": "{{user `vpc_id`}}",
      "subnet_id": "{{user `subnet_id`}}",
      "region": "{{user `region`}}",
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key`}}",
      "ami_name": "base-host-{{ isotime | clean_ami_name }}",
      "ssh_username": "ubuntu"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "execute_command": "sudo -H -S sh '{{.Path}}'",
      "script": "setup_salt.sh"
    }
  ]
}

config.json

{
    "source_ami": "ami-65579105",
    "vpc_id": "vpc-XXXXXXXX",
    "subnet_id": "subnet-XXXXXXXX",
    "aws_access_key": "FOOBAR",
    "aws_secret_key": "FOOBAR"
}

setup_salt.sh

#!/bin/sh
# import the SaltStack repository key
wget -O - https://repo.saltstack.com/apt/ubuntu/14.04/amd64/2016.3/SALTSTACK-GPG-KEY.pub | apt-key add -
# point apt at the official saltstack repo
echo "deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/2015.5 trusty main" > /etc/apt/sources.list.d/saltstack.list
# ensure PPAs are active and install the salt minion!
apt-get update
apt-get install -y salt-minion
# disable the service until configured
service salt-minion stop

# create a mock module for testing
mkdir -p /srv/salt/_modules/
cat <<EOF > /srv/salt/_modules/foobar.py
#!/usr/bin/env python

def test():
    print("hello test")
EOF
ls -alh /srv/salt/*
salt-call --local saltutil.sync_all

# pause to ensure there's no overlap
sleep 10
ls -alh /var/cache/salt/minion/extmods/*
salt-call --local foobar.test