How to use different Hiera/Eyaml keys for different environments using the AWS Parameter Store to store the encryption keys for Hiera/Eyaml.

gistfile1.md

KMS key policy to allow the role to decrypt and describe the KMS key(s)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE>"
        ]
      },
      "Action": [
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    }
  ]
}

Role policy to allow getting the parameter

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:DescribeParameters"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameters"
      ],
      "Resource": [
        "arn:aws:ssm:<REGION>:<ACCOUNT_ID>:parameter/<LOCATION>.*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt"
      ],
      "Resource": [
        "<KMS_KEY_ARN>"
      ]
    }
  ]
}

That policy is then attached to the role marked as in the key policy. And the role is then attached to the/a instance profile and that profile is attached to the instance.

Retrieving the parameter/eyaml keys

describe_tag() {
    ec2-describe-tags \
            --region "$(head -n1 /etc/puppet_region.conf | sed 's@[a-z]$@@')" \
            --filter "resource-type=instance" \
            --filter "resource-id=$(head -n1 /etc/puppet_instance.conf)" \
            --filter "key=${1}" | \
            sed 's@.*\t@@'
}

# Extract tags from the instance.
ec2metadata --availability-zone > /etc/puppet_region.conf
ec2metadata --instance-id       > /etc/puppet_instance.conf
describe_tag environment        > /etc/puppet_environment.conf
describe_tag location           > /etc/puppet_location.conf

# Get the Hiera/eyaml keys from the 'Parameter Store'.
mkdir -p /etc/apache2/puppet
chown puppet:puppet /etc/apache2/puppet
chmod 0750 /etc/apache2/puppet

for type in public private; do
    aws ssm get-parameters --names "$(cat /etc/puppet_location.conf).hierakey.${type}" --with-decryption --region <REGION> | \
            jq -r '.Parameters[0].Value' \
            > /etc/apache2/puppet/$(cat /etc/puppet_environment.conf)-$(cat /etc/puppet_location.conf)_${type}.pem
done

So my Hiera/Eyaml key parameters are named like so: <LOCATION>.hierakey.[public|private] but I could technically retrieve anything that's named <LOCATION>.* (because of the policy above that states <LOCATION>.*).

This means that one could have:

auth.hierakey.public
auth.hierakey.private
jenkins.hierakey.public
jenkins.hierakey.private
rabbitmq.hierakey.public
rabbitmq.hierakey.private

etc, etc, all with different Hiera/Eyaml keys.

Then it's "just" a matter of tagging the instance(s) with the correct values and have the bootstrap script create the /etc/puppet_*.conf files.

In my example code snippet above, I have simply choosen to call these tags "environment" and "location" for simplicity.

Because of the way the role policy is written, any instance with the 'location=auth' tag will only be allowed to read those keys. Meaning, that my Jenkins instance(s) (with 'location=jenkins' tag) will NOT be able to read the auth.* keys etc.

Technically, this could be circumvented, but by tweaking the KMS key policy (by attaching individual roles for the different environments instances), this could be futher limited. Meaning, that you can create multiple keys (one per environment/location), which is only usable by one role (so one per environment/location in this case as well).

Upload Hiera/Eyaml key to the Parameter Store

To upload/update the Hiera/Eyaml keys when/if they need to be recycled, is somewhat more complex. This because you first have to decrypt all Hiera/Eyaml secrets with the old keys, create new keys and encrypt all secrets again with the new keys.

Then the old keys would have to be cleaned up and new keys downloaded on the host(s).

Decrypt the file(s) using the old keys.

eyaml decrypt <file(s)>

Create New keys:

eyaml createkeys

Reencrypt the file(s) with the new keys:

eyaml encrypt <file(s)>

Upload new keys to the Parameter Store:

aws ssm put-parameter --overwrite --name <LOCATION>.hierakey.<TYPE> --type SecureString --value "$(cat "<KEYFILE>")"