Skip to content

Instantly share code, notes, and snippets.

@tmehlinger
Created May 3, 2016 20:47
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save tmehlinger/9c2d40d9fbfe3487e547c34a0ef3334a to your computer and use it in GitHub Desktop.
ext_pillar module for decrypting secrets with AWS KMS
# -*- coding: utf-8 -*-
'''
Decrypt secrets in pillar data using AWS KMS.
This module uses boto3 to connect to KMS and decrypt secrets in pillar data
using keys stored in KMS. It will recurse the supplied pillar data, scanning
for any keys marked by the configured delimiter. If it finds any interesting
keys, it will communicate with KMS to decrypt the respective values with the
suppled key ID.
This relies on the way Salt processes pillars by overwriting interesting
values in the pillar with the decrypted data. The KMS ext_pillar looks for a
configured delimiter (``'$'`` by default) on all keys in the pillar to
determine which values it should decrypt.
Salt Master KMS Configuration
=============================
This module shares no configuration with the Salt master.
Configuring the KMS ext_pillar
==============================
The ``key_id`` parameter must be supplied to ``ext_pillar``; this is they ID
of the key it should use to decrypt secrets. The additional authentication
parameters, if given, must all be present, though using them is not
recommended (it is better to use instance profiles or let boto3 load
configuration from ``~/.aws``).
Example:
.. code-block:: yaml
ext_pillar:
- kms:
key_id: 00000000-0000-0000-0000-000000000000
aws_access_key_id: <my access key ID>
aws_secret_access_key: <my secret key>
region_name: us-east-1
Using the KMS ext_pillar
========================
Secrets to be decrypted using KMS must be stored as base64-encoded strings. If
you use the AWS CLI to encrypt secrets, the value returned by the ``kms
encrypt`` command can be stored in pillar. If using boto3 directly, the binary
data it returns from the ``encrypt`` API call must be base64 encoded.
For longer encrypted values, it may look nicer to store value as multi-line
strings. The KMS ext_pillar will handle this gracefully as long as values are
passed through the ``yaml_encode`` template filter. This will ensure that the
rendered SLS is properly formed.
For more information on handling multi-line strings in templates, see
`this issue <https://github.com/saltstack/salt/issues/5480#issuecomment-212985324>`_`
Example pillar:
.. code-block:: yaml
# my_pillar.sls
my_database:
host: my-database.example.com
user: db-user
$password: <short base64-encoded value)
my_web_server:
cert: /etc/ssl/my_cert.crt
$key: |
<multi-line>
<base64-encoded>
<data>
.. code-block:: yaml
# my_state.sls
configure_database:
file.managed:
- name: /etc/my-app/db.conf
- template: jinja
configure_web_server:
file.managed:
- name: /etc/ssl/my_cert.key
# yaml_encode is required to correctly handle multi-line strings!
- contents: {{ pillar['my_web_server']['$key'] | yaml_encode }}
.. code-block:: yaml
# db.conf
username = {{ pillar['my_database']['user'] }}
password = {{ pillar['my_database']['$password'] }}
'''
from __future__ import absolute_import
import base64
import logging
import re
import six
log = logging.getLogger(__name__)
try:
import boto3
HAS_BOTO3 = True
except ImportError:
HAS_BOTO3 = False
def __virtual__():
if not HAS_BOTO3:
return False
return 'kms'
__opts__ = {'kms.key'}
def _recurse_pillar(pillar, delimiter, kms):
'''Recurse the pillar data, replacing the values associated with any
interesting keys. This will return a tree with only the keys and values we
updated with decrypted data from KMS so it can be safely merged into the
pillar.
'''
if isinstance(pillar, list):
log.debug('found a list, checking for interesting values')
ret = []
for item in pillar:
item = _recurse_pillar(item, delimiter, kms)
if item:
log.debug('found an interesting value in list, storing it')
ret.append(item)
return ret
if isinstance(pillar, dict):
log.debug('found a dict, checking for interesting values')
ret = {}
for k, value in six.iteritems(pillar):
if k.startswith(delimiter):
log.debug('found %s, decrypting it', k)
if isinstance(value, six.string_types):
# Values must all be base64 encoded but they may end up
# with whitespace in them because of how Salt handles
# multi-line strings. Since it's not valid in a base64-
# encoded string anyway, we can safely strip it out.
blob = base64.b64decode(re.sub(r'\s+', '', value))
result = kms.decrypt(CiphertextBlob=blob)
ret[k] = result['Plaintext']
continue
else:
log.warning('value for %s is not a string type, ignoring',
k)
item = _recurse_pillar(value, delimiter, kms)
if item:
log.debug('found an interesting value in dict, storing it')
ret[k] = item
return ret
# For anything that isn't a collection type, return None so it will get
# filtered from the final result. This will avoid clobbering uninteresting
# values in pillar.
return None
def ext_pillar(minion_id, pillar, key_id=None, delimiter='$', **kw):
'''Decrypt secrets using keys from AWS KMS.
Parameters:
* `key_id`: The ID of the key in KMS to use for decrypting secrets.
* `delimiter`: (optional) The delimiter at the beginning of a key in
pillar which indicates its data should be decrypted.
* `aws_access_key_id`: (optional) Access key to override the one
stored in ``~/.aws/credentials``.
* `aws_secret_access_key`: (optional) Secret access key to override
the one stored in ``~/.aws/credentials``.
* `region_name`: (optional) The region in which the given access key
and secret access key are valid.
If any of ``aws_access_key_id``, ``aws_secret_access_key``, or
``region_name`` are given, they must *all* be given. Using this method of
authentication is *not* recommended; it is far better to use instance
profiles in EC2 or let boto3 load configuration from ``~/.aws``.
'''
if not key_id:
raise RuntimeError('KMS pillar requires a KMS key ID')
log.info('%s is decrypting secrets with KMS', minion_id)
if (kw and set(kw.keys()) != set(['aws_access_key_id',
'aws_secret_access_key',
'region_name'])):
log.error('incomplete AWS credentials given to KMS pillar')
return {}
kms = boto3.client('kms', **kw)
return _recurse_pillar(pillar, delimiter, kms)
@tmehlinger
Copy link
Author

tmehlinger commented May 3, 2016

There is probably a better way to configure it, and I know the test for AWS config options is not right. For instance, using instance profiles, whatever user Salt is running as must have a ~/.aws/config with a region specified, otherwise boto will barf... but there's nothing you can do with the current code because either all or none of the options are required.

Regardless, it works for me and it's a good starting point for others who might find this useful.

@jonathansd1
Copy link

I've gotten this to work and am actively using it. This is a super nice find, so thanks for posting this.

My only question is this -- should we need to designate the KMS key ID? In theory, boto3 uses ciphertext to determine what key needs to be used for decryption, then it's just a matter of managing your IAM/KMS permissions proper so the saltmaster can decrypt the data.

Additionally, I don't see it being used anywhere in here? Just a thought. In the end, this is pretty damn useful. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment