Skip to content

Instantly share code, notes, and snippets.

@hemebond
Last active March 13, 2022 11:51
Show Gist options
  • Save hemebond/4b48df43721adb283e0df404501a4b59 to your computer and use it in GitHub Desktop.
Save hemebond/4b48df43721adb283e0df404501a4b59 to your computer and use it in GitHub Desktop.
A SaltStack AWS Auto Scaling Solution

A SaltStack AWS Auto Scaling Solution

Overview

The AWS Auto Scaling Goup, configured with a customised Cloud-Init file, sends a notification to an SNS Topic, which in turn passes it onto an SQS queue that the Salt Master is subscribed to. A Reactor watches for the auto scaling events and pre-approves the new minion based on its Auto Scaling group name and instance ID.

Salt Master Configuration

These changes require the Salt Master be restarted.

SQS Engine

/etc/salt/master.d/engines.conf

my_sqs_profile:
  region: us-east-1
  message_format: json
  # Your AWS access key ID and secret key
  # for the account that will access the SQS queue
  keyid: YOURAWSACCESSID
  key: Your4WSS3cr3tK3y1D

engines:
# Add more sqs_events list items to watch more SQS queues
- sqs_events:
    queue: my_sqs_queue
    profile: my_sqs_profile
    tag: salt/engine/sqs/autoscaling

Reactor

/etc/salt/master.d/reactor.conf

reactor:
  # Autoscaling reactor based on the SQS engine events
  - 'salt/engine/sqs/autoscaling':
    - '/srv/salt/reactors/my_ec2_autoscaling.sls'

Reactor

/srv/salt/reactors/my_ec2_autoscaling.sls

#!py
import json

def run():
	'''
	Run the reactor
	'''
	ret = {}

	sns = json.loads(data['message'])
	message = json.loads(sns['Message'])
	details = message['Details']

	group_name = str(message['AutoScalingGroupName'])
	instance_id = str(message['EC2InstanceId'])

	if 'launch' in sns['Subject']:
		# Fire off an event to wait for the machine
		ret = {
			'ec2_autoscale_autosign': {
				'runner.my_runner.autosign': [{'name': group_name + instance_id}]
			}
		}
	elif 'termination' in sns['Subject']:
		ret = {
			'ec2_autoscale_termination': {
				'wheel.key.delete': [
					{'match': group_name + instance_id},
				]
			}
		}

	return ret

Custom Runner

This is the custom runner to pre-approve the new minions. Update your master configuration like so:

module_dirs:
  - /srv/salt/extmods

To allow the use of custom runners (and other custom modules).

/srv/salt/extmods/runners/my_runner.py

'''
A custom runner for Salt Master tasks
'''
import os.path

def autosign(name, output=True):
    '''
    Create a file in minions_autosign to pre-approve a minion
    '''
    ret = {}
    autosign_key = os.path.join(__opts__['pki_dir'], 'minions_autosign', name)
    open(autosign_key, 'a').close()

    ret['key'] = autosign_key

    return ret

AWS Configuration

Auto Scaling Group

When configuring the Auto Scaling Group, use the cloud-init file at the bottom of the article as the userdata for new instances, replacing the _GRP_ placeholder with the name of your group. New instances will be expected to have their instance ID prefixed with the group name as their hostname and minion ID.

SQS Queue

Create the SQS queue that your Salt Master will connect to:

$ aws sqs create-queue --queue-name my_sqs_queue
{
    "QueueUrl": "https://queue.amazonaws.com/xxxxxxxxxxxx/my_sqs_queue"
}

We will need to know the ARN for the new queue:

$ aws sqs get-queue-attributes --queue-url https://queue.amazonaws.com/xxxxxxxxxxxx/my_sqs_queue --attribute-names QueueArn
{
    "Attributes": {
        "QueueArn": "arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue"
    }
}

SNS Topic

Create an SNS Topic to receive notifications from your Auto Scaling Group:

$ aws sns create-topic --name my_sns_topic
{
    "TopicArn": "arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic"
}

Auto Scaling Notifications

Using the new SNS Topic ARN, add notification configurations to your Auto Scaling Group:

$ aws autoscaling put-notification-configuration --auto-scaling-group-name my_auto_group --topic-arn arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic --notification-type autoscaling:EC2_INSTANCE_LAUNCH
$ aws autoscaling put-notification-configuration --auto-scaling-group-name my_auto_group --topic-arn arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic --notification-type autoscaling:EC2_INSTANCE_TERMINATE

Subscription

Now we need to subscribe the new SQS Queue to the SNS Topic:

$ aws sns subscribe --topic-arn arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic --protocol sqs --notification-endpoint arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue
{
    "SubscriptionArn": "arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

Permissions

Grant the SNS Topic permission to push notifications to the SQS Queue. This requires explicitly setting the policy document so we can set a conditional. You will also need the ARN for the user that the Salt Master will be connecting as. The resulting policy document will be:

{
  "Version": "2012-10-17",
  "Id": "arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue/SQSDefaultPolicy",
  "Statement": [
    {
      "Sid": "FromMySNSTopicToMySQSQueue",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic"
        }
      }
    },
    {
      "Sid": "SaltStackReadAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxxxxxxxxx:user/saltstack.user"
      },
      "Action": [
        "SQS:ReceiveMessage",
        "SQS:DeleteMessage",
	"SQS:GetQueueUrl"
      ],
      "Resource": "arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue"
    }
  ]
}
#cloud-config
# Set hostname to match the instance ID, rather than the
# automatic hostname based on the IP address.
# In these three commands _GRP_ is a placeholder and
# should be changed to your Auto Scaling Group name.
bootcmd:
# Set the hostname and hosts file entry
- "cloud-init-per instance my_set_hostname sh -xc \"echo _GRP_$INSTANCE_ID > /etc/hostname; hostname -F /etc/hostname\""
- "cloud-init-per instance my_etc_hosts sh -xc \"sed -i -e '/^127.0.1.1/d' etc/hosts; echo 127.0.1.1 _GRP_$INSTANCE_ID >> /etc/hosts\""
- "mkdir -p /etc/salt/; echo _GRP_$INSTANCE_ID > /etc/salt/minion_id"
# Preserve the hostname file since we've had to manually edit it
preserve_hostname: true
# Don't let cloud-init update the hosts file since we have edited it manually
manage_etc_hosts: false
# Add the official Saltstack Debian repo and key
apt_sources:
- source: deb http://repo.saltstack.com/apt/debian/8/amd64/latest jessie main
key: |-
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v2
mQENBFOpvpgBCADkP656H41i8fpplEEB8IeLhugyC2rTEwwSclb8tQNYtUiGdna9
m38kb0OS2DDrEdtdQb2hWCnswxaAkUunb2qq18vd3dBvlnI+C4/xu5ksZZkRj+fW
tArNR18V+2jkwcG26m8AxIrT+m4M6/bgnSfHTBtT5adNfVcTHqiT1JtCbQcXmwVw
WbqS6v/LhcsBE//SHne4uBCK/GHxZHhQ5jz5h+3vWeV4gvxS3Xu6v1IlIpLDwUts
kT1DumfynYnnZmWTGc6SYyIFXTPJLtnoWDb9OBdWgZxXfHEcBsKGha+bXO+m2tHA
gNneN9i5f8oNxo5njrL8jkCckOpNpng18BKXABEBAAG0MlNhbHRTdGFjayBQYWNr
YWdpbmcgVGVhbSA8cGFja2FnaW5nQHNhbHRzdGFjay5jb20+iQE4BBMBAgAiBQJT
qb6YAhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRAOCKFJ3le/vhkqB/0Q
WzELZf4d87WApzolLG+zpsJKtt/ueXL1W1KA7JILhXB1uyvVORt8uA9FjmE083o1
yE66wCya7V8hjNn2lkLXboOUd1UTErlRg1GYbIt++VPscTxHxwpjDGxDB1/fiX2o
nK5SEpuj4IeIPJVE/uLNAwZyfX8DArLVJ5h8lknwiHlQLGlnOu9ulEAejwAKt9CU
4oYTszYM4xrbtjB/fR+mPnYh2fBoQO4d/NQiejIEyd9IEEMd/03AJQBuMux62tjA
/NwvQ9eqNgLw9NisFNHRWtP4jhAOsshv1WW+zPzu3ozoO+lLHixUIz7fqRk38q8Q
9oNR31KvrkSNrFbA3D89uQENBFOpvpgBCADJ79iH10AfAfpTBEQwa6vzUI3Eltqb
9aZ0xbZV8V/8pnuU7rqM7Z+nJgldibFk4gFG2bHCG1C5aEH/FmcOMvTKDhJSFQUx
uhgxttMArXm2c22OSy1hpsnVG68G32Nag/QFEJ++3hNnbyGZpHnPiYgej3FrerQJ
zv456wIsxRDMvJ1NZQB3twoCqwapC6FJE2hukSdWB5yCYpWlZJXBKzlYz/gwD/Fr
GL578WrLhKw3UvnJmlpqQaDKwmV2s7MsoZogC6wkHE92kGPG2GmoRD3ALjmCvN1E
PsIsQGnwpcXsRpYVCoW7e2nW4wUf7IkFZ94yOCmUq6WreWI4NggRcFC5ABEBAAGJ
AR8EGAECAAkFAlOpvpgCGwwACgkQDgihSd5Xv74/NggA08kEdBkiWWwJZUZEy7cK
WWcgjnRuOHd4rPeT+vQbOWGu6x4bxuVf9aTiYkf7ZjVF2lPn97EXOEGFWPZeZbH4
vdRFH9jMtP+rrLt6+3c9j0M8SIJYwBL1+CNpEC/BuHj/Ra/cmnG5ZNhYebm76h5f
T9iPW9fFww36FzFka4VPlvA4oB7ebBtquFg3sdQNU/MmTVV4jPFWXxh4oRDDR+8N
1bcPnbB11b5ary99F/mqr7RgQ+YFF0uKRE3SKa7a+6cIuHEZ7Za+zhPaQlzAOZlx
fuBmScum8uQTrEF5+Um5zkwC7EXTdH1co/+/V/fpOtxIg4XO4kcugZefVm5ERfVS
MA==
=dtMN
-----END PGP PUBLIC KEY BLOCK-----
apt_update: True
packages:
- wget
- python-pip
# Use en_US because it's easier
locale: en_US.UTF-8
# timezone: always use UTC for the timezone
timezone: UTC
# This will install salt-minion with these configuration settings
salt_minion:
conf:
master: salt
transport: tcp
hash_type: sha256

SQS Message

An example of the message the Salt Master will receive from the SQS queue:

salt/engine/sqs/my_sqs_queue	{
    "_stamp": "2016-10-11T03:44:57.968671", 
    "message": "..."
}

SNS Message

The "message" attribute of the SQS message needs to be parsed as JSON and will result in:

{
  "Message": "...",
  "MessageId": "xxxx...xxxx",
  "SignatureVersion": "1",
  "SigningCertURL": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService...",
  "Timestamp": "2016-10-11T01:32:41.674Z",
  "UnsubscribeURL": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic:xxxx...xxxx"
}

And in turn the SNS Message attribute also needs to be parsed as JSON and will have:

{
  "AccountId": "xxxxxxxxxxxx",
  "ActivityId": "xxxx...xxxx",
  "AutoScalingGroupARN": "arn:aws:autoscaling:us-east-1:xxxxxxxxxxxx:autoScalingGroup:xxxx...xxxx:autoScalingGroupName/my_auto_group",
  "AutoScalingGroupName": "my_auto_group",
  "Cause": "At 2016-10-11T01:29:33Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 0 to 1.",
  "Description": "Launching a new EC2 instance: i-xxxxxxxx",
  "Details": {
    "Availability Zone": "us-east-1a",
    "Subnet ID": "subnet-xxxxxxxx"
  },
  "EC2InstanceId": "i-xxxxxxxx",
  "EndTime": "2016-10-11T01:32:41.620Z",
  "Event": "autoscaling:EC2_INSTANCE_LAUNCH",
  "Progress": 50,
  "RequestId": "xxxx...xxxx",
  "Service": "AWS Auto Scaling",
  "StartTime": "2016-10-11T01:29:34.649Z",
  "StatusCode": "InProgress",
  "StatusMessage": "",
  "Time": "2016-10-11T01:32:41.620Z"
}
@hemebond
Copy link
Author

The permissions on the SQS have been updated. The Saltstack SQS engine requires permission to both receive and delete messages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment