Skip to content

Instantly share code, notes, and snippets.

@hatt
Last active March 13, 2017 05:35
Show Gist options
  • Save hatt/6552408 to your computer and use it in GitHub Desktop.
Save hatt/6552408 to your computer and use it in GitHub Desktop.
Dynamically scoping Redis with AWS Elasticache to use the local availability zone. Batch writing is used for the shipper to reduce usage costs.
$redis_host = $::ec2_placement_availability_zone ? {
'us-west-1a' => 'logging-redis-a.<redacted>.0001.usw1.cache.amazonaws.com',
'us-west-1b' => 'logging-redis-b.<redacted>.0001.usw1.cache.amazonaws.com'
}
logstash::output::redis { 'redis-shipper':
batch => true,
host => $redis_host,
key => 'logstash',
data_type => 'list'
}
logstash::input::redis { 'redis-indexer':
host => $redis_host,
key => 'logstash',
data_type => 'list'
}
@brainstorm
Copy link

Valuable insight from this config by @hatt:

you could do something like set a variable (maybe $region_id) to just be a or b or something
and interpolate it for entire hostgroups as an array so if you had 6 redis servers, 3 in each 
AZ for whatever reason, just pass the one var and popular the host array

btw, I can confirm that using only the local AZ in LS/redis/elasticache saves $1/100GB of 
traffic

what'd be really cool is if you say, had a listener for AZ health or something, and if a 
particular AZ had elasticache issues you could swap LS out temporarily to a different 
node by triggering a puppet run, it'd cost you a bit but it;d prevent LS backing up :)

Thanks Mr Hatt!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment