Skip to content

Instantly share code, notes, and snippets.

@nickcarenza

nickcarenza/batch-update-firehose.sh

Created February 1, 2016 18:17
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nickcarenza/aab2946dbe75980b6011 to your computer and use it in GitHub Desktop.
Save nickcarenza/aab2946dbe75980b6011 to your computer and use it in GitHub Desktop.
Download ZIP
Batch update AWS Kinesis Firehose
Raw
batch-update-firehose.sh
# This requires the aws cli and jq
# First pull down existing firehose streams
# This will create files with the name of the stream in a 'streams' folder
mkdir streams
aws firehose list-delivery-streams --limit 100 | jq '.DeliveryStreamNames[]' | xargs -L 1 -I {} sh -c "aws firehose describe-delivery-stream --delivery-stream-name {} > streams/{}.json"
# Make your changes to the files...
# This will push the copy options from each file to firehose
find streams -type f | \
xargs -L 1 \
cat | \
jq -r '.DeliveryStreamDescription | .Destinations[0] as $dest | .DeliveryStreamName + " " + .VersionId + " " + $dest.DestinationId' | \
xargs -t -p -L 1 sh -c 'aws firehose update-destination --delivery-stream-name $0 --current-delivery-stream-version-id $1 --destination-id $2 --redshift-destination-update "$(jq -c ".DeliveryStreamDescription.Destinations[0].RedshiftDestinationDescription|{CopyCommand}" < streams/$0.json)"'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment