Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
If you ever want to debug a kinesis stream, copy this bash one liner.
On a mac, `brew install awscli gnu-sed` before.
streamname=staging;aws kinesis describe-stream --stream-name $streamname --output text | grep SHARDS | awk '{print $2}' | while read shard; do aws kinesis get-shard-iterator --stream-name $streamname --shard-id $shard --shard-iterator-type LATEST --output text | while read iterator; do while output=`aws kinesis get-records --shard-iterator $iterator --output text`; do iterator=`echo "$output" | head -n1`; echo "$output" | gsed 1d | grep RECORDS | while read record; do echo $record | awk '{print $2}' | base64 -D; done; done; done; done
@vre

This comment has been minimized.

Copy link

@vre vre commented Jun 16, 2016

This fail on aws-cli/1.10.38 at least for me, working onliner:

streamname=staging; aws kinesis describe-stream --stream-name $streamname --output text | grep SHARDS | awk '{print $2}' | while read shard; do aws kinesis get-shard-iterator --stream-name $streamname --shard-id $shard --shard-iterator-type LATEST --output text | while read iterator; do while output=`aws kinesis get-records --shard-iterator $iterator --output text`; do iterator=`echo "$output" | head -n1 | awk '{print $2}'`; echo "$output" | gsed 1d | grep RECORDS | while read record; do echo $record | awk '{print $3}' | base64 -D; done; done; done; done
@cjwebb

This comment has been minimized.

Copy link

@cjwebb cjwebb commented Jul 15, 2016

Thanks for this. Very useful!

I made one small amendment, by swapping out gsed 1d for awk 'NR > 1'. Therefore, you only need to brew install awscli

@AaronHarris

This comment has been minimized.

Copy link

@AaronHarris AaronHarris commented Jan 27, 2017

I had to use base64 -id to get it to work

@emmanuelnk

This comment has been minimized.

Copy link

@emmanuelnk emmanuelnk commented Oct 29, 2018

Thanks guys, I used a combination of all your answers to get mine (ubuntu 18.04 | aws-cli/1.14.44 Python/3.6.5 Linux/4.15.0-36-generic botocore/1.8.48):

  1. use vre's one-liner
  2. switched out gsed 1d for awk 'NR > 1'
  3. switched out base64 -D for base64 -id
    Result:
streamname=my-dev-stream-name;aws kinesis describe-stream --stream-name $streamname --output text | grep SHARDS | awk '{print $2}' | while read shard; do aws kinesis get-shard-iterator --stream-name $streamname --shard-id $shard --shard-iterator-type LATEST --output text | while read iterator; do while output=`aws kinesis get-records --shard-iterator $iterator --output text`; do iterator=`echo "$output" | head -n1 | awk '{print $2}'`; echo "$output" | awk 'NR > 1' | grep RECORDS | while read record; do echo $record | awk '{print $3}' | base64 -id; done; done; done; done
@jeffvandyke

This comment has been minimized.

Copy link

@jeffvandyke jeffvandyke commented Oct 26, 2020

We got a Unable to open 'd': No such file or directory error, on Mac, and it looks like the base64 command works if you just use base64 -d. Great script!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment