Get Kinesis Records By Shard In Bash One Liner

get-kinesis-records-by-shard-id-in-bash-one-liner.markdown

One line to get all records from a stream since its StartingSequenceNumber. Just specify the shard, endpoint, and stream at the start of the script.

shard="shardId-000000000001"; endpoint="http://localhost:4567"; stream="test-stream";

See here an example pointed at kinesalite running locally.

shard="shardId-000000000001"; endpoint="http://localhost:4567"; stream="test-stream"; \
    sequence=$( \
      aws --endpoint-url=$endpoint kinesis describe-stream --stream-name $stream \
      | jq --raw-output --arg id "$shard" '.StreamDescription.Shards[] | select(.ShardId == $id) | .SequenceNumberRange.StartingSequenceNumber' \
    ); \
    iter=$( \
      aws --endpoint-url=$endpoint kinesis get-shard-iterator --stream-name $stream --shard-id $shard --shard-iterator AT_SEQUENCE_NUMBER --starting-sequence-number $sequence \
      | jq --raw-output .ShardIterator \
    ); \
    aws --endpoint-url=$endpoint kinesis get-records --shard-iterator "$iter"