Skip to content

Instantly share code, notes, and snippets.

@jamesmoey
Created June 26, 2015 01:46
Show Gist options
  • Save jamesmoey/24c030121e11da0dfa97 to your computer and use it in GitHub Desktop.
Save jamesmoey/24c030121e11da0dfa97 to your computer and use it in GitHub Desktop.
De duplicate message in AWS SQS
declare -A messages;
for i in {1..10000};
do
msg=$(aws --region ap-southeast-2 sqs receive-message \
--queue-url $1 \
--max-number-of-messages 1);
msgMd5=$(echo $msg | jq -r '.Messages[].MD5OfBody');
msgId=$(echo $msg | jq -r '.Messages[].MessageId');
if [ -n "${messages[$msgMd5]}" -a "${messages[$msgMd5]}" != "$msgId" ];
then
receiptHandler=$(echo $msg | jq -r '.Messages[].ReceiptHandle');
aws --region ap-southeast-2 sqs delete-message \
--queue-url $1 \
--receipt-handle $receiptHandler;
echo "Removing" $msgMd5;
else
messages[$msgMd5]=$msgId;
fi;
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment