S3: Dump keys from AWS S3 Bucket with pagination using bash

dump_s3_keys.md

Dump keys from AWS S3 Bucket with pagination

$ export url='https://BUCKET_NAME.s3.amazonaws.com/?list-type=2'
$ function dump_urls { cat data.xml \
  | sed 's/<Contents>/\n/g' \
  | sed -ne 's/.*<Key>\(.*\)<\/Key>.*/\1/p' >> urls.txt; \
  }
$ rm -f urls.txt data.xml
$ curl $url -o data.xml
$ dump_urls
$ while (grep "<IsTruncated>true</IsTruncated>" data.xml > /dev/null;) do \
  curl "$url&start-after=$(tail -n1 urls.txt)" -o data.xml; \
  dump_urls; \
done

The urls are dumped to urls.txt