Skip to content

Instantly share code, notes, and snippets.

Last active May 31, 2016 18:57
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
What would you like to do?
Download Papertrail archives, upload to S3
# Downloads 10 days of logs at a time, asynchronously.
# Env vars:
# MAX_DAYS: Number of days of logs to download
# PAPERTRAIL_API_KEY: Self explanitory, get from PT "Profile" page
# BUCKET_NAME: S3 Bucket to upload to
# LOG_PATH: Path inside S3 Bucket to save logs to
for i in $(seq 0 ${MAX_DAYS})
date -u -v-${i}d +%Y-%m-%d | \
xargs -I {} curl --progress-bar -f --no-include --create-dirs -o dt={}/{}.tsv.gz \
-L -H "X-Papertrail-Token: ${PAPERTRAIL_API_KEY}"{}/download &
if (( $i % 10 == 0 )); then wait; fi
s3cmd put --recursive . s3://${BUCKET_NAME}/${LOG_PATH}/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment