Skip to content

Instantly share code, notes, and snippets.

@frosforever

frosforever/manifest.sh

Created October 18, 2017 13:14
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save frosforever/501a1713404a279b479fd4f93743f393 to your computer and use it in GitHub Desktop.
Save frosforever/501a1713404a279b479fd4f93743f393 to your computer and use it in GitHub Desktop.
Download ZIP
create Redshift manifest from prefix ignoring `SUCCESS` files
Raw
manifest.sh
aws s3api list-objects --bucket BUCKET_NAME --prefix WTVR/PREFIX/ --query 'Contents[?!contains(Key,`SUCCESS`)].{Key:Key}' --output json | jq '[.[] | .["url"] = "s3://BUCKET_NAME/" + .Key | .["mandatory"] = true | del(.Key)] | { entries: .}'
@frosforever
Copy link
Author

can then pipe to s3 cp to put back on s3:

... | aws s3 cp - s3://BUCKET/manifest.json

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment