Skip to content

Instantly share code, notes, and snippets.

@cwvhogue

cwvhogue/MapReduce_manta_image_identify

Created August 26, 2013 21:15
Show Gist options
  • Save cwvhogue/6346744 to your computer and use it in GitHub Desktop.
Save cwvhogue/6346744 to your computer and use it in GitHub Desktop.
Download ZIP
Open Getty Image verification on Joyent Manta using ImageMagick "identify" command in a one-line MapReduce job.
Raw
MapReduce_manta_image_identify
# MapReduce ImageMagick 'identify' command run over 4,599 jpg files in /public/art Manta directory
$ mfind /$MANTA_USER/public/art | mjob create -w -m 'identify $MANTA_INPUT_FILE' -r cat
added 1000 inputs to 6f2e6ac8-b6a5-4ed5-bad4-8371aea010dd
added 1000 inputs to 6f2e6ac8-b6a5-4ed5-bad4-8371aea010dd
added 1000 inputs to 6f2e6ac8-b6a5-4ed5-bad4-8371aea010dd
added 1000 inputs to 6f2e6ac8-b6a5-4ed5-bad4-8371aea010dd
added 599 inputs to 6f2e6ac8-b6a5-4ed5-bad4-8371aea010dd
# Retrieve the output file to my local machine
$ mget $(mjob outputs 6f2e6ac8-b6a5-4ed5-bad4-8371aea010dd) > image_identify.txt
...bee1-c32536dd4fae [=======================>] 100% 466.54KB
# look for lines that do not contain the string 'JPEG'
$ grep -v -e 'JPEG' image_identify.txt
/manta/$MANTA_USER/public/art/03901101.jpg SVG 0x0 16-bit sRGB 231B 0.000u 0:00.010
/manta/$MANTA_USER/public/art/00660201.jpg SVG 0x0 16-bit sRGB 231B 0.010u 0:00.010
/manta/$MANTA_USER/public/art/10821901.jpg SVG 0x0 16-bit sRGB 231B 0.000u 0:00.050
# Check Manta for any errors - where identify returned 1
$ mjob errors 6f2e6ac8-b6a5-4ed5-bad4-8371aea010dd
# The three files listed above were inspected, broken at the Getty data source.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment