Skip to content

Instantly share code, notes, and snippets.

View cwvhogue's full-sized avatar

Christopher W.V. Hogue cwvhogue

11 followers · 1 following
View GitHub Profile
@cwvhogue
cwvhogue / mdnload_draft.sh
Created October 21, 2013 18:55
Draft - retrieve a Manta Directory Hierarchy and File content on a Unix/Linux platform. Works with spaces in path names.
# Phase 1 Retrieves directories, creates hierarcy from current working directory.
mfind -t d ~~/stor/winfoo | sed "s,/$MANTA_USER/stor/,," | xargs -I {} mkdir -p "{}"
# This retrieves object paths, splits into download file and target local directory file
# Phase 2 retrieves objects, splits into source/target directories, calls mget
mfind -t o ~~/stor/winfoo | tee dnldfiles.txt | sed -e "s,/$MANTA_USER/stor/,," | xargs -I {} dirname "{}" > dnlddirs.txt
# the dnlddirs.txt file can be prepended with a target directory or windows-ified
cat dnldfiles.txt | xargs -I {} basename "{}" > dnldnames.txt
@cwvhogue
cwvhogue / manta_hierarchy_remove.sh
Created October 21, 2013 17:42
I use this to remove Windows hierarchies made on Manta - pathnames with spaces e.g. "/New folder (2)/"
mfind -t o ~~/stor/winfoo | xargs -I {} mrm "{}"
mfind -t d ~~/stor/winfoo | sort -r | xargs -I {} mrmdir "{}"
@cwvhogue
cwvhogue / convert_parameters.sh
Created October 16, 2013 21:43
Inner loop mjob for image conversion script, which re-uses a list of input files saved earlier with a single mfind call.
echo /${MANTA_USER}/${USER_PATH}/${USER_ORIGINALS}/input_list.txt | \
mjob create ${MJOB_ARGS} "xargs mcat" -m "${CONVERT_JOB} && ${MPIPE_STR} < ${PDIR_STR}"
@cwvhogue
cwvhogue / save_mfind.sh
Created October 16, 2013 21:40
saving mfind results as a file to a manta object
mjob create -w -r "mfind /${MANTA_USER}/${USER_PATH}/${USER_ORIGINALS} -n 'jpg$' > /var/tmp/out.txt && \
mput -f /var/tmp/out.txt /${MANTA_USER}/${USER_PATH}/${USER_ORIGINALS}/input_list.txt" < /dev/null
@cwvhogue
cwvhogue / r_manta_api_test.R
Created October 2, 2013 19:57
First cut of a working code block that authenticates from R to Joyent Manta.
# install.packages(c("RCurl","RJSONIO"), dependencies = TRUE)
require(RCurl); require(RJSONIO);
ACTION="/cwvhogue/stor"
MANTA_USER<-Sys.getenv("MANTA_USER");
MANTA_KEY_ID<-Sys.getenv("MANTA_KEY_ID");
MANTA_URL<-Sys.getenv("MANTA_URL");
USER_HOME<-Sys.getenv("HOME");
USER_KEY<-"/.ssh/id_rsa";
key_location<-paste(USER_HOME,USER_KEY,sep="");
openssl_cmd="openssl";
@cwvhogue
cwvhogue / Manta_Getty_Remove.sh
Created September 27, 2013 17:14
Delete files in a Joyent Manta directory:
mfind ~~/public/originals | mjob create 'mrm $MANTA_INPUT_OBJECT'
@cwvhogue
cwvhogue / Manta_Getty_Originals.sh
Created September 27, 2013 17:12
Script to populate a Joyent Manta account with Getty Open Content originals, capped at 20.
#!/bin/bash
SITE="https://us-east.manta.joyent.com/mantademo/public/images/getty-open/"
DEST="/$MANTA_USER/public/"
# For more files change "-20" below to a larger number.
# For the complete download, remove "head -20 |" below
mmkdir -p ${DEST}/originals
curl -ksL ${SITE}/filelist.txt | head -20 | mput ${DEST}/filelist.txt
echo ${DEST}/filelist.txt | mjob create -m "xargs -I {} sh -c 'curl -ksL ${SITE}/originals/{} | mput ${DEST}/originals/{}'"
@cwvhogue
cwvhogue / Getty_Originals_dnld.sh
Created September 27, 2013 17:08
Script to download the Getty Open Originals from Joyent Manta
#!/bin/bash
curl -k https://us-east.manta.joyent.com/mantademo/public/images/getty-open/filelist.txt > filelist.txt
cat filelist.txt | xargs -I {} curl -k https://us-east.manta.joyent.com/mantademo/public/images/getty-open/originals/{} -o {}
@cwvhogue
cwvhogue / Manta_resize_color_preserve.sh
Created September 27, 2013 16:56
Image resize job with color preservation on Joyent Manta.
time mfind /$MANTA_USER/public/art -n '1.jpg$' | mjob create --memory 2048 -w -m 'convert $MANTA_INPUT_FILE -colorspace RGB -resize 250000@ -colorspace sRGB -quality 80 /var/tmp/out.jpg && mpipe ${MANTA_INPUT_OBJECT%.*}_s.jpg < /var/tmp/out.jpg'
@cwvhogue
cwvhogue / Local_resize_color_preserve.sh
Last active December 24, 2015 02:39
Unix command to process resize operation with color preservation
find . -name '*.jpg' -exec sh -c 'convert "{}" -colorspace RGB -resize 250000@ -colorspace sRGB -quality 80 `basename "{}" .jpg`_250.jpg' ';'