Skip to content

Instantly share code, notes, and snippets.

Christopher W.V. Hogue cwvhogue

Block or report user

Report or block cwvhogue

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View worksforme-D20
* Lenovo ThinkStation D20 4158CM1
** Dual Intel Xeon X5650 2.67GHz - Total 24 Cores.
** Flashed MB Bios to latest (01/14/14).
** Disable CStates (all!). Disable NUMA-aware! Enable Hyperthreading.
** Onboard Broadcom NICs are functioning.
** 48GB RAM (Kingston KTL-TS316S/8G, 240pin Reg ECC Single Rank DDR3-1600MHz PC3-12800 1.5V)
** 6 Sticks in CPU1 bank - CPU2 bank does not work with this RAM
** 96GB is unattainable with cheap RAM
** LSI MegaRAID 9211-8I 8-Port 6Gb/S PCIEx8 SAS/SATA HBA
** Disable onboard SAS from BIOS. Flash LSI MegaRAID BIOS to IT mode driver
View ascp_tests.txt
#TESTING ASPERA CONNECT:
# Download
# 64-bit
curl -O http://download.asperasoft.com/download/sw/connect/3.5/aspera-connect-3.5.1.92523-linux-64.sh
chmod +x ./aspera-connect-3.5.1.92523-linux-64.sh
./aspera-connect-3.5.1.92523-linux-64.sh
# Download
View manta.cow
cwvhogue@manta # cowsay -f /tmp/manta.cow hello there humans
____________________
< hello there humans >
--------------------
\
\
_||___||_
_|{_______}|_
_/ -- -- \_
_ / -- -- \ _
@cwvhogue
cwvhogue / recompress_bz2.sh
Created Mar 14, 2014
Recompress bzip2 files with pbzip2 in place - mjob map phase contents
View recompress_bz2.sh
bzip2 -dc $MANTA_INPUT_FILE | pbzip2 -c > /var/tmp/file.tmp && mpipe ${MANTA_INPUT_OBJECT} < /var/tmp/file.tmp
@cwvhogue
cwvhogue / mdnload_draft.sh
Created Oct 21, 2013
Draft - retrieve a Manta Directory Hierarchy and File content on a Unix/Linux platform. Works with spaces in path names.
View mdnload_draft.sh
# Phase 1 Retrieves directories, creates hierarcy from current working directory.
mfind -t d ~~/stor/winfoo | sed "s,/$MANTA_USER/stor/,," | xargs -I {} mkdir -p "{}"
# This retrieves object paths, splits into download file and target local directory file
# Phase 2 retrieves objects, splits into source/target directories, calls mget
mfind -t o ~~/stor/winfoo | tee dnldfiles.txt | sed -e "s,/$MANTA_USER/stor/,," | xargs -I {} dirname "{}" > dnlddirs.txt
# the dnlddirs.txt file can be prepended with a target directory or windows-ified
cat dnldfiles.txt | xargs -I {} basename "{}" > dnldnames.txt
@cwvhogue
cwvhogue / manta_hierarchy_remove.sh
Created Oct 21, 2013
I use this to remove Windows hierarchies made on Manta - pathnames with spaces e.g. "/New folder (2)/"
View manta_hierarchy_remove.sh
mfind -t o ~~/stor/winfoo | xargs -I {} mrm "{}"
mfind -t d ~~/stor/winfoo | sort -r | xargs -I {} mrmdir "{}"
@cwvhogue
cwvhogue / convert_parameters.sh
Created Oct 16, 2013
Inner loop mjob for image conversion script, which re-uses a list of input files saved earlier with a single mfind call.
View convert_parameters.sh
echo /${MANTA_USER}/${USER_PATH}/${USER_ORIGINALS}/input_list.txt | \
mjob create ${MJOB_ARGS} "xargs mcat" -m "${CONVERT_JOB} && ${MPIPE_STR} < ${PDIR_STR}"
@cwvhogue
cwvhogue / save_mfind.sh
Created Oct 16, 2013
saving mfind results as a file to a manta object
View save_mfind.sh
mjob create -w -r "mfind /${MANTA_USER}/${USER_PATH}/${USER_ORIGINALS} -n 'jpg$' > /var/tmp/out.txt && \
mput -f /var/tmp/out.txt /${MANTA_USER}/${USER_PATH}/${USER_ORIGINALS}/input_list.txt" < /dev/null
@cwvhogue
cwvhogue / r_manta_api_test.R
Created Oct 2, 2013
First cut of a working code block that authenticates from R to Joyent Manta.
View r_manta_api_test.R
# install.packages(c("RCurl","RJSONIO"), dependencies = TRUE)
require(RCurl); require(RJSONIO);
ACTION="/cwvhogue/stor"
MANTA_USER<-Sys.getenv("MANTA_USER");
MANTA_KEY_ID<-Sys.getenv("MANTA_KEY_ID");
MANTA_URL<-Sys.getenv("MANTA_URL");
USER_HOME<-Sys.getenv("HOME");
USER_KEY<-"/.ssh/id_rsa";
key_location<-paste(USER_HOME,USER_KEY,sep="");
openssl_cmd="openssl";
@cwvhogue
cwvhogue / Manta_Getty_Remove.sh
Created Sep 27, 2013
Delete files in a Joyent Manta directory:
View Manta_Getty_Remove.sh
mfind ~~/public/originals | mjob create 'mrm $MANTA_INPUT_OBJECT'
You can’t perform that action at this time.