Skip to content

Instantly share code, notes, and snippets.

View cwvhogue's full-sized avatar

Christopher W.V. Hogue cwvhogue

11 followers · 1 following
View GitHub Profile
@cwvhogue
cwvhogue / worksforme-D20
Last active August 29, 2015 14:12
* Lenovo ThinkStation D20 4158CM1
** Dual Intel Xeon X5650 2.67GHz - Total 24 Cores.
** Flashed MB Bios to latest (01/14/14).
** Disable CStates (all!). Disable NUMA-aware! Enable Hyperthreading.
** Onboard Broadcom NICs are functioning.
** 48GB RAM (Kingston KTL-TS316S/8G, 240pin Reg ECC Single Rank DDR3-1600MHz PC3-12800 1.5V)
** 6 Sticks in CPU1 bank - CPU2 bank does not work with this RAM
** 96GB is unattainable with cheap RAM
** LSI MegaRAID 9211-8I 8-Port 6Gb/S PCIEx8 SAS/SATA HBA
** Disable onboard SAS from BIOS. Flash LSI MegaRAID BIOS to IT mode driver
@cwvhogue
cwvhogue / ascp_tests.txt
Last active August 29, 2015 14:12
#TESTING ASPERA CONNECT:
# Download
# 64-bit
curl -O http://download.asperasoft.com/download/sw/connect/3.5/aspera-connect-3.5.1.92523-linux-64.sh
chmod +x ./aspera-connect-3.5.1.92523-linux-64.sh
./aspera-connect-3.5.1.92523-linux-64.sh
# Download
@cwvhogue
cwvhogue / manta.cow
Created June 16, 2014 18:40
cwvhogue@manta # cowsay -f /tmp/manta.cow hello there humans
____________________
< hello there humans >
--------------------
\
\
_||___||_
_|{_______}|_
_/ -- -- \_
_ / -- -- \ _
@cwvhogue
cwvhogue / recompress_bz2.sh
Created March 14, 2014 22:04
Recompress bzip2 files with pbzip2 in place - mjob map phase contents
bzip2 -dc $MANTA_INPUT_FILE | pbzip2 -c > /var/tmp/file.tmp && mpipe ${MANTA_INPUT_OBJECT} < /var/tmp/file.tmp
@cwvhogue
cwvhogue / mdnload_draft.sh
Created October 21, 2013 18:55
Draft - retrieve a Manta Directory Hierarchy and File content on a Unix/Linux platform. Works with spaces in path names.
# Phase 1 Retrieves directories, creates hierarcy from current working directory.
mfind -t d ~~/stor/winfoo | sed "s,/$MANTA_USER/stor/,," | xargs -I {} mkdir -p "{}"
# This retrieves object paths, splits into download file and target local directory file
# Phase 2 retrieves objects, splits into source/target directories, calls mget
mfind -t o ~~/stor/winfoo | tee dnldfiles.txt | sed -e "s,/$MANTA_USER/stor/,," | xargs -I {} dirname "{}" > dnlddirs.txt
# the dnlddirs.txt file can be prepended with a target directory or windows-ified
cat dnldfiles.txt | xargs -I {} basename "{}" > dnldnames.txt
@cwvhogue
cwvhogue / manta_hierarchy_remove.sh
Created October 21, 2013 17:42
I use this to remove Windows hierarchies made on Manta - pathnames with spaces e.g. "/New folder (2)/"
mfind -t o ~~/stor/winfoo | xargs -I {} mrm "{}"
mfind -t d ~~/stor/winfoo | sort -r | xargs -I {} mrmdir "{}"
@cwvhogue
cwvhogue / convert_parameters.sh
Created October 16, 2013 21:43
Inner loop mjob for image conversion script, which re-uses a list of input files saved earlier with a single mfind call.
echo /${MANTA_USER}/${USER_PATH}/${USER_ORIGINALS}/input_list.txt | \
mjob create ${MJOB_ARGS} "xargs mcat" -m "${CONVERT_JOB} && ${MPIPE_STR} < ${PDIR_STR}"
@cwvhogue
cwvhogue / save_mfind.sh
Created October 16, 2013 21:40
saving mfind results as a file to a manta object
mjob create -w -r "mfind /${MANTA_USER}/${USER_PATH}/${USER_ORIGINALS} -n 'jpg$' > /var/tmp/out.txt && \
mput -f /var/tmp/out.txt /${MANTA_USER}/${USER_PATH}/${USER_ORIGINALS}/input_list.txt" < /dev/null
@cwvhogue
cwvhogue / r_manta_api_test.R
Created October 2, 2013 19:57
First cut of a working code block that authenticates from R to Joyent Manta.
# install.packages(c("RCurl","RJSONIO"), dependencies = TRUE)
require(RCurl); require(RJSONIO);
ACTION="/cwvhogue/stor"
MANTA_USER<-Sys.getenv("MANTA_USER");
MANTA_KEY_ID<-Sys.getenv("MANTA_KEY_ID");
MANTA_URL<-Sys.getenv("MANTA_URL");
USER_HOME<-Sys.getenv("HOME");
USER_KEY<-"/.ssh/id_rsa";
key_location<-paste(USER_HOME,USER_KEY,sep="");
openssl_cmd="openssl";
@cwvhogue
cwvhogue / Manta_Getty_Remove.sh
Created September 27, 2013 17:14
Delete files in a Joyent Manta directory:
mfind ~~/public/originals | mjob create 'mrm $MANTA_INPUT_OBJECT'