Skip to content

Instantly share code, notes, and snippets.

View concretevitamin's full-sized avatar

Zongheng Yang concretevitamin

View GitHub Profile
storage.backend=com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager
storage.dynamodb.client.credentials.class-name=com.amazonaws.auth.DefaultAWSCredentialsProviderChain
storage.dynamodb.client.credentials.constructor-args=
storage.dynamodb.client.endpoint=https://dynamodb.us-west-1.amazonaws.com
storage.connection-timeout=100000
storage.dynamodb.prefix=mygraph
storage.dynamodb.force-consistent-read=true
storage.dynamodb.max-self-throttled-retries=60
@concretevitamin
concretevitamin / gist:865773865d5672c76990
Last active September 16, 2015 07:38 — forked from evanye/gist:aaca8e65dc3cfd2d992b
titan thrift frame size
com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44)
at com.thinkaurelius.titan.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:428)
at com.thinkaurelius.titan.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:2
53)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.edgeQuery(StandardTitanGraph.java:3
65)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx$7$3.get(StandardTitanTx.java:1093)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx$7$3.get(StandardTitanTx.java:1090)
at com.thinkaurelius.titan.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:47)

Latency numbers every programmer should know

L1 cache reference ......................... 0.5 ns
Branch mispredict ............................ 5 ns
L2 cache reference ........................... 7 ns
Mutex lock/unlock ........................... 25 ns
Main memory reference ...................... 100 ns             
Compress 1K bytes with Zippy ............. 3,000 ns  =   3 µs
Send 2K bytes over 1 Gbps network ....... 20,000 ns  =  20 µs
SSD random read ........................ 150,000 ns  = 150 µs

Read 1 MB sequentially from memory ..... 250,000 ns = 250 µs

# Mirror matrix (left-right)
mirror.matrix <- function(x) {
xx <- as.data.frame(x);
xx <- rev(xx);
xx <- as.matrix(xx);
xx;
}
createImage <- function(x) {
mirror.matrix(matrix(x, nrow=28))
d <- 784 # dimension of digits
NTrain <- 60000
NTest <- 10000
D <- 900 # number of random features
gamma <- 4e-4
readMatrix <- function(split, part, d) {
matList <- sapply(part, function(line) {
as.numeric(strsplit(line, ",", fixed=TRUE)[[1]])
@concretevitamin
concretevitamin / MnistSovler.R
Created November 21, 2014 22:06
MNIST Demo
options(java.parameters="-Xmx20g")
library(SparkR)
library(Matrix)
library(caret)
source("MnistUtils.R")
source("ImageUtils.R")
sc <- sparkR.init("local[8]")
file <- textFile(sc,
"/data/train-mnist-dense-with-labels.data", 8)
root@ip-172-31-5-222 SparkR-pkg]$ YARN_CONF_DIR=/root/ephemeral-hdfs/conf/ MASTER=yarn-client ./sparkR
...
14/10/30 05:42:53 INFO YarnClientImpl: Submitted application application_1414639986366_0017
Error in .jcall("RJavaTools", "Ljava/lang/Object;", "invokeMethod", cl, :
org.apache.spark.SparkException: Yarn application already ended,might be killed or not able to launch application master.)
"Executor task launch worker-0" daemon prio=10 tid=0x00007f32ec003800 nid=0xaac waiting on condition [0x00007f33f4428000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f3e0d7198e8> (a scala.concurrent.impl.Promise$CompletionLatch)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:202)
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:218)
/ignore -channels #channel * JOINS PARTS QUITS NICKS
http://blog.htbaa.com/news/irssi-ignore-joins-parts-quits-and-nicks-messages
~/workplace/spark-covi(branch:estimates*) » javap -c NoCase\$\$anonfun\$1
Compiled from "NoCase.scala"
public final class NoCase$$anonfun$1 extends scala.runtime.AbstractFunction1$mcVI$sp implements scala.Serializable {
public final void apply(int);
Code:
0: aload_0
1: iload_1
2: invokevirtual #17 // Method apply$mcVI$sp:(I)V
5: return