Skip to content

Instantly share code, notes, and snippets.

View aorjoa's full-sized avatar

Bhuridech Sudsee aorjoa

84 followers · 48 following
View GitHub Profile
@aorjoa
aorjoa / gist:e13b16e1330179b0c99ab14f6b42a3f3
Created May 16, 2017 14:46
#!/bin/bash
echo "rdd-WordCount-sparktest-sha256-java-checkpoint-with-cache-in-one-test"
testname="rdd-WordCount-sparktest-sha256-java-checkpoint-with-cache-in-one-test"
echo -e '''
package th.ac.sut.aiyara.sparktest
import org.apache.spark.cprdd.{ImplicitDatasetCheckpoint, ImplicitRDDCheckpoint}
import com.bloomberg.sparkflow
import org.scalatest.FunSuite
import th.ac.sut.aiyara.sparktest.utils.{ConfigSpark, MathFunc}
@aorjoa
aorjoa / gist:c608f0fee2aaa06c0f2b8e92597e252e
Created May 13, 2017 05:50
ftp://ftp.ecn.purdue.edu/puma/kmeans_30GB.tar.bz2
@aorjoa
aorjoa / testOnCluster
Created March 26, 2017 05:25
Sparktest with sbt custom task
AorJoa@aorjoa:~/Dropbox/repository/sparktest$ sbt 'testOnCluster'
[info] Loading project definition from /Users/AorJoa/Dropbox/repository/sparktest/project
[info] Set current project to sparktest (in build file:/Users/AorJoa/Dropbox/repository/sparktest/)
[info] Including from cache: curator-recipes-2.4.0.jar
[info] Including from cache: curator-framework-2.4.0.jar
[info] Including from cache: curator-client-2.4.0.jar
[info] Including from cache: zookeeper-3.4.5.jar
[info] Including from cache: guava-14.0.1.jar
[info] Including from cache: scala-library-2.11.8.jar
[info] Including from cache: javax.servlet-api-3.1.0.jar
@aorjoa
aorjoa / gist:67300f0887b79cd98e5486f25767937b
Created March 21, 2017 16:53
Bytecode for method apply
L0
LINENUMBER 16 L0
ALOAD 0
ILOAD 1
INVOKEVIRTUAL th/ac/sut/aiyara/sparktest/CheckpointingTest$$anonfun$1$$anonfun$2.apply$mcZI$sp (I)Z
IRETURN
L1
LOCALVARIABLE this Lth/ac/sut/aiyara/sparktest/CheckpointingTest$$anonfun$1$$anonfun$2; L0 L1 0
LOCALVARIABLE x$1 I L0 L1 1
@aorjoa
aorjoa / .gitignore
Created November 28, 2016 06:52
.gitignore for Spring Roo
.settings
.classpath
.project
target
annotations/src/main/java/org
bin
*/bin
*.ipr
*.iml
*.iws
@aorjoa
aorjoa / gist:05b7fb200b75b8dd1026c13690687814
Last active October 28, 2016 15:07
#!/bin/bash
for II in `seq 11 110`;
do
docker-machine create -d digitalocean --digitalocean-access-token $DO --digitalocean-size 1gb Aiyara-SwarmZilla-$II
docker-machine ssh Aiyara-SwarmZilla-$II "echo net.ipv4.neigh.default.gc_thresh1 = 30000 >> /etc/sysctl.conf ;"
docker-machine ssh Aiyara-SwarmZilla-$II "echo net.ipv4.neigh.default.gc_thresh2 = 32000 >> /etc/sysctl.conf ;"
docker-machine ssh Aiyara-SwarmZilla-$II "echo net.ipv4.neigh.default.gc_thresh3 = 32768 >> /etc/sysctl.conf ;"
docker-machine ssh Aiyara-SwarmZilla-$II "sysctl -p;"
docker-machine ssh Aiyara-SwarmZilla-$II "docker swarm join --advertise-addr $(docker-machine ip Aiyara-SwarmZilla-$II) --token SWMTKN-1-29tx6z2k8zmsbt9z2c2ay54jc9ce2l94ixhmmagveifv3pp4fa-5e8i9gugpb9b4bmi9vrp7m5su 67.205.160.45:2377"
@aorjoa
aorjoa / gist:9b110adfa2da3b1f257a9e24877dec8e
Created October 7, 2016 09:39
AorJoa@Dekcom:~$ cat /mnt/ceph/logging/app-20161007163514-0008
{"Event":"SparkListenerLogStart","Spark Version":"1.6.2"}
{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor ID":"driver","Host":"192.168.1.200","Port":54465},"Maximum Memory":480116736,"Timestamp":1475832914204}
{"Event":"SparkListenerExecutorAdded","Timestamp":1475832916404,"Executor ID":"6","Executor Info":{"Host":"192.168.1.109","Total Cores":4,"Log Urls":{"stdout":"http://192.168.1.109:8081/logPage/?appId=app-20161007163514-0008&executorId=6&logType=stdout","stderr":"http://192.168.1.109:8081/logPage/?appId=app-20161007163514-0008&executorId=6&logType=stderr"}}}
{"Event":"SparkListenerExecutorAdded","Timestamp":1475832916417,"Executor ID":"8","Executor Info":{"Host":"192.168.1.101","Total Cores":4,"Log Urls":{"stdout":"http://192.168.1.101:8081/logPage/?appId=app-20161007163514-0008&executorId=8&logType=stdout","stderr":"http://192.168.1.101:8081/logPage/?appId=app-20161007163514-0008&executorId=8&logType=stderr"}}}
{"Ev
@aorjoa
aorjoa / gist:11d909491430552ee977b110c1ac49ad
Last active October 3, 2016 17:28
Test generate dataset GroupByTest Spark
import java.util.Random
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.SparkContext._
var numMappers = 100
var numKVPairs = 10000
var valSize = 1000
var numReducers = 36
@aorjoa
aorjoa / gist:cb2a06b4f19547559452bb7b5ef42271
Created July 29, 2016 08:22
Ceph Log Temporary File
lcod 0'0 peering m=11] enter Started/Primary/Peering/WaitUpThru
-12> 2016-07-29 08:12:44.982390 a9017a40 1 -- 192.168.4.108:6807/2311 <== osd.2 192.168.4.106:6802/5966 40 ==== pg_info(1 pgs e99:2.1f) v4 ==== 901+0+0 (394638346 0 0) 0xbd9415a0 con 0xbd746c80
-11> 2016-07-29 08:12:44.982529 a9017a40 5 -- op tracker -- seq: 129, time: 2016-07-29 08:12:44.982526, event: started, op: pg_info(1 pgs e99:2.1f)
-10> 2016-07-29 08:12:44.982712 a9017a40 5 -- op tracker -- seq: 129, time: 2016-07-29 08:12:44.982710, event: done, op: pg_info(1 pgs e99:2.1f)
-9> 2016-07-29 08:12:44.983309 a1817a40 5 osd.4 pg_epoch: 99 pg[2.28( v 72'101 lc 72'33 (0'0,72'101] local-les=99 n=44 ec=64 les/c/f 99/88/0 98/98/64) [1,4,0] r=1 lpr=98 pi=64-97/18 luod=0'0 crt=72'101 lcod 0'0 active m=34] exit Started/ReplicaActive/RepNotRecovering 0.060583 3 0.000749
-8> 2016-07-29 08:12:44.983387 a1817a40 5 osd.4 pg_epoch: 99 pg[2.28( v 72'101 lc 72'33 (0'0,72'101] local-les=99 n=44 ec=64 les/c/f 99/88/0 98/98/64) [1,4,0] r=1
@aorjoa
aorjoa / Ceph_on_ARM.md
Last active May 29, 2016 09:04
Install Ceph on ARM and start

Change source.list to yakkety and then apt-get update like these.

# cat /etc/apt/sources.list 
#------------------------------------------------------------------------------#

###### Ubuntu Main Repos
deb [arch=armhf] http://ports.ubuntu.com/ yakkety main restricted universe 
deb-src [arch=armhf] http://ports.ubuntu.com/ yakkety main 

###### Ubuntu Update Repos