This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Dispatcher passes the following command to mesos: | |
18/06/13 10:30:45 INFO MesosClusterScheduler: Command for the driver (test):./bin/spark-submit --name test --master mesos://zk://master.mesos:2181/mesos --driver-cores 1.0 --driver-memory 1000M --class org.apache.spark.examples.SparkPi --conf spark.executor.extraJavaOptions="-Dexecutor.test.param1=\"value 1\" -Dexecutor.test.param2=value\\ 2 -Dexecutor.test.path=\$PATH" --conf spark.mesos.containerizer=mesos --conf spark.driver.supervise=false --conf spark.app.name=test --conf spark.driver.memory=1000M --conf spark.mesos.executor.docker.image=skonto/spark-test:test --conf spark.driver.cores=1.0 --conf spark.driver.extraJavaOptions="-Dspark.test.param1=\"value 1\" -Dspark.test.param2=value\\ 2 -Dspark.test.path=\$PATH" $MESOS_SANDBOX/spark-examples_2.11-2.3.0.jar 10000 | |
spark-class will run this at the executor side: | |
exec /usr/lib/jvm/jre1.8.0_152/bin/java -cp '/opt/spark/dist/conf/:/opt/spark/dist/jars/*:/etc/hadoop/' '-Dexecutor.test.param1=value 1' '-Dexecu |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import org.apache.hadoop.fs.FileSystem | |
import org.apache.hadoop.fs.Path | |
import org.apache.hadoop.mapred.JobConf | |
import java.util.ServiceLoader | |
val path = new Path(readURL) | |
val c= new JobConf() | |
val sl = ServiceLoader.load(classOf[FileSystem]) | |
import scala.collection.JavaConversions._ | |
sl.iterator.toList.foreach { fs => |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
"cmd": "export JAVA_HOME=$MESOS_SANDBOX/jre1.8.0_112 && cp $MESOS_SANDBOX/core-site.xml hadoop-2.6.0-cdh5.9.1/etc/hadoop && cp $MESOS_SANDBOX/hdfs-site.xml hadoop-2.6.0-cdh5.9.1/etc/hadoop && hadoop-2.6.0-cdh5.9.1/bin/hadoop fs -copyToLocal hdfs://hdfs/spark-kafka-drainer-0.0.1-deploy.jar spark-kafka-drainer-0.0.1-deploy.jar && $JAVA_HOME/bin/java -cp spark-kafka-drainer-0.0.1-deploy.jar com.lightbend.fdp.spark.KafkaDrainerDStream", | |
"cpus":1, | |
"mem":2048, | |
"env":{ | |
"DRAINER_BOOTSTRAP_SERVERS": "broker.kafka.l4lb.thisdcos.directory:9092", | |
"DRAINER_KAFKA_SOURCE_TOPIC":"test", | |
"DRAINER_OUTPUT_HDFS_FILE":"hdfs://hdfs/drainer-data", | |
"DRAINER_ZOOKEEPER_KAFKA_URL":"master.mesos:2181/dcos-service-kafka", | |
"DRAINER_BATCH_DURATION": "10" | |
}, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Run against the normal package, not the beta one. | |
DATA=1 GB | |
Teragen: | |
17/05/24 15:30:57 INFO mapreduce.Job: Counters: 21 | |
File System Counters | |
FILE: Number of bytes read=276327 | |
FILE: Number of bytes written=565835 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Available commands | |
# Usage: hadoop fs [generic options] | |
# [-appendToFile <localsrc> ... <dst>] | |
# [-cat [-ignoreCrc] <src> ...] | |
# [-checksum <src> ...] | |
# [-chgrp [-R] GROUP PATH...] | |
# [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...] | |
# [-chown [-R] [OWNER][:[GROUP]] PATH...] | |
# [-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>] | |
# [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"Outputs":{ | |
"PublicAgentDNSName":{ | |
"Value":{ | |
"Fn::GetAtt":[ | |
"PublicAgentLoadBalancer", | |
"DNSName" | |
] | |
}, | |
"Description":"Public Agent DNS Name" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"builders": [ | |
{ | |
"type": "virtualbox-iso", | |
"name": "mybox", | |
"guest_os_type": "Ubuntu_64", | |
"iso_url": "file:///home/stavros/Downloads/ubuntu-16.04-server-amd64.iso", | |
"iso_checksum": "23e97cd5d4145d4105fbf29878534049", | |
"iso_checksum_type": "md5", | |
"ssh_username": "user", |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"builders": [ | |
{ | |
"type": "virtualbox-iso", | |
"name": "mybox", | |
"guest_os_type": "Ubuntu_64", | |
"iso_url": "file:///home/stavros/Downloads/ubuntu-16.04-server-amd64.iso", | |
"iso_checksum": "23e97cd5d4145d4105fbf29878534049", | |
"iso_checksum_type": "md5", | |
"ssh_username": "user", |
NewerOlder