skonto

{
  "builders": [
    {
      "type": "virtualbox-iso",
      "name": "mybox",
      "guest_os_type": "Ubuntu_64",
      "iso_url": "file:///home/stavros/Downloads/ubuntu-16.04-server-amd64.iso",
      "iso_checksum": "23e97cd5d4145d4105fbf29878534049",
      "iso_checksum_type": "md5",
      "ssh_username": "user",

skonto

{
  "builders": [
    {
      "type": "virtualbox-iso",
      "name": "mybox",
      "guest_os_type": "Ubuntu_64",
      "iso_url": "file:///home/stavros/Downloads/ubuntu-16.04-server-amd64.iso",
      "iso_checksum": "23e97cd5d4145d4105fbf29878534049",
      "iso_checksum_type": "md5",
      "ssh_username": "user",

skonto

{
   "Outputs":{
      "PublicAgentDNSName":{
         "Value":{
            "Fn::GetAtt":[
               "PublicAgentLoadBalancer",
               "DNSName"
            ]
         },
         "Description":"Public Agent DNS Name"

skonto

# Available commands
# Usage: hadoop fs [generic options]
# 	[-appendToFile <localsrc> ... <dst>]
# 	[-cat [-ignoreCrc] <src> ...]
# 	[-checksum <src> ...]
# 	[-chgrp [-R] GROUP PATH...]
# 	[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
# 	[-chown [-R] [OWNER][:[GROUP]] PATH...]
# 	[-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>]
# 	[-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]

skonto

Run against the normal package, not the beta one.

DATA=1 GB

Teragen:

17/05/24 15:30:57 INFO mapreduce.Job: Counters: 21
	File System Counters
		FILE: Number of bytes read=276327
		FILE: Number of bytes written=565835

skonto

  "cmd": "export JAVA_HOME=$MESOS_SANDBOX/jre1.8.0_112 && cp $MESOS_SANDBOX/core-site.xml hadoop-2.6.0-cdh5.9.1/etc/hadoop && cp $MESOS_SANDBOX/hdfs-site.xml hadoop-2.6.0-cdh5.9.1/etc/hadoop &&  hadoop-2.6.0-cdh5.9.1/bin/hadoop fs -copyToLocal hdfs://hdfs/spark-kafka-drainer-0.0.1-deploy.jar spark-kafka-drainer-0.0.1-deploy.jar  &&  $JAVA_HOME/bin/java -cp spark-kafka-drainer-0.0.1-deploy.jar com.lightbend.fdp.spark.KafkaDrainerDStream",
  "cpus":1,
  "mem":2048,
  "env":{
    "DRAINER_BOOTSTRAP_SERVERS": "broker.kafka.l4lb.thisdcos.directory:9092",
    "DRAINER_KAFKA_SOURCE_TOPIC":"test",
    "DRAINER_OUTPUT_HDFS_FILE":"hdfs://hdfs/drainer-data",
    "DRAINER_ZOOKEEPER_KAFKA_URL":"master.mesos:2181/dcos-service-kafka",
    "DRAINER_BATCH_DURATION": "10"
  },

skonto

     import org.apache.hadoop.fs.FileSystem
     import org.apache.hadoop.fs.Path
     import org.apache.hadoop.mapred.JobConf
     import java.util.ServiceLoader

     val path = new Path(readURL)
     val c= new JobConf()
     val sl = ServiceLoader.load(classOf[FileSystem])
import scala.collection.JavaConversions._
     sl.iterator.toList.foreach { fs =>

skonto

Dispatcher passes the following command to mesos:

18/06/13 10:30:45 INFO MesosClusterScheduler: Command for the driver (test):./bin/spark-submit --name test --master mesos://zk://master.mesos:2181/mesos --driver-cores 1.0 --driver-memory 1000M --class org.apache.spark.examples.SparkPi --conf spark.executor.extraJavaOptions="-Dexecutor.test.param1=\"value 1\" -Dexecutor.test.param2=value\\ 2 -Dexecutor.test.path=\$PATH" --conf spark.mesos.containerizer=mesos --conf spark.driver.supervise=false --conf spark.app.name=test --conf spark.driver.memory=1000M --conf spark.mesos.executor.docker.image=skonto/spark-test:test --conf spark.driver.cores=1.0 --conf spark.driver.extraJavaOptions="-Dspark.test.param1=\"value 1\" -Dspark.test.param2=value\\ 2 -Dspark.test.path=\$PATH" $MESOS_SANDBOX/spark-examples_2.11-2.3.0.jar 10000
spark-class will run this at the executor side:

exec /usr/lib/jvm/jre1.8.0_152/bin/java -cp '/opt/spark/dist/conf/:/opt/spark/dist/jars/*:/etc/hadoop/' '-Dexecutor.test.param1=value 1' '-Dexecu

skonto

so they earliest msg is:
```
E0721 23:50:08.788032      52 kubelet_volumes.go:140] Orphaned pod "9213b43a-8c6e-11e8-a405-02941124cb54" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
E0721 23:50:10.785026      52 kubelet_volumes.go:140] Orphaned pod "9213b43a-8c6e-11e8-a405-02941124cb54" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
E0721 23:50:12.787966      52 kubelet_volumes.go:140] Orphaned pod "9213b43a-8c6e-11e8-a405-02941124cb54" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
E0721 23:50:14.785313      52 kubelet_volumes.go:140] Orphaned pod "9213b43a-8c6e-11e8-a405-02941124cb54" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
E0721 23:50:16.787653      52 kubelet_volumes.g

skonto

I0315 19:45:37.845666      11 webhook.go:136] Serving admission request
I0315 19:45:37.847765      11 webhook.go:290] Pod spark-pi-driver in namespace spark is subject to mutation
I0315 19:45:37.865038      11 spark_pod_eventhandler.go:41] Pod spark-pi-driver added in namespace spark.
I0315 19:45:37.865066      11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:37.865370      11 spark_pod_eventhandler.go:52] Pod spark-pi-driver updated in namespace spark.
I0315 19:45:37.865400      11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:37.876064      11 spark_pod_eventhandler.go:52] Pod spark-pi-driver updated in namespace spark.
I0315 19:45:37.876092      11 spark_pod_eventhandler.go:77] Enqueuing SparkApplication spark/spark-pi for app update processing.
I0315 19:45:38.225406      11 controller.go:595] SparkApplication spark/spark-pi has been submitted
I0315 19:45:38.225482      11 spark