Spark On Mesos

sparkOnMesos.md

###RUN SPARK ON MESOS

###单机版 ####模式一(测试通过）

• Mesos运行在host上
• Spark driver 和 executor运行docker上

#####1.搭建Mesos单机环境 详情 #####2.运行Spark demo 启动spark driver container

docker run -it --net host registry.dataman.io/centos7/spark:1.5.1 bash 

修改spark配置文件

cd $SPARK_HOME
vi conf/spark-defaults.conf

########################
spark.mesos.executor.home /spark-1.5.1-bin-hadoop2.6
spark.mesos.executor.docker.image registry.dataman.io/centos7/spark:1.5.1
########################

vi conf/spark-env.sh

#####################
export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:jre/bin/java::")
export MASTER=mesos://$MESOS_MASTER_IP:5050    #将MESOS_MASTER_IP替换为ip地址
export SPARK_HOME=/spark-1.5.1-bin-hadoop2.6
export SPARK_LOCAL_IP=`ifconfig eth0 | awk '/inet addr/{print substr($2,6)}'
export SPARK_LOCAL_HOSTNAME=`ifconfig eth0 | awk '/inet addr/{print substr($2,6)}'
#####################

启动Spark shell

bin/spark-shell

运行demo

sc.parallelize(1 to 1000) count

###模式二（测试不通过）

• Mesos通过数人云发布
• spark driver和executor运行在container里

####1.通过数人云发布mesos

####2.运行spark（同模式一步骤2）

[root@omegamaster1 spark-1.5.1-bin-hadoop2.6]# bin/spark-shell
/spark-1.5.1-bin-hadoop2.6/conf/spark-env.sh: line 63: ifconfig: command not found
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties
To adjust logging level use sc.setLogLevel("INFO")
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.5.1
      /_/

Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_85)
Type in expressions to have them evaluated.
Type :help for more information.
15/10/14 13:36:07 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
I1014 21:36:07.617255   141 sched.cpp:157] Version: 0.23.0
I1014 21:36:07.622272   137 sched.cpp:254] New master detected at master@10.3.11.2:5050
I1014 21:36:07.622850   137 sched.cpp:264] No credentials provided. Attempting to register without authentication

---一直在hang住。单机版都不可以---
解决办法：让spark找到master ip

registry.dataman.io/centos7/spark:1.5.1 镜像中没有ifconfig,
解决办法：
yum install net-tools

export SPARK_LOCAL_IP=ifconfig eth0 | awk '/inet addr/{print substr($2,6)}' export SPARK_LOCAL_HOSTNAME=ifconfig eth0 | awk '/inet addr/{print substr($2,6)}'
缺最后一个符号`

这个镜像问题太多。直接使用https://hub.docker.com/r/mesosphere/spark/ 测试

docker run -it --net host mesosphere/spark:1.5.0-hadoop2.6.0 bash

root@omegamaster1:/opt/spark/dist# cat conf/spark-defaults.conf

spark.mesos.coarse=true
spark.mesos.executor.home /opt/spark/dist
spark.mesos.executor.docker.image  mesosphere/spark:1.5.0-hadoop2.6.0

然后问题就解决了。