Mesosphere Cluster with mesos-dns, hdfs and spark on GCE

gistfile1.md

Intro

this screen cast will demo how to setup an mesosphere cluster for the purposes of analytics. We will show how to provision mesosphere on Google Compute Platform along with installing Mesos-DNS, HDFS and Spark.

We will start with setting up mesosphere on GCE by directing our browser to google.mesosphere.com

GCE setup

1. setup through wizard
2. download and run openvpn
3. see mesos ui
4. see marathon ui

Mesos-DNS setup

there are 2 steps to setting up mesos-dns. The first is to run mesos-dns at the master. The second is to make setup each slave to reference the master as a name server.

In order to accomplish the first step you may want to review https://mesosphere.github.io/mesos-dns/docs/tutorial-gce.html. However we we deviate from the instructions by using the docker image, which allows us to avoid the build step.

Master Setup

ssh into master ssh -i ~/.ssh/gce jclouds@104.154.57.8

setup config.json

sudo mkdir /etc/mesos-dns
sudo vi /etc/mesos-dns/config.json

change zk url to use the master ip

{
  "zk": "zk://10.41.40.151:2181/mesos",
  "refreshSeconds": 60,
  "ttl": 60,
  "domain": "mesos",
  "port": 53,
  "resolvers": ["169.254.169.254","10.0.0.1"],
  "timeout": 5,
  "email": "root.mesos-dns.mesos"
}

run docker image

sudo docker pull mesosphere/mesos-dns
sudo docker run --net=host -d -v "/etc/mesos-dns/config.json:/config.json" mesosphere/mesos-dns /mesos-dns -config=/config.json

test mesos dns

sudo docker run --net=host tutum/dnsutils dig @10.205.30.161 google.com
sudo docker run --net=host tutum/dnsutils dig @10.205.30.161 master.mesos

# strangeloop name server
sudo sed -i "1s/^/nameserver $(hostname -i)\n/" /etc/resolv.conf
sudo sed -i "1s/^/prepend domain-name-servers $(hostname -i);\n/" /etc/dhcp/dhclient.conf

# test that the master resolves to its dns first
sudo docker run --net=host tutum/dnsutils dig master.mesos

Enable GCE Network for DNS

add firewall rule to network

• name: dns
• ip: 10.0.0.0/8
• protocol: tcp:53;udp:53

Make Slave Configuration

Pick a slave: know it's name and ip (example development-6420-20d 104.197.95.128 10.48.29.83)

*ssh into the slave

change the resolv.conf

sudo sed -i '1s/^/nameserver 10.205.30.161\n /' /etc/resolv.conf

change the dhcp config

sudo sed -i '1s/^/prepend domain-name-servers 10.205.30.161;\n /' /etc/dhcp/dhclient.conf

test configuration

sudo docker run --net=host tutum/dnsutils dig master.mesos

Create Slave VM Image

The initial slave sizes are not sufficient for running analytics. so we need to create slaves of bigger size. This can be accomplished by following instructions for Adding a slave to mesosphere cluster on GCE

create snapshot

create slave vm

• name: development-6420-slavex
• machine: n1-standard-8
• boot: new disk from snapshot
• snapshot: slave-snapshot
• network: mesosphere-development-6420
• keep in same zone as other slaves
• metadata: jclouds-group: development-6420, jclouds-delete-boot-disk: true
• tag: mesosphere-development-6462-port-22 Replace development-6462 with your cluster name

update slave information and restart

#ssh in 
echo $( hostname -i) | sudo tee /etc/mesos-slave/ip
echo $(hostname -f) | sudo tee /etc/mesos-slave/attributes/host
sudo service mesos-slave restart

CREATE 5 of these

HDFS Setup

Remove Hadoop Configuation

The default provisioning of a mesosphere cluster on GCE installs hadoop. We will be running a self provisioning hdfs service on the cluster. To avoid version and configuration differences we need to remove that installation on all nodes including the master.

ssh into each node remove hadoop sudo aptitude purge hadoop hadoop-yarn hadoop-hdfs hadoop-hdfs-namenode hadoop-hdfs-datanode hadoop-0.20-mapreduce hadoop-0.20-mapreduce-jobtracker hadoop-0.20-mapreduce-tasktracker hadoop-mapreduce remove installation directories sudo rm -rf /etc/hadoop /mnt/hdfs /var/lib/hadoop* /var/log/hadoop*

Build HDFS

clone the project git clone https://github.com/mesosphere/hdfs

There are a couple of ways of making the configuration changes we need. For this screencast, we will make the changes prior to the build.

copy all of the XML files from hdfs/example-conf/mesosphere-dcos/ to hdfs/conf

modify conf/mesos-site.xml

• set mesos.hdfs.native-hadoop-binaries to false
• set mesos.native.library to /usr/local/lib/libmesos.so

example:

<property>
    <name>mesos.hdfs.native-hadoop-binaries</name>
    <value>false</value>
    <description>DCOS comes with pre-distributed HDFS binaries in a single-tenant environment</description>
  </property>
    <property>
    <name>mesos.native.library</name>
    <value>/usr/local/lib/libmesos.so</value>
  </property>

from hdfs directory, build ./bin/build-hdfs

scp the tarball to the master scp -i ~/.ssh/gce hdfs-mesos-0.1.1.tgz jclouds@104.154.57.8:~

ssh to the master ssh -i ~/.ssh/gce jclouds@104.154.57.8

untar hdfs tar zxvf hdfs-mesos-*.tgz

start HDFS cd hdfs-mesos-0.1.1 ./bin/hdfs-mesos

need to add maxClientCnxns=60 to /etc/zookeeper/conf/zoo.cfg