You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Guide: Deploy a DC/OS Mesos Cluster on Google Compute Engine
DC/OS sets up a cluster and deploys pre-configured components services needed to complete a task on hand. You don’t have to entirely understand the complexity of the infrastructure and how to set it up, DC/OS helps you creating necessary abstractions. Once complete, you will have a running cluster with interactive research notebook (container of Jupyter Python Notebook with Apache Spark) and distributed file system (HDFS), ready to tackle any large-scale data processing task.
Getting Started
This guide explains the process for setting up a DC/OS Mesos cluster on Google Cloud Platform.
Steps:
Create dcos-bootstrap CE instance to manage DC/OS installations
Create a Mesos cluster consisting of 3 master nodes and 5 agent nodes
See configuration details at the end of this document
After completing the setup of the cluster using the bootstrap instance, you'll see the generated master and agent nodes in Google Cloud Console. In order to interact with DC/OS via the UI or CLI, you'll need to enable webserver (HTTP/HTTPS) on your leader master node. You'll also need to assign a static IP address to the master leader node in order to access and use the DC/OS dashboard UI.
Enable HTTP/HTTPS
In the Google Cloud Console for Compute Engine, navigate to the settings for the leader master. Click "Edit" and check the boxes for HTTP and HTTPS under the "Firewall" section, then save your changes.
Add SSH keys to master instances
bootstrap node ssh public key
your machines ssh public key
Set Static IP
Next, assign a static IP address to the master leader node by creating a static IP in the networking settings of Google Cloud Platform for the project.
After setting up the DC/OS cluster on Google, install the DC/OS CLI on your local machine. Then, login and set the IP urls of dcos and mesos. The dcos_url corresponds to the public IP address of the master leader node.
# login to dcos using Google OAuth
dcos auth login
# set `dcos_url` to public external IP of master leader node
dcos config set core.dcos_url http://150.155.55.250
# set `mesos_master_url` to internal IP of master leader node
dcos config set core.mesos_master_url http://10.128.0.3:5050
This repository contains scripts to configure a DC/OS cluster on Google Compute Engine.
A bootstrap node is required to run the scripts and to bootstrap the DC/OS cluster.
PLEASE READ THE ENTIRE DOCUMENT. YOU MUST MAKE CHANGES FOR THE SCRIPTS TO WORK IN YOUR GCE ENVIRONMENT.
##Bootstrap node configuration
YOU MUST CREATE A PROJECT using the google cloud console. The author created a project called trek-treckr
You can create the bootstrap node using the google cloud console. The author used a n1-standard-1 instance running centos 7 with a 10 GB persistent disk in
zone europe-west1-c. The bootstrap node must have "Allow full access to all Cloud APIs" in the Identity and API access section. Also enable Block project-wide SSH keys in the SSH Keys section. Create the instance.
After creating the boot instance run the following from the shell
Note: Delta RPMs disabled because /usr/bin/applydeltarpm not installed.$ sudo yum install deltarpm
You need to create the rsa public/private keypairs to allow passwordless logins via SSH to the nodes of the DC/OS cluster. This is required by ansible to create the cluster nodes and install DC/OS on the nodes.
Run the following to generate the keys
ssh-keygen -t rsa -f ~/.ssh/id_rsa -C ajazam
PLEASE REPLACE ajazam with your username. Do not eneter a password when prompted
sudo tee /etc/yum.repos.d/docker.repo <<-'EOF'[dockerrepo]name=Docker Repositorybaseurl=https://yum.dockerproject.org/repo/main/centos/7/enabled=1gpgcheck=1gpgkey=https://yum.dockerproject.org/gpgEOF
install the docker package
sudo yum install docker-engine-1.11.2
Add following changes to /usr/lib/systemd/system/docker.service
start_id=0001 and end_id=0002 specify the range of id's that are appended to the hostname "agent" to create unique agent names. If start_id is not specified then a default of 0001 is used.
If the end_id is not specified then a default of 0001 is used.
When specifying start_id or end_id via CLI, the leading zeroes must be dropped for any agent id higher than 7 or ansible will throw a format error.
File './hosts' is an ansible inventory file. Text wrapped by [] represents a group name and individual entries after the group name represent hosts in that group.
The [masters] group contains node names and IP addresses for the master nodes. In the supplied file the host name is master0 and the ip address 10.132.0.3 is assigned to
master0. YOU MUST CHANGE the IP address for master0 for your network. You can create multiple entries e.g. master1, master2 etc. Each node must have a unique IP address.
The [agents] group has one entry. It specifies the names of all the agents one can have in the DC/OS cluster. The value specifies that agent0000 to agent9999, a
total of 10,000 agents are allowed. This really is an artificial limit because it can easily be changed.
The [bootstrap] group has the name of the bootstrap node.
File './group_vars/all' contains miscellaneous parameters that will change the behaviour of the installation scripts. The parameters are split into two groups. Group 1 parameters must be changed to reflect your environment. Group 2 parameters can optionally be changed to change the behaviour of the scripts.
###Group 1 parameters YOU MUST CHANGE for your environment
project
Your project id. Default: trek-trackr
subnet
Your network. Default: default
login_name
The login name used for accessing each GCE instance. Default: ajazam
bootstrap_public_ip
The bootstrap nodes public IP. Default: 10.132.0.2
zone
You may change this to your preferred zone. Default: europe-west1-c
###Group 2 parameters which optionally change the behaviour of the installation scripts
master_boot_disk_size:
The size of the master node boot disk. Default 10 GB
master_machine_type
The GCE instance type used for the master nodes. Default: n1-standard-2
master_boot_disk_type
The master boot disk type. Default: pd-standard
agent_boot_disk_size
The size of the agent boot disk. Default 10 GB
agent_machine_type
The GCE instance type used for the agent nodes. Default: n1-standard-2
agent_boot_disk_type
The agent boot disk type. Default: pd-standard
agent_instance_type
Allows agents to be preemptible. If the value is "MIGRATE" then they are not preemptible. If the value is '"TERMINATE" --preemptible' then the instance is preemptible. Default: "MIGRATE"
agent_type
Can specify whether an agent is "public" or "private". Default: "private"
start_id
The number appended to the text agent is used to define the hostname of the first agent. e.g. agent0001. Intermediate agents between start_id and end_id will be created if required. Default: 0001
end_id
The number appended to the text agent is used to define the hostname of the last agent. e.g. agent0001. Intermediate agents between start_id and end_id will be created if required. Default: 0001
gcloudbin
The location of the gcloudbin binary. Default: /usr/local/bin/gcloud
image
The disk image used on the master and agent. Default: /centos-cloud/centos-7-v20161027
bootstrap_public_port
The port on the bootstrap node which is used to fetch the dcos installer from each of the master and agent nodes. Default: 8080
cluster_name
The name of the DC/OS cluster. Default: cluster_name
scopes
Don't change this. Required by the google cloud SDK
dcos_installer_filename
The filename for the DC/OS installer. Default dcos_generate_config.sh
dcos_installer_download_path
The location of where the dcos installer is available from dcos.io. Default: https://downloads.dcos.io/dcos/stable/{{ dcos_installer_filename }} The value of {{ dcos_installer_file }} is described above.
home_directory
The home directory for your logins. Default: /home/{{ login_name }} The value of {{ login_name }} is described above.
downloads_from_bootstrap
The concurrent downloads of the dcos installer to the cluster of master and agent nodes. You may need to experiment with this to get the best performance. The performance will be a function of the machine type used for the bootstrap node. Default: 2
dcos_bootstrap_container
Holds the name of the dcos bootstrap container running on the bootstrap node. Default: dcosinstaller