Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Deploy Hadoop, Spark and Flink clusters on Amazon EC2 with this
name: HadoopSparkFlinkCluster
ec2:
region: us-west-2
type: m3.xlarge
cookbooks:
hadoop:
branch: master
github: karamel-lab/apache-hadoop-chef
spark:
branch: master
github: ashansa/spark-chef
flink:
branch: master
github: ashansa/flink-chef
attrs:
hadoop:
mr:
user: ubuntu
yarn:
user: ubuntu
hdfs:
user: ubuntu
spark:
user: ubuntu
driver_memory: 8192m
executor_memory: 100g
flink:
user: ubuntu
jobmanager:
heap_mbs: '8192'
taskmanager:
heap_mbs: '102400'
groups:
datanodes:
size: 2
ec2:
region: us-west-2
type: i2.4xlarge
price: 0.5
recipes:
- spark::worker
- hadoop::dn
- hadoop::nm
- flink::taskmanager
namenodes:
size: 1
ec2:
region: us-west-2
type: m3.xlarge
price: 0.15
recipes:
- flink::jobmanager
- hadoop::nn
- spark::master
- hadoop::rm
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.