Skip to content

Instantly share code, notes, and snippets.

@zackteo

zackteo/hdfs-spark-ec2.yaml Secret

Last active Nov 30, 2020
Embed
What would you like to do?
Cloudformation template for EC2 cluster for Hadoop and Spark
---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'AWS CloudFormation Sample Template EC2InstanceWithSecurityGroupSample:
Create an Amazon EC2 instance running the Amazon Linux AMI. The AMI is chosen based
on the region in which the stack is run. This example creates an EC2 security group
for the instance to give you SSH access. **WARNING** This template creates an Amazon
EC2 instance. You will be billed for the AWS resources used if you create a stack
from this template.'
Parameters:
KeyName:
Description: Name of an existing EC2 KeyPair to enable SSH access to the instance
Type: AWS::EC2::KeyPair::KeyName
ConstraintDescription: must be the name of an existing EC2 KeyPair.
InstanceType:
Description: WebServer EC2 instance type
Type: String
Default: t2.small
AllowedValues:
- t2.nano
- t2.micro
- t2.small
- t2.medium
- t2.large
- t2.xlarge
- t2.2xlarge
ConstraintDescription: must be a valid EC2 instance type.
SSHLocation:
Description: The IP address range that can be used to SSH to the EC2 instances
Type: String
MinLength: '9'
MaxLength: '18'
Default: 0.0.0.0/0
AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
ConstraintDescription: must be a valid IP CIDR range of the form x.x.x.x/x.
Mappings:
AWSInstanceType2Arch:
t2.nano:
Arch: HVM64
t2.micro:
Arch: HVM64
t2.small:
Arch: HVM64
t2.medium:
Arch: HVM64
t2.large:
Arch: HVM64
t2.xlarge:
Arch: HVM64
t2.2xlarge:
Arch: HVM64
AWSInstanceType2NATArch:
t2.nano:
Arch: NATHVM64
t2.micro:
Arch: NATHVM64
t2.small:
Arch: NATHVM64
t2.medium:
Arch: NATHVM64
t2.large:
Arch: NATHVM64
t2.xlarge:
Arch: NATHVM64
t2.2xlarge:
Arch: NATHVM64
AWSRegionArch2AMI:
ap-east-1:
HVM64: ami-f85b1989
HVMG2: NOT_SUPPORTED
ap-northeast-1:
HVM64: ami-0b2c2a754d5b4da22
HVMG2: ami-09d0e0e099ecabba2
ap-northeast-2:
HVM64: ami-0493ab99920f410fc
HVMG2: NOT_SUPPORTED
ap-northeast-3:
HVM64: ami-01344f6f63a4decc1
HVMG2: NOT_SUPPORTED
ap-south-1:
HVM64: ami-03cfb5e1fb4fac428
HVMG2: ami-0244c1d42815af84a
ap-southeast-1:
HVM64: ami-0ba35dc9caf73d1c7
HVMG2: ami-0e46ce0d6a87dc979
ap-southeast-2:
HVM64: ami-0ae99b503e8694028
HVMG2: ami-0c0ab057a101d8ff2
us-east-1:
HVM64: ami-0885b1f6bd170450c # Ubuntu Server 20.04 LTS
HVMG2: ami-0aeb704d503081ea6
us-east-2:
HVM64: ami-027cab9a7bf0155df
HVMG2: NOT_SUPPORTED
us-west-1:
HVM64: ami-088c153f74339f34c
HVMG2: ami-0a7fc72dc0e51aa77
us-west-2:
HVM64: ami-01fee56b22f308154
HVMG2: ami-0fe84a5b4563d8f27
Resources:
EC2Instance1:
Type: AWS::EC2::Instance
DependsOn:
- InstanceSecurityGroup
- PrivateSubnet1
Properties:
InstanceType:
Ref: InstanceType
# SecurityGroups:
# - Ref: InstanceSecurityGroup
KeyName:
Ref: KeyName
ImageId:
Fn::FindInMap:
- AWSRegionArch2AMI
- Ref: AWS::Region
- Fn::FindInMap:
- AWSInstanceType2Arch
- Ref: InstanceType
- Arch
NetworkInterfaces:
- PrivateIpAddress: '172.31.23.4'
SubnetId: !Ref PrivateSubnet1
DeviceIndex: '0'
GroupSet:
- !Ref InstanceSecurityGroup
AssociatePublicIpAddress: true
UserData: !Base64
'Fn::Join':
- ''
- - |
#!/bin/bash -xe
- |
cd /home/ubuntu/
- |
mkdir setup && cd setup
- |
curl https://gist.githubusercontent.com/zackteo/0924304ec7616a92689dc90dcceabb25/raw/5e42aae45ba2c48a0fe6a235132c56734e9a481a/namenode.sh > setup.sh
- |
chmod +x setup.sh
- |
#./setup.sh
- |+
EC2Instance2:
Type: AWS::EC2::Instance
DependsOn:
- InstanceSecurityGroup
- PrivateSubnet1
Properties:
InstanceType:
Ref: InstanceType
KeyName:
Ref: KeyName
ImageId:
Fn::FindInMap:
- AWSRegionArch2AMI
- Ref: AWS::Region
- Fn::FindInMap:
- AWSInstanceType2Arch
- Ref: InstanceType
- Arch
NetworkInterfaces:
- PrivateIpAddress: '172.31.23.5'
SubnetId: !Ref PrivateSubnet1
DeviceIndex: '0'
GroupSet:
- !Ref InstanceSecurityGroup
AssociatePublicIpAddress: true
# NetworkInterfaces:
# # PrivateIpAddress: '172.31.23.5'
# #SubnetId: !Ref PrivateSubnet1
# - DeviceIndex: '1'
# NetworkInterfaceId: !Ref EC2Instance1
# GroupSet:
# - !Ref InstanceSecurityGroup
# #AssociatePublicIpAddress: true
UserData: !Base64
'Fn::Join':
- ''
- - |
#!/bin/bash -xe
- |
cd /home/ubuntu/
- |
mkdir setup && cd setup
- |
curl https://gist.githubusercontent.com/zackteo/0924304ec7616a92689dc90dcceabb25/raw/5e42aae45ba2c48a0fe6a235132c56734e9a481a/datanode.sh > setup.sh
- |
chmod +x setup.sh
- |
#./setup.sh
- |+
EC2Instance3:
Type: AWS::EC2::Instance
DependsOn:
- InstanceSecurityGroup
- PrivateSubnet1
Properties:
InstanceType:
Ref: InstanceType
KeyName:
Ref: KeyName
ImageId:
Fn::FindInMap:
- AWSRegionArch2AMI
- Ref: AWS::Region
- Fn::FindInMap:
- AWSInstanceType2Arch
- Ref: InstanceType
- Arch
NetworkInterfaces:
- PrivateIpAddress: '172.31.23.6'
SubnetId: !Ref PrivateSubnet1
DeviceIndex: '0'
GroupSet:
- !Ref InstanceSecurityGroup
AssociatePublicIpAddress: true
UserData: !Base64
'Fn::Join':
- ''
- - |
#!/bin/bash -xe
- |
cd /home/ubuntu/
- |
mkdir setup && cd setup
- |
curl https://gist.githubusercontent.com/zackteo/0924304ec7616a92689dc90dcceabb25/raw/5e42aae45ba2c48a0fe6a235132c56734e9a481a/datanode.sh > setup.sh
- |
chmod +x setup.sh
- |
./setup.sh
- |+
EC2Instance4:
Type: AWS::EC2::Instance
DependsOn:
- InstanceSecurityGroup
- PrivateSubnet1
Properties:
InstanceType:
Ref: InstanceType
KeyName:
Ref: KeyName
ImageId:
Fn::FindInMap:
- AWSRegionArch2AMI
- Ref: AWS::Region
- Fn::FindInMap:
- AWSInstanceType2Arch
- Ref: InstanceType
- Arch
NetworkInterfaces:
- PrivateIpAddress: '172.31.23.7'
SubnetId: !Ref PrivateSubnet1
DeviceIndex: '0'
GroupSet:
- !Ref InstanceSecurityGroup
AssociatePublicIpAddress: true
UserData: !Base64
'Fn::Join':
- ''
- - |
#!/bin/bash -xe
- |
cd /home/ubuntu/
- |
mkdir setup && cd setup
- |
curl https://gist.githubusercontent.com/zackteo/0924304ec7616a92689dc90dcceabb25/raw/5e42aae45ba2c48a0fe6a235132c56734e9a481a/datanode.sh > setup.sh
- |
chmod +x setup.sh
- |
./setup.sh
- |+
PrivateSubnet1:
Type: 'AWS::EC2::Subnet'
Properties:
CidrBlock: 172.31.23.0/24
MapPublicIpOnLaunch: true
VpcId: !Ref VPCloud
DependsOn:
- VPCloud
VPCGatewayAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref VPCloud
InternetGatewayId: !Ref InternetGateway
InternetGateway:
Type: 'AWS::EC2::InternetGateway'
VPCloud:
Type: 'AWS::EC2::VPC'
Properties:
CidrBlock: 172.31.23.0/16
EnableDnsHostnames: true
EnableDnsSupport: true
InstanceSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Enable SSH access via port 22
VpcId: !Ref VPCloud
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: '22'
ToPort: '22'
CidrIp:
Ref: SSHLocation
SecurityGroupEgress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
RouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPCloud
InternetRoute:
Type: AWS::EC2::Route
DependsOn: VPCGatewayAttachment
Properties:
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref InternetGateway
RouteTableId: !Ref RouteTable
SubnetARouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref RouteTable
SubnetId: !Ref PrivateSubnet1
Outputs:
InstanceId:
Description: InstanceId of the newly created EC2 instance
Value:
Ref: EC2Instance1
AZ:
Description: Availability Zone of the newly created EC2 instance
Value:
Fn::GetAtt:
- EC2Instance1
- AvailabilityZone
PublicDNS:
Description: Public DNSName of the newly created EC2 instance
Value:
Fn::GetAtt:
- EC2Instance1
- PublicDnsName
PublicIP:
Description: Public IP address of the newly created EC2 instance
Value:
Fn::GetAtt:
- EC2Instance1
- PublicIp
PublicDNS2:
Description: Public DNSName of the newly created EC2 instance
Value:
Fn::GetAtt:
- EC2Instance2
- PublicDnsName
PublicDNS3:
Description: Public DNSName of the newly created EC2 instance
Value:
Fn::GetAtt:
- EC2Instance3
- PublicDnsName
PublicDNS4:
Description: Public DNSName of the newly created EC2 instance
Value:
Fn::GetAtt:
- EC2Instance4
- PublicDnsName
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment