demand concept for creating your own cluster with lambda and ec2

_Lambda_ec2_cluster.md

Infos

AMI.ecs-optimized: ami-084ab95c0cbe247e5

IAM Role in ec2 https://docs.aws.amazon.com/de_de/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html set Credentials in docker https://cameroneckelberry.co/words/getting-aws-credentials-into-a-docker-container-without-hardcoding-it

Requirements

1. process which can automate generating AMIs based on git pushes for CI/CD

2. AMIs which has model + start script when launching

3. lambda which can read the newest AMI-ID

4. lambda which can run 0-100 ec2 and scale down to 100-0 3.a CF-Template which creates

   • IAM Role for ec2 execution
   • SecurityGroup for ec2 instance
   • Subnet for ec2 instance

5. lambda which also monitors if something is wrong

1. Process which can automate generating AMIs

Create AMI with Cloudformation https://medium.com/poka-techblog/managing-amis-using-cloudformation-a097f86a3622 https://github.com/PokaInc/cloudformation-ami

write custom lambda for creating aws ami images

1. create ec2 instance

2. run instance with user_data

3. stop instance

4. create ami

ec2.create_image(InstanceId=instance_id, NoReboot=True, Name="abc")

5. terminate instance
6. create CF Stack with AMI-ID

2. lambda which can read the newest AMI-ID

Idea is to create AMI with CDK or cloudformation and output AMI-ID, in Lambda just reading this and starting the instances. So its dynamic and everytime the newest. Maybe add some ENV var to override the output for using custom AMI-ID

3. Lambda which can start and stop ec2 instances

Stopping based on InstanceID

ec2.instances.filter(InstanceIds=ids).terminate()

Starting new Images

easy start

ec2.create_instances(ImageId='<ami-image-id>', MinCount=1, MaxCount=5)

start with without subnet

import boto3

ec2 = boto3.resource('ec2')

user_data = '''#!/bin/bash
docker run xx'''

instance = ec2.create_instances(ImageId='ami-abcd1234', MinCount=1, MaxCount=1,
KeyName='my-key', SecurityGroupIds=['sg-abcd1234'], UserData=user_data, 
InstanceType='t2.nano',
IamInstanceProfile={
'Name': 'ExampleInstanceProfile'
})

complete parameter with subnet

min Parameter for starting ec2 instance

• user data = start execution script
• run python3 app.py

import boto3

ec2 = boto3.resource('ec2')

user_data = '''#!/bin/bash
echo 'test' > /tmp/hello'''

instance = ec2.create_instances(ImageId='ami-abcd1234', MinCount=1, MaxCount=1,
KeyName='my-key', SecurityGroupIds=['sg-abcd1234'], UserData=user_data, 
InstanceType='t2.nano', SubnetId='subnet-abcd1234',
IamInstanceProfile={
'Arn': 'arn:aws:iam::123456789012:instanceprofile/ExampleInstanceProfile'
'Name': 'ExampleInstanceProfile'
})

Required Parameter/Ressources

• securitygroup
• subnet
• vpc

4. Lambda monitoring which ec2 is running

 instances = ec2.instances.filter(
   Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
 for instance in instances:
   print(instance.id, instance.instance_type)

gets back instance.id for starting and stoping ec2 instances

Using filters so get specific ec2 instances

 filters = [{'Name':'tag:environment', 'Values':[Env]},
          {'Name':'tag:role', 'Values':[Role]}
         ]
 instances = ec2.instances.filter(
   Filters=filters)

Links

• automating things with python3 https://stackabuse.com/automating-aws-ec2-management-with-python-and-boto3/
• boto3 for ec2 https://boto3.amazonaws.com/v1/documentation/api/latest/guide/migrationec2.html
• ec2 user data start script https://stackoverflow.com/questions/44190556/ec2-user-data-not-working-via-python-boto-command
• tool to build amis https://github.com/mickep76/docker-build-ami https://pypi.org/project/docker-build-ami/
• Create AMI with Cloudformation https://medium.com/poka-techblog/managing-amis-using-cloudformation-a097f86a3622 https://github.com/PokaInc/cloudformation-ami/blob/master/README.md#installation

a_handler.py

# lambda for creating custom ami with docker image in it
import boto3 

user_data = '''#!/bin/bash
docker pull hello-world'''

def handler():
    instance={}
    ## creates instance
    instance['instance_id'] = create_instance()
    ## waiter for finishing user_data script
    wait_for_ec2(instance['instance_id'] )
    ## stops instance 
    stop_ami_instance(instance['instance_id'])
    ## create ami 
    instance['image_id'] = create_ami_image(instance_id=instance['instance_id'],name="test")
    ## termnaite ec2 instance 
    terminate_ami(instance['instance_id'])
    ## expose new ami_image_id
    print(instance['image_id'] )
    ## creates CF stack with Output
    create_cloudformation(image_id=instance['image_id'],cf_stack='AMI-Creation')
    return instance['image_id'] 

def create_instance():
    ec2 = boto3.resource('ec2', region_name='eu-central-1')
    try:
        instance = ec2.create_instances(
            BlockDeviceMappings=[
                {
                    'DeviceName': '/dev/sdh',
                    'VirtualName': 'ephemeral0',
                    'Ebs': {
                        'DeleteOnTermination': True,
                        'VolumeSize': 10,
                        'VolumeType': 'gp2',
                        'Encrypted': False,
                    },
                },
            ],
            ImageId='ami-084ab95c0cbe247e5',
            InstanceType='t2.micro',
            MaxCount=1,
            MinCount=1,
            UserData=user_data,
        )
        return instance[0].instance_id
    except:
        raise ValueError('couldn´t start ec2 instance')



def create_ami_image(instance_id,name):
    ec2_client = boto3.client('ec2', region_name='eu-central-1')
    try:
        image = ec2_client.create_image(InstanceId=instance_id, NoReboot=True, Name=name)
        return image['ImageId']
    except:
        raise ValueError('couldn´t create ec2 Image')



def stop_ami_instance(instance_id):
    ec2 = boto3.resource('ec2', region_name='eu-central-1')
    try:
        instance = ec2.Instance(instance_id)
        instance.stop()
        return True
    except:
        raise ValueError('couldn´t stop ec2 instance')


def terminate_ami(instance_id):
    ec2 = boto3.resource('ec2', region_name='eu-central-1')
    try:
        instance = ec2.Instance(instance_id)
        instance.terminate()
        return True
    except:
        raise ValueError('couldn´t terminate ec2 instance')


def wait_for_ec2(instance_id):
    ec2_client = boto3.client('ec2', region_name='eu-central-1')
    try:
        waiter = ec2_client.get_waiter('instance_status_ok')
        waiter.wait(InstanceIds=[instance_id])
        return True
    except:
        raise ValueError('couldn´t wait for ec2 instance')

def create_cloudformation(image_id,cf_stack):
    template =  {
        "AWSTemplateFormatVersion": "2010-09-09",
        "Description": "Stack 1",
        "Conditions": {
            "Never": {  
                "Fn::Equals": [
                    'true', 'false'
                ]} 
          
        },
        "Resources": {
            "NullResource": {
                "Type": "Custom::Null",
                "Condition":"Never"
        }
        },
        "Outputs": {
            "Nothing": {
                "Description": "Empty placeholder",
                "Value": image_id,
                "Export": {
                    "Name": "AMI-ID"
                }
            }
        }
    }
    template_json = json.dumps(template)
    cf_conn = boto3.client('cloudformation', region_name='eu-central-1')
    try:
        cf_conn.create_stack(
            StackName=cf_stack,
            TemplateBody=template_json,
        )
    except:
        cf_conn.update_stack(
            StackName=cf_stack,
            TemplateBody=template_json,
        )
    return True
   
handler()

cf-security-group.yaml

# aws --profile xxx cloudformation create-stack --stack-name stackName --template-body file://<(cfn-flip cfn-create-group.yaml) --parameters ParameterKey=STAGE,ParameterValue=qa
# aws --profile xxx  cloudformation update-stack --stack-name stackName --template-body file://<(cfn-flip cfn-create-group.yaml) --parameters ParameterKey=STAGE,ParameterValue=qa

# aws cloudformation create-stack --stack-name Sec-Group --template-body file://<(cfn-flip cfn-create-group.yml)
# aws cloudformation update-stack --stack-name Sec-Group --template-body file://<(cfn-flip cfn-create-group.yml)

AWSTemplateFormatVersion: 2010-09-09
Description: "Creates Security-Groups for ec2"

Parameters:
  # AMIImage:
  #   Type: String
  #   Default: "x"
  VpcId:
    Type: String
    Default: "vpc-7a4abd10"
  GroupName:
    Type: String
    Default: "launch-wizard-1"
  DeployRegion:
    Type: String
    Default: "eu-central-1"

Resources:
  LaunchWizard:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Ref GroupName
      GroupDescription: Launch-Group-Win10
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        - IpProtocol: tcp
          Description: GIT ohne SSL
          FromPort: 8080
          ToPort: 8080
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          Description: GIT ohne SSL
          FromPort: 8080
          ToPort: 8080
          CidrIpv6: ::/0
        - IpProtocol: tcp
          Description: Nexus SSL
          FromPort: 8441
          ToPort: 8441
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          Description: Nexus SSL
          FromPort: 8441
          ToPort: 8441
          CidrIpv6: ::/0
        - IpProtocol: tcp
          Description: TAC SSL
          FromPort: 8443
          ToPort: 8443
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          Description: TAC SSL
          FromPort: 8443
          ToPort: 8443
          CidrIpv6: ::/0
        - IpProtocol: tcp
          Description: Talend Command Line ohne SSL
          FromPort: 8002
          ToPort: 8002
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          Description: Talend Command Line ohne SSL
          FromPort: 8002
          ToPort: 8002
          CidrIpv6: ::/0
        - IpProtocol: tcp
          Description: Talend Data Catalog SSL
          FromPort: 11480
          ToPort: 11480
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          Description: Talend Data Catalog SSL
          FromPort: 11480
          ToPort: 11480
          CidrIpv6: ::/0
        - IpProtocol: tcp
          Description: Remote Desktop Connection
          FromPort: 3389
          ToPort: 3389
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          Description: Remote Desktop Connection
          FromPort: 3389
          ToPort: 3389
          CidrIpv6: ::/0
        - IpProtocol: tcp
          Description: MDM ohne SSL
          FromPort: 9080
          ToPort: 9080
          CidrIpv6: ::/0
        - IpProtocol: tcp
          Description: MDM ohne SSL
          FromPort: 9080
          ToPort: 9080
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          Description: Spotfire Server
          FromPort: 4444
          ToPort: 4444
          CidrIpv6: ::/0
        - IpProtocol: tcp
          Description: Spotfire Server
          FromPort: 4444
          ToPort: 4444
          CidrIp: 0.0.0.0/0
      # SecurityGroupEgress:
      #   IpProtocol: -1
      #   CidrIp: 0.0.0.0/0
Outputs:
  # Export SQS-data
  InsightDynamoDbName:
    Description: "Description"
    Value: !Ref LaunchWizard
    Export:
      Name: SECURITY-GROUP-NAME

create_cf.py

import boto3 
import json
import time
def create_cloudformation(image_id):
    template =  {
        "AWSTemplateFormatVersion": "2010-09-09",
        "Description": "Stack 1",
        "Conditions": {
            "Never": {  
                "Fn::Equals": [
                    'true', 'false'
                ]} 
          
        },
        "Resources": {
            "NullResource": {
                "Type": "Custom::Null",
                "Condition":"Never"
        }
        },
        "Outputs": {
            "Nothing": {
                "Description": "Empty placeholder",
                "Value": image_id,
                "Export": {
                    "Name": "AMI-ID"
                }
            }
        }
    }
    template_json = json.dumps(template)
    cf_conn = boto3.client('cloudformation', region_name='us-east-1')
    try:
        cf_conn.create_stack(
            StackName="EC2-AMI-STACK",
            TemplateBody=template_json,
        )
    except:
        cf_conn.update_stack(
            StackName="EC2-AMI-STACK",
            TemplateBody=template_json,
        )
    time.sleep(5)
    stack = cf_conn.describe_stacks(StackName="EC2-AMI-STACK")
    print(stack['Stacks'][0]['Outputs'][0]['OutputValue'])

create_cloudformation('123')