AWS Lambda AMI Backups

lambdaAMIBackups.py

# Automated AMI Backups
#
# @author Bobby Kozora
#
# This script will search for all instances having a tag with the name "backup"
# and value "Backup" on it. As soon as we have the instances list, we loop
# through each instance
# and create an AMI of it. Also, it will look for a "Retention" tag key which
# will be used as a retention policy number in days. If there is no tag with
# that name, it will use a 7 days default value for each AMI.
#
# After creating the AMI it creates a "DeleteOn" tag on the AMI indicating when
# it will be deleted using the Retention value and another Lambda function

import boto3
import collections
import datetime
import sys
import pprint

ec = boto3.client('ec2')
#image = ec.Image('id')


def lambda_handler(event, context):

    reservations = ec.describe_instances(Filters=[
        {
            'Name': 'tag-key',
            'Values': ['backup', 'Backup']
        },
    ]).get('Reservations', [])

    instances = sum([[i for i in r['Instances']] for r in reservations], [])

    print("Found %d instances that need backing up" % len(instances))

    to_tag = collections.defaultdict(list)

    for instance in instances:
        try:
            retention_days = [
                int(t.get('Value')) for t in instance['Tags']
                if t['Key'] == 'Retention'
            ][0]
        except IndexError:
            retention_days = 7

            #for dev in instance['BlockDeviceMappings']:
            #    if dev.get('Ebs', None) is None:
            #        continue
            #    vol_id = dev['Ebs']['VolumeId']
            #    print "Found EBS volume %s on instance %s" % (
            #        vol_id, instance['InstanceId'])

            #snap = ec.create_snapshot(
            #    VolumeId=vol_id,
            #)

            #create_image(instance_id, name, description=None, no_reboot=False, block_device_mapping=None, dry_run=False)
            # DryRun, InstanceId, Name, Description, NoReboot, BlockDeviceMappings
            create_time = datetime.datetime.now()
            create_fmt = create_time.strftime('%Y-%m-%d')

            AMIid = ec.create_image(
                InstanceId=instance['InstanceId'],
                Name="Lambda - " + instance['InstanceId'] + " from " +
                create_fmt,
                Description="Lambda created AMI of instance " +
                instance['InstanceId'] + " from " + create_fmt,
                NoReboot=True,
                DryRun=False)

            pprint.pprint(instance)
            #sys.exit()
            #break

            #to_tag[retention_days].append(AMIid)

            to_tag[retention_days].append(AMIid['ImageId'])

            print("Retaining AMI %s of instance %s for %d days" % (
                AMIid['ImageId'],
                instance['InstanceId'],
                retention_days,
            ))

    print(to_tag.keys())

    for retention_days in to_tag.keys():
        delete_date = datetime.date.today() + datetime.timedelta(
            days=retention_days)
        delete_fmt = delete_date.strftime('%m-%d-%Y')
        print("Will delete %d AMIs on %s" %
              (len(to_tag[retention_days]), delete_fmt))

        #break

        ec.create_tags(Resources=to_tag[retention_days],
                       Tags=[
                           {
                               'Key': 'DeleteOn',
                               'Value': delete_fmt
                           },
                       ])

Hey guys... so... I saw this and all your struggling and it kinda made me weep for the internet as a whole. So, I've been needing this a while... I finally got around to it and made it into a micro example/open source serverless stack. It's here on my github, called AWS Automated Daily Snapshots. It will...

• Automatically scan through EVERY AWS region (by default, you can also narrow the list to speed it up)
• Will automatically deploy to your AWS account with restrictive permissions and set a CloudWatch event to run daily
• Will automatically backup instances to AMIs and then delete those AMIs as they become stale
• Will preserve ALL tags that the instance had, on the AMI (and adding a few for our purposes, to rotate)
• Has a bit of error checking... but it could have more. :P Haven't hit any errors yet that I'm not catching but if you guys find any, let me know I'll gladly fix it.
• And is totally open source and simple as heck, so feel free to hack on it yourselves. Forks/MRs welcome.
• Is very loosely based on the logic you guys had here above, but really rewritten from the ground up, as you can tell. Some of the flaws in the linked delete script everyone had is fixed in my project.

Enjoy y'all! :)

I have a windows instance,
how do I have it stop the instance before it creates the AMI and start afterward?
Also does it need any waits?

@AndrewFarley
Thank you for your contribution. It worked like a charm.
two questions:

• How can we make sure the AMI-Name timestamps are based on our current timezone?
• How can we tag the snapshots using original volume tags

@AndrewFarley
Thanks for your project. Have you tried it in an account with 80+ instances? I am using @bkozora 's code which I tweaked and find that each function (creation then cleanup) runs for at least a minute each. Sometimes it fails due to boto3 errors (max retries reached) but Lambda re-runs it automatically.

Just want to know how scalable your serverless app is.

Thanks @AndrewFarley (and @bkozora for this gist). I just implemented the https://github.com/AndrewFarley/AWSAutomatedDailyInstanceAMISnapshots project to replace a previous CM-based solution we had, and it works great! I just submitted a P/R for adding to the README with examples of scheduling the daily backups via cron, and aws cli query filtering on tags, to help folks visualize and manage AMI's, etc.
This project is a gem and saved me a lot of time reinventing another wheel. Thanks again!

Anyone tweak this or the addition that @cmcconnell1 made to do snapshots. Not AMI's but of volumes, to replace the sensitive and kind of clunky AWS CLI ec2-automate-backup ?

I've just tried implementing this script for the first time and am getting the following error - can anyone provide some guidance please?


09:34:25
START RequestId: 568053dd-cf94-11e8-8b3a-a7435442769d Version: $LATEST

09:34:25
module initialization error: name 'instances' is not defined

09:34:25
END RequestId: 568053dd-cf94-11e8-8b3a-a7435442769d

09:34:25
REPORT RequestId: 568053dd-cf94-11e8-8b3a-a7435442769d Duration: 557.09 ms Billed Duration: 600 ms Memory Size: 128 MB Max Memory Used: 56 MB

09:34:25
module initialization error name 'instances' is not defined

@DJ-Laker
Copy the raw text to preserve the indentation.
https://stackoverflow.com/questions/49618760/aws-lambda-error-module-initialization-error-name-dynamodb-is-not-defined-w

some created images are not getting tags,
so , they are not deleting using delete function.

Hi i need instance names tagging with AMI as following

AMIid = ec.create_image(InstanceId=instance['InstanceId'], Name="Lambda - " + instance_Name + " from " + create_fmt, Description="Lambda created AMI of instance " + instance_Name + " from " + create_fmt, NoReboot=True, DryRun=False)

Can any one help me?
@sstalon your modification mentioned @ramesher is not working

My execution failed. Can u help me
Response:
{
"errorMessage": "Syntax error in module 'lambda_function': invalid syntax (lambda_function.py, line 39)",
"errorType": "Runtime.UserCodeSyntaxError",
"stackTrace": [
" File "/var/task/lambda_function.py" Line 39\n print "Found %d instances that need backing up" % len(instances)\n"
]
}

Request ID:
"f95badef-b2be-4716-8f57-ac650e051c03"

Function Logs:
START RequestId: f95badef-b2be-4716-8f57-ac650e051c03 Version: $LATEST
[ERROR] Runtime.UserCodeSyntaxError: Syntax error in module 'lambda_function': invalid syntax (lambda_function.py, line 39)
Traceback (most recent call last):
  File "/var/task/lambda_function.py" Line 39
        print "Found %d instances that need backing up" % len(instances)
END RequestId: f95badef-b2be-4716-8f57-ac650e051c03
REPORT RequestId: f95badef-b2be-4716-8f57-ac650e051c03 Duration: 36.79 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 55 MB

execution failed plz help me

Response:
{
"errorMessage": "Syntax error in module 'lambda_function': invalid syntax (lambda_function.py, line 39)",
"errorType": "Runtime.UserCodeSyntaxError",
"stackTrace": [
" File "/var/task/lambda_function.py" Line 39\n print "Found %d instances that need backing up" % len(instances)\n"
]
}

Request ID:
"f95badef-b2be-4716-8f57-ac650e051c03"

Function Logs:
START RequestId: f95badef-b2be-4716-8f57-ac650e051c03 Version: $LATEST
[ERROR] Runtime.UserCodeSyntaxError: Syntax error in module 'lambda_function': invalid syntax (lambda_function.py, line 39)
Traceback (most recent call last):
  File "/var/task/lambda_function.py" Line 39
        print "Found %d instances that need backing up" % len(instances)
END RequestId: f95badef-b2be-4716-8f57-ac650e051c03
REPORT RequestId: f95badef-b2be-4716-8f57-ac650e051c03 Duration: 36.79 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 55 MB

execution failed plz help me

Response:
{
"errorMessage": "Syntax error in module 'lambda_function': invalid syntax (lambda_function.py, line 39)",
"errorType": "Runtime.UserCodeSyntaxError",
"stackTrace": [
" File "/var/task/lambda_function.py" Line 39\n print "Found %d instances that need backing up" % len(instances)\n"
]
}

Request ID:
"f95badef-b2be-4716-8f57-ac650e051c03"

Function Logs:
START RequestId: f95badef-b2be-4716-8f57-ac650e051c03 Version: $LATEST
[ERROR] Runtime.UserCodeSyntaxError: Syntax error in module 'lambda_function': invalid syntax (lambda_function.py, line 39)
Traceback (most recent call last):
  File "/var/task/lambda_function.py" Line 39
        print "Found %d instances that need backing up" % len(instances)
END RequestId: f95badef-b2be-4716-8f57-ac650e051c03
REPORT RequestId: f95badef-b2be-4716-8f57-ac650e051c03 Duration: 36.79 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 55 MB

Your probably using higher than python 2.7. Looks like his code only works on Python 2.7.

I've updated the code to be syntactically compliant with Python 3, only having to wrap print in parenthesis. This will prob be the last update I make, however. AWS has evolved a great deal and now offers automated backups using Systems Manager.

https://docs.aws.amazon.com/en_pv/systems-manager/latest/userguide/automation-aws-createimage.html

https://console.aws.amazon.com/systems-manager/documents/AWS-CreateImage/content?region=us-east-1

The script works great. I want to send an email of the list of instances for which the AMI's are created. How can I get that.

hi bkozora,
Firstly let me thank you for your contribution. I have been using this script for auto AMI backup since a year ago and it works great. However since python 2 is ending support, I have to change to use the latest updated script (python 3.8). The Lambda runs and there is no specific error, I'm only getting the below message from cloudwatch but the AMI backups will not appear. Can you please help?


06:45:10
START RequestId: e87b6746-8746-4bac-ab31-6afda9ee6515 Version: $LATEST

06:45:10
Found 14 instances that need backing up

06:45:10
dict_keys([])

Hi Guys,
I took reference of this script and did ami deletion based on the tag .
But the requirement is Based on tag as well , that Tagged instance backup should be retained 30 days if more than 30 days it should delete.

Could you please help me here with logic

Code below
++++++++++
import os
import re
from datetime import datetime, timedelta
import boto3
import collections
import datetime
import logging
import time
import sys

ec = boto3.client('ec2',)
ec2 = boto3.resource('ec2',)
images = ec2.images.filter(Owners=["self"])

def lambda_handler(event, context):

to_tag = collections.defaultdict(list)
date = datetime.datetime.now()
date_fmt = date.strftime('%Y-%m-%d')

imagesList = []

for image in images:

       try:
            if image.tags is not None:
                deletion_date = [
                    t.get('Value') for t in image.tags
                    if t['Key'] == 'backup'
                ][0]
                imagesList.append(image.id)
                
                
       except IndexError:
           deletion_date = False
           delete_date = False

       today_time = datetime.datetime.now().strftime('%m-%d-%Y')
        # today_fmt = today_time.strftime('%m-%d-%Y')
       today_date = time.strptime(today_time, '%m-%d-%Y')

                     
       
print ("=============")

print ("About to process the following AMIs:")
print ('imagesList')

myAccount = boto3.client('sts').get_caller_identity()['Account']

snapshots = ec.describe_snapshots(MaxResults=1000, OwnerIds=[myAccount])['Snapshots']

# loop through list of image IDs
for image in imagesList:
    try:
        print ("deregistering image %s" % image)
        amiResponse = ec.deregister_image(
        DryRun=False,
        ImageId=image,
       )
    except ClientError:
        print ("ClientError, likely AMI not found")
        continue

    for snapshot in snapshots:
        if snapshot['Description'].find(image) > 0:
            snap = ec.delete_snapshot(SnapshotId=snapshot['SnapshotId'])
            print ("Deleting snapshot " + snapshot['SnapshotId'])

print ("-------------")
+++++++++++++++

How to tag snapshot by using ami id created by this script

Thank you for the nice script. :)

Hi thanks for this nice script. I have made a small enhancement on the script, so it also supports instances with multiple EBS volumes attached.

On line 42 i have added a new empty list, that will hold all instances that where already done :
inDone = []

And the line 67 should be surrounded by a check. Just replace the line 67 with:

            if str(instance['InstanceId']) not in inDone:

              AMIid = ec.create_image(InstanceId=instance['InstanceId'], Name="Lambda - " + instance['InstanceId'] + " from " + create_fmt, Description="Lambda created AMI of instance " + instance['InstanceId'] + " from " + create_fmt, NoReboot=True, DryRun=False)

              inDone.insert(0,str(instance['InstanceId']))

              print "Created AMI %s of instance %s " % (AMIid['ImageId'], instance['InstanceId'])

            else:

              print "We already got an AMI of instance %s " % (instance['InstanceId'])

Thanks again!
Regards, Frank

Hello Frank,

Can you paste me the full script along with modifications you made