Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
AWS Lambda Function to Delete AMIs and Snapshots
# Automated AMI and Snapshot Deletion
#
# @author Robert Kozora <bobby@kozora.me>
#
# This script will search for all instances having a tag with "Backup" or "backup"
# on it. As soon as we have the instances list, we loop through each instance
# and reference the AMIs of that instance. We check that the latest daily backup
# succeeded then we store every image that's reached its DeleteOn tag's date for
# deletion. We then loop through the AMIs, deregister them and remove all the
# snapshots associated with that AMI.
import boto3
import collections
import datetime
import time
import sys
ec = boto3.client('ec2', 'us-east-1')
ec2 = boto3.resource('ec2', 'us-east-1')
images = ec2.images.filter(Owners=["self"])
def lambda_handler(event, context):
reservations = ec.describe_instances(
Filters=[
{'Name': 'tag-key', 'Values': ['backup', 'Backup']},
]
).get(
'Reservations', []
)
instances = sum(
[
[i for i in r['Instances']]
for r in reservations
], [])
print "Found %d instances that need evaluated" % len(instances)
to_tag = collections.defaultdict(list)
date = datetime.datetime.now()
date_fmt = date.strftime('%Y-%m-%d')
imagesList = []
# Set to true once we confirm we have a backup taken today
backupSuccess = False
# Loop through all of our instances with a tag named "Backup"
for instance in instances:
imagecount = 0
# Loop through each image of our current instance
for image in images:
# Our other Lambda Function names its AMIs Lambda - i-instancenumber.
# We now know these images are auto created
if image.name.startswith('Lambda - ' + instance['InstanceId']):
# print "FOUND IMAGE " + image.id + " FOR INSTANCE " + instance['InstanceId']
# Count this image's occcurance
imagecount = imagecount + 1
try:
if image.tags is not None:
deletion_date = [
t.get('Value') for t in image.tags
if t['Key'] == 'DeleteOn'][0]
delete_date = time.strptime(deletion_date, "%m-%d-%Y")
except IndexError:
deletion_date = False
delete_date = False
today_time = datetime.datetime.now().strftime('%m-%d-%Y')
# today_fmt = today_time.strftime('%m-%d-%Y')
today_date = time.strptime(today_time, '%m-%d-%Y')
# If image's DeleteOn date is less than or equal to today,
# add this image to our list of images to process later
if delete_date <= today_date:
imagesList.append(image.id)
# Make sure we have an AMI from today and mark backupSuccess as true
if image.name.endswith(date_fmt):
# Our latest backup from our other Lambda Function succeeded
backupSuccess = True
print "Latest backup from " + date_fmt + " was a success"
print "instance " + instance['InstanceId'] + " has " + str(imagecount) + " AMIs"
print "============="
print "About to process the following AMIs:"
print imagesList
if backupSuccess == True:
myAccount = boto3.client('sts').get_caller_identity()['Account']
snapshots = ec.describe_snapshots(MaxResults=1000, OwnerIds=[myAccount])['Snapshots']
# loop through list of image IDs
for image in imagesList:
print "deregistering image %s" % image
amiResponse = ec.deregister_image(
DryRun=False,
ImageId=image,
)
for snapshot in snapshots:
if snapshot['Description'].find(image) > 0:
snap = ec.delete_snapshot(SnapshotId=snapshot['SnapshotId'])
print "Deleting snapshot " + snapshot['SnapshotId']
print "-------------"
else:
print "No current backup found. Termination suspended."
@esong01

This comment has been minimized.

Copy link

commented Sep 1, 2016

how did you manage to configure that triggers on your AWS console, or did you not manage this python on your lambda? I am trying to set up using Lambda AWS.

@bkozora

This comment has been minimized.

Copy link
Owner Author

commented Dec 19, 2016

I just set it up as a cron-like Lambda trigger http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html

@Fzomerdijk

This comment has been minimized.

Copy link

commented Jan 31, 2017

Thanks for this nice script, I have made a small enhancement.

The line 20 can be replaced by:
images = ec2.images.filter(Owners=["self"])

And the line 100 can be replaced with:

    myAccount = boto3.client('sts').get_caller_identity()['Account']

    snapshots = ec.describe_snapshots(MaxResults=1000, OwnerIds=[myAccount])['Snapshots']

Thanks,
Frank

@raakey

This comment has been minimized.

Copy link

commented May 22, 2017

Getting Error while executing the script

"errorMessage": "Syntax error in module 'lambda_function'"

Syntax error in module 'lambda_function': inconsistent use of tabs and spaces in indentation (lambda_function.py, line 49)

And also it's pointing the function error, where print command in script.

@YorkITGuy

This comment has been minimized.

Copy link

commented Oct 3, 2017

When I run this function to delete AMIs and snapshots; it never deletes anything. I updated the xxxx with my AWS account ID. The backup function is creating the AMIs and they have the DeleteOn Tag with the proper date. The CloudWatch log of the Delete function says "instance "instance ID" has 0 AMIs" for all of the EC2 instances.

@bkozora

This comment has been minimized.

Copy link
Owner Author

commented Oct 11, 2017

@Fzomerdijk thanks so much! Updating it now.

@bkozora

This comment has been minimized.

Copy link
Owner Author

commented Oct 11, 2017

Anyone with syntax errors: Make sure to copy/paste from the raw source to retain proper punctuation. These scripts run fine when copied correctly.

@bkozora

This comment has been minimized.

Copy link
Owner Author

commented Oct 11, 2017

@YorkITGuy Are you using my other Python AWS Lambda function to create the AMIs? https://gist.github.com/bkozora/724e01903a9ad481d21e The two scripts are meant to work together.

@srivivino

This comment has been minimized.

Copy link

commented Nov 16, 2017

@bkozora The Given script is executing fine without any error. It is Deregistering the only AMIs which is having 'Backup' tag, But the script is not deleting the corresponding snapshots. Request you to help me on the same.

@AkshatGhai123

This comment has been minimized.

Copy link

commented Dec 13, 2017

how can we modify it to search in more than one region?

@aboarya

This comment has been minimized.

Copy link

commented Jan 17, 2018

Imagine if you had 100 instances and your retention period is 14 days. You are looping through 1400 images, 100 times. I am trying to workout a better way. Either remove image from images in the inner loop, since an AMI is only for one instance, or a better filter. Also, why use resource AND client?

@akdiegorsantos

This comment has been minimized.

Copy link

commented Jan 18, 2018

This project is very nice. I also recommend using AWS Ops Automator: https://docs.aws.amazon.com/solutions/latest/ops-automator/overview.html

@muralisheri

This comment has been minimized.

Copy link

commented Jan 26, 2018

@bkozore even though i have the delete on tag, it gives a message "instance i-bxxxxx5 has 0 AMIs". I have used your own script to create AMI
https://gist.github.com/bkozora/724e01903a9ad481d21e
and i also noticed when it creating the AMIs it is not updating the Deleton Tag for all the AMIs

@santhosh261190

This comment has been minimized.

Copy link

commented Feb 13, 2018

@muralisheri
Ensure you have updated below lines with your instance own instance region {us-east-1}
ec = boto3.client('ec2', 'us-east-1')
ec2 = boto3.resource('ec2', 'us-east-1')

@santhosh261190

This comment has been minimized.

Copy link

commented Feb 13, 2018

@bkozora
I have the ami's listed for de-register while running the deletion script but it does not remove the AMI's with proper "Delete on" date tag.
I'm getting below message after listing AMI's ready for retention.
""
No current backup found. Termination suspended

""

@lkonradi

This comment has been minimized.

Copy link

commented Mar 7, 2018

@santhosh261190 have you found a resolution to the issue "No current backup found. Termination suspended"?

@mohanish12

This comment has been minimized.

Copy link

commented Mar 12, 2018

hi,

The script is running fine for AMI deletion but the corresponding snapshots are not getting deleted.
I checked the "for" loop for snapshot deletion, I guess in the If condition it checks for AMI name in the snapshot.
My snapshots have AMI name in the description so it should work but its not working.

Anyway I can troubleshoot what is the cause of the failure or any help

@mcalr3

This comment has been minimized.

Copy link

commented Apr 9, 2018

@mohanish12 For those of you who changed line 65 in the backup function script:

create_fmt = create_time.strftime('%Y-%m-%d')
To
create_fmt = create_time.strftime('%Y-%m-%d--%H-%M-%S')

The cleanup function is looking for if image.name.endswith(date_fmt): on line 86. Since the strftime variable does not now end with today's date (now time), then it will throw error.

My workaround was to put the time in front of the date, like this:
create_fmt = create_time.strftime('%H-%M-%S(UTC)--on--%Y-%m-%d')

Hope this helps someone.

@kishoretheknight

This comment has been minimized.

Copy link

commented May 4, 2018

HI @bkozora

I am getting this error while testing. How to resolve this? Thanks in advance

{
"stackTrace": [
[
"/var/task/lambda_function.py",
79,
"lambda_handler",
"if delete_date <= today_date:"
]
],
"errorType": "UnboundLocalError",
"errorMessage": "local variable 'delete_date' referenced before assignment"
}

@kishoretheknight

This comment has been minimized.

Copy link

commented May 21, 2018

Can someone help to resolve the above issue?

@mcalr3

This comment has been minimized.

Copy link

commented May 22, 2018

Having implemented this function to back up over 80 instances, it keeps timing out even when function is set to 5 minute maxiumum with 512MB RAM. Looking at the cloudwatch logs, most of the time is spent looping through each instance to find the AMI's assigned and check if there is a backup today.

My first thought was to decouple the function into two, so parsing the imagesList list to something like SQS or S3 and then a second function to read the list and deregister the AMIs and delete the snapshots.

Would anyone be able to help with this? I am relatively new to python coming from an Ops background.

@mcalr3

This comment has been minimized.

Copy link

commented Jun 4, 2018

I found the solution to poor performance and timeouts.
First, I modified the backup script to create a tag of InstanceID on the AMI. This way we can filter the AMIs on the For instance in instances loop to only list the AMIs relating to that instance instead of it looping through every AMI in the account (in our account we have over 4000 and 80 instances. So it would loop through 4000 AMIs 80 times!!).

Here is the code. See my comment on the bkozora's backup script to see the code inserted to add the tag to instances.

    # Loop through all of our instances with a tag named "Backup"
    for instance in instances:
	imagecount = 0
	instance_ID = instance['InstanceId']
    
        # Loop through each image of our current instance
        images = ec2.images.filter(
        Owners=["self"],
        Filters=[
        {
            'Name': 'tag:InstanceID',
            'Values': [instance_ID
                ,
            ]
        },
        ],
        )
        for image in images:
@muralikrishna2basa

This comment has been minimized.

Copy link

commented Jul 23, 2018

@bkozora The Given script is executing fine without any error. It is Deregistering the only AMIs which is having 'Backup' tag, But the script is not deleting the corresponding snapshots. Request you to help me on the same.

Hi @bkozora, Still it is open, could you please help me on this?

@dcr18

This comment has been minimized.

Copy link

commented Aug 16, 2018

The 3 snapshots are not getting deleted though the script is picking up the correct snapshot. The AMIs are deleted daily. I have to manually delete the snapshots.

I have specified AWS Account Number in the place of "XXXXX" for both snapshots and AMI in the cleanup script.

DeleteOn date for AMIs are set to 1 day.

Cleanup Log:

21:00:20
START RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Version: $LATEST

21:00:21
Found 3 instances that need evaluated
Found 3 instances that need evaluated

21:00:21
Present date and time:10-08-2018:21.08.1533934821
Present date and time:10-08-2018:21.08.1533934821

21:00:21
Latest backup from 10-08-2018 was a success
Latest backup from 10-08-2018 was a success

21:00:21
instance i-0c11ae923979330b4 has 2 AMIs
instance i-0c11ae923979330b4 has 2 AMIs

21:00:21
Latest backup from 10-08-2018 was a success
Latest backup from 10-08-2018 was a success

21:00:21
instance i-0f92fc4815a37f6d7 has 2 AMIs
instance i-0f92fc4815a37f6d7 has 2 AMIs

21:00:21
Latest backup from 10-08-2018 was a success
Latest backup from 10-08-2018 was a success

21:00:21
instance i-07472214fe53e242f has 2 AMIs
instance i-07472214fe53e242f has 2 AMIs

21:00:21

=============

21:00:21
About to process the following AMIs:
About to process the following AMIs:

21:00:21
['ami-0ad6753ac9acd5a2f', 'ami-01ff3bbb8ce3d1dbb', 'ami-02157a595bf3b1e4b']
['ami-0ad6753ac9acd5a2f', 'ami-01ff3bbb8ce3d1dbb', 'ami-02157a595bf3b1e4b']

21:00:21
Ignore Index Error:Ebs
Ignore Index Error:Ebs

21:00:21
Deregistering image ami-0ad6753ac9acd5a2f
Deregistering image ami-0ad6753ac9acd5a2f

21:00:22
Ignore Index Error:Ebs
Ignore Index Error:Ebs

21:00:22
Deregistering image ami-01ff3bbb8ce3d1dbb
Deregistering image ami-01ff3bbb8ce3d1dbb

21:00:22
Deregistering image ami-02157a595bf3b1e4b
Deregistering image ami-02157a595bf3b1e4b

21:00:22

=============

21:00:22
About to process the following Snapshots associated with above Images:
About to process the following Snapshots associated with above Images:

21:00:22
['snap-0324d1ee46d02d068', 'snap-0a1b2be8066efca9d', 'snap-0ffb018ae4bfaf0a0']
['snap-0324d1ee46d02d068', 'snap-0a1b2be8066efca9d', 'snap-0ffb018ae4bfaf0a0']

21:00:22
The timer is started for 10 seconds to wait for images to deregister before deleting the snapshots associated to it
The timer is started for 10 seconds to wait for images to deregister before deleting the snapshots associated to it

21:00:30
END RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46
END RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46

21:00:30
REPORT RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Duration: 10010.39 ms Billed Duration: 10000 ms Memory Size: 128 MB Max Memory Used: 61 MB
REPORT RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Duration: 10010.39 ms Billed Duration: 10000 ms Memory Size: 128 MB Max Memory Used: 61 MB

21:00:30
2018-08-10T21:00:30.564Z 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Task timed out after 10.01 seconds
2018-08-10T21:00:30.564Z 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Task timed out after 10.01 seconds


21:01:31
START RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Version: $LATEST
START RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Version: $LATEST

21:01:32
Found 3 instances that need evaluated
Found 3 instances that need evaluated

21:01:32
Present date and time:10-08-2018:21.08.1533934892
Present date and time:10-08-2018:21.08.1533934892

21:01:32
local variable 'delete_date' referenced before assignment: UnboundLocalError Traceback (most recent call last): File "/var/task/lambda_function.py", line 89, in lambda_handler if delete_date <= today_date: UnboundLocalError: local variable 'delete_date' referenced before assignment
local variable 'delete_date' referenced before assignment: UnboundLocalError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 89, in lambda_handler
if delete_date <= today_date:
UnboundLocalError: local variable 'delete_date' referenced before assignment


21:01:32
END RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46
END RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46

21:01:32
REPORT RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Duration: 866.40 ms Billed Duration: 900 ms Memory Size: 128 MB Max Memory Used: 50 MB
REPORT RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Duration: 866.40 ms Billed Duration: 900 ms Memory Size: 128 MB Max Memory Used: 50 MB

21:03:22
START RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Version: $LATEST
START RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Version: $LATEST

21:03:23
Found 3 instances that need evaluated
Found 3 instances that need evaluated

21:03:23
Present date and time:10-08-2018:21.08.1533935003
Present date and time:10-08-2018:21.08.1533935003

21:03:23
Latest backup from 10-08-2018 was a success
Latest backup from 10-08-2018 was a success

21:03:23
instance i-0c11ae923979330b4 has 1 AMIs
instance i-0c11ae923979330b4 has 1 AMIs

21:03:23
Latest backup from 10-08-2018 was a success
Latest backup from 10-08-2018 was a success

21:03:23
instance i-0f92fc4815a37f6d7 has 1 AMIs
instance i-0f92fc4815a37f6d7 has 1 AMIs

21:03:23
Latest backup from 10-08-2018 was a success
Latest backup from 10-08-2018 was a success

21:03:23
instance i-07472214fe53e242f has 1 AMIs
instance i-07472214fe53e242f has 1 AMIs

21:03:23

=============

21:03:23
About to process the following AMIs:
About to process the following AMIs:

21:03:23
[]
[]

21:03:23

=============

21:03:23
About to process the following Snapshots associated with above Images:
About to process the following Snapshots associated with above Images:

21:03:23
[]
[]

21:03:23
The timer is started for 10 seconds to wait for images to deregister before deleting the snapshots associated to it
The timer is started for 10 seconds to wait for images to deregister before deleting the snapshots associated to it

21:03:32
END RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46
END RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46

21:03:32
REPORT RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Duration: 10010.22 ms Billed Duration: 10000 ms Memory Size: 128 MB Max Memory Used: 50 MB
REPORT RequestId: 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Duration: 10010.22 ms Billed Duration: 10000 ms Memory Size: 128 MB Max Memory Used: 50 MB

21:03:32
2018-08-10T21:03:32.781Z 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Task timed out after 10.01 seconds
2018-08-10T21:03:32.781Z 63dec014-9ce0-11e8-919b-ab3d3a2b3f46 Task timed out after 10.01 seconds

@tomoliveri

This comment has been minimized.

Copy link

commented Sep 5, 2018

#! /usr/bin/env python

# Automated AMI and Snapshot Deletion
#
# @author Robert Kozora <bobby@kozora.me>

import boto3
import collections
import datetime
import time
import sys

ec = boto3.client('ec2',)
ec2 = boto3.resource('ec2',)
images = ec2.images.filter(Owners=["self"])

def lambda_handler(event, context):

    to_tag = collections.defaultdict(list)

    date = datetime.datetime.now()
    date_fmt = date.strftime('%Y-%m-%d')

    imagesList = []

    for image in images:

            try:
                if image.tags is not None:
                    deletion_date = [
                        t.get('Value') for t in image.tags
                        if t['Key'] == 'DeleteOn'][0]
                    delete_date = time.strptime(deletion_date, "%m-%d-%Y")
            except IndexError:
                deletion_date = False
                delete_date = False

            today_time = datetime.datetime.now().strftime('%m-%d-%Y')
            # today_fmt = today_time.strftime('%m-%d-%Y')
            today_date = time.strptime(today_time, '%m-%d-%Y')

            # If image's DeleteOn date is less than or equal to today,
            # add this image to our list of images to process later
            if delete_date <= today_date:
                imagesList.append(image.id)

    print "============="

    print "About to process the following AMIs:"
    print imagesList

    myAccount = boto3.client('sts').get_caller_identity()['Account']

    snapshots = ec.describe_snapshots(MaxResults=1000, OwnerIds=[myAccount])['Snapshots']

    # loop through list of image IDs
    for image in imagesList:
        try:
            print "deregistering image %s" % image
            amiResponse = ec.deregister_image(
            DryRun=False,
            ImageId=image,
           )
        except ClientError:
            print "ClientError, likely AMI not found"
            continue

        for snapshot in snapshots:
            if snapshot['Description'].find(image) > 0:
                snap = ec.delete_snapshot(SnapshotId=snapshot['SnapshotId'])
                print "Deleting snapshot " + snapshot['SnapshotId']
print "-------------"
@glekoliveira

This comment has been minimized.

Copy link

commented Feb 22, 2019

Hi,

Anyone found a resolution to the issue "No current backup found. Termination suspended"? The script found the AMI but can´t delete it.

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.