Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
Programmatically manipulate AWS resources with boto3 - a quick hands on

boto3 quick hands-on

This documentation aims at being a quick-straight-to-the-point-hands-on AWS resources manipulation with boto3.

First of all, you'll need to install boto3. Installing it along with awscli is probably a good idea as

  • awscli is boto-based
  • awscli usage is really close to boto's
  • boto3 will use the same configuration files

A convenient method consists in installing them in a python virtualenv:

$ virtualenv awscli
$ . awscli/bin/activate
(awscli)$ pip install awscli boto3

From now on I will assume you already have an AWS account setup, best practices suggest that you must NOT use your master account but instead create an account which might have adequate privileges.

$ aws configure
AWS Access Key ID [None]: your_key_id
AWS Secret Access Key [None]: your_access_key
Default region name [None]: eu-west-1
Default output format [None]: json

In order switch between regions more easily, I suggest you make aliases on the awscli configuration files:

$ cat ~/.aws/credentials 
aws_access_key_id = your_key_id
aws_secret_access_key = your_access_key

aws_access_key_id = your_key_id
aws_secret_access_key = your_access_key

$ cat ~/.aws/config
output = json
region = eu-west-1

[profile ireland]

You should then be able to summon awscli by region:

$ aws ec2 describe-instances --profile ireland
    [some json output]

We are now able to dig around with boto3 itself. In order to learn its usage, we'll use the fantastic ipython. First import boto3:

$ ipython3
In [1]: import boto3

Then initiate a boto session:

In [2]: s = boto3.Session(profile_name='ireland')

From there, initiate an object corresponding to the resource you want to manipulate, for instance ec2:

In [3]: ec2 = s.resource('ec2')

And witness the methods at your disposal:

In [4]: ec2.[press tab]
ec2.DhcpOptions                    ec2.create_snapshot
ec2.Image                          ec2.create_subnet
ec2.Instance                       ec2.create_tags
ec2.InternetGateway                ec2.create_volume
ec2.KeyPair                        ec2.create_vpc
ec2.NetworkAcl                     ec2.create_vpc_peering_connection
ec2.NetworkInterface               ec2.dhcp_options_sets
ec2.PlacementGroup                 ec2.disassociate_route_table
ec2.RouteTable                     ec2.images
ec2.RouteTableAssociation          ec2.import_key_pair
ec2.SecurityGroup                  ec2.instances
ec2.Snapshot                       ec2.internet_gateways
ec2.Subnet                         ec2.key_pairs
ec2.Tag                            ec2.meta
ec2.Volume                         ec2.network_acls
ec2.Vpc                            ec2.network_interfaces
ec2.VpcPeeringConnection           ec2.placement_groups
ec2.create_dhcp_options            ec2.register_image
ec2.create_instances               ec2.route_tables
ec2.create_internet_gateway        ec2.security_groups
ec2.create_key_pair                ec2.snapshots
ec2.create_network_acl             ec2.subnets
ec2.create_network_interface       ec2.volumes
ec2.create_placement_group         ec2.vpc_peering_connections
ec2.create_route_table             ec2.vpcs

While those methods permit to execute operations, they also give access to objects themselves, for example:

In [5]: for i in ec2.instances.all(): print(i)

Let's pick one of those:

In [6]: i = ec2.Instance(id='i-ea39b240')

And discover its methods:

In [7]: i.[press tab]
i.ami_launch_index          i.private_ip_address
i.architecture              i.product_codes
i.attach_classic_link_vpc   i.public_dns_name
i.attach_volume             i.public_ip_address
i.block_device_mappings     i.ramdisk_id
i.client_token              i.reboot
i.console_output            i.reload
i.create_image              i.report_status
i.create_tags               i.reset_attribute
i.describe_attribute        i.reset_kernel
i.detach_classic_link_vpc   i.reset_ramdisk
i.detach_volume             i.reset_source_dest_check
i.ebs_optimized             i.root_device_name
i.hypervisor                i.root_device_type
i.iam_instance_profile      i.security_groups                        i.source_dest_check
i.image                     i.spot_instance_request_id
i.image_id                  i.sriov_net_support
i.instance_id               i.start
i.instance_lifecycle        i.state
i.instance_type             i.state_reason
i.kernel_id                 i.state_transition_reason
i.key_name                  i.stop
i.key_pair                  i.subnet
i.launch_time               i.subnet_id
i.load                      i.tags
i.meta                      i.terminate
i.modify_attribute          i.unmonitor
i.monitor                   i.virtualization_type
i.monitoring                i.volumes
i.network_interfaces        i.vpc
i.password_data             i.vpc_id
i.placement                 i.wait_until_exists
i.placement_group           i.wait_until_running
i.platform                  i.wait_until_stopped
i.private_dns_name          i.wait_until_terminated

For example:

In [8]: i.hypervisor
Out[8]: 'xen'

Here's an example of the filter method that many objects use to match only certain criterias:

In [9]: filter = {'Name': 'name', 'Values' : ['debian*amd64*ebs']}
In [10]: for i in ec2.images.filter(Filters = [filter]): print(i)

Accessing services through resource is called the high level method, and not everything is yet reachable through this consistent interface. The other, low-level method is client:

In [11]: ec2c = s.client('ec2')
In [12]: ec2c.[press tab]
Display all 188 possibilities? (y or n)

As a usage example, no high-level resource is available to list a region Availability Zones, but a client can achieve this:

In [13]: ec2c.describe_availability_zones()
{'AvailabilityZones': [{'Messages': [],
   'RegionName': 'eu-west-1',
   'State': 'available',
   'ZoneName': 'eu-west-1a'},
  {'Messages': [],
   'RegionName': 'eu-west-1',
   'State': 'available',
   'ZoneName': 'eu-west-1b'},
  {'Messages': [],
   'RegionName': 'eu-west-1',
   'State': 'available',
   'ZoneName': 'eu-west-1c'}],
 'ResponseMetadata': {'HTTPStatusCode': 200,
  'RequestId': '46e474a5-7e77-4716-a9a1-010990d066ee'}}

Every single resource and client methods are documented in boto3's documentation.

Last but not least, I wrote a convenient little module in order to ease boto3 usage. It is available in my GitHub repository.

Its usage is pretty straightforward:

In [14]: from session import Aws
In [15]: ec2 = Aws('ireland', 'ec2')

The instanciated ec2 objects holds both resource and client methods along with helper functions and variables:

In [16]: ec2.[press tab]
ec2.change_nsrecord      ec2.getamis              ec2.profile
ec2.client               ec2.getinst              ec2.region
ec2.create_tag           ec2.gettagval            ec2.resource
ec2.dmesg                ec2.lsinstances          ec2.session
ec2.get_id_from_nametag  ec2.lsinstnames          ec2.tags2dict
ec2.getall               ec2.mktags               
ec2.getami               ec2.mkuserdata

Every helper is nicely documented:

In [17]: ec2.change_nsrecord??
    def change_nsrecord(self, action, dnsrecord):
        '''Create, delete or modify a DNS record

        :param str action: One of ``CREATE``, ``DELETE`` or ``UPSERT``
        :param dict dnsrecord: A dict describing the DNS record to change

And uses boto3 functions behind the scenes:

In [18]: ec2.dmesg('i-ea39b240').split('\n')[0]
Out[18]: u' wheezy/main debconf-utils all 1.5.49 [55.8 kB]\r'

Let's finish this quick hands on with a very basic EC2 instance creation and termination using boto3:

In [19]: rc = ec2.resource.create_instances(
    ImageId = ec2.getami('NetBSD*64*6.1.5*'),
    MinCount = 1,
    MaxCount = 1,
    KeyName = 'mysshpemkey',
    InstanceType = 'm3.medium',
    PrivateIpAddress = '',
    SubnetId = ec2.get_id_from_nametag('subnets', 'examplesubnet')
In [20]: print(rc[0].id)
In [21]: rc[0].terminate()
{'ResponseMetadata': {'HTTPStatusCode': 200,
  'RequestId': '45738798-812c-4ea4-9c15-edb7ddcd9950'},
 u'TerminatingInstances': [{u'CurrentState': {u'Code': 32,
    u'Name': 'shutting-down'},
   u'InstanceId': 'i-b1774f1b',
   u'PreviousState': {u'Code': 16, u'Name': 'running'}}]}

In this example, we retrieve the AMI id through my module's getami() function, and the SubnetId using my helper function get_id_from_nametag() which supposes that you gave a tag, examplesubnet here, to the subnet you intend to spawn this instance in.

The create_instances function returns an array of instances objects that are immediately ready for usage, and as a matter of fact, in this example, we show the instance id and terminate it.

Hope that quick hands on has been informative, do not hesitate to comment and share it!

Emile iMil Heitor --


This comment has been minimized.

Copy link

@bwhaley bwhaley commented Aug 19, 2015

Helpful intro to boto3, which has very confusing docs with not enough examples. Thanks!


This comment has been minimized.

Copy link

@ikrauchanka ikrauchanka commented Oct 10, 2015

also nice option in boto3 is DryRun=True|False


This comment has been minimized.

Copy link

@pas256 pas256 commented Oct 21, 2015

Why is tags2dict not part of boto3? Seems so ridiculously helpful.


This comment has been minimized.

Copy link

@shdobxr shdobxr commented Jan 12, 2016

@bwhaley couldn't agree more. Not enough examples. Specifically regarding security groups description.


This comment has been minimized.

Copy link

@emcenrue emcenrue commented Mar 16, 2016

fantastic tutorial, you would think with all that money, AWS would actually make more tutorials, but i guess obfuscating information when you're on top generates more overhead for everyone which increases vendor lockin, c'est la vie...


This comment has been minimized.

Copy link

@j2kun j2kun commented May 11, 2016

I can create and terminate instances just fine, but how do I actually launch a process on an instance?


This comment has been minimized.

Copy link

@sukenneth sukenneth commented May 20, 2016


I am new to boto3, thanks for your great information.
I have one question where I could find the original document such as the method of ec2.volumes or ec2.Volume ?

Appreciate your answer, thanks!


This comment has been minimized.

Copy link

@blather blather commented Aug 31, 2016

This is top notch.


This comment has been minimized.

Copy link

@RvKmR-WaGh RvKmR-WaGh commented Sep 8, 2016

how to get scheduled events of an instance ?


This comment has been minimized.

Copy link

@ZhiyuSun ZhiyuSun commented Oct 17, 2016

It helps me a lot.Thanks!


This comment has been minimized.

Copy link

@Rob-B1 Rob-B1 commented Nov 29, 2016

Yeah this is good. Just using python instead of ipython so no tabbing the methods.


This comment has been minimized.

Copy link

@Nmishin Nmishin commented Jan 26, 2017

Seems to i.start/i.stop methods don't work:
In [11]: i.start
Out[11]: <bound method ResourceFactory._create_action..do_action of ec2.Instance(id='i-something')>
Tested on two machines.


This comment has been minimized.

Copy link

@redi-mediboinap redi-mediboinap commented Mar 3, 2017

Is there anyway to pass a tag / name to an instance when, being created. The create_instances does not have Name / Tag option.


This comment has been minimized.

Copy link

@eyekelly eyekelly commented Oct 25, 2017

This is really useful, thanks so much.


This comment has been minimized.

Copy link

@ms4720 ms4720 commented May 3, 2018

A very handy, bookmarked, resource. Thanks


This comment has been minimized.

Copy link

@dhillonfarms dhillonfarms commented Aug 14, 2018

Awesome resource. Thanks for the compilation and examples.


This comment has been minimized.

Copy link

@peacengell peacengell commented Aug 24, 2018

nice resources thanks man.

Great tips and tricks


This comment has been minimized.

Copy link

@Ibrahimous Ibrahimous commented Oct 11, 2018

I wondered how to use an existing Keypair, and boto3 doc was nastily confusing. A thousand thanks !


This comment has been minimized.

Copy link

@MichalTaratuta MichalTaratuta commented Oct 11, 2018

I have been looking for something like this for some time now, thanks 👍


This comment has been minimized.

Copy link

@vennemp vennemp commented Jan 21, 2019

You do a better job of documentation than Amazon themselves. They should be ashamed of their boto3 resources.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment