Skip to content

Instantly share code, notes, and snippets.

@iMilnb
Last active October 19, 2022 09:15
Star You must be signed in to star a gist
Save iMilnb/0ff71b44026cfd7894f8 to your computer and use it in GitHub Desktop.
Programmatically manipulate AWS resources with boto3 - a quick hands on

boto3 quick hands-on

This documentation aims at being a quick-straight-to-the-point-hands-on AWS resources manipulation with boto3.

First of all, you'll need to install boto3. Installing it along with awscli is probably a good idea as

  • awscli is boto-based
  • awscli usage is really close to boto's
  • boto3 will use the same configuration files

A convenient method consists in installing them in a python virtualenv:

$ virtualenv awscli
$ . awscli/bin/activate
(awscli)$ pip install awscli boto3

From now on I will assume you already have an AWS account setup, best practices suggest that you must NOT use your master account but instead create an account which might have adequate privileges.

$ aws configure
AWS Access Key ID [None]: your_key_id
AWS Secret Access Key [None]: your_access_key
Default region name [None]: eu-west-1
Default output format [None]: json

In order switch between regions more easily, I suggest you make aliases on the awscli configuration files:

$ cat ~/.aws/credentials 
[default]
aws_access_key_id = your_key_id
aws_secret_access_key = your_access_key

[ireland]
aws_access_key_id = your_key_id
aws_secret_access_key = your_access_key

$ cat ~/.aws/config
[default]
output = json
region = eu-west-1

[profile ireland]
output=json
region=eu-west-1

You should then be able to summon awscli by region:

$ aws ec2 describe-instances --profile ireland
{
    [some json output]
}

We are now able to dig around with boto3 itself. In order to learn its usage, we'll use the fantastic ipython. First import boto3:

$ ipython3
In [1]: import boto3

Then initiate a boto session:

In [2]: s = boto3.Session(profile_name='ireland')

From there, initiate an object corresponding to the resource you want to manipulate, for instance ec2:

In [3]: ec2 = s.resource('ec2')

And witness the methods at your disposal:

In [4]: ec2.[press tab]
ec2.DhcpOptions                    ec2.create_snapshot
ec2.Image                          ec2.create_subnet
ec2.Instance                       ec2.create_tags
ec2.InternetGateway                ec2.create_volume
ec2.KeyPair                        ec2.create_vpc
ec2.NetworkAcl                     ec2.create_vpc_peering_connection
ec2.NetworkInterface               ec2.dhcp_options_sets
ec2.PlacementGroup                 ec2.disassociate_route_table
ec2.RouteTable                     ec2.images
ec2.RouteTableAssociation          ec2.import_key_pair
ec2.SecurityGroup                  ec2.instances
ec2.Snapshot                       ec2.internet_gateways
ec2.Subnet                         ec2.key_pairs
ec2.Tag                            ec2.meta
ec2.Volume                         ec2.network_acls
ec2.Vpc                            ec2.network_interfaces
ec2.VpcPeeringConnection           ec2.placement_groups
ec2.create_dhcp_options            ec2.register_image
ec2.create_instances               ec2.route_tables
ec2.create_internet_gateway        ec2.security_groups
ec2.create_key_pair                ec2.snapshots
ec2.create_network_acl             ec2.subnets
ec2.create_network_interface       ec2.volumes
ec2.create_placement_group         ec2.vpc_peering_connections
ec2.create_route_table             ec2.vpcs
ec2.create_security_group          

While those methods permit to execute operations, they also give access to objects themselves, for example:

In [5]: for i in ec2.instances.all(): print(i)
ec2.Instance(id='i-ea39b240')
ec2.Instance(id='i-54c4fafe')
ec2.Instance(id='i-0d0533a7')
ec2.Instance(id='i-44c5fbee')
ec2.Instance(id='i-9405333e')
ec2.Instance(id='i-b105331b')
ec2.Instance(id='i-04fc68ae')

Let's pick one of those:

In [6]: i = ec2.Instance(id='i-ea39b240')

And discover its methods:

In [7]: i.[press tab]
i.ami_launch_index          i.private_ip_address
i.architecture              i.product_codes
i.attach_classic_link_vpc   i.public_dns_name
i.attach_volume             i.public_ip_address
i.block_device_mappings     i.ramdisk_id
i.client_token              i.reboot
i.console_output            i.reload
i.create_image              i.report_status
i.create_tags               i.reset_attribute
i.describe_attribute        i.reset_kernel
i.detach_classic_link_vpc   i.reset_ramdisk
i.detach_volume             i.reset_source_dest_check
i.ebs_optimized             i.root_device_name
i.hypervisor                i.root_device_type
i.iam_instance_profile      i.security_groups
i.id                        i.source_dest_check
i.image                     i.spot_instance_request_id
i.image_id                  i.sriov_net_support
i.instance_id               i.start
i.instance_lifecycle        i.state
i.instance_type             i.state_reason
i.kernel_id                 i.state_transition_reason
i.key_name                  i.stop
i.key_pair                  i.subnet
i.launch_time               i.subnet_id
i.load                      i.tags
i.meta                      i.terminate
i.modify_attribute          i.unmonitor
i.monitor                   i.virtualization_type
i.monitoring                i.volumes
i.network_interfaces        i.vpc
i.password_data             i.vpc_id
i.placement                 i.wait_until_exists
i.placement_group           i.wait_until_running
i.platform                  i.wait_until_stopped
i.private_dns_name          i.wait_until_terminated

For example:

In [8]: i.hypervisor
Out[8]: 'xen'

Here's an example of the filter method that many objects use to match only certain criterias:

In [9]: filter = {'Name': 'name', 'Values' : ['debian*amd64*ebs']}
In [10]: for i in ec2.images.filter(Filters = [filter]): print(i)
ec2.Image(id='ami-61e56916')
ec2.Image(id='ami-879e4ff0')
ec2.Image(id='ami-8bf29ffc')
ec2.Image(id='ami-971a65e0')
ec2.Image(id='ami-99f39eee')
ec2.Image(id='ami-c935cbbe')
ec2.Image(id='ami-e31a6594')
ec2.Image(id='ami-e7e66a90')

Accessing services through resource is called the high level method, and not everything is yet reachable through this consistent interface. The other, low-level method is client:

In [11]: ec2c = s.client('ec2')
In [12]: ec2c.[press tab]
Display all 188 possibilities? (y or n)
ec2c.accept_vpc_peering_connection
ec2c.allocate_address
ec2c.assign_private_ip_addresses
ec2c.associate_address
ec2c.associate_dhcp_options
ec2c.associate_route_table
ec2c.attach_classic_link_vpc
ec2c.attach_internet_gateway
ec2c.attach_network_interface
ec2c.attach_volume
ec2c.attach_vpn_gateway
ec2c.authorize_security_group_egress
ec2c.authorize_security_group_ingress
ec2c.bundle_instance
ec2c.can_paginate
[...]

As a usage example, no high-level resource is available to list a region Availability Zones, but a client can achieve this:

In [13]: ec2c.describe_availability_zones()
Out[13]:
{'AvailabilityZones': [{'Messages': [],
   'RegionName': 'eu-west-1',
   'State': 'available',
   'ZoneName': 'eu-west-1a'},
  {'Messages': [],
   'RegionName': 'eu-west-1',
   'State': 'available',
   'ZoneName': 'eu-west-1b'},
  {'Messages': [],
   'RegionName': 'eu-west-1',
   'State': 'available',
   'ZoneName': 'eu-west-1c'}],
 'ResponseMetadata': {'HTTPStatusCode': 200,
  'RequestId': '46e474a5-7e77-4716-a9a1-010990d066ee'}}

Every single resource and client methods are documented in boto3's documentation.

Last but not least, I wrote a convenient little module in order to ease boto3 usage. It is available in my GitHub repository.

Its usage is pretty straightforward:

In [14]: from session import Aws
In [15]: ec2 = Aws('ireland', 'ec2')

The instanciated ec2 objects holds both resource and client methods along with helper functions and variables:

In [16]: ec2.[press tab]
ec2.change_nsrecord      ec2.getamis              ec2.profile
ec2.client               ec2.getinst              ec2.region
ec2.create_tag           ec2.gettagval            ec2.resource
ec2.dmesg                ec2.lsinstances          ec2.session
ec2.get_id_from_nametag  ec2.lsinstnames          ec2.tags2dict
ec2.getall               ec2.mktags               
ec2.getami               ec2.mkuserdata

Every helper is nicely documented:

In [17]: ec2.change_nsrecord??
[...]
    def change_nsrecord(self, action, dnsrecord):
        '''Create, delete or modify a DNS record

        :param str action: One of ``CREATE``, ``DELETE`` or ``UPSERT``
        :param dict dnsrecord: A dict describing the DNS record to change
[...]

And uses boto3 functions behind the scenes:

In [18]: ec2.dmesg('i-ea39b240').split('\n')[0]
Out[18]: u'ian.net/debian/ wheezy/main debconf-utils all 1.5.49 [55.8 kB]\r'

Let's finish this quick hands on with a very basic EC2 instance creation and termination using boto3:

In [19]: rc = ec2.resource.create_instances(
    ImageId = ec2.getami('NetBSD*64*6.1.5*'),
    MinCount = 1,
    MaxCount = 1,
    KeyName = 'mysshpemkey',
    InstanceType = 'm3.medium',
    PrivateIpAddress = '10.10.0.1',
    SubnetId = ec2.get_id_from_nametag('subnets', 'examplesubnet')
)
In [20]: print(rc[0].id)
i-b1774f1b
In [21]: rc[0].terminate()
Out[21]: 
{'ResponseMetadata': {'HTTPStatusCode': 200,
  'RequestId': '45738798-812c-4ea4-9c15-edb7ddcd9950'},
 u'TerminatingInstances': [{u'CurrentState': {u'Code': 32,
    u'Name': 'shutting-down'},
   u'InstanceId': 'i-b1774f1b',
   u'PreviousState': {u'Code': 16, u'Name': 'running'}}]}

In this example, we retrieve the AMI id through my module's getami() function, and the SubnetId using my helper function get_id_from_nametag() which supposes that you gave a tag, examplesubnet here, to the subnet you intend to spawn this instance in.

The create_instances function returns an array of instances objects that are immediately ready for usage, and as a matter of fact, in this example, we show the instance id and terminate it.

Hope that quick hands on has been informative, do not hesitate to comment and share it!

Emile iMil Heitor -- imil@NetBSD.org

@bwhaley
Copy link

bwhaley commented Aug 19, 2015

Helpful intro to boto3, which has very confusing docs with not enough examples. Thanks!

@ikrauchanka
Copy link

also nice option in boto3 is DryRun=True|False

@pas256
Copy link

pas256 commented Oct 21, 2015

Why is tags2dict not part of boto3? Seems so ridiculously helpful.

@shdobxr
Copy link

shdobxr commented Jan 12, 2016

@bwhaley couldn't agree more. Not enough examples. Specifically regarding security groups description.

@ejcer
Copy link

ejcer commented Mar 16, 2016

fantastic tutorial, you would think with all that money, AWS would actually make more tutorials, but i guess obfuscating information when you're on top generates more overhead for everyone which increases vendor lockin, c'est la vie...

@j2kun
Copy link

j2kun commented May 11, 2016

I can create and terminate instances just fine, but how do I actually launch a process on an instance?

@sukenneth
Copy link

Hello,

I am new to boto3, thanks for your great information.
I have one question where I could find the original document such as the method of ec2.volumes or ec2.Volume ?

Appreciate your answer, thanks!

@blather
Copy link

blather commented Aug 31, 2016

This is top notch.

@RvKmR-WaGh
Copy link

how to get scheduled events of an instance ?

@ZhiyuSun
Copy link

It helps me a lot.Thanks!

@Rob-B1
Copy link

Rob-B1 commented Nov 29, 2016

Yeah this is good. Just using python instead of ipython so no tabbing the methods.

@Nmishin
Copy link

Nmishin commented Jan 26, 2017

Seems to i.start/i.stop methods don't work:
In [11]: i.start
Out[11]: <bound method ResourceFactory._create_action..do_action of ec2.Instance(id='i-something')>
Tested on two machines.

@redi-mediboinap
Copy link

Is there anyway to pass a tag / name to an instance when, being created. The create_instances does not have Name / Tag option.

@eyekelly
Copy link

This is really useful, thanks so much.

@ms4720
Copy link

ms4720 commented May 3, 2018

A very handy, bookmarked, resource. Thanks

@dhillonfarms
Copy link

Awesome resource. Thanks for the compilation and examples.

@peacengell
Copy link

nice resources thanks man.

Great tips and tricks

@Ibrahimous
Copy link

I wondered how to use an existing Keypair, and boto3 doc was nastily confusing. A thousand thanks !

@MichalTaratuta
Copy link

I have been looking for something like this for some time now, thanks 👍

@vennemp
Copy link

vennemp commented Jan 21, 2019

You do a better job of documentation than Amazon themselves. They should be ashamed of their boto3 resources.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment