Created

Embed URL

HTTPS clone URL

SSH clone URL

You can clone with HTTPS or SSH.

Download Gist

A script to query the Amazon Web Services (S3/EC2/etc) usage reports programmatically.

View aws_usage.py
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129
#!/usr/bin/env python
 
"""
A script to query the Amazon Web Services usage reports programmatically.
Ideally this wouldn't exist, and Amazon would provide an API we can use
instead, but hey - that's life.
Basically takes your AWS account username and password, logs into the
website as you, and grabs the data out. Always gets the 'All Usage Types'
report for the specified service.
Requirements:
* Mechanize: http://wwwsearch.sourceforge.net/mechanize/
You can install this via pip/easy_install
Run with -h to see the available options.
"""
 
import os
import sys
from datetime import date
import time
 
import mechanize
 
FORMATS = ('xml', 'csv')
PERIODS = ('hours', 'days', 'months')
SERVICES = ('AmazonS3', 'AmazonEC2', 'AmazonCloudFront', 'AmazonSimpleDB', 'AWSQueueService', 'IngestionService', 'AmazonVPC',)
 
FORM_URL = "https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=usage-report"
 
def get_report(service, date_from, date_to, username, password, format='csv', period='days', debug=False):
br = mechanize.Browser()
br.set_handle_robots(False)
 
if debug:
# Log information about HTTP redirects and Refreshes.
br.set_debug_redirects(True)
# Log HTTP response bodies (ie. the HTML, most of the time).
br.set_debug_responses(True)
# Print HTTP headers.
br.set_debug_http(True)
br.addheaders = [
# the login process 404s if you leave Python's UA string
('User-Agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1) Gecko/20090701 Ubuntu/9.04 (jaunty) Firefox/3.5'),
('Accept', 'text/html, application/xml, */*'),
]
# login
print >>sys.stderr, "Logging in..."
try:
resp = br.open(FORM_URL)
br.select_form(name="signIn")
br["email"] = username
br["password"] = password
resp = br.submit() # submit current form
except Exception, e:
print >>sys.stderr, "Error logging in to AWS"
raise
# service selector
print >>sys.stderr, "Selecting service %s..." % service
br.select_form(name="usageReportForm")
br["productCode"] = [service]
resp = br.submit()
# report selector
print >>sys.stderr, "Building report..."
br.select_form(name="usageReportForm")
br["timePeriod"] = ["Custom date range"]
br["startYear"] = [str(date_from.year)]
br["startMonth"] = [str(date_from.month)]
br["startDay"] = [str(date_from.day)]
br["endYear"] = [str(date_to.year)]
br["endMonth"] = [str(date_to.month)]
br["endDay"] = [str(date_to.day)]
br["periodType"] = [period]
resp = br.submit("download-usage-report-%s" % format)
return resp.read()
if __name__ == "__main__":
from optparse import OptionParser
USAGE = (
"Usage: %prog [options] -s SERVICE DATE_FROM DATE_TO\n\n"
"DATE_FROM and DATE_TO should be in YYYY-MM-DD format (eg. 2009-01-31)\n"
"Username and Password can also be specified via AWS_USERNAME and AWS_PASSWORD environment variables.\n"
"\n"
"Available Services: " + ', '.join(SERVICES)
)
parser = OptionParser(usage=USAGE)
parser.add_option('-s', '--service', dest="service", type="choice", choices=SERVICES, help="The AWS service to query")
parser.add_option('-p', '--period', dest="period", type="choice", choices=PERIODS, default='days', metavar="PERIOD", help="Period of report entries")
parser.add_option('-f', '--format', dest="format", type="choice", choices=FORMATS, default='csv', metavar="FORMAT", help="Format of report")
parser.add_option('-U', '--username', dest="username", metavar="USERNAME", help="Email address for your AWS account")
parser.add_option('-P', '--password', dest="password", metavar="PASSWORD")
parser.add_option('-d', '--debug', action="store_true", dest="debug", default=False)
opts, args = parser.parse_args()
if len(args) < 2:
parser.error("Missing date range")
date_range = [date(*time.strptime(args[i], '%Y-%m-%d')[0:3]) for i in range(2)]
if date_range[1] < date_range[0]:
parser.error("End date < start date")
if not opts.service:
parser.error("Specify a service to query!")
if not opts.username and not os.environ.get('AWS_USERNAME'):
parser.error("Must specify username option or set AWS_USERNAME")
if not opts.password and not os.environ.get('AWS_PASSWORD'):
parser.error("Must specify password option or set AWS_PASSWORD")
kwopts = {
'service': opts.service,
'date_from': date_range[0],
'date_to': date_range[1],
'format': opts.format,
'period': opts.period,
'username': opts.username or os.environ.get('AWS_USERNAME'),
'password': opts.password or os.environ.get('AWS_PASSWORD'),
'debug': opts.debug,
}
print get_report(**kwopts)
Owner
rcoup commented

Simplest usage:

python check_aws_usage.py AmazonS3 2010-05-01 2010-05-31 -U aws@example.com -P mypassword

Any comments are welcome here, or drop me an email at robert@coup.net.nz

maxpar commented

Hi,
I tried using your script, but I get this error:

Traceback (most recent call last):
File "log-amazon.py", line 27, in
import mechanize
ImportError: No module named mechanize

why ?

thekev commented
maxpar commented

Ok,
I installed Mechanize module, insert: python log-amazon.py AmazonCloudFront 2012-05-01 2012-05-06 -U pippo@pippo.com -P pippo

I get this error:

Traceback (most recent call last):
File "log-amazon.py", line 173, in
date_range = [date(*time.strptime(args[i], '%Y-%m-%d')[0:3]) for i in range(2)]
File "/usr/lib/python2.6/_strptime.py", line 454, in _strptime_time
return _strptime(data_string, format)[0]
File "/usr/lib/python2.6/_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data 'AmazonCloudFront' does not match format '%Y-%m-%d'

Owner
rcoup commented

@maxpar, --help is your friend.

I suspect you want python log-amazon.py -s AmazonCloudFront 2012-05-01 2012-05-06 -U pippo@pippo.com -P pippo

maxpar commented

yeaaaa it works! Thank you.

Thanks for the script, works great!

is it posible to log in with access key, secret key and amazon id? so we dont need to use aws credential login? many thanks.

any idea, why i got this error?

Error logging in to AWS
Traceback (most recent call last):
  File "aws_usage.py", line 129, in <module>
    print get_report(**kwopts)
  File "aws_usage.py", line 56, in get_report
    br.select_form(name="signIn")
  File "build/bdist.macosx-10.8-intel/egg/mechanize/_mechanize.py", line 499, in select_form
  File "build/bdist.macosx-10.8-intel/egg/mechanize/_html.py", line 544, in __getattr__
  File "build/bdist.macosx-10.8-intel/egg/mechanize/_html.py", line 557, in forms
  File "build/bdist.macosx-10.8-intel/egg/mechanize/_html.py", line 237, in forms
  File "build/bdist.macosx-10.8-intel/egg/mechanize/_form.py", line 844, in ParseResponseEx
  File "build/bdist.macosx-10.8-intel/egg/mechanize/_form.py", line 981, in _ParseFileEx
  File "build/bdist.macosx-10.8-intel/egg/mechanize/_form.py", line 760, in feed
mechanize._form.ParseError: unexpected '\\' char in declaration

This was working great for me until 4/17. I'm now running into the same issue as jackbit. Anyone have any ideas?

And as of 4/29 it's working again... I didn't change a thing. :-/

Bouke commented

Running into a similar issue as jackbit. I'm able to signin, but it fails when trying to process the usage report form:

File "aws_usage.py", line 115, in get_report
    br.select_form(name="usageReportForm")
  File "/mechanize/_mechanize.py", line 499, in select_form
    global_form = self._factory.global_form
  File "/mechanize/_html.py", line 544, in __getattr__
    self.forms()
  File "/mechanize/_html.py", line 557, in forms
    self._forms_factory.forms())
  File "/mechanize/_html.py", line 237, in forms
    _urlunparse=_rfc3986.urlunsplit,
  File "/mechanize/_form.py", line 844, in ParseResponseEx
    _urlunparse=_urlunparse,
  File "/mechanize/_form.py", line 981, in _ParseFileEx
    fp.feed(data)
  File "/mechanize/_form.py", line 760, in feed
    raise ParseError(exc)
mechanize._form.ParseError: unexpected '\\' char in declaration

When debugging I see the last response (that should contain the usage report form), this is included in the body:

<div id="message_warning" class="message warning">
    <span></span>
    <h6>Important Message!</h6>
    <p>

            To better protect your account, please re-enter your password and then enter the characters as they are shown in the image below.<br />

    </p>
</div>

So the script probably fails to login, or loses its session along the way.

Bouke commented

In my fork I've updated the login procedure to catch login failures. It will also prompt for captcha inputs. The code is not very nice, but it works for me. See: https://gist.github.com/Bouke/5719905

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.