A microgem to query the Amazon Web Services (S3/EC2/etc) usage reports.

README.md

Aws Usage

Download AWS usage reports, in XML, CSV or parsed CSV. Based on the Python gist A script to query the Amazon Web Services (S3/EC2/etc) usage reports programmatically..

require 'aws_usage'

# Yield rows from report
aws_usage = AwsUsage.new('my@email.com', 'mypassword')

# Get a report by line, with default options
aws_usage.report 'AmazonCloudFront' do |row|
  puts row['UsageValue']
end

# Get the full report as a csv string, with custom options
puts aws_usage.raw_report 'AmazonS3', 'csv',
  time_period: 'aws-portal-last-week',
  granularity: 'hours'

# Get the full report as a csv string
puts aws_usage.raw_report 'AmazonS3', 'xml',
  time_period: 'aws-portal-last-week',
  granularity: 'days'

Look at the method comments in the code for a full list of possible parameters and options.

Running the tests

No tests, it's just some code made for an internal tool that we've been using for a few days. Don't expect any maintenance.

Not implemented

• not possible to choose a value for the field "Usage type". It will be "All Usage Types".
• not possible to choose a value for the field "All operations". It will be "All operations".
• no re-login if the session expired

Notes

• It probably does not work on Ruby 1.8 (because of 'csv').
• No exception handling. For example, it will raise exotic errors if credentials are invalid.
• If you think it could be a good idea to maintain this gem, contact me. I'll create a repo in GitHub and publish the gem at rubygems.org. Then I'll ride my moutain bike, and hope to see tons of pull requests when I'm back.

aws_usage.gemspec

Gem::Specification.new do |s|
  s.name         = 'aws_usage'
  s.summary      = 'A microgem to query the Amazon Web Services (S3/EC2/etc) usage reports programmatically.'
  s.description  = 'Please Amazon, make an API. Web scraping makes me sad.'
  s.version      = '0.1'
  s.platform     = Gem::Platform::RUBY

  s.files        = ['aws_usage.rb']
  s.require_path = '.'

  s.author      = 'Nicolas Fouché'
  s.email       = ''
  s.homepage    = 'http://about.me/nfo'

  s.add_dependency 'mechanize', '~> 2.0'
end

aws_usage.rb

# encoding: UTF-8

require 'csv'
require 'mechanize'

class AwsUsage

  attr_reader :aws_email, :aws_password, :agent

  def initialize(aws_email, aws_password)
    @aws_email = aws_email
    @aws_password = aws_password

    @agent = Mechanize.new
    # require 'logger'
    # @agent.log = Logger.new STDERR
    @agent.user_agent_alias = 'Mac Safari' # Making sure that the AWS website does not reject us because we don't use a "normal" browser
  end

  # Download an AWS usage report in the a raw format, CSV or XML
  # NOTE: Very large usage reports may be truncated. Please check the last row of the downloaded file for warnings or error messages.
  #       If you see a message indicating the report was truncated, you can try downloading smaller reports by requesting a shorter
  #       time period or decreasing the granularity from hourly to daily or monthly. 
  # @param [String] :service 'AWSDirectConnect', 'IngestionService', 'AWSStorageGateway', 'AmazonCloudFront', 'AmazonCloudSearch',
  #   'AmazonDynamoDB', 'AmazonElastiCache', 'AmazonEC2', 'ElasticMapReduce', 'AmazonGlacier', 'AmazonRDS', 'AmazonRoute53',
  #   'AmazonSES', 'AmazonSNS', 'AWSQueueService', 'AmazonS3', 'AmazonSWF', 'AmazonSimpleDB', 'AmazonVPC'
  # @param [Hash] :options
  #   :granularity 'hours' (default), 'days', 'months'
  #   :time_period 'aws-portal-current-bill-period' (default), 'aws-portal-prev-bill-period', 'aws-portal-last-week', 'aws-portal-last-month', 'aws-portal-custom-date-range'
  #   :from a Date object, used if :time_period is set to 'aws-portal-custom-date-range'
  #   :to a Date object, used if :time_period is set to 'aws-portal-custom-date-range'
  # @yield [CSV::Row] You can access columns with the `[]()` method. e.g. row['UsageValue']. Possible column names: 
  #   'Service', 'Operation', 'UsageType', 'Resource', 'StartTime', 'EndTime', 'UsageValue'
  # @return [Array] An array of CSV::Row objects.
  def report(service, options = {})
    csv_report = raw_report(service, 'csv', options)

    # CSV columns: 'Service', 'Operation', 'UsageType', 'Resource', 'StartTime', 'EndTime', 'UsageValue'
    if block_given?
      CSV.parse(csv_report, :headers => true) do |row|
        yield row
      end
    else
      CSV.parse(csv_report, :headers => true)
    end
  end

  # Download an AWS usage report in the a raw format, CSV or XML
  # NOTE: Very large usage reports may be truncated. Please check the last row of the downloaded file for warnings or error messages.
  #       If you see a message indicating the report was truncated, you can try downloading smaller reports by requesting a shorter
  #       time period or decreasing the granularity from hourly to daily or monthly. 
  # @param [String] :service 'AWSDirectConnect', 'IngestionService', 'AWSStorageGateway', 'AmazonCloudFront', 'AmazonCloudSearch',
  #   'AmazonDynamoDB', 'AmazonElastiCache', 'AmazonEC2', 'ElasticMapReduce', 'AmazonGlacier', 'AmazonRDS', 'AmazonRoute53',
  #   'AmazonSES', 'AmazonSNS', 'AWSQueueService', 'AmazonS3', 'AmazonSWF', 'AmazonSimpleDB', 'AmazonVPC'
  # @param [String] :format 'csv' or 'xml'
  # @param [Hash] :options
  #   :granularity 'hours', 'days', 'months'
  #   :time_period 'aws-portal-current-bill-period', 'aws-portal-prev-bill-period', 'aws-portal-last-week', 'aws-portal-last-month', 'aws-portal-custom-date-range'
  #   :from a Date object, used if :time_period is set to 'aws-portal-custom-date-range'
  #   :to a Date object, used if :time_period is set to 'aws-portal-custom-date-range'
  # @return [String] A CSV or XML formatted string
  #   CSV headers: Service, Operation, UsageType, Resource, StartTime, EndTime, UsageValue
  def raw_report(service, format, options = {})
    login
    form = @main_form_page.form_with(:name => 'usageReportForm')

    # Fill in the service name and submit
    form.field_with(:name => 'productCode').value = service
    page = agent.submit(form)

    # Fill in the report options
    form = page.form_with(:name => 'usageReportForm')
    form.field_with(:name => 'timePeriod').value = options[:time_period] if options[:time_period]
    if options[:time_period] == 'aws-portal-custom-date-range'
      form.field_with(:name => 'startMonth').value = options[:from].month
      form.field_with(:name => 'startDay').value = options[:from].day
      form.field_with(:name => 'startYear').value = options[:from].year
      form.field_with(:name => 'endMonth').value = options[:to].month
      form.field_with(:name => 'endDay').value = options[:to].day
      form.field_with(:name => 'endYear').value = options[:to].year
    end
    form.field_with(:name => 'periodType').value = options[:granularity] if options[:granularity]

    # Submit to download the report
    button = form.button_with(:name => "download-usage-report-#{format}")
    page = agent.submit(form, button)

    page.body # a CSV or XML formatted string
  end

  private

  def login
    return @logged_in if @logged_in

    page = @agent.get 'https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=usage-report'
    form = page.form_with :name => 'signIn'
    form.field_with(:id => "ap_email").value = aws_email
    form.field_with(:id => "ap_password").value = aws_password
    @main_form_page = @agent.submit(form)

    @logged_in = true
  end
end