Backup database and store to AWS S3 with fog-aws

DB backup and store to S3.md

Backup database and store to AWS S3 with fog-aws

Today I had a task to take backup of database and upload to S3. Taking the backup to local machine was straight forward job. Uploading to S3 with fog-aws gem was time consuming due to lack of documentation. May be this article may help you with Uploading file to S3 with fog. Since the application is already using fog-aws gem, I preferred to use the same gem over aws-sdk.

Prerequisites

Make sure Gemfile has

gem 'fog-aws'

Code

DB backup

  config   = Rails.configuration.database_configuration[Rails.env]
  file_name = "#{config['database']}_#{Time.now.strftime('%d_%m_%Y_%H_%M_%S')}.dump"
  path = "#{Rails.root}/#{file_name}"
  `PGPASSWORD=#{config['password']} pg_dump  -Fc --no-acl --no-owner --host localhost --username=#{config['username']} #{config['database']} > #{path}`

Upload to S3

  require 'fog/aws'
  connection = Fog::Storage.new(
    provider: 'AWS',
    aws_access_key_id: 'xxxxxxxxxxxxxxxxxx',
    aws_secret_access_key: 'xxxxxxxxxxxxxxxxxxxxxxxxx'
  )

  bucket = connection.directories.get("bucket_name")
  s3_file = bucket.files.create(:key => "backups/#{Rails.env}/#{file_name}",  :body => File.open(path), :public => true)
  puts "Done!! Created backup file: #{file_name} on S3 at Url: #{s3_file.public_url}"
  `rm #{path}`
  puts "Removed local copy of dump."

Console example

  require 'fog/aws'
  connection = Fog::Storage.new(
    provider: 'AWS',
    aws_access_key_id: 'xxxxxxxxxxxxxxxxxx',
    aws_secret_access_key: 'xxxxxxxxxxxxxxxxxxxxxxxxx'
  )

  x = connection.directories.get("bucket_name", prefix: "backups/production").files
  <Fog::Storage::AWS::Files
    common_prefixes=[],
    delimiter=nil,
    directory=    <Fog::Storage::AWS::Directory
      key="bucket_name",
      creation_date=nil,
      location="us-east-1"
    >,
    is_truncated=false,
    marker=nil,
    max_keys=1000,
    prefix="backups/production"
    [
                  <Fog::Storage::AWS::File
        key="backups/production/db_name_production_09_05_2018_23_39_29.dump",
        cache_control=nil,
        content_disposition=nil,
        content_encoding=nil,
        content_length=0,
        content_md5=nil,
        content_type=nil,
        etag="d41d8cd98f00b204e9800998ecf8427e",
        expires=nil,
        last_modified=2018-05-09 18:09:34 UTC,
        metadata={},
        owner={:display_name=>"bojan", :id=>"8ab40532aea07641bb168370a84258a84445e4b803c69dd892586d0571409e7d"},
        storage_class="STANDARD",
        encryption=nil,
        encryption_key=nil,
        version=nil,
        kms_key_id=nil
      >    
    ]
  > 

References

https://stackoverflow.com/questions/15955991/how-to-list-all-files-in-an-s3-folder-using-fog-in-ruby/15955992#15955992

https://stackoverflow.com/questions/12288125/how-to-create-an-s3-object-in-a-bucket-with-one-fog-call