Skip to content

Instantly share code, notes, and snippets.

@aashish
Last active May 9, 2018 19:13
Show Gist options
  • Save aashish/890c718a812c73646adda6bb5adedfa8 to your computer and use it in GitHub Desktop.
Save aashish/890c718a812c73646adda6bb5adedfa8 to your computer and use it in GitHub Desktop.
Backup database and store to AWS S3 with fog-aws

Backup database and store to AWS S3 with fog-aws

Today I had a task to take backup of database and upload to S3. Taking the backup to local machine was straight forward job. Uploading to S3 with fog-aws gem was time consuming due to lack of documentation. May be this article may help you with Uploading file to S3 with fog. Since the application is already using fog-aws gem, I preferred to use the same gem over aws-sdk.

Prerequisites

Make sure Gemfile has

gem 'fog-aws'

Code

DB backup

  config   = Rails.configuration.database_configuration[Rails.env]
  file_name = "#{config['database']}_#{Time.now.strftime('%d_%m_%Y_%H_%M_%S')}.dump"
  path = "#{Rails.root}/#{file_name}"
  `PGPASSWORD=#{config['password']} pg_dump  -Fc --no-acl --no-owner --host localhost --username=#{config['username']} #{config['database']} > #{path}`

Upload to S3

  require 'fog/aws'
  connection = Fog::Storage.new(
    provider: 'AWS',
    aws_access_key_id: 'xxxxxxxxxxxxxxxxxx',
    aws_secret_access_key: 'xxxxxxxxxxxxxxxxxxxxxxxxx'
  )

  bucket = connection.directories.get("bucket_name")
  s3_file = bucket.files.create(:key => "backups/#{Rails.env}/#{file_name}",  :body => File.open(path), :public => true)
  puts "Done!! Created backup file: #{file_name} on S3 at Url: #{s3_file.public_url}"
  `rm #{path}`
  puts "Removed local copy of dump."

Console example

  require 'fog/aws'
  connection = Fog::Storage.new(
    provider: 'AWS',
    aws_access_key_id: 'xxxxxxxxxxxxxxxxxx',
    aws_secret_access_key: 'xxxxxxxxxxxxxxxxxxxxxxxxx'
  )

  x = connection.directories.get("bucket_name", prefix: "backups/production").files
  <Fog::Storage::AWS::Files
    common_prefixes=[],
    delimiter=nil,
    directory=    <Fog::Storage::AWS::Directory
      key="bucket_name",
      creation_date=nil,
      location="us-east-1"
    >,
    is_truncated=false,
    marker=nil,
    max_keys=1000,
    prefix="backups/production"
    [
                  <Fog::Storage::AWS::File
        key="backups/production/db_name_production_09_05_2018_23_39_29.dump",
        cache_control=nil,
        content_disposition=nil,
        content_encoding=nil,
        content_length=0,
        content_md5=nil,
        content_type=nil,
        etag="d41d8cd98f00b204e9800998ecf8427e",
        expires=nil,
        last_modified=2018-05-09 18:09:34 UTC,
        metadata={},
        owner={:display_name=>"bojan", :id=>"8ab40532aea07641bb168370a84258a84445e4b803c69dd892586d0571409e7d"},
        storage_class="STANDARD",
        encryption=nil,
        encryption_key=nil,
        version=nil,
        kms_key_id=nil
      >    
    ]
  > 

References

https://stackoverflow.com/questions/15955991/how-to-list-all-files-in-an-s3-folder-using-fog-in-ruby/15955992#15955992

https://stackoverflow.com/questions/12288125/how-to-create-an-s3-object-in-a-bucket-with-one-fog-call

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment