Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Download data from Ruby on Rails production database to development server using Replicate
# OVERVIEW:
# Replicate [1] is a Gem that lets you dump and load relational objects between
# Ruby/Ruby on Rails environments, e.g. dump data from your production database
# and load it in your development database.
# The examples given in the README [2] are of shell scripts being used to do
# this. I wanted to use a Capistrano task because it keeps everything related to
# the production site in one place (particularly the config).
# Note: This is tested on a relatively small data set yet it still took about 20
# seconds to run - YMMV!
# [1]: https://github.com/rtomayko/replicate
# [2]: https://github.com/rtomayko/replicate/blob/master/README.md
# SETUP:
# - Add the code below to your config/deploy.rb file
# - Alter the "User.all; Project.all" line to list all the data you want dumped
# by default
# USAGE:
# Download default data from the production site:
# cap db:replicate
# Download specified data only:
# cap db:replicate DATA='User.find(1); Page.find(1)'
# Dependant records are automatically downloaded by Replicate, e.g. this
# downloads both the Project records and the User record:
# cap db:replicate DATA='User.find_by_name("dave").projects'
# To empty the local database first, set CLEAR=1:
# cap db:replicate CLEAR=1
# cap db:replicate CLEAR=1 DATA='User.all'
# See the full usage instructions:
# cap -e db:replicate
# POSSIBLE FUTURE IMPROVEMENTS:
# - Steam the data directly from the dumper to the loader instead of holding it
# in a temporary variable
# - Automatically dump all tables instead of requiring them to be listed
# manually
# - Improve the DATA variable parsing in case there's a semi-colon that needs
# escaping
namespace :db do
desc <<DESC
Replicates data from production site database to the local database.
By default all database objects are downloaded:
cap db:replicate
Set the DATA variable to specify the objects to download:
cap db:replicate DATA='User.find(1); Page.find(1)'
Dependant records are automatically downloaded by Replicate, e.g. this downloads
both the Project records and the User record:
cap db:replicate DATA='User.find_by_name("dave").projects'
To empty the local database first, set CLEAR=1:
cap db:replicate CLEAR=1
cap db:replicate CLEAR=1 DATA='User.all'
See https://github.com/rtomayko/replicate for more details about Replicate.
DESC
task :replicate, roles: :db, only: { primary: true } do
# Download data from the production database
# NOTE: YOU NEED TO ALTER THE LINE BELOW TO LIST ALL THE OBJECTS YOU WANT TO DUMP BY DEFAULT
code = ENV["DATA"] || "User.all; Project.all"
code = code.split(";").map{|c| c.strip.shellescape }.join(" ")
data = capture("cd #{latest_release} && RAILS_ENV=production bundle exec replicate -q -r '#{latest_release}/config/environment' -d #{code}")
# Clear the local database
system "rake db:schema:load" if ENV["CLEAR"] == "1"
# Load the data into the local database
require File.dirname(__FILE__) + '/../config/environment'
require 'replicate'
Replicate::Loader.new do |loader|
loader.log_to $stdout
loader.read StringIO.new(data)
end
end
end
@ghost

This comment has been minimized.

Copy link

commented Aug 11, 2012

Totally awesome. Thanks.

@mcfiredrill

This comment has been minimized.

Copy link

commented Aug 9, 2013

Have you ever gotten this error?

/home/tony/.rvm/gems/ruby-2.0.0-p195/gems/replicate-1.5.1/lib/replicate/loader.rb:56:in `load': dump format error(0xa) (ArgumentError)
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/replicate-1.5.1/lib/replicate/loader.rb:56:in `read'
        from ./config/deploy.rb:166:in `block (3 levels) in load'
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/replicate-1.5.1/lib/replicate/emitter.rb:9:in `initialize'
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/replicate-1.5.1/lib/replicate/loader.rb:20:in `initialize'
        from ./config/deploy.rb:164:in `new'
        from ./config/deploy.rb:164:in `block (2 levels) in load'
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/capistrano-2.15.5/lib/capistrano/configuration/execution.rb:138:in `instance_eval'
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/capistrano-2.15.5/lib/capistrano/configuration/execution.rb:138:in `invoke_task_directly'
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/capistrano-2.15.5/lib/capistrano/configuration/callbacks.rb:25:in `invoke_task_directly_with_callbacks'
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/capistrano-2.15.5/lib/capistrano/configuration/execution.rb:89:in `execute_task'
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/capistrano-2.15.5/lib/capistrano/configuration/execution.rb:101:in `find_and_execute_task'
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/capistrano-2.15.5/lib/capistrano/cli/execute.rb:46:in `block in execute_requested_actions'
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/capistrano-2.15.5/lib/capistrano/cli/execute.rb:45:in `each'
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/capistrano-2.15.5/lib/capistrano/cli/execute.rb:45:in `execute_requested_actions'
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/capistrano-2.15.5/lib/capistrano/cli/help.rb:19:in `execute_requested_actions_with_help'
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/capistrano-2.15.5/lib/capistrano/cli/execute.rb:34:in `execute!'
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/capistrano-2.15.5/lib/capistrano/cli/execute.rb:14:in `execute'
        from /home/tony/.rvm/gems/ruby-2.0.0-p195/gems/capistrano-2.15.5/bin/cap:4:in `<top (required)>'
        from ./bin/cap:16:in `load'
        from ./bin/cap:16:in `<main>'
@r00takaspin

This comment has been minimized.

Copy link

commented Nov 14, 2013

have the same problem as mcfiredrill

@tcrayford

This comment has been minimized.

Copy link

commented Jan 29, 2014

@mcfiredrill/ @r00takaspin: you might want to look to see if your rails app outputs anything on stdout before starting to dump replicate info.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.