This guide is for Rails 5.1 or lower. Starting in 5.2, all these issues have been made irrelevant.
In general, Active Record will automagically manage database connections in an efficient manner. See Configuring a Database and Database pooling for information about the basics of optimally configuring your application to use available resources.
Sometimes, you will need to manage Active Record connections and/or connection pools yourself. These situations are:
- Configuring a multi-process web server
- Spawning processes
In previous version of rails, you needed to think about managing connections when spawning threads, but this is no longer the case (I'm not sure in which version this was changed).
Each process in a multi-process web server needs its own database connection pool. If the server is also multi-threaded, the threads within each process will automatically share that processes' connection pool.
Here is how to configure a server to set up its pools while booting your app:
Before the server forks, you need to disconnect the pool so that the forked processes don't use the same connections as the parent process. Here's how to do that in a puma config:
before_fork do
ActiveRecord::Base.connection_pool.disconnect!
end
After a process forks and has no connections, it needs to establish connections. Here's how to do that in a puma config:
c.on_worker_boot do
ActiveRecord::Base.establish_connection
end
When spawning processes, you must do a small amount of manual database pool management.
Let's say you you have a script that you are going to run as a worker using rails runner
,
my_worker.rb. In this script, you are going to start two long-running processes.
Here's how you would go about doing that:
ActiveRecord::Base.connection_pool.disconnect!
thing1_pid = Process.fork do
ActiveRecord::Base.establish_connection
Thing1.new.run
end
thing2_pid = Process.fork do
ActiveRecord::Base.establish_connection
Thing2.new.run
end
# optional: If you want to do other DB operations in the parent thread,
# you will need to now reconnect like this:
ActiveRecord::Base.establish_connection
Process.wait thing1_pid
Process.wait thing2_pid
Note that each of the forked processes will have their own connection pool with
the number of connections configured in your app. So if database.yml specified a
connection pool of 5, then running rails runner my_worker.rb
will use up to 10
connections.
Now, one might imagine that, in the above example (which is exemplary of typical code), because...
- A completely new pool of connections is established in each child process.
- No DB operations are done in the children before the child pools are established.
- No DB operations are done in the parent before the child pools are established.
...then maybe the disconnect in the parent isn't necessary, since no resources are attempted to be shared between processes at the same time?
However, this is not the case. There is no mechanism in Rails for "forgetting" the old pool without destroying it. The parent pool must always be disconnected in the parent process first.
Rails will automatically allow threads to share connections from a connection pool. As long as your app is configured to have at least as many connections as there are threads running at the same time, you won't have to worry about managing connections.
You can experiment with this behavior in the console:
500.times{ Thread.new{print User.count}.join }
# succeeds
500.times{ Thread.new{print User.count; sleep 1} }
# after a few successful threads, raises "could not obtain a connection from the pool"
- http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/AbstractAdapter.html
- http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html
- https://github.com/puma/puma-heroku/blob/master/lib/puma/plugin/heroku.rb
- https://github.com/grosser/parallel#activerecord
- https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
- https://devcenter.heroku.com/articles/concurrency-and-database-connections#threaded-servers
- https://tenderlovemaking.com/2011/10/20/connection-management-in-activerecord.html
The main thing to look at is this: rails/rails#29807
In particular this comment: rails/rails#29807 (comment)
"I believe I've eliminated the need for all of this guidance: it should finally all Just Work without anyone needing to think about it."
no
probably not
maybe you found a bug. maybe you could try making a minimal rails app which reproduces the problem
hm, interesting. seems unlikely. are you able to experiment?
if you share your code here maybe i'll be able to recommend something
oh okay - seems like all good then? what a wild ride 🎢 😅