This post will provide some guidance for configuration of unicorn-worker-killer for the Heroku environment.
Set memory_limit_max to (memory/worker_threads)*0.8
Set memory_limit_min to (memory/worker_threads)*0.75
Enable MaxRequests worker killing, with values equal to the number of requests each thread handles every 2-4 hours.
Example for a very busy site:
Set max_requests_min to 10000
Set max_requests_max to 20000
Heroku recommends Unicorn. When a rails app is configured to use it, unicorn-worker-killer is a gem that adds functionality to periodically kill the worker threads serving content for your application.
In order to return memory to the system and improve overall performance for you application.
Ruby never frees memory back to the operating system. Ruby allocates memory as needed, then manages that internally. When it runs out of memory to use, it performs garbage collection to free structures associated with unused objects, and then if it still needs more it allocates more from the OS.
Each allocation is 1.8 times the size of the previous one [reference]. This is done to reduce the number of malloc calls to the OS, because they're expensive. Ruby performs Garbage Collection (GC) using an algorithm called mark and sweep. This has evolved through Ruby 1.8 to 2.1. [reference, see Ruby source, reference].
Since memory is never returned to the operating system, the amount of memory held by the worker thread is a "high water" mark of the largest amount required by any single request. Memory use can be increased dramatically by serving a single web page, as is well described in this post:
"This second type of trigger is a very common cause of large processes. Imagine you have some code that queries a database with a query that pulls a large number of records. Maybe it does something cool, like pulling two sets of records, and then uses Ruby's set facilities to get a union of the two sets. It's all very slick, and works just fine. But then you notice that when the code runs, your process size immediately jumps by many megabytes, and it never goes down. What 's happened is that your queries created a very large number of temporary objects, and they exceeded the available space in the Ruby heap, so a new heap allocation was performed."
Even if you only rarely have requests that cause high memory demand (say once an hour), eventually each of your worker threads will have handled one, and memory held by the threads will be pinned at that high mark forever.
Enter unicorn-worker-killer. This gem kills worker threads if their memory has grown above a certain threshhold, or after processing a certain number of requests. To use unicorn-worker-killer, add the 'unicorn-worker-killer' gem to your gemfile and set up your configuration to use it.
Unicorn-worker-killer will only kill a worker thread after it's done processing a request, so it will not affect the serving of requests.
While running an application on Heroku, memory limits are applied which vary according to the size of the dyno ). If your application exceeds these limits, performance of your application will be impacted, and R14 messages will be sent as described here.
At first glance, it might seem that the memory limits should be set so that the worker threads consume all available memory, i.e. divide the total memory by the number of worker threads. However, memory is also consumed by other parts of your application other than the worker threads, and this can eventually result in exceeding your memory allocation when all worker threads get near the unicorn-worker-killer threshold, yet before they trigger it and are replaced by new worker threads.
A more sensible configuration is to replace worker threads when they reach 75-80% of their share of memory. Here’s an example configuration (contents of config.ru file):
# This file is used by Rack-based servers to start the application.
puts 'configuring from config.ru'
if ENV['RAILS_ENV'] == 'production'
num_workers = (ENV["WEB_CONCURRENCY"] || 3).to_i
puts 'Setting up worker killer'
require 'unicorn/worker_killer'
# Set worker kill memory limits
dyno_memory_limit = (ENV['HEROKU_DYNO_MEMORY_LIMIT_MB'] || 2048).to_i * (1024 ** 2)
mem_per_worker = dyno_memory_limit.to_f / num_workers
kill_mem_min = (0.75 * mem_per_worker).round
kill_mem_max = (0.80 * mem_per_worker).round
use Unicorn::WorkerKiller::Oom, kill_mem_min, kill_mem_max
# Set worker kill request number limits
kill_requests_min = 1000
kill_requests_max = 1000
use Unicorn::WorkerKiller::MaxRequests, kill_requests_min, kill_requests_max
end
require ::File.expand_path('../config/environment', __FILE__)
run Rails.applicationand for completeness, here's the config/unicorn.rb file from the same app:
# config/unicorn.rb
puts 'configuring from unicorn.rb'
num_workers = (ENV["WEB_CONCURRENCY"] || 3).to_i
worker_processes num_workers
timeout 30
preload_app true
before_fork do |server, worker|
Signal.trap 'TERM' do
puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
Process.kill 'QUIT', Process.pid
end
defined?(ActiveRecord::Base) and
ActiveRecord::Base.connection.disconnect!
end
after_fork do |server, worker|
Signal.trap 'TERM' do
puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
end
defined?(ActiveRecord::Base) and
ActiveRecord::Base.establish_connection
end