Unicorn.rb for keen.io configuration

unicorn.rb

worker_processes 3 # amount of unicorn workers to spin up
timeout 30         # restarts workers that hang for 30 seconds

preload_app true

GC.respond_to?(:copy_on_write_friendly=) and
  GC.copy_on_write_friendly = true

before_fork do |server,worker|
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!
end

after_fork do |server,worker|
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
    
  if defined?(EventMachine)
    unless EventMachine.reactor_running? && EventMachine.reactor_thread.alive?
      if EventMachine.reactor_running?
        EventMachine.stop_event_loop
        EventMachine.release_machine
        EventMachine.instance_variable_set("@reactor_running",false)
      end
      Thread.new { EventMachine.run }
    end
  end

  Signal.trap("INT") { EventMachine.stop }
  Signal.trap("TERM") { EventMachine.stop }
end

I'm using this on Heroku and occasionally seeing this error: An EventMachine loop must be running to use publish_async calls.

Have you seen anything like this? I assume that for some reason the EventMachine crashed but the dyno remained alive. Do you have any suggestions for how to handle that case? Could I restart the EventMachine from inside one of my unicorn forks?

I have also seen the same error. Did you find a solution? Also, do you see this: Error R12 (Exit timeout) -> At least one process failed to exit within 10 seconds of SIGTERM

This only started happening after I added this code.

The issues commented above were solved ? If yes, how ? Thanks in advance.

To @arches - it is possible that EM can crash while the dyno still is up (even when EM is restarted after forks). A quick way to avoid this is to ensure the loop is running before each call:

def ensure_em
  unless EventMachine.reactor_running? && EventMachine.reactor_thread.alive?
    Thread.new { EventMachine.run }
    sleep 1
  end
end

# put this before calls
ensure_em
Keen.publish_async(...)

To @elihuu - hard to say what that might be. It could be a stray event loop, but stop should be getting called. If something is going wrong in the event loop, you might be able to find it by registering a global error handler.

EventMachine.error_handler{ |e|
  puts "Error raised during event loop: #{e.message}"
}

http://rubydoc.info/github/eventmachine/eventmachine/EventMachine.error_handler

Above configuration works great for Unicorn + EventMachine, but every time either new deploy or restart heroku process, it prints out a lot of errors in the log https://gist.github.com/samnang/8536882bc16eebfdcc29

Same issue with Heroku here.

Same thing here with Heroku + Rails

WARNING: Detected 2 Thread(s) started in app boot:
2015-03-26T00:07:54.952008+00:00 app[web.1]: [3] ! #<Thread:0x007f020a082d08 sleep> - /app/vendor/bundle/ruby/2.1.0/gems/newrelic_rpm-3.9.9.275/lib/new_relic/agent/event_loop.rb:118:in select' 2015-03-26T00:07:54.952030+00:00 app[web.1]: [3] ! #<Thread:0x007f020ac92cb0 sleep> - /app/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.7/lib/eventmachine.rb:187:inrun_machine'

issue still happens