Create a gist now

Instantly share code, notes, and snippets.

@arches /Procfile
Last active Aug 17, 2016

What would you like to do?
A little hack to restart Heroku web dynos when they hit 1000MB of memory
# NOTE!! For this to work you need to enable the log-runtime-metrics beta feature on Heroku
# See
require 'heroku-api'
require 'open-uri'
require 'pony'
require 'keen'
STDOUT.sync = true
app_name = ENV['MONITOR_APP_NAME']
puts "[monitor] Starting to monitor #{app_name}"
# you can delete/replace all this stats stuff if you aren't using Keen
class SafeStats
attr_accessor :client
def initialize
self.client = => ENV['KEEN_PROJECT_ID'], :write_key => ENV['KEEN_WRITE_KEY'], :read_key => ENV['KEEN_READ_KEY'])
def method_missing(name, *args, &blk)
client.send(name, *args, &blk)
rescue Keen::HttpError => ex
puts "[monitor] Keen::HttpError during Stats #{name}(#{args.join(", ")})"
puts "[monitor] #{ex.message}"
heroku = => API_KEY)
# make sure this range will cover your web dynos.
dynos = (1..20).to_a
previous_timestamps =
restarts =
while true
puts "[monitor] Fetching log"
response = open(heroku.get_logs(app_name, :ps => "web", 'num' => '1000').body){|l| l.include? "memory_total"}
rescue => ex
puts "[monitor] Unable to fetch log (#{ex.message})"
if response
dynos.each do |i|
restarts[i] ||=
dyno_response = response.detect{|l| l.include? "[web.#{i}]"}
next unless dyno_response
size = dyno_response.match(/val=(\d*)/)[1]
timestamp = dyno_response.match(/^(\d*)-(\d*)-(\d*)T(\d*):(\d*):(\d*)/)
unless previous_timestamps[i].to_s == timestamp.to_s
puts "[monitor] web.#{i} #{timestamp} dyno-size=#{size}MB"
STATS.publish("monitor", {type: "dyno-size", process: "web.#{i}", timestamp: timestamp, size: size.to_i})
if size.to_i > 1000 and ( - restarts[i]) > 600
restarts[i] =
puts "RESTARTING WEB.#{i}" if size.to_i > 1000
puts heroku.post_ps_restart(app_name, "ps" => "web.#{i}")
STATS.publish("monitor", {type: "dyno-restart", reason: "memory-overflow", process: "web.#{i}", timestamp: timestamp})
previous_timestamps[i] = timestamp
sleep 10
rescue SignalException => ex
# SIGTERM is normal, we don't need an email about it
raise ex unless ex.signm == 'SIGTERM'
rescue Exception => ex
Pony.options = {
:via => :smtp,
:via_options => {
:address => '',
:port => '587',
:domain => '',
:user_name => ENV['SENDGRID_USERNAME'],
:password => ENV['SENDGRID_PASSWORD'],
:authentication => :plain,
:enable_starttls_auto => true
m = Pony.mail(to: ENV['MONITOR_ALERT_EMAIL'], from: ENV['MONITOR_ALERT_EMAIL'], subject: "[monitor] fatal exception occurred", body: "#{ex.message}\n\n#{ex.backtrace.join("\n")}")
puts m.inspect.to_s
monitor: bundle exec ruby monitor.rb

arches commented Aug 8, 2013

Not a gorgeous implementation, but it works :) Hopefully it'll get nicer over time (eg, use the new Heroku API instead of legacy, maybe gem it, etc)


arches commented Aug 8, 2013

This could be modified to work without log-runtime-metrics by looking for R14 errors instead of directly at dyno size.

Hey, is there an updated version of this or other solutions that solve the issue?

Hey, is there an updated version of this or other solutions that solve the issue?

+1 I'm also interested if there's another solution--or if you used @arches solution here--to do this, @alexkravets ... Or @arches, have you built anything else--does this still work well, two years later?

(I have a sneaking suspicion that Heroku is letting their dynos deteriorate faster these days, so this'll be a more wanted thing increasingly. At least that's my anecdotal experience, with no hard evidence to back it up.)

UPDATE: Doh! Just found this: I assume that's your latest thinking on the subject Chris. Thanks and never mind.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment