A quick rake task to reboot a heroku dyno if it's approaching your memory limit. For some reason when heroku cycles them, with our unicorn setup, we have some downtime - but when we reboot them, they don't get traffic until they are back up- so no downtime.

check_memory_at_heroku_and_reboot.rake

task :web_restart_based_on_memory => :environment do
  begin
    @client = Heroku::API.new(:api_key => ENV['HEROKU_TOKEN'])
    app_name = 'HEROKU_APP_NAME_HERE'

    dynos_status = @client.get_ps(app_name).body

    #get some recent lines that are from heroku referencing web.X 
    response = open(@client.get_logs(app_name, ps: "web", source: 'heroku', 'num' => '1500').body).readlines
    reboot_count = 0
    rebooted_dynos = []

    response.each do |line|
      if line.include?("memory_rss")
        begin
          dyno = line.split('heroku[')[1].split(']')[0]
          memory_size = line.split("memory_rss=")[1].split("MB")[0].to_i
          uptime_minutes = dynos_status.find {|i| i['process'] == dyno }['elapsed']/60.0
          puts "DEBUG: #{dyno} #{memory_size}MB #{uptime_minutes} minutes"
          if uptime_minutes > 45 and memory_size > 900 and !rebooted_dynos.include?(dyno)  #set 900MB for 2X dynos
            puts "REBOOTING #{dyno}"
            rebooted_dynos << dyno
            @client.post_ps_restart('HEROKU_APP_NAME', ps: dyno)
            reboot_count += 1
          end
          break if reboot_count == 2  #let's not reboot ALL our dynos.. (we run  more than 2)
        rescue
          next
        end
      end
    end

    api_token = 'LITABOT-TOKEN-HERE' #lita bot - notify the development room a reboot happened
    client = HipChat::Client.new(api_token, :api_version => 'v2')
    client['Development'].send('Lita Bot', "Just restarted these dynos for being over 900MB memory: #{rebooted_dynos.join(',')}") if !rebooted_dynos.empty?

  rescue => ex
    to = 'troy@SOMEEMAIL.com'
    from = 'help@SOMEEMAIL.com'
    subject = "[ProdMemoryRebooter]  - #{Time.zone.now.to_date.to_s}"
    message = "Ran into an exception, it was: #{ex} \n\n#{ex.backtrace}"
    AccountMailer.contact_mailer(to,from,subject,message).deliver
  end

end

This is meant to be run from some other app, using heroku scheduler, we are running it every 10 minutes.