Skip to content

Instantly share code, notes, and snippets.

@serg-kovalev
Last active August 29, 2015 14:04
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save serg-kovalev/71af1ccd121fba7ef56e to your computer and use it in GitHub Desktop.
Save serg-kovalev/71af1ccd121fba7ef56e to your computer and use it in GitHub Desktop.
Phusion Passenger Monitor library: it allows you to automatically gracefully kill/reload passenger instances, the end user will not see any errors. See another gist that shows how to use it: https://gist.github.com/serg-kovalev/a201e89e69ee5a86ad79
# Finds bloating passengers and try to kill them gracefully.
# @example:
# PassengerMonitor.run
require 'logger'
class PassengerMonitor
# How much memory (MB) single Passenger instance can use
DEFAULT_MEMORY_LIMIT = 450
# Log file name
DEFAULT_LOG_FILE = 'passenger_monitoring.log'
# How long should we wait after graceful kill attempt, before force kill
WAIT_TIME = 10
# Number of attempts before force kill
NUMBER_OF_ATTEMPTS = 5
def self.run(params = {})
new(params).check
end
# Set up memory limit, log file and logger
def initialize(params = {})
@memory_limit = params[:memory_limit] || DEFAULT_MEMORY_LIMIT
@log_file = params[:log_file] || DEFAULT_LOG_FILE
@wait_time = params[:wait_time] || WAIT_TIME
@attempts = params[:attempts] || NUMBER_OF_ATTEMPTS
# log rotation - 30 days, 20 megabytes
@logger = Logger.new(@log_file, 30, 20971520)
@logger.level = Logger::WARN
@pids_with_time = []
end
# Check all the Passenger processes
def check
@logger.info 'Checking for bloated Passenger workers'
`passenger-memory-stats`.each_line do |line|
# next unless (line =~ /RackApp: / || line =~ /Rails: /)
next unless (line =~ /RackApp:/)
pid, memory_usage = extract_stats(line)
# If a given passenger process is bloated try to
# kill it gracefully and if it fails, force killing it
if bloated?(pid, memory_usage)
was_killed = kill(pid)
kill!(pid) if !was_killed and process_running?(pid)
end
end
@logger.info 'Finished checking for bloated Passenger workers'
kill_bad_processes
end
private
# Check if a given process is still running
def process_running?(pid)
Process.getpgid(pid) != -1
rescue Errno::ESRCH
false
end
# Wait for process to be killed
def wait(pid = nil)
@logger.info "Waiting for worker (#{pid}) to shutdown..."
sleep(@wait_time)
end
# Kill it gracefully
def kill(pid)
@pids_with_time << [pid, Time.now]
was_killed = false
# NUMBER_OF_ATTEMPTS to kill it gracefully
(1..@attempts).each do |n|
@logger.info "Attempt #{n}. Trying to kill #{pid} gracefully..."
was_killed = true unless (`passenger-config detach-process #{pid}` =~ /#{pid} detached/i).nil?
break if was_killed
wait(pid)
end
# returns true if process is detached
was_killed
end
def kill_bad_processes
@logger.info 'Start checking for processes that are still running'
# sort by time
@pids_with_time.sort_by{|e| e.last}.each do |p|
pid = p.first
run_at = p.last
if process_running?(pid)
while (run_at + 120) > Time.now
wait(pid)
end
kill!(pid) if process_running?(pid)
end
end
@pids_with_time = []
@logger.info 'Finished checking for processes that are still running'
end
# Kill it with fire
def kill!(pid)
@logger.error "Force kill: #{pid}"
Process.kill('TERM', pid)
end
# Extract pid and memory usage of a single Passenger
def extract_stats(line)
stats = line.split
return stats[0].to_i, stats[3].to_f
end
# Check if a given process is exceeding memory limit
def bloated?(pid, size)
bloated = size > @memory_limit
@logger.warn "Found bloated worker: #{pid} - #{size}MB" if bloated
bloated
end
end
#!/usr/bin/env ruby
require File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib', 'passenger_monitor'))
# Set logger to log into Rails project /log directory and start monitoring
PassengerMonitor.run(
:log_file => File.join(File.dirname(__FILE__), '..', 'log', 'passenger_monitor.log'),
:memory_limit => 400, # this is in MB
:wait_time => 10, # this in seconds (timeout before attempts)
:attempts => 5 # number of attempts
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment