Skip to content

Instantly share code, notes, and snippets.

@sathishmanohar
Forked from andynu/examples.md
Created February 2, 2024 17:19
Show Gist options
  • Save sathishmanohar/30841e719852f9741b022c02fa213c01 to your computer and use it in GitHub Desktop.
Save sathishmanohar/30841e719852f9741b022c02fa213c01 to your computer and use it in GitHub Desktop.
Assorted rake tasks for orienting and evaluating the use of a rails app.

Overview:

rails stats:controllers  # Show basic controller usage stats
rails stats:models       # Show basic model usage stats
rails stats:trends       # show frequency trends for a date column; MODEL=x COLUMN=y
rails stats:unroutable   # Find routes that will raise a routing error when requested
rails stats:users        # Show basic details of users

bin/rails stats:controllers # Show basic controller usage stats

parses the logs, so you need a log/production.log file.

+------------------------------+----------+------------+---------------------------+------------+
| Action                       | Requests | Earliest   | Latest                    | Latest_rel |
+------------------------------+----------+------------+---------------------------+------------+
| MyController#create          | 2        | 2023-11-28 | 2024-01-29 11:34:58 -0500 | 3 days     |
| MyController#destroy         | No Log   |            |                           |            |
| MyController#edit            | 20       | 2023-07-12 | 2024-01-29 11:57:35 -0500 | 3 days     |
| MyController#generate        | 13       | 2023-09-06 | 2024-01-29 11:34:53 -0500 | 3 days     |

bin/rails stats:models - Show basic model usage stats

Model Stats:
+-------------------------------+--------+------------+------------+----------------+
| Model                         | Count  | Earliest   | Latest     | Latest_rel     |
+-------------------------------+--------+------------+------------+----------------+
| MyModelA                      | 688    | 2021-08-26 | 2024-01-31 | 1 day          |
| MyModelB                      | 9835   | 2019-12-03 | 2024-02-01 | about 4 hours  |
...


Life Stats: over 6 years (2017-2024)

Latest record (MyModelC): 2024-02-01 - 1 minute ago

rails stats:trends # show frequency trends for a date column; MODEL=x COLUMN=y

This one especially would go poorly for a huge database.

❯ bin/rails stats:trends MODEL=CayuseUsername COLUMN=created_at
+---------------------------+---+---+---+---+---+---+---+------+------+------+----+------+----+-------+-------+----+----+------+------+----+----+----+----+----+
| hour                      | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8    | 9    | 10   | 11 | 12   | 13 | 14    | 15    | 16 | 17 | 18   | 19   | 20 | 21 | 22 | 23 | 24 |
+---------------------------+---+---+---+---+---+---+---+------+------+------+----+------+----+-------+-------+----+----+------+------+----+----+----+----+----+
| hour                      | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8    | 9    | 10   | 11 | 12   | 13 | 14    | 15    | 16 | 17 | 18   | 19   | 20 | 21 | 22 | 23 | 24 |
+---------------------------+---+---+---+---+---+---+---+------+------+------+----+------+----+-------+-------+----+----+------+------+----+----+----+----+----+
| counts                    | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 3    | 61   | 1    | 0  | 69   | 0  | 1439  | 1315  | 0  | 0  | 106  | 105  | 0  | 0  | 0  | 0  | 0  |
| pct (total=3099)          | - | - | - | - | - | - | - | 0.10 | 1.97 | 0.03 | -  | 2.23 | -  | 46.43 | 42.43 | -  | -  | 3.42 | 3.39 | -  | -  | -  | -  | -  |
| pct (total_with_nil=3099) | - | - | - | - | - | - | - | 0.10 | 1.97 | 0.03 | -  | 2.23 | -  | 46.43 | 42.43 | -  | -  | 3.42 | 3.39 | -  | -  | -  | -  | -  |
+---------------------------+---+---+---+---+---+---+---+------+------+------+----+------+----+-------+-------+----+----+------+------+----+----+----+----+----+
+---------------------------+---+---+---+---+---+---+---+---+---+----+----+----+----+----+-------+----+----+----+----+----+----+----+----+----+------+------+------+----+----+----+----+
| day                       | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15    | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25   | 26   | 27   | 28 | 29 | 30 | 31 |
+---------------------------+---+---+---+---+---+---+---+---+---+----+----+----+----+----+-------+----+----+----+----+----+----+----+----+----+------+------+------+----+----+----+----+
| day                       | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15    | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25   | 26   | 27   | 28 | 29 | 30 | 31 |
+---------------------------+---+---+---+---+---+---+---+---+---+----+----+----+----+----+-------+----+----+----+----+----+----+----+----+----+------+------+------+----+----+----+----+
| counts                    | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0  | 0  | 0  | 0  | 0  | 2965  | 0  | 0  | 0  | 0  | 0  | 0  | 0  | 0  | 0  | 2    | 63   | 69   | 0  | 0  | 0  | 0  |
| pct (total=3099)          | - | - | - | - | - | - | - | - | - | -  | -  | -  | -  | -  | 95.68 | -  | -  | -  | -  | -  | -  | -  | -  | -  | 0.06 | 2.03 | 2.23 | -  | -  | -  | -  |
| pct (total_with_nil=3099) | - | - | - | - | - | - | - | - | - | -  | -  | -  | -  | -  | 95.68 | -  | -  | -  | -  | -  | -  | -  | -  | -  | 0.06 | 2.03 | 2.23 | -  | -  | -  | -  |
+---------------------------+---+---+---+---+---+---+---+---+---+----+----+----+----+----+-------+----+----+----+----+----+----+----+----+----+------+------+------+----+----+----+----+
+---------------------------+---+---+---+---+-------+------+---+---+---+------+----+----+
| month                     | 1 | 2 | 3 | 4 | 5     | 6    | 7 | 8 | 9 | 10   | 11 | 12 |
+---------------------------+---+---+---+---+-------+------+---+---+---+------+----+----+
| month                     | 1 | 2 | 3 | 4 | 5     | 6    | 7 | 8 | 9 | 10   | 11 | 12 |
+---------------------------+---+---+---+---+-------+------+---+---+---+------+----+----+
| counts                    | 0 | 0 | 0 | 0 | 2965  | 132  | 0 | 0 | 0 | 2    | 0  | 0  |
| pct (total=3099)          | - | - | - | - | 95.68 | 4.26 | - | - | - | 0.06 | -  | -  |
| pct (total_with_nil=3099) | - | - | - | - | 95.68 | 4.26 | - | - | - | 0.06 | -  | -  |
+---------------------------+---+---+---+---+-------+------+---+---+---+------+----+----+
+---------------------------+-------+------+
| year                      | 2022  | 2023 |
+---------------------------+-------+------+
| year                      | 2022  | 2023 |
+---------------------------+-------+------+
| counts                    | 2967  | 132  |
| pct (total=3099)          | 95.74 | 4.26 |
| pct (total_with_nil=3099) | 95.74 | 4.26 |
+---------------------------+-------+------+

rails stats:unroutable # Find routes that will raise a routing error when requested

❯ bin/rails stats:unroutable
            Prefix Verb  URI Pattern                              Controller#Action
    new_auth_users GET   /auth/admin/auth_users/new(.:format)  auth_users#new
   edit_auth_users GET   /auth/admin/auth_users/edit(.:format) auth_users#edit
                   PATCH /auth/admin/auth_users(.:format)      auth_users#update
                   PUT   /auth/admin/auth_users(.:format)      auth_users#update
          username GET   /usernames/:id(.:format)              usernames#show
5 unroutable routes

rails stats:users # Show basic details of users

Our user records usually have either a User#role or User#roles. We're also using auth_logic, so have some last-login-time audit columns. This could be easily tweaked for devise.

This example output lacks the terminal color coding in the activity section.

User count: 2

Role method: role
Roles: user

Roles => Users
  user: bill, ted

Users by Activity: Request in within week, month, year, beyond, unknown (these are terminal color coded)
year: bill, ted
# pre-reqs
# gem 'rainbow'
# gem 'terminal-table'
require 'terminal-table'
require 'action_view'
require 'action_view/helpers'
include ActionView::Helpers::DateHelper
namespace :stats do
desc 'Show basic details of users'
task :users => :environment do
role_method = if User.new.respond_to? :roles
:roles
elsif User.new.respond_to? :role
:role
end
puts "User count: #{User.count}"
puts
if role_method.present?
puts "Role method: #{role_method}"
puts "Roles: #{User.all.map(&role_method).flatten.uniq.join(', ')}"
puts
puts 'Roles => Users'
User.all.group_by(&role_method).each do |role, users|
puts " #{role}: #{users&.map(&:username)&.join(', ')}"
end
else
puts "Users: #{User.pluck(:usenrame).join(', ')}"
end
puts
colors = { week: :red, month: :yellow, year: :blue, beyond: :gray, unknown: :cyan }
puts 'Users by Activity: Request in within %s' % colors.map{|lbl, clr| Rainbow(lbl.to_s).send(clr) }.join(', ')
users_with_recency = User.order('last_request_at desc').map do |user|
delta = Time.now - user.last_request_at rescue :unknown
recency = if delta == :unknown
:unknown
elsif delta > 1.year
:beyond
elsif delta > 1.month
:year
elsif delta > 1.week
:month
else
:week
end
OpenStruct.new(username: user.username, recency: recency, color: colors[recency])
end
users_with_recency.group_by(&:recency).each_pair do |recency, users|
puts "#{recency}: #{users.map{|u| Rainbow(u.username).send(u.color) }.join(', ')}"
end
puts
end
desc 'Show basic model usage stats'
task :models => :environment do
puts 'Model Stats:'
dates = []
min_date = nil
max_date = nil
max_model = nil
table = Terminal::Table.new headings: %w[Model Count Earliest Latest Latest_rel]
Zeitwerk::Loader.eager_load_all
ApplicationRecord.descendants.each do |model|
next unless model.attribute_names.include? 'created_at'
count = model.count
earliest_date = model.minimum('created_at')
latest_date = model.maximum('created_at')
if max_date.nil? || (latest_date.present? && latest_date > max_date)
max_date = latest_date
max_model = model.name
end
if min_date.nil? || (earliest_date.present? && earliest_date < min_date)
min_date = earliest_date
end
dates.concat [earliest_date, latest_date]
# puts "%15s: %6d (%s, %s)" % [model.name, count, latest_date.to_date, time_ago_in_words(latest_date)]
table << [
model.name,
count,
earliest_date&.to_date,
latest_date&.to_date,
latest_date.nil? ? nil : time_ago_in_words(latest_date)
]
end
puts table
puts
puts "Life Stats: #{distance_of_time_in_words(min_date, max_date)} (#{min_date.year}-#{max_date.year})"
puts
puts "Latest record (#{max_model}): #{max_date.to_date} - #{time_ago_in_words(max_date)} ago"
puts
end
desc 'Show basic controller usage stats'
task :controllers => :environment do
logfiles = Dir['log/%s.log*' % Rails.env].sort
logs_gz, logs_txt = logfiles.partition{|f| Pathname.new(f).extname == '.gz' }
results = `ag Started -A 1 #{logs_txt.join(' ')}`
unless logs_gz.empty?
results << `zcat #{logs_gz.join(' ')} |ag Started -A 1`
end
Event = Struct.new(:http_method, :uri_path, :client_ip, :requested_at_str, :controller_name, :controller_action, :format) do
def requested_at
Chronic.parse(requested_at_str)
end
def action_fmt
[controller_name, controller_action].join('#')
end
end
events_raw = results.lines.each_cons(2)
events = events_raw.map{|lines|
next unless lines.first =~ /Started/ && lines.last =~ /Processing/
# Started GET "/auth/signin" for 10.9.9.182 at 2018-07-30 09:22:15 -0400 ()
# Processing by UserSessionController#new as HTML ()
started_m = /Started (?<http_method>\w+) "(?<uri_path>[^ ]*)" for (?<client_ip>\d+\.\d+\.\d+\.\d+) at (?<requested_at_str>[^ ]+ [^ ]+ [^ ]+)(?: .*)?/.match(lines.first)&.named_captures
processing_m = /Processing by (?<controller_name>[^ ]+)#(?<controller_action>\w+) as (?<format>[^ ]+)(?: .*)?/.match(lines.last)&.named_captures
next if started_m.nil?
next if processing_m.nil?
e = Event.new(*started_m.merge(processing_m).values)
# p started_m.merge(processing_m).values
# p e
# puts "==="
e
}.compact
logged_controller_action_events = events.group_by(&:action_fmt)
defined_controller_actions = []
Zeitwerk::Loader.eager_load_all
ApplicationController.descendants.each do |clazz|
methods = clazz.instance_methods - ApplicationController.instance_methods
methods.each do |method|
defined_controller_actions << OpenStruct.new(controller_name: clazz.name, controller_action: method, action_fmt: "#{clazz.name}##{method}")
end
end
all_controller_actions = (logged_controller_action_events.keys + defined_controller_actions.map(&:action_fmt)).compact.uniq.sort
table = Terminal::Table.new headings: %w[Action Requests Earliest Latest Latest_rel]
all_controller_actions.each do |controller_action|
events = logged_controller_action_events[controller_action]
if events.present?
dates = events.map(&:requested_at)
min_date = dates.min.to_date
max_date = dates.max
#puts "%d\t%s (earliest: %s, latest: %s (%s)" % [events.count, controller_action, min_date, max_date, time_ago_in_words(max_date)]
table << [controller_action, events.count, min_date, max_date, time_ago_in_words(max_date)]
else
table << [controller_action, Rainbow('No Log').gray, nil, nil, nil]
end
end
puts table
end
desc 'show frequency trends for a date column; MODEL=x COLUMN=y'
task :trends => :environment do
model_str = ENV.fetch('MODEL', nil)
col_str = ENV.fetch('COLUMN', nil)
Zeitwerk::Loader.eager_load_all
models = ApplicationRecord.descendants.index_by{|model| model.to_s}
model = models[model_str]
if model.nil?
warn "Unkown model class #{model_str}"
exit 1
end
unless col_str.in?(model.attribute_names)
warn "Unknown column name '#{col_str}' for #{model_str}"
exit 1
end
col_def = model.columns.find{|c| c.name == col_str}
unless col_def.type == :datetime
warn "Column '#{col_str}' is not a datetime. type=#{col_def.type}"
exit 1
end
dates = model.pluck(col_str)
total_with_nil = dates.count
dates.compact!
total = dates.count
# hrs
freq_table = ->(field, keys) do
counts = dates.map(&field).tally
if keys == :min_max
keys = (counts.keys.min..counts.keys.max).to_a
end
table = Terminal::Table.new headings: [field] + keys
values = counts.values
mean = values.sum(0.0) / counts.size
sum = values.sum(0.0) { |el| (el - mean)**2 }
variance = sum / (values.size - 1)
stddev = Math.sqrt(variance)
headings_row = [field]
counts_row = [:counts]
pct_row = ["pct (total=#{total})"]
pct_nil_row = ["pct (total_with_nil=#{total_with_nil})"]
keys.each do |key|
count = counts[key]
# color coded by quartiles
# color = if count.nil?
# :white
# elsif count <= stats[:q1]
# :gray
# elsif count <= stats[:q2]
# :blue
# elsif count <= stats[:q3]
# :orange
# else
# :red
# end
color = :white
# color coded by standard deviation
color = if count.nil?
:white
elsif count <= mean - stddev
:gray
elsif count <= mean + stddev
:blue
else
:red
end
headings_row << Rainbow(key).send(color)
counts_row << Rainbow(count || 0).send(color)
pct_row << if count.blank?
'-'
else
'%0.2f' % (count.to_f / total * 100)
end
pct_nil_row << if count.blank?
'-'
else
'%0.2f' % (count.to_f / total_with_nil * 100)
end
end
table << headings_row
table << :separator
table << counts_row
table << pct_row
table << pct_nil_row
puts table
end
freq_table.call(:hour, (1..24).to_a)
freq_table.call(:day, (1..31).to_a)
freq_table.call(:month, (1..12).to_a)
freq_table.call(:year, :min_max)
end
desc 'Find routes that will raise a routing error when requested'
task :unroutable => :environment do
# Written by Nate Matykiewicz from the "Ruby on Rails Link" Slack.
# https://gist.github.com/natematykiewicz/521bb5c9ec1f02c69a282bee21af1031
#
# A lot of this code was taken from how `rake routes` works
# https://github.com/rails/rails/blob/f95c0b7e96eb36bc3efc0c5beffbb9e84ea664e4/railties/lib/rails/commands/routes/routes_command.rb
require 'action_dispatch/routing/inspector'
unroutables = Rails.application.routes.routes
.map { |r| ActionDispatch::Routing::RouteWrapper.new(r) }
.reject { |r| r.internal? || r.engine? || r.path.starts_with?('/rails/') || !r.controller }
.reject do |r|
controller = "#{r.controller}_controller".classify.safe_constantize
controller && (
controller.method_defined?(r.action) ||
Rails.root.join('app/views', r.controller).glob("#{r.action}.*").any?
)
end
.map { |r| r.__getobj__ }
if unroutables.present?
inspector = ActionDispatch::Routing::RoutesInspector.new(unroutables)
formatter = if ENV['EXPANDED']
ActionDispatch::Routing::ConsoleFormatter::Expanded.new
else
ActionDispatch::Routing::ConsoleFormatter::Sheet.new
end
puts inspector.format(formatter)
end
puts "#{unroutables.size} unroutable routes"
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment