Assorted rake tasks for orienting and evaluating the use of a rails app.

examples.md

Overview:

rails stats:controllers  # Show basic controller usage stats
rails stats:models       # Show basic model usage stats
rails stats:trends       # show frequency trends for a date column; MODEL=x COLUMN=y
rails stats:unroutable   # Find routes that will raise a routing error when requested
rails stats:users        # Show basic details of users

bin/rails stats:controllers # Show basic controller usage stats

parses the logs, so you need a log/production.log file.

+------------------------------+----------+------------+---------------------------+------------+
| Action                       | Requests | Earliest   | Latest                    | Latest_rel |
+------------------------------+----------+------------+---------------------------+------------+
| MyController#create          | 2        | 2023-11-28 | 2024-01-29 11:34:58 -0500 | 3 days     |
| MyController#destroy         | No Log   |            |                           |            |
| MyController#edit            | 20       | 2023-07-12 | 2024-01-29 11:57:35 -0500 | 3 days     |
| MyController#generate        | 13       | 2023-09-06 | 2024-01-29 11:34:53 -0500 | 3 days     |

bin/rails stats:models - Show basic model usage stats

Model Stats:
+-------------------------------+--------+------------+------------+----------------+
| Model                         | Count  | Earliest   | Latest     | Latest_rel     |
+-------------------------------+--------+------------+------------+----------------+
| MyModelA                      | 688    | 2021-08-26 | 2024-01-31 | 1 day          |
| MyModelB                      | 9835   | 2019-12-03 | 2024-02-01 | about 4 hours  |
...


Life Stats: over 6 years (2017-2024)

Latest record (MyModelC): 2024-02-01 - 1 minute ago

rails stats:trends # show frequency trends for a date column; MODEL=x COLUMN=y

This one especially would go poorly for a huge database.

❯ bin/rails stats:trends MODEL=CayuseUsername COLUMN=created_at
+---------------------------+---+---+---+---+---+---+---+------+------+------+----+------+----+-------+-------+----+----+------+------+----+----+----+----+----+
| hour                      | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8    | 9    | 10   | 11 | 12   | 13 | 14    | 15    | 16 | 17 | 18   | 19   | 20 | 21 | 22 | 23 | 24 |
+---------------------------+---+---+---+---+---+---+---+------+------+------+----+------+----+-------+-------+----+----+------+------+----+----+----+----+----+
| hour                      | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8    | 9    | 10   | 11 | 12   | 13 | 14    | 15    | 16 | 17 | 18   | 19   | 20 | 21 | 22 | 23 | 24 |
+---------------------------+---+---+---+---+---+---+---+------+------+------+----+------+----+-------+-------+----+----+------+------+----+----+----+----+----+
| counts                    | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 3    | 61   | 1    | 0  | 69   | 0  | 1439  | 1315  | 0  | 0  | 106  | 105  | 0  | 0  | 0  | 0  | 0  |
| pct (total=3099)          | - | - | - | - | - | - | - | 0.10 | 1.97 | 0.03 | -  | 2.23 | -  | 46.43 | 42.43 | -  | -  | 3.42 | 3.39 | -  | -  | -  | -  | -  |
| pct (total_with_nil=3099) | - | - | - | - | - | - | - | 0.10 | 1.97 | 0.03 | -  | 2.23 | -  | 46.43 | 42.43 | -  | -  | 3.42 | 3.39 | -  | -  | -  | -  | -  |
+---------------------------+---+---+---+---+---+---+---+------+------+------+----+------+----+-------+-------+----+----+------+------+----+----+----+----+----+
+---------------------------+---+---+---+---+---+---+---+---+---+----+----+----+----+----+-------+----+----+----+----+----+----+----+----+----+------+------+------+----+----+----+----+
| day                       | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15    | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25   | 26   | 27   | 28 | 29 | 30 | 31 |
+---------------------------+---+---+---+---+---+---+---+---+---+----+----+----+----+----+-------+----+----+----+----+----+----+----+----+----+------+------+------+----+----+----+----+
| day                       | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15    | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25   | 26   | 27   | 28 | 29 | 30 | 31 |
+---------------------------+---+---+---+---+---+---+---+---+---+----+----+----+----+----+-------+----+----+----+----+----+----+----+----+----+------+------+------+----+----+----+----+
| counts                    | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0  | 0  | 0  | 0  | 0  | 2965  | 0  | 0  | 0  | 0  | 0  | 0  | 0  | 0  | 0  | 2    | 63   | 69   | 0  | 0  | 0  | 0  |
| pct (total=3099)          | - | - | - | - | - | - | - | - | - | -  | -  | -  | -  | -  | 95.68 | -  | -  | -  | -  | -  | -  | -  | -  | -  | 0.06 | 2.03 | 2.23 | -  | -  | -  | -  |
| pct (total_with_nil=3099) | - | - | - | - | - | - | - | - | - | -  | -  | -  | -  | -  | 95.68 | -  | -  | -  | -  | -  | -  | -  | -  | -  | 0.06 | 2.03 | 2.23 | -  | -  | -  | -  |
+---------------------------+---+---+---+---+---+---+---+---+---+----+----+----+----+----+-------+----+----+----+----+----+----+----+----+----+------+------+------+----+----+----+----+
+---------------------------+---+---+---+---+-------+------+---+---+---+------+----+----+
| month                     | 1 | 2 | 3 | 4 | 5     | 6    | 7 | 8 | 9 | 10   | 11 | 12 |
+---------------------------+---+---+---+---+-------+------+---+---+---+------+----+----+
| month                     | 1 | 2 | 3 | 4 | 5     | 6    | 7 | 8 | 9 | 10   | 11 | 12 |
+---------------------------+---+---+---+---+-------+------+---+---+---+------+----+----+
| counts                    | 0 | 0 | 0 | 0 | 2965  | 132  | 0 | 0 | 0 | 2    | 0  | 0  |
| pct (total=3099)          | - | - | - | - | 95.68 | 4.26 | - | - | - | 0.06 | -  | -  |
| pct (total_with_nil=3099) | - | - | - | - | 95.68 | 4.26 | - | - | - | 0.06 | -  | -  |
+---------------------------+---+---+---+---+-------+------+---+---+---+------+----+----+
+---------------------------+-------+------+
| year                      | 2022  | 2023 |
+---------------------------+-------+------+
| year                      | 2022  | 2023 |
+---------------------------+-------+------+
| counts                    | 2967  | 132  |
| pct (total=3099)          | 95.74 | 4.26 |
| pct (total_with_nil=3099) | 95.74 | 4.26 |
+---------------------------+-------+------+

rails stats:unroutable # Find routes that will raise a routing error when requested

❯ bin/rails stats:unroutable
            Prefix Verb  URI Pattern                              Controller#Action
    new_auth_users GET   /auth/admin/auth_users/new(.:format)  auth_users#new
   edit_auth_users GET   /auth/admin/auth_users/edit(.:format) auth_users#edit
                   PATCH /auth/admin/auth_users(.:format)      auth_users#update
                   PUT   /auth/admin/auth_users(.:format)      auth_users#update
          username GET   /usernames/:id(.:format)              usernames#show
5 unroutable routes

rails stats:users # Show basic details of users

Our user records usually have either a User#role or User#roles. We're also using auth_logic, so have some last-login-time audit columns. This could be easily tweaked for devise.

This example output lacks the terminal color coding in the activity section.

User count: 2

Role method: role
Roles: user

Roles => Users
  user: bill, ted

Users by Activity: Request in within week, month, year, beyond, unknown (these are terminal color coded)
year: bill, ted

stats.rake

# pre-reqs
# gem 'rainbow'
# gem 'terminal-table'

require 'terminal-table'
require 'action_view'
require 'action_view/helpers'
include ActionView::Helpers::DateHelper

namespace :stats do
  desc 'Show basic details of users'
  task :users => :environment do
    role_method = if User.new.respond_to? :roles
                    :roles
                  elsif User.new.respond_to? :role
                    :role
                  end
    puts "User count: #{User.count}"
    puts
    if role_method.present?
      puts "Role method: #{role_method}"
      puts "Roles: #{User.all.map(&role_method).flatten.uniq.join(', ')}"
      puts
      puts 'Roles => Users'
      User.all.group_by(&role_method).each do |role, users|
        puts "  #{role}: #{users&.map(&:username)&.join(', ')}"
      end
    else
      puts "Users: #{User.pluck(:usenrame).join(', ')}"
    end
    puts

    colors = { week: :red, month: :yellow, year: :blue, beyond: :gray, unknown: :cyan }
    puts 'Users by Activity: Request in within %s' % colors.map{|lbl, clr| Rainbow(lbl.to_s).send(clr) }.join(', ')

    users_with_recency = User.order('last_request_at desc').map do |user|
      delta = Time.now - user.last_request_at rescue :unknown
      recency = if delta == :unknown
                  :unknown
                elsif delta > 1.year
                  :beyond
                elsif delta > 1.month
                  :year
                elsif delta > 1.week
                  :month
                else
                  :week
                end
      OpenStruct.new(username: user.username, recency: recency, color: colors[recency])
    end
    users_with_recency.group_by(&:recency).each_pair do |recency, users|
      puts "#{recency}: #{users.map{|u| Rainbow(u.username).send(u.color) }.join(', ')}"
    end
    puts

  end


  desc 'Show basic model usage stats'
  task :models => :environment do
    puts 'Model Stats:'
    dates = []
    min_date = nil
    max_date = nil
    max_model = nil
    table = Terminal::Table.new headings: %w[Model Count Earliest Latest Latest_rel]
    Zeitwerk::Loader.eager_load_all
    ApplicationRecord.descendants.each do |model|
      next unless model.attribute_names.include? 'created_at'
      count = model.count
      earliest_date = model.minimum('created_at')
      latest_date = model.maximum('created_at')
      if max_date.nil? || (latest_date.present? && latest_date > max_date)
        max_date = latest_date
        max_model = model.name
      end
      if min_date.nil? || (earliest_date.present? && earliest_date < min_date)
        min_date = earliest_date
      end
      dates.concat [earliest_date, latest_date]
      # puts "%15s: %6d (%s, %s)" % [model.name, count, latest_date.to_date, time_ago_in_words(latest_date)]
      table << [
        model.name,
        count,
        earliest_date&.to_date,
        latest_date&.to_date,
        latest_date.nil? ? nil : time_ago_in_words(latest_date)
      ]
    end
    puts table


    puts
    puts "Life Stats: #{distance_of_time_in_words(min_date, max_date)} (#{min_date.year}-#{max_date.year})"
    puts

    puts "Latest record (#{max_model}): #{max_date.to_date} - #{time_ago_in_words(max_date)} ago"
    puts
  end

  desc 'Show basic controller usage stats'
  task :controllers => :environment do
    logfiles = Dir['log/%s.log*' % Rails.env].sort
    logs_gz, logs_txt = logfiles.partition{|f| Pathname.new(f).extname == '.gz' }
    results = `ag Started -A 1 #{logs_txt.join(' ')}`
    unless logs_gz.empty?
      results << `zcat #{logs_gz.join(' ')} |ag Started -A 1`
    end
    Event = Struct.new(:http_method, :uri_path, :client_ip, :requested_at_str, :controller_name, :controller_action, :format) do
      def requested_at
        Chronic.parse(requested_at_str)
      end
      def action_fmt
        [controller_name, controller_action].join('#')
      end
    end
    events_raw = results.lines.each_cons(2)
    events = events_raw.map{|lines|
      next unless lines.first =~ /Started/ && lines.last =~ /Processing/
      # Started GET "/auth/signin" for 10.9.9.182 at 2018-07-30 09:22:15 -0400 ()
      # Processing by UserSessionController#new as HTML ()
      started_m = /Started (?<http_method>\w+) "(?<uri_path>[^ ]*)" for (?<client_ip>\d+\.\d+\.\d+\.\d+) at (?<requested_at_str>[^ ]+ [^ ]+ [^ ]+)(?: .*)?/.match(lines.first)&.named_captures
      processing_m = /Processing by (?<controller_name>[^ ]+)#(?<controller_action>\w+) as (?<format>[^ ]+)(?: .*)?/.match(lines.last)&.named_captures
      next if started_m.nil?
      next if processing_m.nil?
      e = Event.new(*started_m.merge(processing_m).values)
      # p started_m.merge(processing_m).values
      # p e
      # puts "==="
      e
    }.compact

    logged_controller_action_events = events.group_by(&:action_fmt)

    defined_controller_actions = []
    Zeitwerk::Loader.eager_load_all
    ApplicationController.descendants.each do |clazz|
      methods = clazz.instance_methods - ApplicationController.instance_methods
      methods.each do |method|
        defined_controller_actions << OpenStruct.new(controller_name: clazz.name, controller_action: method, action_fmt: "#{clazz.name}##{method}")
      end
    end

    all_controller_actions = (logged_controller_action_events.keys + defined_controller_actions.map(&:action_fmt)).compact.uniq.sort

    table = Terminal::Table.new headings: %w[Action Requests Earliest Latest Latest_rel]
    all_controller_actions.each do |controller_action|
      events = logged_controller_action_events[controller_action]
      if events.present?
        dates = events.map(&:requested_at)
        min_date = dates.min.to_date
        max_date = dates.max
        #puts "%d\t%s (earliest: %s, latest: %s (%s)" % [events.count, controller_action, min_date, max_date, time_ago_in_words(max_date)]
        table << [controller_action, events.count, min_date, max_date, time_ago_in_words(max_date)]
      else
        table << [controller_action, Rainbow('No Log').gray, nil, nil, nil]
      end
    end
    puts table
  end

  desc 'show frequency trends for a date column; MODEL=x COLUMN=y'
  task :trends => :environment do
    model_str = ENV.fetch('MODEL', nil)
    col_str = ENV.fetch('COLUMN', nil)

    Zeitwerk::Loader.eager_load_all
    models = ApplicationRecord.descendants.index_by{|model| model.to_s}
    model = models[model_str]
    if model.nil?
      warn "Unkown model class #{model_str}"
      exit 1
    end

    unless col_str.in?(model.attribute_names)
      warn "Unknown column name '#{col_str}' for #{model_str}"
      exit 1
    end

    col_def = model.columns.find{|c| c.name == col_str}
    unless col_def.type == :datetime
      warn "Column '#{col_str}' is not a datetime. type=#{col_def.type}"
      exit 1
    end

    dates = model.pluck(col_str)
    total_with_nil = dates.count
    dates.compact!
    total = dates.count

    # hrs
    freq_table = ->(field, keys) do
      counts = dates.map(&field).tally
      if keys == :min_max
        keys = (counts.keys.min..counts.keys.max).to_a
      end
      table = Terminal::Table.new headings: [field] + keys

      values = counts.values
      mean = values.sum(0.0) / counts.size
      sum = values.sum(0.0) { |el| (el - mean)**2 }
      variance = sum / (values.size - 1)
      stddev = Math.sqrt(variance)

      headings_row = [field]
      counts_row = [:counts]
      pct_row = ["pct (total=#{total})"]
      pct_nil_row = ["pct (total_with_nil=#{total_with_nil})"]

      keys.each do |key|
        count = counts[key]

        # color coded by quartiles
        # color = if count.nil?
        #           :white
        #         elsif count <= stats[:q1]
        #           :gray
        #         elsif count <= stats[:q2]
        #           :blue
        #         elsif count <= stats[:q3]
        #           :orange
        #         else
        #           :red
        #         end
        color = :white

        # color coded by standard deviation
        color = if count.nil?
                  :white
                elsif count <= mean - stddev
                  :gray
                elsif count <= mean + stddev
                  :blue
                else
                  :red
                end

        headings_row << Rainbow(key).send(color)

        counts_row << Rainbow(count || 0).send(color)

        pct_row << if count.blank?
                     '-'
                   else
                     '%0.2f' % (count.to_f / total * 100)
                   end

        pct_nil_row << if count.blank?
                         '-'
                       else
                         '%0.2f' % (count.to_f / total_with_nil * 100)
                       end
      end

      table << headings_row
      table << :separator
      table << counts_row
      table << pct_row
      table << pct_nil_row

      puts table
    end

    freq_table.call(:hour, (1..24).to_a)
    freq_table.call(:day, (1..31).to_a)
    freq_table.call(:month, (1..12).to_a)
    freq_table.call(:year, :min_max)
  end

  desc 'Find routes that will raise a routing error when requested'
  task :unroutable => :environment do
    # Written by Nate Matykiewicz from the "Ruby on Rails Link" Slack.
    # https://gist.github.com/natematykiewicz/521bb5c9ec1f02c69a282bee21af1031
    #
    # A lot of this code was taken from how `rake routes` works
    # https://github.com/rails/rails/blob/f95c0b7e96eb36bc3efc0c5beffbb9e84ea664e4/railties/lib/rails/commands/routes/routes_command.rb

    require 'action_dispatch/routing/inspector'

    unroutables = Rails.application.routes.routes
                       .map { |r| ActionDispatch::Routing::RouteWrapper.new(r) }
                       .reject { |r| r.internal? || r.engine? || r.path.starts_with?('/rails/') || !r.controller }
                       .reject do |r|
                          controller = "#{r.controller}_controller".classify.safe_constantize

                          controller && (
                            controller.method_defined?(r.action) ||
                            Rails.root.join('app/views', r.controller).glob("#{r.action}.*").any?
                          )
                        end
                       .map { |r| r.__getobj__ }

    if unroutables.present?
      inspector = ActionDispatch::Routing::RoutesInspector.new(unroutables)

      formatter = if ENV['EXPANDED']
                    ActionDispatch::Routing::ConsoleFormatter::Expanded.new
                  else
                    ActionDispatch::Routing::ConsoleFormatter::Sheet.new
                  end

      puts inspector.format(formatter)
    end

    puts "#{unroutables.size} unroutable routes"
  end

end