Find which tests always run before a given test when the suite fails, in order to detect ordering issues in your test suite.

inspector.rb

#!/usr/bin/env ruby
#
# Find repeated output across runs of a process.
#
# Usage:
#
#   ./inspector "./minitest-runner some_failing_test"
#
# The main purpose for this is to find tests that pollute the environment making
# other tests fail erratically, for test suites that run in a random order.
#
# The command will be run multiple times, and every time it fails, its STDOUT
# will be broken down into lines and compared to the STDOUT of previous runs,
# keeping only those lines that have been present in _all_ runs of the process.
#
# This terminates when it gets down to 1 entry, but that might not happen (yay
# randomness...) If at any point you send an INT or TERM signal to the process
# it will print out the best it could do in the time it had. (FIXME: Add a
# timeout option.)

require "set"

command = ARGV.fetch(0)

possible_polluters = Set.new

finish = ->(*) do
  puts possible_polluters.to_a.sort.join if possible_polluters.any?
  exit 0
end

Signal.trap "SIGINT",  &finish
Signal.trap "SIGTERM", &finish

loop do
  status = nil

  IO.popen(command) do |io|
    tests = io.readlines
    next if status = $?.to_i.zero?

    if possible_polluters.empty?
      possible_polluters.merge(tests)
    else
      possible_polluters &= tests
    end
  end

  $stderr.print status ? "." : "F"

  finish.call if possible_polluters.size == 1
end

minitest-runner.rb

#!/usr/bin/env ruby
#
# List tests that run previous to a specific test.
#
# Usage:
#
#    ./minitest-runner some_test_pattern
#
# The intent is to filter out which tests run before a certain test. This is
# useful to debug a test that fails erratically due to a previous tests that is
# contamining the environment. Run this enough times and you'll be able to
# compare the list of failing tests to find which ones always run before this
# test, reducing the amount of tests to look through.

test_name = ARGV.fetch(0) { abort "Please pass a test name" }

output = nil
IO.popen("rake TESTOPTS=--verbose") { |io| output = io.readlines }

status = output.last
success = status[/(\d+) failures/].to_i.zero?
success &&= status[/(\d+) errors/].to_i.zero?

test = output.grep(/#{test_name}/).first
abort "Can't find a test matching `#{test_name}`" if test.nil?

line_with_sought_test = output.index(test)

# Remove the test itself from the output, since it isn't interesting.
output.delete_at(line_with_sought_test)

# Remove anything _after_ the test, as it isn't relevant.
output[line_with_sought_test, output.size - line_with_sought_test] = []

# MiniTest prints a 4 line header. Remove that.
output[0, 4] = []

# Massage MiniTest output's to only print test names. We don't care how much it
# took to run each individual test.
puts output.map { |line| line.sub(/\s.*/, "") }.join

exit_status = success ? 0 : 1
exit exit_status

oh wow this is gold

The gem minitest-bisect does a much more rigorous job of this.

https://github.com/seattlerb/minitest-bisect/