Examples of trying to set test order to be sorted. Note that I'm looking at the order that test files are being run as well. Perhaps that isn't controlled here.

sorted_test_order.rb

# So far I have tried the following code in test_helper.rb

# Based on documentation here: http://api.rubyonrails.org/v4.2/classes/ActiveSupport/TestCase.html

class ActiveSupport::TestCase
  def test_order
    :sorted
  end
end

# Realizing that it is a class method I changed this to

ActiveSupport::TestCase.test_order = :sorted

# Regardless in Rails 4 the default test order is supposed to be :sorted 
# while in Rails 5 it changes to :random
#
# Also checking manually using RAILS_ENV=test rails c I can see that 
# ActiveSupport::TestCase is fact returning :sorted

# Moving on to MiniTest directly

class MiniTest::Test
  def test_order
    :sorted
  end
end

class MiniTest::Test
  def self.test_order
    :sorted
  end
end

# Both of these seem to have no impact as well. Both the order the test files
# are executed and the order of tests within the files are still randomized.
# What I have had luck with is fixing the seed value when running locally.

# I have also tried adding `i_suck_and_my_tests_are_order_dependent!` to the
# TestCase base class for good measure, but I have no intention of checking
# in any code like that.

# Also considered that perhaps I was setting this too late and moved the 
# code to `config/environment/test.rb`. I can confirm via pry breakpoint 
# that MiniTest::Test.test_order == :sorted but tests continue to run in 
# random order.

Fixing the seed value when running tests works fine for me. Though I did waste a bit of time trying these options first. Also it may be worth noting that I'm happy with running tests in random order on the CI server where I get the benefits of random ordering with zero productivity cost.

When running locally however it just seems to be a usability inconvenience traded off against protection from a problem I quite frankly can't actually recall ever needing protection from (i.e., order dependent tests). Perhaps I've just been very fortunate with that myself, I do get that with the tendency towards testing against the database in Rails applications, though even then assuming the test setup is done correctly this shouldn't be a problem.

@chrisnicola what version of minitest are you using? I would have thought putting ActiveSupport::TestCase.test_order = :sorted in the test_helper.rb at the end like here would be enough.

@jrafanie. Interestingly I'm using Rails 4.2 so :sorted is the default for ActiveSupport::TestCase.test_order. Good point though, I'll update and check gem versions though.

Looks like I was already using 5.9.0

OK. So line 6 is wrong because it is an instance method, not a class method.

Line 13 should work afaik.

Line 24 is the same as line 6, needs to be a class method. Ah. Except that it is Minitest, but that should still work fine for now.

Line 30 should work fine... but rails 5 has made a lot of changes to their overrides of minitest so I'm not entirely sure (we're still on rails 4 so I haven't poked)...

Lines 13 and 30 should be exactly the same thing, but might not in rails. I would think that 13 would take higher priority because it is subclassing Minitest::Test.

Line 6 should be the same as line 13 if it were a class method.

Either should be fine and should exist in your test/test_helper.rb near the top but after the requires.

I only included the class method stuff because it was trying things randomly in case I was blind. I do realize it should be an instance method though. I am currently testing this with Rails 4.

I've tried setting the class method in both test_helper.rb and config/environment/test.rb and on both MiniTest::Test and ActiveSupport::TestCase (which I have verified already defaults to :sorted as it is Rails 4). I have no idea why it is having no effect and the test order remains random.

I'm going to try with a clean rails project and see if I see the same results. Perhaps another gem in interfering.

Just to convince myself I'm not insane I added the following to the beginning of a test file and tested it specifically:

  before :each do
    puts "Test Order: #{self.class.test_order}"
  end

It returns sorted every time, but the tests are output in random order anyways. I've also checked before the tests. This is the top of my test_helper.rb:

ENV['RAILS_ENV'] ||= 'test'
require File.expand_path('../../config/environment', __FILE__)
class ActiveSupport::TestCase
  def self.test_order
    :sorted
  end
end

class MiniTest::Test
  def self.test_order
    :sorted
  end
end

puts MiniTest::Test.test_order

I've checked this without :sorted and :alpha. Anyways all that is left it to test it with a clean Rails project.

Ok I've tracked this down to minitest-spec-rails. I'm not entirely sure why it breaks the override behaviour, but I was able to get consistent test order without it in a new project using both ActiveSupport::TestCase and MiniTest::Test based tests.

I've opened a bug here metaskills/minitest-spec-rails#76

Cool, thanks for tracking it down @chrisnicola. It would be nice to update minitest-spec-rails's readme if there's something you tripped on that could save others the same headache down the road, or that the very least, having your issue in the repo could help others. Thanks for taking the time to dig into it.