Alternative expectation interface for MiniTest and RSpec

expectations.md

Expectations

I took the ideas presented here and built a gem called Mustard. Check it out!

There are several expectation/assertion interfaces available for writing tests/specs. Here are some issues I have with them.

Test::Unit/MiniTest

• The order of assert_equals feels backwards
• Oh wait, that should be assert_equal (that too)
• I like to write the subject first

MiniTest::Spec

• Adds 30+ methods to Object
• Any matchers you want to add are another method on Object
• Still too few matchers, so it's ugly: foo.must_be :<, 10

Bacon

• too.many.method.calls.to.simulate.english

RSpec Should

• The space makes for some awkward syntaxes that cause warnings (when used with ==, >, etc.)
• eq() and such aren't bad, but may need parenthesis
• Pollutes the spec context with methods for each matcher

I haven't used the RSpec expect() interface enough to know cons.

Proposed Solution

# should returns an object which can have matchers called on it
5.should.eq 5
5.should.equal 5

# should_not can be used to do a reverse match
5.should_not.equal 7

# some comparison matchers
5.should.be_less_than 8
5.should.be_greater_than 3
5.should.be_lt 8
5.should.be_gt 3
5.should.be_gte 5
true.should.be_true
false.should.be_false

# call the method and see if it is true
[].should.be :empty?
5.should.be :between?, 3, 7

# add matchers
Should.matcher :be_empty do # any args will be passed into block
  empty? # runs in the context of the object and returns true/false
end
[].should.be_empty
[].should_not.be_empty # message: Expected [] to not be empty

# add matcher through class
Should.matcher :be_empty, BeEmptyMatcher
class BeEmptyMatcher
  # methods in here for defining behavior and messages.
end

I like this because it does not pollute the Object space much (just shouldand should_not). Also every method called after it is simply a matcher. No crazy method chains.

What do you think? If there's enough interest I'll write this up as a plugin for both RSpec-Core and MiniTest.

@zenspider I had considered proposing additional matchers to minitest/spec but feel it is adding too much to Object already. The motivation for my solution is moving minitest/spec style matchers off of Object. Surely 2 methods on Object is better than 30+?

Also, this solution has a consistent should_not variation so each matcher automatically has an inverse. No need to make two matchers each time.

@ryanb,

eq() and such aren't bad, but may need parenthesis

RSpec's eq does not need parentheses. My team regularly useseq` without parens.

You might be referring to the statement from Myron and Ilya that == causes warnings in verbose mode. I've actually never been able to reproduce that warning so I'm not sure what they're talking about.

# rspectest.rb 
describe 'thing' do
  it 'should thingy' do
    3.should == 3
  end
end

$ ruby -w -W2 `which rspec` rspectest.rb 
.

Finished in 0.00029 seconds
1 example, 0 failures

No warning in the output above. Tested in 1.9.3, 1.8.7, and REE.

@BMorearty,

Try this:

describe 'thing' do
  it 'should thingy' do
    3.should == 3
    3.should == 3
  end
end

@ryanb,

I've never had any problems with MiniTest's pollutions. Methods on the test class all start with assert_ or refute_; methods on Object all start with must_ or wont_. Never had any collisions.

One of the best thing with Test::Unit-style assertions is that it's so simple to add custom assertions: You just write Ruby. def assert_response is all you need.

I also appreciate how assert_ allows you to write custom failure messages.

I completely agree @ryanb, object pollution is bad. I have tried out expect().to syntax, and it felt right (sorry for xUnit advocates, I hate .that syntax, but it's OK, I don't feel I know better). Maybe both styles could be merged like expect(true).to.be_false. This way you could add a sinfle method to 'it' namespace, two methods to expect's returning namespace, and you could put matchers inside another one. Full separation.

I kind of never really liked RSpec. One ugly column of assertions in Test::Unit would result in one ugly long line of RSpec. Would be great to have something in the middle with a beautiful codebase and stil being crazy fast.

@ryanb be does essentially the same as it does in other frameworks, it provides syntax sugar. In Bacon, be has a few extra useful side effects:

  5.should.be -> (n) { n == 5 }
  5.should.be  { |n|   n == 5 }
  5.should.not.be.nil?
  5.should.not.nil?

  5.should.be("the product of some complex math") do |n|
    # do some complex math with n
    false
  end
  # Bacon::Error: the product of some complex math

The advantage of the code being so short, is that it's very readable.

If you want rspec style matchers, just include methods into the Should class and you're away. It's no more complex than that.

I find it somewhat confusing that you claim you cannot understand what some of bacons methods are doing, when the rspec equivalents are doing much much more in order to achieve their logic. The matcher proxies are MUCH more complicated, I really cannot believe anyone would describe them as being simpler than the bacon implementation.

In bacon almost all assertions you write, in normal usage (e.g. your original examples) execute in this class:

https://github.com/chneukirchen/bacon/blob/master/lib/bacon.rb#L290-L351

That's all of it. ALL OF IT. There is no more.

By contrast, in rspec, the same stuff goes through:

https://github.com/rspec/rspec-core/blob/master/lib/rspec/core/subject.rb#L52-L79
https://github.com/rspec/rspec-expectations/blob/master/lib/rspec/expectations/handler.rb#L15-L61
https://github.com/rspec/rspec-expectations/blob/master/lib/rspec/matchers.rb#L227-L231
https://github.com/rspec/rspec-expectations/blob/master/lib/rspec/matchers/built_in/be.rb#L48-L92

And that's NOT all of it. That's just the parts that you really care about.
There's just no way your comments are based on reading the source here. RSpecs be implementation is very very similar, just much more complicated and indirect.

Now I use (regularly) RSpec, Bacon, MiniTest and Test::Unit. None of them are perfect (there are tradeoffs, I would not "blame" authors for any shortcomings in general use a testing system), but I really can't let the claims stand that minitest or bacon are in any way "harder to understand" than the other two. If you've ever had to debug the libraries themselves, or you've spent some time reading their internals (not just implementing a few matchers here and there), then there is no way that you can describe RSpec or Test::Unit as being "simpler" or "easier to understand", it's just not real.

Onto matchers a little more, if you study what is available, and the way that bacon actually works, you'll realize that you can also implement local scope matchers, that won't even cross-pollute your tests:

module AwesomeMatcher
  def awesome
    -> (o) { o.awesome? }
  end
end

extend AwesomeMatcher

should "be awesome" do
  5.should.be awesome
end

Alternatively, you can just implement a regular method on Should:

class Should
  def awesome?
    @object.awesome?
  end
end

should "be awesome" do
  5.should.awesome
  5.should.be.awesome
end

If you want be_ matchers, you can do that too:

module AwesomeMatcher
  def be_awesome
    -> (o) { o.awesome? }
  end
end

extend AwesomeMatcher

should "be awesome" do
  5.should be_awesome
end

See how easy that is, to be different? I cannot stress enough how this is a product of simplicity. And that simplicity of implementation actually goes further...

Proof? Ok, lets implement generic be_ (that's space be_) matchers then:

module BeDownUnder
  def method_missing(name, *args, *block)
    return super unless name =~ /^be_(.*)$/
    target = $1
    -> (o) { o.send(target) }
  end
end

extend BeDownUnder

should "be awesome" do
  5.should be_awesome
end

Does that feel closer to what you want?

Ah yes, should_not, okay:

class Object; def should_not(*args, &block); Should.new(self).not.be(*args, &block); end; end

Lets review:

• be_ matchers, that you can use space separation: check.
• should_not: check.
• no crazy method chains: check.
• under 500 LoC still: check.
• no warnings: check.
• 1.8.6 -> 2.0.0: check.

So I've implemented what you discussed, and your desires.

I dare you 😜 to implement it in a gist comment, in 10 minutes, for RSpec.

Have fun!

While we are on this topic I want to express my confusion with third-person verb usage in MiniTest. I don't understand it:

assert_equal
assert_includes
assert_match
assert_raises
assert_respond_to
assert_throws

@zenspider I had considered proposing additional matchers to minitest/spec but feel it is adding too much to Object already.

If you feel that the language of minitest is lacking, you should say so. They get implemented as assertions anyhow and just glued into Object.

The motivation for my solution is moving minitest/spec style matchers off of Object. Surely 2 methods on Object is better than 30+?

Well no, actually. If your argument is against global pollution, then anything > 0 is bad. You can't have it both ways. As pointed out elsewhere, I always scope my methods on object to must_* and wont_* so it really isn't any worse than your should/should_not.

What IS worse is the extra method calls and levels of indirection.

Also, as I've pointed out, you can disable the expectation methods in minispec. Clearly 0 methods infecting Object is better than your 2. Right? I mean... I do get to use your own logic against you, right?

Also, this solution has a consistent should_not variation so each matcher automatically has an inverse. No need to make two matchers each time.

I only make two expectations when it make sense to have both. This is the whole:

def assert_nothing_raised
  yield
end

thing. must_not_be_empty is just as useless so why provide it? Your need for consistency actually encourages bad testing and a false sense of security.

Finally (sorry)... I walked through bacon's method flow in my "Size isn't Everything" talk at cascadia ruby conf last year with a nice illustration and everything. You might want to check it out. It is a very beautiful little framework.

Compare your proposed:

Should.matcher :be_empty do
  empty?
end

With raggi's (and what would be very similar in minitest) bacon:

def be_empty(*args)
  @object.empty?
end

(comments stripped on both)

Do you see the main difference there?

In bacon/minitest you only need to know ruby. You only need to know one object system. You only need to know one way to create behavior. Pretty much anyone can look at the code and simply understand what is going on.

In rspec and your example, you need to know 2: ruby and the testing system's notion of objects, classes, and methods. I have no idea what that empty? sitting in the middle of nowhere is supposed to do to anything. I'd bet that 90% of the ppl out there give hand-wavey explanations on how rspec and your proposal actually work.

I will never wrap my head around why there is an entirely new class hierarchy, inheritance, and module system in rspec and your proposal. I will never wrap my head around why you want Should.matcher name do to be some obtuse alias for def name. Levels of indirection just make things more confusing, harder to debug and slower. But most importantly, it is much harder to teach to the people who need it most.

I mentioned this on Twitter this morning but thought I would be worth putting my thoughts here too (and over more than 140 chars).

---

I played around with similar earlier this year in https://github.com/gshutler/mustard. I never published it but it seems I had similar thoughts.

Things that motivated me:

Lack of pollution

Mustard extends Object with must and must_not, everything else hangs off the Mustard proxy.

Intuitive

RSpec and similar matchers have often tripped me up as I couldn't remember which way around spaces and statements needed to be, when parantheses became mandatory, how the be_ statement was written, etc.

Mustard works with regular boolean expressions which you would use with Ruby so there's no new syntax to learn. If your regular boolean statement is foo.vaguely_sensible? :bar then foo.must.vaguely_sensible? :bar is a working assertion.

Readability != "Is English"

We're developers, we read code for a living. I know that <= means less than or equal to so why no just use that rather than added another set of vocabulary to learn/forget?

More assertive verb

Referencing RFC 2119 SHOULD has weak connotations whereas MUST is assertive. Given that tests fail when your expectations aren't met must seems a better fit.

---

All that being said I was doing it more as a thought experiment rather than a serious effort to create an alternative and I've continued to use RSpec matchers day-to-day. However, if there would be wider interest in refining this (I stopped at the point where there were diminishing returns between effort and interest) I'd be happy to pick it up again.

I went ahead and coded my ideas out into a gem. You can find it at:

https://github.com/ryanb/musts

For me it strikes a nice balance of simplicity and utility. Feel free to use it if you like it too. If you don't, you don't have to use it. :)

Renamed the project to Mustard. It can be found here:

https://github.com/ryanb/mustard

I would prefer:

[].must.be_empty

I think is better than be :empty? What if something was supposed to be a symbol and you are explicitly looking for the symbol :empty? ??

If you're still considering interfaces have you looked at https://github.com/sconover/wrong ?

I have similar objections to the test frameworks you call out. Unfortunately I don't care for expectations either. Here's my attempt to answer the problem. MicroTest

+1 on checking Wrong out (https://github.com/sconover/wrong). I think it addresses a lot of the issues you listed.