Note: Hi Jani! It's a pleasure to make your acquaintance today! I'm going to lean hard into this feedback, because I care an awful lot about this stuff (I guess that should be obvious if I'd name a company after it!). So below I'm just going to default to quoting all of your prose and then commenting blow-by-blow
When unit testing real-world code, there are many situations that make tests hard to write. How do you check if a function was called? How do you test an ajax call? Or code using
setTimeout? That's when you use test doubles - replacement code that makes hard to test things easy to test.
This framing of why people use test doubles has always been problematic to me. If something is hard to test, then I think test doubles are usually the wrong answer, and I'd sooner rework the design of the code so that it's easier to test. If testing an AJAX call, then I'd often suggest running a fake server which stubs responses. If using tricky natives like setTimeout, I'd recommend wrapping that function in one I own that isn't painful to work with in tests.
Where test doubles do come in handy for me is in unit testing functions whose responsibility is to delegate work among various dependencies. "If the job of sendInvoice(client) is to load billable hours, calculate prices, and fire email, then I want to stub loadHours(client) with 40, stub calculatePrice(client, 40) with 3000, and verify we call sendEmail(client, "Pay me $3000!"). That's because the role of sendInvoice (despite the name) isn't to actually send invoices, it's to parcel out the work of sending invoices, and its unit test helps me figure out how it should do that job (i.e. it makes me realize I need loadHours, calculatePrice, and sendEmail functions with those signatures)..
Before diving in I wanted to clear that up first, because I think the vast majority of Sinon use in the wild is "ugh this is hard to test, let's just poke holes in reality until the hurting stops", which leads to two problems: (a) hard to test code is inherently hard to use code, and so using a sinon-like tool for that reason enables us to keep writing hard to use code, and (b) poking holes in reality based on what's hard about testing usually results in tests that are hard to understand later ("what's real/fake? what's the value of the test?" etc).
Okay, I'll get off my soapbox now.
For many years, Sinon.js has been the de-facto standard in JavaScript tests for creating test doubles. It's a must-have tool for any JavaScript developer writing tests, as without it, writing tests for real applications would be nigh impossible.
The only other thing I might point to is Jasmine spies, which predate Sinon by several years, I believe. These "spies" are more accurately termed as spies than Sinon's abuse of the word "spy", but I'll circle back to that.
Recently, a new library aptly named testdouble.js has been making waves. It boasts a similar featureset as Sinon.js, with a few differences here and there.
"Making waves" is a very kind turn of phrase. It's been a very long and winding road!
In this article, we'll look into what both Sinon.js and testdouble.js offer, and compare their respective pros and cons. Will Sinon.js remain the superior choice, or will the challenger take the prize?
Note: If you're unfamiliar with test doubles, I recommend reading my Sinon.js tutorial first. It will help you better understand the concepts we'll be talking about here.
Other background on test doubles:
- Gerard Meszaros coined the term in his book/wiki XUnitPatterns
- My own definition of the term is here
In order to ensure it's easy to understand what is being discussed, a quick overview on the terminology used:
- A test double is a replacement for a function used during a test. It can refer to any of the three types mentioned below.
- A spy is a type of test double which does not affect the behavior of the original function
- A stub is a type of test double which replaces the original function completely
- A mock is similar to a stub which can also be used for verification
The terminology around test doubles is often confused and problematic, and it's one reason why you see testdouble.js avoid using traditional test double jargon. Not because the words' meaning doesn't matter, but because it does, and because tools like Sinon have so thoroughly confused it.
As linked above, XUnitPatterns in the mid-aughts defined most of these terms definitively for the first time (they've been floating around the extreme programming community since 1999).
I'll lay out the definitions in a different order, because the history matters for why each word is used:
- Your definition of "test double" is fine. It's the superset, meant to evoke th idea of a "stunt double" for tests
- A "stub" is any test double configured to return a value when called by the subject. (XUnit)
- A "mock" is any test double that requires preemptive expectation of every call, and those calls may or may not also be stubbed (e.g. In your test setup you might say
expect(foo).toBeCalledWith(5).thenReturn('blah')and iffoois never called with(5), then the test blows up.). Key to the definition of a mock is that these verifications have to happen before you call the subject, and they were viewed by people in the community like myself because they violate the natural Arrange-Act-Assert ordering people understand best - A "spy" is not a real thing that doesn't affect original behavior, as Sinon's API erroneously refers to it. Spies were an innovation of Mockito's author (Xunit) that were designed to be more natural to use than mocks. Rather than blow up preemptively if not called in any way, and rather than tangle up the concept of stubbing a return value and simultaneously verifying a call, a spy is a totally fake test double that just quietly listens to all of its invocations so you can verify a subject called your test double correctly after you call your subject in the "Assert" phase (a step should either be setup or assertion, to be both woulud be to overspecify in the test).
Yes, this means that sinon.spy() is emphatically not a spy, but every td.function() can serve as either a stub or a spy.
So what is a Sinon spy? Unfortunately, because no one imagined an equivalent to Sinon spies (that is "a real thing that is silently listened to for verification purposes"), it didn't get a name early on that was agreed to. The first implementation I know of to do this was rr for Ruby, and it called these things "proxies". Because sinon spies can also be used to stub return values, I view them as very dangerous (if the call changes in any way the stubbing might fail and call through to the real code, yielding quietly surprising results) and I would call the Sinon spy a Partial Mock antipattern.
It should be noted that one of the goals of testdouble.js is to reduce the confusion between this type of terminology.
LOL, yep!
Let's begin with a look at how Sinon.js and testdouble.js compare in basic use.
Sinon has three separate concepts for test doubles: Spies, stubs and mocks. The idea is that each represent a different usage scenario. This makes the library more familiar to those coming from other languages or who have read books using the same terminology, such as xUnit Test Patterns. But the other side is that these three concepts can also make Sinon more difficult to understand when first using it.
In this article, I don't know how you disambiguate what Sinon calls things vs what they actually are, but that's indeed the problem with Sinon's verbiage—even discussions like this one quickly descend into word salad.
Here is a basic example of Sinon usage:
//Here's how we can see a function call's parameters:
var spy = sinon.spy(Math, 'abs');
Math.abs(-10);
console.log(spy.firstCall.args); //output: [ -10 ]
spy.restore();
//Here's how we can control what a function does:
var stub = sinon.stub(document, 'createElement');
stub.returns('not an html element');
var x = document.createElement('div');
console.log(x); //output: 'not an html element'
stub.restore();testdouble.js opts for an API which is more straightforward. Instead of using concepts like spies or stubs, it uses language much more familiar to JavaScript developers, such as
td.function,td.objectandtd.replace. With this, testdouble can be easier to pick up at first, and certain tasks are much easier to do using it. But on the other hand, some more advanced uses may not be possible at all (which is sometimes intentional).
Here's what testdouble.js looks in use:
//Here's how we can see a function call's parameters:
var abs = td.replace(Math, 'abs');
Math.abs(-10);
var explanation = td.explain(abs);
console.log(explanation.calls[0].args); //output: [ -10 ]
//Here's how we can control what a function does:
var createElement = td.replace(document, 'createElement');
td.when(createElement(td.matchers.anything())).thenReturn('not an html element');
var x = document.createElement('div');
console.log(x); //output: 'not an html element'
//testdouble resets all testdoubles with one call, no need for separate cleanup
td.reset();You'll notice the language used by testdouble is more straightforward. We "replace" a function instead of "stubbing" it. We ask testdouble to "explain" a function to get information from it. Other than this, so far it's fairly similar to Sinon.
This also extends to creating "anonymous" test doubles:
var x = sinon.stub();
//vs
var x = td.function();The biggest difference you'll notice relates to how you set up your stubs and verifications. With Sinon, you chain commands after a stub, and use an assertion to verify the result. testdouble.js simply has you show it how you want the function to be called - or "rehearse" the function call.
I think this section is very good. All I might add is that td.explain(abs) actually tells you a whole heckuva lot, especially in complex cases, and for which Sinon has no equivalent. I might also mention that td.func('.nameOfThing') leads to much better messages which help you track things down later.
var x = sinon.stub();
x.withArgs('hello', 'world').returns(true);
var y = sinon.stub();
sinon.assert.calledWith(y, 'foo', 'bar');
//vs...
var x = td.function();
td.when(x('hello', 'world')).thenReturn(true);
var y = td.function();
td.verify(y('foo', 'bar'));This can make testdouble's API easier to understand, since you don't need to know what operations you can chain and when.
👍
On a high level both libraries are reasonably similar. But what about common testing tasks that you might need to do in a real project? Let's take a look at a few cases where the differences are starting to show.
The first thing to note is testdouble.js has no concept of a "spy". While Sinon.js allows us to replace a function call so that we get information from it, while keeping the function's default behavior, this it not possible at all with testdouble.js. When you replace a function with testdouble, it always loses its default behavior.
Based on the conversation above, I really take issue with this use of the word "spy" here, because every testdouble.js function is definitely a spy by its proper definition. I'd say testdouble.js goes out of its way to prevent users from conflating realness & fakeness as Sinon does with its poorly named "spy()" function. Because test doubles are so often confusing to folks, it's really important to know whether you're looking at a real or fake thing, and in over 10 years of using test doubles, I've never once run into a legitimate case where spying on a real API was (IMO) a good idea in a unit test.
This is not necessarily a problem however. The most common usage for spies would be using them to verify callbacks were called, which is easily doable with
td.function:
var spy = sinon.spy();
myAsyncFunction(spy);
sinon.assert.calledOnce(spy);
//vs...
var spy = td.function();
myAsyncFunction(spy);
td.verify(spy());While not a big issue, it's still good to know this difference exists between the two, as otherwise you may be surprised if you expect to be able to use spies in some more specific manner with testdouble.js.
Completely valid point. This is something that people used to Sinon need to understand, and I need to figure out how to document this for Sinon users. I have started an issue for this here
The second difference you'll run into is testdouble is stricter about inputs.
Both Sinon's stubs and assertions allow you to be imprecise about what parameters are given. This is easiest illustrated by an example:
My argument for this has always been that in an isolated unit test (which is where I believe test doubles ought to be used), the test is almost always in control of exactly what gets passed to a test double, so the default specification should be exact. In the example below I could change my stub('hello', 'world') call in my production code to stub('hello', 'Jani') and the test wouldn't fail—what is the likelihood that's what a tester would intend? What's the likelihood that wouldn't break in production while keeping the test green? I would argue almost 0%!!
You do touch on this below, I just (as is apparent by now) care a great deal about this.
var stub = sinon.stub();
stub.withArgs('hello').returns('foo');
console.log(stub('hello', 'world')); //output: 'foo'
sinon.assert.calledWith(stub, 'hello'); //no error
//vs...
var stub = td.function();
td.when(stub('hello')).thenReturn('foo');
console.log(stub('hello', 'world')); //output: undefined
td.verify(stub('hello')); //throws error!By default, Sinon doesn't care how many extra parameters are given to a function. While it provides functions such as
sinon.assert.calledWithExactly, those are not suggested as the default in the documentation. Functions likestub.withArgsalso do not come with an "exactly" variant.testdouble.js on the other hand defaults to requiring the exact parameters specified. This is by design. The idea is that if a function is given some other parameters unspecified in the test, it's potentially a bug, and should fail the test.
It is possible to allow specifying arbitrary parameters in testdouble.js, it just isn't the default:
👍
//tell td to ignore extra arguments entirely
td.when(stub('hello'), { ignoreExtraArgs: true }).thenReturn('foo');With
ignoreExtraArgs: truethe behavior is similar to Sinon.js
It may or may not be worth pointing out that the default Sinon stub has no arg-checking whatsoever. If you don't call withArgs, you can pass literally anything and it'll satisfy the stubbing. This has always bewildered me for the same reasons as above (you may as well just pass function () { return 5 }).
While using promises with Sinon.js is not complicated, testdouble.js has built-in methods for returning and rejecting promises.
var stub = sinon.stub();
stub.returns(Promise.resolve('foo'));
//or
stub.returns(Promise.reject('foo'));
//vs...
var stub = td.function();
td.when(stub()).thenResolve('foo');
//or
td.when(stub()).thenReject('foo');Note: it's possible to include similar convenience functions in Sinon 1.x using sinon-as-promised. Sinon 2.0 and newer include promise support in the form of
stub.resolvesandstub.rejects
There are some interesting new configurations in testdouble@2.x that Sinon lacks with respect to Promises, which is that you can set the delay like td.when(stub(), {delay: 40}).thenResolve('foo'); this can be really really handy for logic that aggregates promises and does things based on when multiple promises resolve.
Additionally, this article doesn't touch on it, but test double has really robust support for callback APIs which I haven't seen anywhere else, and which also supports options like delay and defer.
In addition to being able to replace functions using
td.replace, testdouble lets you replace entire modules.This is mainly useful in situations where you have a module which directly exports a function which you need to replace:
module.exports = function() {
//do something
};If we want to replace this with testdouble, we can use
td.replace('path/to/file'), for example...
var td = require('testdouble');
//assuming the above function is in ../src/myFunc.js
var myFunc = td.replace('../src/myFunc');
myFunc();
td.verify(myFunc());I think the above might be more impactful to illustrate what this looks like in practice, because the module replacement affects the subject code:
var myFunc = td.replace('../src/myFunc')
var subject = require('../src/doubleIt')
td.when(myFunc()).thenReturn(5)
var result = subject()
assert.equal(10, result)With subject
var myFunc = require('./myFunc')
module.exports = function () { return myFunc() * 2 }I worry the light bulb doesn't really go off on how clever/magical/terse the module replacement feature feels without a full example of both test & subject.
While Sinon.js can replace functions which are members of some object, it cannot replace a module in a similar fashion to this. To do this when using Sinon, you need to use another module such as proxyquire or rewire
var sinon = require('sinon');
var proxyquire = require('proxyquire');
var myFunc = proxyquire('../src/myFunc', sinon.stub());Part of why I suggested the full example above is because proxyquire isn't actually as magical as the shorthand above suggests. You have to actually pass all at once everything:
var myFunc = sinon.stub()
var subject = proxyquire('../src/doubleIt', {'./myFunc': myFunc});This is a lot more error-prone! You have to think in terms of relative-to-the-test pathing for the first arg and relative-to-the-subject pathing for each dependency you replace. That always felt like madness to me. Also it's more redundant because you have to first create the stub (so you have a reference to it), then pass it again to the proxyquire call.
Another thing worth noticing about module replacement is testdouble.js replaces the entire module automatically. If it's a function export like in the example here, it replaces the function. If it's an object containing several functions, it replaces all of them. Both proxyquire and rewire require you to individually specify what to replace and how.
We even support constructors/classes! :P
testdouble's module replacer has one fairly large downside though: It doesn't support replacing modules from node_modules. This is at least partially due to the design philosophy of testdouble, and the reasoning is explained in more detail in testdouble's documentation.
Thanks for this reminder. I waffle on this point because there are OK cases for it (example: internal modules you own). In general we want to encourage people to wrap their third-party dependencies as discussed in don't mock what you don't own
If you're using Sinon's fake timers, fake XMLHttpRequest or fake server, you'll notice they are missing from testdouble.
Fake timers are available as a plugin, but XMLHttpRequests and Ajax functionality needs to be done in a different way.
This is valid. I always felt like offering these things in a test double lib would be to encourage people to use test doubles in integration tests, which we really do not intend to do. There are better tools that are focused on just these problems like pretender
One easy solution is to replace the ajax function you're using, such as
$.post:
//replace $.post so when it gets called with 'some/url',
//it will call its callback with variable `someData`
td.replace($, 'post');
td.when($.post('some/url')).thenCallback(someData);Oh neat, you did know about thenCallback!
A common stumbling block for beginners with Sinon.js tends to be cleaning up spies and stubs. The fact that Sinon provides three different ways of doing it doesn't help.
You're telling me. This is one of the things I couldn't believe when I tried pairing with teams using Sinon.
it('should test something...', function() {
var stub = sinon.stub(console, 'log');
stub.restore();
});
//or...
describe('something', function() {
var sandbox;
beforeEach(function() {
sandbox = sinon.sandbox.create();
});
afterEach(function() {
sandbox.restore();
});
it('should test something...', function() {
var stub = sandbox.stub(console, 'log');
});
});
//or...
it('should test something...', sinon.test(function() {
this.stub(console, 'log');
//with sinon.test, the stub cleans up automatically
}));Typically the sandbox and sinon.test methods are recommended in practice, as otherwise it's very easy to accidentally leave stubs or spies in place, which can then cause problems in other tests. This can result in hard to track cascading failures.
testdouble.js only provides on way of cleaning up your test doubles:
td.reset(). The recommended way is to call it in anafterEachhook:
describe('something', function() {
afterEach(function() {
td.reset();
});
it('should test something...', function() {
td.replace(console, 'log');
//the replaced log function gets cleaned up in afterEach
});
});This greatly simplifies both set up of test duobles and cleaning up after tests, reducing the likelihood of hard to track bugs.
We've looked at the functionality in both libraries now. They both offer a similar feature set, but they have a somewhat different design philosophy from each other. Can we break this down into pros and cons?
Let's first talk about Sinon.js. It provides some additional features over testdouble.js, and some aspects of it are more configurable. This affords it some increased flexibility in more special testing scenarios. Sinon.js also uses language more familiar to those coming from other languages - concepts such as spies, stubs and mocks exist in varying libraries and are discussed in testing related books as well.
If you say Sinon has additional features on testdouble, I'd argue testdouble also has additional features Sinon lacks. Examples:
- td.callback/thenCallback
- deep matcher support (you can use any test double matcher on a deeply nested property, like
td.verify(foo({name: 'Joe', age: td.matchers.isA(Number)))) - commonJS module replacement
- argument captors
- introspection of a test double function via td.explain
As for "more configurable", do you have particular examples of configurations sinon offers that testdouble.js lacks?
The downside of this is added complexity. While its flexibility allows experts to do more things, it also means some tasks are more complicated than in testdouble.js. For those new to the concept of test doubles, it can also have a steeper learning curve. In fact, even someone like me who is very familiar with it can have trouble elaborating some of the differences between
sinon.stubandsinon.mock!testdouble.js instead opts for a somewhat simpler interface. Most of it is reasonably straightforward to use, and feels more intuitive for JavaScript, while Sinon.js can sometimes feel like it was designed with some other language in mind. Thanks to this and some of its design principles, it can be easier to pick up for beginners, and even experienced testers will find many tasks simpler to do. For example, testdouble uses the same API for both setting up test doubles and verifying the results. It can also be less error prone due to its simpler clean-up mechanism.
testdouble's biggest problems are caused by some of its design principles. For example, the total lack of spies can make it unusable for some who prefer using them instead of stubs. This is something that is very much a matter of opinion, and you may not find a problem at all. Besides this, testdouble.js is very competitive with Sinon.js despite being a much more recent entry.
Again, nitpick with the use of "spies", but it's true that if you like using them, we don't offer them.
Below is a feature by feature comparison:
| Feature | Sinon.js | testdouble.js |
|---|---|---|
| Spies | Yes | No |
| Stubs | Yes | Yes |
| Delayed stub results | No | Yes |
| Mocks | Yes | Yes (can replace whole object) |
| Promise support | Yes (in 2.0+) | Yes |
| Time helpers | Yes | Yes (via plugin) |
| Ajax helpers | Yes | No (replace function instead) |
| Module replacement | No | Yes |
| Built-in assertions | Yes | Yes |
| Matchers | Yes | Yes |
| Custom matchers | Yes | Yes |
Only issue with this table is the terminology of spies/stubs/mocks. I would say td.js does not offer mocks at all (there's just no way to do it), however you'd be right to highlight the extent to which td.js can imitate a function, object, or constructor (plus their properties) and whether Sinon can (I think it has a createStubInstance, but this is a little narrower in focus)
Both Sinon.js and testdouble.js provide a fairly similar set of functionality. Neither of them is clearly superior in this sense.
The biggest differences between the two are in their API. Sinon.js is perhaps slightly more verbose, while providing a lot of options on how to do things. This can be both its blessing and curse. testdouble.js has a more streamlined API, which can make it easier to learn and use, but due to its more opinionated design, some may find it problematic.
So which should you pick then?
Do you agree with testdouble's design principles? If yes, then there's no reason not to use it. I've used Sinon.js in many projects, and I can safely say testdouble.js does at least 95% of everything I've done with Sinon.js, and the remaining 5% is probably doable via some easy workaround.
If you've found Sinon.js difficult to use, or are looking for a more "JavaScripty" way to do test doubles, then testdouble.js might also be for you. Even as someone who has spent a lot of time learning to use Sinon, I'm inclined to recommend trying testdouble.js and seeing if you like it.
Certain aspects of testdouble.js can however give headaches to those who know Sinon.js or are veteran testers otherwise. For example the total lack of spies can be a deal breaker. For experts and those who want the maximum amount of flexibility, Sinon.js is still a great choice.
Obviously, I can't disagree with your opinions! Thanks for sharing them, I apprecaite it.