Principles of perfect testing
A testing manifesto
The perfect test is concise, unambiguous, safe, with easily understandable logging.
- It is named to concatenate
- It flows from 'it'
- Describe what it does, not what it should do
- It asserts semantically
- When useful, it favours custom matchers
- It keeps state on context, shunning local variables
- It is mostly traditional functions(){}
- One expect per test
- Prefer tree-shaped specs over list-shaped specs
- it is ok to generate the tests
- it checks invariants at all times
It is named to concatenate
Most test runners refer to a test by concatenating any describe labels and the it label. If you choose names that concatenate well your logs will benefit from sensible descriptions.
Don't write:
describe( 'good dog suite', function() {
describe( 'test giving it cookies', function() {
it( 'supports eating them with glee', function() {
expect( this.dog ).toBeADoggoWith( {eating: cookies, mood: glee} )In a log this message reads poorly:
Error: good dog suite test giving it cookies supports eating them with glee
✘ expected mood to be glee but was unappreciative
Instead, write:
describe( 'A good dog', function() {
describe( 'when given cookies', function() {
it( 'eats gleefully', function() {
expect( this.dog ).toBeADoggoWith( {eating: cookies, mood: glee} )This gives a much better failure message:
A good dog when given cookies eats gleefully
✘ expected mood to be glee but was unappreciative
It flows from 'it'
The test name should flow cleanly from the it.
Don't do:
describe('a good dog', function() {
it('check that it rolls over', function() {This does not read cleanly because in English 'it check that it rolls over' is not valid.
Instead do:
describe('a good dog', function() {
it('rolls over', function() {In English, 'it rolls over' is valid, precise, and concise.
Describe what it does, not what it should do
Don't write:
it('should do a merry dance', function () {
it('is expected to do a merry dance', function () {
it('supports merry dancing', function () {
it('can do a merry dance', function () {By using terms like 'should', 'can', or 'expected':
- It is ambiguous if this behaviour is optional or required
- You're writing unnecessary boilerplate
It is more precise and concise to write:
it('dances merrily', function() {It asserts semantically
Avoid asserting directly on comparisons between primitives:
expect( this.dog.mood ).toBe( 'glee' )If this fails the error message isn't great:
✘ expected 'sad' to be 'glee'Why would you expect sad to be glee? These are opposites! This problem gets worse with assertions on booleans:
expect( this.dog.isHappy ).toBe( true )The error message expected true to be false is absurd. True can never be false!
Instead, consider writing this:
expect( this.dog ).toHaveProperties( {mood: 'glee'} )By asserting at a higher level of abstraction a richer messages is possible:
✘ expected Dog 'fido' to have property 'mood' of 'glee' but was 'sad'`.
When useful, it favours custom matchers
- jasmine custom asserters
- [mocha custom asserters]
- Chai helpers
It keeps state on context, shunning local variables
Tests should be independent of each other.
Variables allow state can leak between tests.
This can cause unpredictability where tests pass or fail depending on other tests.
describe('flying a drone', function() {
const droneSpeed;
const drone;
beforeEach(function(){
drone = DroneFactory.manufacture('one drone');
})
it('goes fast', function() {
droneSpeed = '10m/s';
drone.flyAt( droneSpeed ).to( anywhere )
expect( drone ).toBeAccelerateUpToASpeedOf( '10m/s' )
});
// this test will pass when the whole suite is run, but fail
// if it is run in isolation because droneSpeed is undefined.
it('spies on people', function() {
drone.flyAt( droneSpeed ).to( 'enemies' )
expect( drone ).canSee( 'enemies' )
});
})The popular Javascript test suites (list them) create a new test context object for each it, which is first into
your beforeEach and it as the context (this). If you place the state on this object it is much more difficult
to create leaky tests.
describe('flying a drone', function() {
beforeEach(function(){
// the beforeEach will be called twice, each time with a different object as 'this'.
this.drone = DroneFactory.manufacture('one drone');
})
it('goes fast', function() {
this.droneSpeed = '10m/s';
this.drone.flyAt( this.droneSpeed ).to( anywhere )
expect( this.drone ).toBeAccelerateUpToASpeedOf( '10m/s' )
});
// this test is still wrong, but now it will fail consistently because
// this.droneSpeed is always undefined.
it('spies on people', function() {
// the 'this' object in here is not the one given to 'it goes fast'
// so this.droneSpeed will be undefined, allowing the mistake of not
// initialising to be easily found
this.drone.flyAt( this.droneSpeed ).to( 'enemies' )
expect( this.drone ).canSee( 'enemies' )
});
})If your functions don't use any external variables, they are now pure functions, because the 'this' context is really just another parameter to your function. A pure function will always behave in the same way when called with the same input, meaning that they are safe to declare and re-use in any context:
describe('flying a drone', function() {
beforeEach(function(){
// the beforeEach will be called twice, each time with a different object as 'this'.
this.drone = DroneFactory.manufacture('one drone');
this.dog = Pets.get('dog');
this.fastSpeed = '10m/s';
})
function itFliesReallyFast() {
it( 'flies really fast', function() {
expect( this.drone ).toBeAtSpeed( this.fastSpeed );
});
}
describe( 'when joystick pushed forwards', function() {
beforeEach( function() {
this.drone.inputFromJoystick( 'go', this.fastSpeed )
} )
itFliesReallyFast();
});
describe( 'when chased by a dog', function() {
beforeEach( function() {
this.dog.chase( this.drone )
} )
itFliesReallyFast();
})
})It is mostly traditional functions(){}
As a gotcha, the this inside describe blocks is not the this you are looking for.
describe('flying a drone', function() {
// the describe function is only called once while the tests are loaded, and never while
// they are ran. Even if it worked, there is no way to store per-test state here
this.does = 'not work';
beforeEach(function(){
this.isWhere = 'initialisation belongs';
})
});As a further gotcha, arrow functions ()=>{} override normal context passing and will
not work here
describe('a bad spec file', () => {
// 'this' here will be bound to the enclosing 'this'
it( 'adds new state', () => {
// Here 'this' will be bound to the same enclosing 'this',
// not the per-it test object.
this.testState = 42;
expect( this ).toHaveProperties( {testState: 42} );
})
it( 'allows that state to influence other tests', () => {
// This test will either pass or fail depending on the order the tests are run in,
// if there is sharding, or if tests are being run in isolation
expect( this ).not.toHaveAPropertyNamed( 'testState' );
})
});Note that we don't care what the context is in the describe blocks since they are run when the spec is loaded rather than when it is executed. So for compactness it is safe to use arrow functions here:
describe('arrow functions are harmless for describe', () => {
it( 'should be long-form functions for "it", "beforeEach", and "afterEach"' function() {
})
});One expect per test
Yeah yeah, write this thing or something
If this seems like a lot of typing, you're probably doing too much in the it and not enough in the beforeEach
The exception to this rule is invariant testing, which should be done constantly.
Prefer tree-shaped specs over list-shaped specs
When testing a state machine (or user journey) bugs relating to routes through state transitions are difficult to comprehensively test.
In a fairly typical user journey you might do step A, and B, and then finish with
steps C1, C2, or C3. With assertions at each step of course. We can write the
possible journeys as [[A, B, C1], [A, B, C2], [A, B, C3]].
In reality there are probably two ways to do B, so now we we have nine cases:
[
[A],
[A, B1],
[A, B1, C1],
[A, B1, C2],
[A, B1, C3],
[A, B2]
[A, B2, C1],
[A, B2, C2],
[A, B2, C3]
]Treating this as a flat list, testing them is a lot of typing:
describe('the ABC machine', function() {
it( 'performs step A', function () {
expect( this.stateMachine.doStepA() ).not.toBeAMisstep();
} )
it( 'performs step B1 after A', function () {
this.stateMachine.doStepA();
expect( this.stateMachine.doStepB1() ).not.toBeAMisstep();
} )
it( 'performs C1 after steps A and B1', function () {
this.stateMachine.doStepA();
this.stateMachine.doStepB1();
expect( this.stateMachine.doStepC1() ).not.toBeAMisstep();
} )
it( 'performs C2 after steps A and B1', function () {
this.stateMachine.doStepA();
this.stateMachine.doStepB1();
expect( this.stateMachine.doStepC2() ).not.toBeAMisstep();
} )
// ... and the other 5 test cases!
})For multi-step tests a tree is a more natural representation:
[A,
[B1,
[C1, C2, C3]
],
[B2,
[C1, C2, C3]
]
]describe and forEach calls can be combined to create a spec that follows the
tree structure of the problem domain:
describe('given step A is done', function() {
beforeEach( function () {
this.stepA = this.stateMachine.doStepA();
} )
it( 'did not-miss-step at A', function () {
expect( this.stepA ).not.toBeAMisstep();
} )
describe('given step B1 is done', function() {
beforeEach( function () {
this.stepB = this.stateMachine.doStepB1();
} )
it( 'did not-miss-step at B1', function () {
expect( this.stepB ).not.toBeAMisstep();
} )
it( 'does not mis-step at C1', function () {
expect (this.stateMachine.doStepC1).not.toBeAMisstep()
} );
it( 'does not mis-step at C2', function () {
expect (this.stateMachine.doStepC2).not.toBeAMisstep()
} );
it( 'does not mis-step at C3', function () {
expect (this.stateMachine.doStepC3).not.toBeAMisstep()
} );
})
describe('given step B2 is done', function() {
beforeEach( function () {
this.stepB = this.stateMachine.doStepB2();
} )
it( 'did not-miss-step at B2', function () {
expect( this.stepB ).not.toBeAMisstep();
} )
it( 'does not mis-step at C1', function () {
expect (this.stateMachine.doStepC1).not.toBeAMisstep()
} );
it( 'does not mis-step at C2', function () {
expect (this.stateMachine.doStepC2).not.toBeAMisstep()
} );
it( 'does not mis-step at C3', function () {
expect (this.stateMachine.doStepC3).not.toBeAMisstep()
} );
})
}it is ok to generate the tests
The above example is still quite long, but because the 'its' are stateless they can be extracted:
// extracted function with common 'it's
function itWorksForAnyCombinationOfC() {
it( 'does not mis-step at C1', function () {
expect (this.stateMachine.doStepC1).not.toBeAMisstep()
} );
it( 'does not mis-step at C2', function () {
expect (this.stateMachine.doStepC2).not.toBeAMisstep()
} );
it( 'does not mis-step at C3', function () {
expect (this.stateMachine.doStepC3).not.toBeAMisstep()
} );
}
describe('given step A is done', function() {
beforeEach( function () {
this.stepA = this.stateMachine.doStepA();
} )
it( 'did not-miss-step at A', function () {
expect( this.stepA ).not.toBeAMisstep();
} )
describe('given step B1 is done', function() {
beforeEach( function () {
this.stepB = this.stateMachine.doStepB1()
} )
it( 'did not-miss-step at B1', function () {
expect( this.stepB ).not.toBeAMisstep();
} )
itWorksForAnyCombinationOfC()
})
describe('given step B2 is done', function() {
beforeEach( function () {
this.stepB = this.stateMachine.doStepB2()
} )
it( 'did not-miss-step at B2', function () {
expect( this.stepA ).not.toBeAMisstep()
} )
itWorksForAnyCombinationOfC()
})
}Recursion, init!
// your-test-utils.js
function itNeverMisstepsOnAnyPathThrough( tree ) {
describe( `given step ${tree.step} is done`, () => {
beforeEach( () => {
this.step = this.stateMachine[`doStep${tree.step}`]();
})
it( `did not miss-step at ${tree.step}`, () => {
expect( this.step ).not.toBeAMisstep();
})
(tree.followedBy || []).forEach( itNeverMisstepsOnAnyPathThrough )
});
}// state-machine.spec.js
itNeverMisstepsOnAnyPathThrough(
{step:'A', followedBy:
[
{step: 'B1', followedBy:
[{step:'C1'}, {step:'C2'}, {step:'C3'}]
, {step: 'B2', followedBy:
[{step:'C1'}, {step:'C2'}, {step:'C3'}]
]
}
)Once you have this kind of framework expanding the tests is easy:
// 54 journeys tested using a compact syntax
itNeverMisstepsOnAnyPathThrough(
{steps:['A1', 'A2', 'A3'], followedBy:
[
{steps: ['B1', 'B2', 'B3'], followedBy:
{steps: ['C1', 'C2', 'C3', 'C4', 'C5', 'C6']}
]
}
)Of course, this spec is quite abstract so let's look at a more realistic example:
describe( 'wizard', function() {
// no problem to mix with 'normal' tests
beforeEach( function() {
} );
itNeverMisstepsOnAnyPathThrough(
{
name:'cast fireball spell',
step: function(){this.player.cast('fireball'},
expects: function(){expect(this.player.magicalPower).toBe(50)},
followedBy:
[
{step: ['at door', 'in north direction'],
expects: function(){expect(this.world.door).toBe('aflame')},
followedBy:
{steps: ['C1', 'C2', 'C3', 'C4', 'C5', 'C6']}
{steps: ['B1', 'B2', 'B3'], followedBy:
{steps: ['C1', 'C2', 'C3', 'C4', 'C5', 'C6']}
]
}
)
});