- Theoretically, what does the perfect test look like? Obviously, the perfect test doesn't actually exist, any more than an ideal gas, or a perfect sphere BUT if it did, what properties would it have?
- It would verify all the things. And by all the things, literally all the things
- every user interaction,
- every workflow,
- every browser
- every level of user ability
- every possible combination and sequence of events in the system.
- Such a perfect test would have an incredible impact in terms of saving our team cost by ensuring that no bugs escaped to production, and that our products worked all the time, every time, for every one.
- with the concept of a perfect test as a maximum, we could write every test on a spectrum, from a throw-away test, up to the theoretical limit implied by our our "perfect" test.
┌─────────────┐
───────────────────────────────────────────────────────────────────▶ │ ∞ (perfect) │
│ Test Size └─────────────┘
│ Small -> Big
worthless
- "normal" test designations fit along this spectrum
- unit, acceptance, integration aren't absolute values
┏━━━━┓ ┏━━━━━━━━━━━┓ ┏━━━━━━━━━━━┓ ┏━━━━━━━━━━━┓
┃Unit┃ ┃Integration┃ ┃Acceptance ┃ ┃ E2E ┃
┗━━━━┛ ┗━━━━━━━━━━━┛ ┗━━━━━━━━━━━┛ ┗━━━━━━━━━━━┛
│ │ │ │
│ │ │ │
▼ ▼ ▼ ▼ ┌─────────────┐
──────0───────────────0─────────────────0─────────────────0────────▶ │ ∞ (perfect) │
│ Test Size └─────────────┘
│ Small -> Big
worthless
- Test designations are fuzzy in meaning and everybody is always arguing about them.
- What distinguishes a unit test from an "acceptance" test isn't a fixed set of attributes, rather it's how big they are in relation to each other.
- So what keeps us from writing the perfect test?
- If we could write a perfect test every time, we'd do that right?
- It's hard. Why?
- creating a simulated environment is hard.
- slow especially if there are API requests involved
- difficult to isolate: symptoms include virtual machines everywhere.
- they take a long time to run
- can be very flaky
- conclusion: The decision isn't 1-dimensional.... Test size comes with a cost
▲
│ ┏━━━━━━━━━━━┓
│ ┃ E2E ┃
│ ┗━━━━━━━━━━━┛
│ │
│ │
│ ▼
│ 0
│
│ ┏━━━━━━━━━━━┓
│ ┃Acceptance ┃
│ ┗━━━━━━━━━━━┛
Time │ │
+ │ │
$$$ │ ┏━━━━━━━━━━━┓ ▼
│ ┃Integration┃ 0
│ ┗━━━━━━━━━━━┛
│ │
│ ┏━━━━┓ │
│ ┃Unit┃ ▼
│ ┗━━━━┛ 0
│ │
│ │
│ ▼
│ 0 ┌─────────────┐
free ──── └──────────────────────────────────────────────────────────────────▶ │ ∞ (perfect) │
│ Test Size └─────────────┘
│ Small -> Big
worthless
- Time is money, and typically, the bigger the test, the bigger the cost.
- time to develop a test suite.
- training developers on how to write tests effectively
- speed of write / run / feedback loop
- time to maintain that test suite
- long running tests cost precious development cycles that can add up to days, weeks and or even months
- flaky tests cost precious development cycles that do the same.
- curve is non-linear, and with current techniques, returns are diminishing
- time to develop a test suite.
- TODO: would love to somehow overlay the cost of not testing onto this graph.... i.e. just unit tests are cheap AF, but introduce cost in the form of regressions and re-work.
- Given infinite resources for testing, we'd write the perfect test.
- with limited resources in both cash, but also time, we write smaller, less valuable tests.
- So what's the best test we can write?
- Answer: The biggest possible test that we can afford (for whatever our budget is).
- If the cost of writing bigger tests were lowered across the board, then it follows that we'd spend a whole lot more time writing bigger tests
- Enter BigTest
- big test is a suite of tools designed to drastically lower the cost of writing big tests.
- host of primitives make it easy to meet your test suite wherever it is. Use one of them. Use them all.
- primitives for asynchrony make all levels of testing more robust and FAST.
- if a test takes 2 seconds to run vs 200ms it costs several orders of magnitude more for all the reasons
- 5 minutes vs 50 minutes is all the difference in the world.
- primitives for user interactions allows tests to run in any browser against any framework.
- primitives for simulating the network
- network is a high cost, for low reward (almost feel like this is a separate blog post)
- separate runtime, language.
- usually needs database + other external services.
- not isolatable, no
yarn install && yarn test - huge cost in terms of speed
- network is a high cost, for low reward (almost feel like this is a separate blog post)
▲
│
│
│
│
│
│
│
│
│
│
│
Time │
+ │
$$$ │
│ ┏━━━━━━━━━━━┓
│ ┏━━━━━━━━━━━┓┃ E2E ┃
│ ┏━━━━━━━━━━━┓ ┃Acceptance ┃┗━━━━━━━━━━━┛
│ ┏━━━━┓ ┃Integration┃ ┗━━━━━━━━━━━┛ │
│ ┃Unit┃ ┗━━━━━━━━━━━┛ │ │
│ ┗━━━━┛ │ │ ▼
│ │ │ ▼ 0
│ │ ▼ 0
│ ▼ 0
│ 0 ┌─────────────┐
free ──── └──────────────────────────────────────────────────────────────────▶ │ ∞ (perfect) │
│ Test Size └─────────────┘
│ Small -> Big
worthless
- BigTest will go wherever it needs to, to lower the cost of testing, so that you can afford to test a whole lot bigger every day.