layered-cake-of-testing.md

The layered cake of testing

1) Static code analysis

This includes linting, type checking, code-flow analysis ... TypeScript, Reason, Flowtype, ESLint, TSLint are all examples of static code analysis.

When to run

These should run as frequently as possible. Optimally, they are continuously running while developing.

Code smells

1. Ambigious types on large objects, like TypeScript's any
2. Overly typing entities, rather than allowing for type inference
3. Not using the compiler's build as a pass-fail mechanism for your pipeline

Coverage

As close to 100% of the codebase should be evaluated.

Recommendation

Although ReasonML is a great solution due to being completely type safe and sound, it's not the most well-known and does have a cognitive challenge to its very FP nature. My default recommendation would be TypeScript, due to it being type safe (though not sound), but is very well-known and is written and feels very much like JavaScript with types, and not a completely different language.

2) Unit testing

These are super light-weight tests that run assertions against the most atomized parts of code, the units. These are just testing the result for a given input for a function. These functions/units are pure stateless functions, so there is no need to mock or stub anything other than the input/arguments provided. In other words, there is nothing external to the unit tested that's needed, no servers, no APIs, no DOM, no HTTP ... nothing.

Critical focus

1. Speed
2. Simplicity
3. Low effort
4. Great dev experience
5. Verifying internal structures

When to run

These should be run, at minimum, at every git commit.

Code smells

1. Stubbing functions or methods that have side-effects
2. Mocking (other than the arguments given) additional objects and global variables
3. beforeAll for testing "setup"
4. afterAll for testing "teardown"
5. You can't just simply run the test node myunit.test.js without running other processes or servers

Coverage

Coverage is a dangerous proposition as it can lead to low-value testing just for the numbers. I think the majority of the app's functionality should have unit tests, but whether that's 60% or 90% is quite arbitrary and quickly surpases the point of diminishing return.

My view is any function that has hand-written logic in it should be both positive and negative testing. Here are some easy rules that apply to all testing:

1. Only test hand-written logic; don't test native functions, or methods from external libraries or frameworks
2. The more complex the hand-written logic, the more priority should be placed on testing
3. The more critical or "hot path" the function, the more priority should be placed on testing

The focus should be on the value (quality) of the tests, not the number of tests (quantity).

Recommendation

My very favorite unit testing library is tape, but it does require quite a bit of manual configuration at scale. For better "out of the box" configuration for scale, Jest is a great alternative that breaks the capability of running the unit test file alone, but the benefits that come with it are too great to ignore.

Unit tests should reside right along the files they test. Example:

- todo-list
  | - todo-filter.ts
  | - todo-filter.test.ts

The Jest library has everything needed for unit testing. There should be no additional libraries or frameworks needed.

3) Integration testing

These are more similar to unit tests than they are other types of tests. These tests are still input-output driven, but since it tests the composition of units, it tests at a higher, more complex level, so mocking/stubbing is almost always required.

Mocking and stubbing should be done without the need for additional processes, servers or APIs. The mocks or stubs should be static and side-effect free.

Critical focus

1. Speed
2. Completeness
3. Good dev experience
4. Verifying internal structures

When to run

These should be run at minimum at every git push.

Code smells

1. beforeAll for testing "setup"
2. afterAll for testing "teardown"
3. You can't just simply run the test node myunit.test.js without running other processes or servers

Coverage

All full suites of units should be tested up to the point of needing to run additional processes, servers or needing the a real DOM. These should test as much surface area as possible with as few tests as possible and should be contained to just testing the local codebase or module. You never test the functionality of external/third-party functionality.

Recommendation

Continuing with the Jest recommendation from above, Jest can also handle integration testing with its ability to mock modules.

4) View-component testing

These are analogous to unit or integration tests for React or Vue components. These can be run along with your unit and integration tests. They are input and output based and do not require additional process, servers or live DOM.

Critical focus

1. Speed
2. Simplicity
3. Low effort
4. Good dev experience
5. Verifying internal structures

When to run

These should be run at minimum for each PR to the main branch, but could be run at every git push.

Code smells

1. You’re testing a native library or framework function
2. You’re testing something that is too simple
3. Heavy use of beforeAll for testing "setup"
4. Heavy use of afterAll for testing "teardown"

Coverage

All view-components that have complexity, logic, derived data or branching should be tested.

Recommendation

Jest will continue our testing capabilities with the addition or JSDom for mocking the DOM in order for us to synthetically render our components for testing.

5) End-to-end testing (aka functional testing/e2e)

This type of testing is to mimic user interaction to the closest degree. It requires a running app and running APIs to simulate the real environment as much as possible. Tests require a Web driver to mimic clicking, typing, navigating and submitting data to a running application against appropriate testing stages.

Critical focus

1. Completeness
2. Predictability
3. Good dev experience
4. Verifying the API contracts between external, dependent systems
5. Replicating real UX flows

When to run

These should be run for each PR to the main branch.

Code smells

1. You’re testing a native library or framework function
2. You’re testing something that is too simple
3. You’re testing the DOM or browser function

Coverage

Critical user flows, security and privacy related features, anything that’s considered high-risk. The intention of a e2e test is to verify the entire user flow within a running environment. Avoid testing functionality that can more easily be tested though unit or integration tests or redundantly testing something that’s already well covered in your unit or integration tests.

Recommendation

TBD

Additional references:

• Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
• Learn the smart, efficient way to test any JavaScript application

You're can't just simply run the test node myunit.test.js without running other processes or servers

This seems to exclude a lot unit testing frameworks since they rely on something external to the test providing globals (test, suite, etc). Also if you take this literally you also can't run any sort of compilation, which rules out using typescript, esmodules, etc. This seems like a huge limitation, so what's the payoff?

They are input and output based and do not require additional process, servers or live DOM.

Do you include JSDOM in your definition of "live DOM"?

@cameron-martin

This seems to exclude a lot unit testing frameworks since they rely on something external to the test providing globals (test, suite, etc).

I actually prefer testing frameworks that don't do this (like tape.js), but Jest has become such a common framework (and a good one at that) that I don't include the testing functions themselves that are passed into the test file when executed. For example: describe and it ... really the point is that the code you're testing (not the test file) doesn't require anything outside of itself. Think Functional Programming and purity.

Also if you take this literally you also can't run any sort of compilation, which rules out using typescript, esmodules, etc. This seems like a huge limitation, so what's the payoff?

Don't take things too literally :) The point is that you reduce the environmental overhead to run tests. Unit, Integration and View Tests should be as lightweight as possible, so they are simple to write and fast to run. At the end of the day, these tests should be executable within nothing more than a single Node process. If you find yourself needing to run additional servers for APIs or a headless browser or anything like that, they are no longer one of the three types of test mentioned. They are now Functional or E2E tests.

Do you include JSDOM in your definition of "live DOM"?

No, not for View Tests. JSDOM is not a live DOM, but a DOM that has been mocked for testing purposes, which is why I feel View Tests that render to a mock DOM are basically integration tests for the rendering components. What I mean by a "live DOM" is like a headless browser or Web Driver, like Selenium.

Makes sense, thanks for clarifying.

What're your thoughts on performance testing, both small-scale (e.g. does this algorithm still run fast on large inputs) and large-scale (e.g. is the time to interactive still acceptable)? This is something that I've never implemented before but thought about a bit, and would be interesting to know your opinions on its usefulness and exactly how it was implemented if you've ever implemented/used it.

@cameron-martin

What're your thoughts on performance testing, both small-scale (e.g. does this algorithm still run fast on large inputs) and large-scale (e.g. is the time to interactive still acceptable)? This is something that I've never implemented before but thought about a bit, and would be interesting to know your opinions on its usefulness and exactly how it was implemented if you've ever implemented/used it.

I've done a limited amount of performance testing within the above declared layers of testing (most perf testing was done by L&P testing infrastructure), but I have implemented performance testing when something was caught and reported as a performance bug. I implemented JS Perf-style unit and integration testing to ensure performance of algorithms did not fall below a threshold. But, I've never preemptively tested something for performance within the automated testing layers, and I'm not sure I'd recommend it as it can easily lead to micro- or pre-optimization.

Being that performance degradation is almost always going to be in your asynchronous, over-the-wire touch points (latency between entities), I'd leave most of it to L&P testing or production performance monitoring. When you get reports of perf issues, and you find it's within an algorithm/some iterative code, that's when you implement said perf tests within your testing layer, IMHO.