Use this guide as a point of reference for tips on how to define your business-driven development test plans, or for any unit tests while building your application functionality.
The code and frameworks below are for JavaScript, but the ideas here can work for any unit test framework or BDD development style.
- Unreliable: cannot map acceptance criteria to tests
- Failures will compound as tests rely on previous ones passing
- Confusing: hard to determine setup steps from action steps, and action steps from validations
- Tests that must pass in order may not actually be tests
- Mixing setup, actions, and validations out-of-order leads to missed/undocumented requirements and defects
- Test plans have less re-usability
- Automated test files are brittle and unreliable
- Hanging data may continue to exist in the environment
- The suite needs to be configured to end execution on first failure in order to avoid additional problems in the environment
A non-isolated set of tests
Story: The checkout flow allows me to create an account after I complete my order
checkout.page.create.account.tests
- Test setup:
- Be a new customer without an account
- Test 1: Add items to cart
- Add the "55-inch TV" to cart
- Add the "Streaming service dongle" to cart
- Click checkout button
- Verify you are on the Checkout page
- Test 2: Enter shipping information
- Complete Test 1
- Submit shipping information on the page
- Verify you are on the Payment page
- Test 3: Enter payment information
- Complete Test 2
- Submit payment information on the page
- Verify you are on the Order Complete page
- Test 4: Verify order was received for a new user
- Complete Test 3
- Verify the backend contains make sure your order was submitted with the correct items
- Verify that the new account button is displayed on the Order Complete page
- Test 5: Create a new account on completion page
- Complete Test 4
- Click the "Create New Account" button
- Fill in account details
- Submit
- Verify you are signed in with your new account
And repeat for many more sets of feature tests with different end-state validations...
- We define a static structure that is used for each test:
- Arrange to prepare for testing a feature
- Act upon the test setup
- Assert the results of the action(s)
- Maintainability: refactoring occurs less and is easier to handle
- Only a single point of failure will exist, as test execution ends with the first step failure
- Failures are individualized and should not directly affect other tests
- Separate tests can map to individual requirements
- Tests and files can be made smaller overall
- You become a better tester by enforcing standards on your plans
Using an isolated set of tests
Story: The checkout flow allows me to create an account after I complete my order checkout.page.tests
(These tests should already exist in the same page for earlier features!)
-
Test: Verify I can add items to the cart
- Add the "55-inch TV" to cart
- Add the "Streaming service dongle" to cart
- Click checkout button
- Verify you are on the Checkout page
-
Test: Validate that shipping information can be submitted successfully
- Add items to cart
- Click checkout button
- Enter all shipping information correctly on the checkout page
- Submit
- Verify you are on the Payment page
- Verify the shipping information was saved on the page as read-only
-
Test: Validate that payment information can be submitted successfully
- Add items to cart
- Checkout
- Complete shipping information during checkout
- Enter all payment information correctly on the Payment page
- Verify the backend contains make sure your order was submitted with the correct items
- Verify that the new account button is displayed on the Order Complete page
-
Test: Validate that payment information can be submitted successfully
- Test setup: Be a new user
- Add items to cart
- Complete checkout to submit order
- Click the "Create New Account" button
- Fill in account details
- Submit
- Verify you are signed in with your new account
Only repeat steps per tests for each new test added!
Low level tests should always be isolated. Most integration tests can be isolated. As you get into higher-levels, you might require fewer rules needed around isolating every test. Still, knowing your test types will help determine how to build your plan!
Test type examples
This is a non-comprehensive list of types, but these are the most common ones!
- Development (low-level):
- Unit/spec: low-level tests to validate individual functionality of the code
- isolated
- easy to automate
- fast
- can use stubs and mocks to replicate external services and requests
- Examples: validating code functions work; error handling; data transformation; items that exist only within the codebase or application
- Unit/spec: low-level tests to validate individual functionality of the code
- Integration (mid-level):
- Functional: tests that the application code meets the technological and business requirements
- isolated
- automate-able
- involves user stories
- Accessibility: a functional test that validates that the application meets accessibility guidelines,
like W3C or A11y
- mostly isolated, can be non-isolated
- partially automate-able
- manual using special tools
- Exploratory: a "deep dive" into the application to interact with it in ways that may expose bugs in the code
- does not need a full plan besides setup and some guidelines
- manual
- Example: clicking a button or reloading a page many times; cancelling network requests; turning data off/on; performing many normal flows; combinations of all of the listed and more
- Functional: tests that the application code meets the technological and business requirements
- High-level:
- End-to-end: a set of validations to ensure the main functionality of the application as a whole within the
environment
- mainly non-isolated
- mainly manual
- slower to perform
- can be used in multiple environments where external services are attached
- end-to-end tests can be considered "isolated" if the entire process is written out per test and individualized
- Acceptance: tests performed by a user/customer to verify that the actual application meets their requirements
- isolated
- automate-able
- involves user stories
- Smoke: high-level validation of the core application functionality is working after changes are made to the
code. Should consist of a small set of the most important functional tests
- isolated
- automate-able
- End-to-end: a set of validations to ensure the main functionality of the application as a whole within the
environment
An simple and ideal QA process would consist of the following environments:
- development: unit tests to ensure each piece of code is functional
- integration/staging: integration (functional, accessibility, etc). to make sure your application meets the requirements
- acceptance: end-to-end testing that ensures all services interact together successfully
- production: light smoke tests or metrics that ensure your application is usable
However, what is ideal is not always what is real for your team. Maybe you don't have an acceptance environment, or maybe you can not clean up data you create. It's easier to shift tests down to a lower environment where you have more control rather than force a higher envrionment to behave as you want! There should be a level of testing in every environment, but know your limitations.
Limitations of test data
- Stubbing/mocking services tests that your application code can use those services, but it doesn't test the actual integration! Useful for unit tests.
- Using fake data with actual service integrations means that the full application is working correctly, but only in that environment! Useful for functional user story-type tests. Using real data with actual service integrations means that the full application is working correctly, but you may not be able to cleanup that data! Useful for end-to-end testing in higher environments.
Limitations of environments
- development environments: useful for testing your development code alone without service dependencies: does not prove that services integrations will be correct!
- lower integration/staging environments: useful for testing the application on its own or with "development" based services: you can test that your application works in that environment with those particular services, but does not ensure higher environments until production will be successful!
- production: mainly untestable, but light testing here proves that your application works with real services. You must depend on tests in lower environments being successful to know that the product is successful!
The testing pyramid allows you to scale your test plan in such a way that fewer tests are needed as you ascend into higher environments closer to a production level:
- Largest set consists of unit tests
- Medium set of integration tests
- Smallest set for high-level types: end-to-end, exploratory, & acceptance
See BrowserStack's guide to the testing pyramid .
This format allows you to organize your steps in order of each step that must be performed to fully complete a test:
- Arrange: set up your test by creating data, prepping the test environment, and getting to a state where you can begin your necessary actions
- Act: perform the essential action(s)
- Assert: validate the expected behavior against the actual behavior of the actions.
Also known as Given/When/Then in Cucumber's Gherkin language.
Use when curating Integration-level tests and above, like functional testing of user stories. Be concise in writing your test steps! Write test steps with only the needed information and keep them simple. A step should accomplish a single goal: for example, it could be just one action, or a flow through a part of an application, depending on how you write your step!
Using simple actions for steps
An existing user logging in goes to their dashboard (Arrange)
- open the URL for "https://dev.my-website.com"
- click the "Login" button to open the form
(Act)
- enter "my-email@host.com" into the email field within the login form
- enter "thisIsAPassW0RD!" into the password field within the login form
- click the submit button
(Assert)
- validate that the page url contains "/dashboard"
- validate that the user's first name appears as "George"
- validate that the user's last name appears as "Washington"
Describing flows as steps in Gherkin
Feature: My application
Scenario: Completing registration takes the user to their dashboard
Given I open the main page of the application
And I navigate to the registration page
When I complete user registration
Then I should be logged into my dashboardWhen writing unit and integration level tests, it's best to write each test as its own unique entity. A test can share steps and functions with others, but should not rely on an earlier test passing in order for itself to pass.
Note that some unit test frameworks allow for certain scoped tests to be run as non-isolated in case you ever need to deviate from the norm!
Don’t use one test to validate the entirety of the acceptance criteria! Adding more tests to validate each part of the acceptance criteria is much more manageable. Each test can get a unique test ID and also reference a callback to the feature ticket ID. A good rule is that business feature or user story can have multiple tests, but one test should not validate multiple features.
- See "Problems with non-isolated tests" for how shared-state/non-isolated tests can repeat themselves across files
- See "Improvements with isolated tests" for an isolated test file example!
Test-driven development means you write tests (or test descriptions) first, and then add your development code second. This way, rules are enforced on the application before code is created.
The process is:
- Write a test that enforces the code: the test will fail immediately.
- Write actual code that will pass the test.
- Refactor code as necessary that still passes the test.
- Repeat 1-3 for additional functionality and features.
BrowserStack has a great post on Test-Driven Development that explains the process in more granular detail. Refer there for a better understanding!
Here, assume we are using MochaJS as our unit test framework. Most unit test frameworks in other languages, like Ruby's RSpec, will behave similarly.
As well, see Codecademy's guide to Mocha objects for more information!
Variables and hooks defined in outer scopes are passed down to inner scopes. This way, you can share state
between describes and contexts while also maintaining isolation for your unit tests! It's a win-win!
Mocha hierarchy example
describe('An outer describe scope', function () {
before(function () {
//before hook code
})
describe(`An inner describe scope`, function () {
beforeEach(function () {
this.foo = 'bar';
//other beforeEach code
})
const hello = 'hello';
context(`Context #1`, function () {
it(`should be true`, function () {
//runs the "before" hook in "An outer describe scope"
//runs the "beforeEach" hook in "An inner describe scope"
////can use `this.foo`
////can use `const hello = 'hello'`
//...test actions
//...assertions
});
it(`should not be false`, function () {
//runs the "before" hook in "An outer describe scope"
//runs the "beforeEach" hook in "An inner describe scope"
////can use `this.foo`
////can use `const hello = 'hello'`
//...different test actions
//...different assertions
});
});
context(`Context #2`, function () {
specify.skip(`A way of writing a unit test`, function () {
//runs the "before" hook in "An outer describe scope"
//This context has no hooks or assigned variables!
//TODO: fill in with test code!
});
})
})
})- Hooks allow for repeatable actions to be performed for both pre- and post-test execution
- Their placement in each scope allows hooks to be shared or separated based on the hierarchy of the scope
For Mocha, and most unit frameworks, these are common hooks available and how they execute:
before: runs once BEFORE an entiredescribeorcontextscopebeforeEach: runs BEFORE each unit test begins- Actual test code occurs here
afterEach: runs AFTER each unit test endsafter: runs once AFTER an entiredescribeorcontextscope
A Context is an object that collects data to be used
throughout the entire lifecycle of a unit test:
Details
Move from variable declaration:describe('a describe scope', function () {
let counter = 0;
const user = {firstName: "Ernest", lastName: "Hemingway"}
const incrementCounter = () => {
counter++;
}
it('can use a test context', function () {
incrementCounter(); // => counter = 1;
console.log(user); // => {"firstName": "Ernest", "lastName": "Hemingway"}
});
})To using contexts:
describe('a describe scope', function () {
beforeEach(function () {
this.counter = 0;
this.user = {firstName: "Ernest", lastName: "Hemingway"}
this.incrementCounter = function () {
this.counter++
}
});
it('can use a context object', function () {
this.incrementCounter(); // => this.counter = 1;
console.log(this.user); // => {"firstName": "Ernest", "lastName": "Hemingway"}
});
it('also has its own context obect', function () {
this.incrementCounter(); // => this.counter = 1;
console.log(this.user); // => {"firstName": "Ernest", "lastName": "Hemingway"}
});
})Integration testing should definitely take use of the Arrange-Act-Assert format as these tests can be used to document business features. This is known as behavior-driven development.
Cucumber is a framework for behavior-driven development (BDD), where tests exist as scenarios representing pieces of a business or application feature or rule. Those scenarios utilize the Gherkin syntax, a form of the "AAA" format that define steps in the test using words like "Given/When/Then". This makes it easier to understand the test in simple language and possible to automate, while also hiding how automated code works underneath. It works very well for writing integration-level tests with or without automation.
Cucumber also has some patterns taken from unit test frameworks: the World object is a context object to contain data
unique to each scenario, before and after hooks exist on a feature, scenario, and step basis, stubbing and mocking
services can be performed, and scenarios can utilize test data fixtures!
Writing isolated tests as scenarios in a Cucumber feature
Feature: Order checkout behaviors for new users
As a new user
I need to be able to interact with the item checkout flow
So I can place new orders for items in the store
Background:
* I am a new user
* I open the url for "the-best-applicance-store.com/products/televisions"
Scenario: A user can view added items in their cart on the Checkout page
Given I add the "55-inch TV" to my cart
And I add the "Streaming Service dongle" to cart
When I click the checkout button
Then I am on the Checkout page
And the following items appear in my basket:
| items |
| 55-inch TV |
| Streaming Service dongle |
Scenario: A user can submit their shipping information correctly
Given I have added the following items to my cart:
| items |
| 55-inch TV |
| Streaming Service dongle |
And I am on the Checkout page
When I enter my shipping information correctly
And I submit my shipping information
Then I am on the Payment page
And my address appears in the form correctly on the Payment page
Scenario: A user can submit Payment information to be processed during checkout
Given I have added the following items to my cart:
| items |
| 55-inch TV |
| Streaming Service dongle |
And I have begun checking out
And I have completed checkout up to submitting my shipping information
And I am on the Payment page
When I enter my credit card details correctly
And I submit my payment information
Then I can see my submitted order on the Order Complete page
And the backend received my submitted order with the following items:
| items |
| 55-inch TV |
| Streaming Service dongle |
And the backend received my submitted order with my shipping information
And the backend received my submitted order with my payment details
Rule: Creating an account
Scenario: After checking out, a new user can create an account
Given I have completed submitting my order for the following items:
| items |
| 55-inch TV |
| Streaming Service dongle |
And I am on the Order Complete page
When I click the "Create New Account" button
And I complete registration for my new account
Then the page displays my new account as being logged inAt this point in the process, end-to-end tests assume that all services are connected together, and tests should be run to validate the entire picture rather than each part. When writing end-to-end tests with isolation principles, try to summarize each step without going into a lot of detail. Treat your steps as accomplishing flows rather than individual actions**. By this phase of the process, a tester should know what actions are needed per step, and items can be described more loosely.