"Why test your code? So you know it can work, and have a way to quickly verify it still works as it evolves." - some engineer on Quora.
For more info: https://en.wikipedia.org/wiki/Test-driven_development
The overriding thought for testing should be "What tests need to pass in order to ensure my code works as expected?". For both services and libraries, arguably the highest priority are interface tests since they are the specific contract between an application and its end-users.
So for a service, test at the endpoint level first. For example, just test that a /users/{id} endpoint returns the expected user payload. Once all endpoints are tested, then begin to traverse down the call stack. Similarly for a library, test all public methods and utilities first, then traverse down the call stack to all helper methods.
Unit tests cover individual units of source code like statements, branches, functions, methods, and interfaces. It's the most basic type of test. You could say all tests are at least unit tests.
Example:
import pytest
def is_str_a_digit_or_letter(char: str) -> str:
if type(char) is not str:
raise ValueError(f'input is not a string: "{str(char)}"')
if char.isalpha():
return 'letter'
elif char.isdigit():
return 'digit'
else:
raise ValueError(f'character is neither digit nor letter: "{char}"')
def test_detect_digit():
assert is_str_a_digit_or_letter('9') is 'digit'
def test_detect_letter():
assert is_str_a_digit_or_letter('B') is 'letter'
def test_detect_neither():
with pytest.raises(ValueError) as excinfo:
is_str_a_digit_or_letter('9B')
assert 'character is neither digit nor letter' in str(excinfo.value)
def test_detect_not_str():
with pytest.raises(ValueError) as excinfo:
is_str_a_digit_or_letter(9)
assert 'input is not a string' in str(excinfo.value)Integration tests cover connectivity (or integration) among multiple components/units. A good example is making a request to some external service.
This is an integration test:
import requests
def is_arturo_svc_healthy(base_url: str):
try:
response = requests.get(f'{base_url}/_mgmt/health')
response.raise_for_status()
return response.json()['status'] == 'UP'
except requests.HTTPError as exc:
logger.error(str(exc))
return False
except Exception as exc:
logger.error(str(exc))
return False
def test_happy_case():
assert is_arturo_svc_healthy('https://dependency-a.arturo.ai') is True
def test_invalid_base_url():
assert is_arturo_svc_healthy('https://blah.arturo.ai') is FalseThis is not an integration test. It's a unit test because connectivity is mocked:
import requests
class MockResponseGood:
def raise_for_status(self):
logger.info('mocked good response')
def json(self):
return {'status': 'UP'}
class MockResponseBad:
def raise_for_status(self):
raise requests.HTTPError('mocked bad response')
# uses pytest-mock package
def test_happy_case_with_mock(mocker):
mock_response_good = MockResponseGood()
mocker.patch.object(requests, 'get', return_value=mock_response_good)
base_url = 'https://blah.arturo.ai'
assert is_arturo_svc_healthy(base_url) is True
requests.get.assert_called_once_with(f'{base_url}/_mgmt/health')
assert mock_response_good.raise_for_status.call_count == 1
assert mock_response_good.json.call_count == 1End-to-end tests cover end-to-end flows, mimicking real-life scenarios and usage.
Example:
def test_create_user_update_profile_add_to_event():
user_name = "Dean"
user = User.create(name=user_name)
assert len(User.find(name=user_name)) == 1
assert len(Notifications.find(type="user_create", user_ids=user.id)) == 1
color = "forest green"
user.update(favorite_color=color)
assert User.find(name=user_name).favorite_color == color
title = "Winter Solstice"
Event.create(title=title, attendees=[user])
assert len(Event.find(title=title, attendees_contains=[user]))
assert len(Notifications.find(type="event_user_add", user_id=user.id)) == 1Interface tests cover interactivity with interfaces (web apps, CLIs, APIs, etc.).
Example for a web app:
it('should display welcome message', () => {
page.navigateTo();
expect(page.getTitleText()).toEqual('Welcome to arturo-app!');
});Example for a REST API:
import requests
def test_user_create():
name = "Dean"
job = "Chief Strategy Officer"
body = {"name": name, "job": job}
response = requests.post(f'https://some-svc.arturo.ai/users', json=body)
res_user = response.json()['user']
assert 'id' in res_user
assert res_user['name'] == name
assert res_user['job'] == jobPerformance testing is the umbrella term for checking stability, availability, scalability—all non-functional requirements meant to assess whether an application remains healthy during anticipated and peak workloads.
Load and stress testing determines the breaking point at which an application can no longer respond to requests within some latency threshold. This type of testing is very useful in determining the metrics and corresponding threshold to scale at.
Soak testing refers to subjecting an application to load over a significant duration of time like days or weeks. It's used to find errors that result in degraded application performance with continued usage.
Example:
One instance of Service A can handle a throughput up to 74 requests/sec before max latency breaches a max 400 ms latency. So to guarantee availability of Service A, we should set autoscaling to some percent of 74 requests/sec, say 37 requests/sec (50%), to give autoscaling some time buffer to spin up new instances of Service A.
Canary tests check for obvious and frequent sources of issues like network connectivity, database status, authentication status, etc. They're basically an extended health check. A common setup involves a cron job running against a live service.
Example:
# schedule: */1 * * * * (every minute)
import requests
def health_check():
try:
response = requests.get('https://service-a.arturo.ai/_mgmt/health')
response.raise_for_status()
except Exception as exc:
logger.error(str(exc))Acceptance testing is a formal type of testing performed by an end user after a feature is delivered by developers. The aim of this is to check if the software conforms to their business needs and to the original requirements. Acceptance tests are normally documented at the beginning of the sprint and is a means for stakeholders and developers to work towards a common understanding and shared business domain knowledge.