Hi 👋, I'm Jonathan, Android Lead at MWM. "You should write tests", that something we can hear on the mobile / tech industry. But the question is Why?. Let's find out if writing tests is something you should invest on.
"Be sure you test the tests written" 🙃. Romain Guy and Chet Haase.
There are several notions to consider before writing tests. About tests, this article contains the Why / What / How and my personnal opinion about test. At the end, past experiences will be detailed.
-
Main reason: confidence 💪
- The main reason of writing tests is to build confidence on your code. The goal is to make you sleep at night. When refactoring, you should not fear of breaking things. When you release to production code, you should not fear the crashes. When you send the build to QA, they will not find issues! 💪
-
Second main reason: find the why of crash 👷
- Test are great to avoid crashes or be sure of the fix. What's great with test and especially Unit test, it's the fact it's a way to understand the Why behind an issue. With test you will spot reason of crash before the crash hit the production.
-
TDD 🧪
- To use tests as a tool to validate your feature before the implementation. When you are designing a REST API, this is great. On Mobile development, this is a great approch for math / logical code where inputs and outputs are driving your code. Based on past experiences at MWM, I would say only 2% of feature in an app could be develop efficiently with TDD. Often code dealing with Math, time or pure logic.
- TDD is also great when it's not easy to run your code otherwise. For example imagine the team that wrote Apollo 11 program. You are not always able to test your code on a shuttle
^^.
-
Non-regression 🧯
- Non-regression check: Test linked with CI will allow you to be more robust to regression.
-
Because you should not rely on QA 🚀 (Quality team testing the product)
- "QA should find nothing". Uncle Bob
-
Rigidity 🔒
- The main reason is because tests will make your code rigid / difficult to change. Every change of feature tested will require extra work to maintain tests.
-
Because it's new and fun 🐣
- It's not because it's new to you as developer that you should embrace it. It's not because you love writing tests that you must put them everywhere. Take a step back, be pragmatic, test only what's make sens. Tests are a tool, like every tool, it'is a trade-off "Cost" VS "Benefit".
-
Because I have to test every single line of my code 🤓
- Are you sure? What's the point? Why should you test every possibility and every line of your code? Tests should be done with this ratio in mind: Gains (like more confidence, spot issues) VS Cost (time spent to write tests, maintenance and project more rigid to change).
-
To test the platform or external libraries 📱
- You should not test the platform or external library with tests. Only your code. Assume external part are working as expected.
-
To test what the code is doing 🙃
- Writing test is not looking your existing code and implementing test runnig what's your code is doing. No. The approach should be: I have a black box. I can give to this black box inputs, what the outputs I should expect with those inputs.
-
To test private methods 🔐
-
You should test the "black box" via the public API and check what the box is doing via the public API. By doing that, with the good architecture, you should be able to test what you want to test. Of course, on rare occasion, no other choice than testing private method directly. On Android for example, the annotation
@VisibleForTestingexists.Don’t test private methods. Test the public methods. It would be very strange if there were parts of the privates that you could not reach from the publics. Uncle Bob
-
-
To have a great Code coverage 🩺
- Code coverage is the percent of your code covered by tests. Like Uncle Bob is telling us on YouTube videos, this metrics should stay internal to the dev team. That's not the quantity of test that matter but the quality of tests and the part that are tested. Only devs should have access to it because only devs know that this percent does not tell anything to the project sanity. So code coverage should not be the reason of writing test
This list of what should you test is non exhaustive.
Reasons:
- You do not want critical parts to fail
- You want to be sure critical parts are still OK after refacto
- Critical parts of a project do not change too often so you will not have to constantly re-update tests
So what are critical part of your project?
- Business logic
- Core feature. For example audio SoundSystem for a DJ app
- Storage and network parsing
Depending on your project, you may consider a lot of part as critical. One approch could be to to test first the part that are not subject to change too often. When you discover an issue, try to write test before fixing the issue.
"Do not trust external inputs". For example, testing storage and network parsing are great to
- check retro compatibility / non regression
- be sure edge cases are supported
- spot external inputs errors asap to not affect the rest of your app
But keep in mind, no need to test external libraries or the Operating System.
With pure mathematical logic like "shortest path finding", test are required. That help the PR/MR reviewer. That allow you to develop the feature by checking every edge cases.
When you spot an issue, you can consider the test covering this bug was missing. So writing the test (before the fix if possible) is a good way to be sure the fix is working well and will never spot again.
As a good first guess, Google often suggests a 70/20/10 split: 70% unit tests, 20% integration tests, and 10% end-to-end tests. The exact mix will be different for each team, but in general, it should retain that pyramid shape. link
On the Android team at MWM, we use to have:
| Name | Description | % in a project |
|---|---|---|
| End to end tests | Navigation through the app | ~2% |
| Connected tests / Integration tests | UI test on one screen or one view | ~8% |
| Unit tests | Test smallest piece of code | ~90% |
Why Unit tests are so high?
Because the sooner issues are spot, the easier it will to fix them. Unit tests are here to identify reasons behind crash or miss behavior.
This pattern is popular in test link. The goal to make your test readable (first rule you should follow as developer)
fun sha1() {
// Given
val message = "Hello World"
// When
val sha1 = hashManager.sha1(message)
// Then
Assert.assertEquals("0a4d55a8d778e5022fab701977c5d840bbc486d0", sha1)
}Why?
Imagine you jump on an unknonw project. The CI is failing because of only one Unit Test. Great, easy you will say. Not se easy if the test method is testing multiple thing at the same time. That the reason why you should always try to test only one thing by having only one assert check by test method.
Duplicate the method with explicit method name to let other dev know that is tested.
class SearchInputViewPresenterTest {
fun searchClickPerformSearch() {
// ...
Assert.true(searchHasBeenPerformed)
}
fun searchClickOpenSearchResultScreen() {
// ...
Assert.true(searchResultScreenHasBeenOpened)
}
}MVP design pattern or equivalent is a good way to have testable code. Indeed, codebase is easier to test if logical parts of your project have been extracted from the platform code (for example from android.view.View). Here an example of an archi designed to support Unit tests.
You will see, that this approach to make your code testable is even more valuable than the tests. What I like to answer when someone ask me if I test my code is: "I test with Unit test only 5% of my app (most critical parts) but I write testable code.".
Here the same point of view:
The "MVP" described above is abstracting the logic from your UI. Like that, you will be able to test business logic and UI logic. In order to do that, mocks allow:
- to give fake inputs to the class you want to test
- to check if outputs of the black box our are testing are OK
class SearchInputViewPresenterTest {
fun onSearchClickedPerformSearch() {
// Given
val searchManager = mockk<SearchManager>()
val screen = mockk<SearchViewContract.Screen> {
every { getSearchInput() } returns "Cat videos"
}
val presenter = SearchViewPresenter(
screen,
searchManager
)
// When
presenter.onSearchClicked()
// Then
verify { searchManager.search("Cat videos") }
}
fun onSearchClickedOpenSearchScreen() {
// Given
val searchManager = mockk<SearchManager>()
val screen = mockk<SearchViewContract.Screen>()
val presenter = SearchViewPresenter(
screen,
searchManager
)
// When
presenter.onSearchClicked()
// Then
verify { searchManager.openSearchResultScreen() }
}
}
If your test environment does not support Mocks, you can always use interface / protocol (swift) and instantiate fake instances to give to the class you want to test.
On Android, the official documentation mentioned serveral "test doubles".
Your test should be automated, with CI, in order to be useful. Every commit should be tested and every PR/MR should be blocked if the CI is failing.
Why? Otherwise you will never check tests.
https://academy.realm.io/posts/kau-jake-wharton-testing-robots/
The same test that is some days ✅ and some days 🔴 will make your whole test suite useless. We are calling tests like that: flacky. Flackyness in tests in worse than no tests at all. Why? Again it's about trust. Developer will not trust tests so tests are not filling the main reason of why we are writing test: confidence.
On small project or team, investing time on connected test is questionable. Keep in mind that the UI is the part of your codebase that will change the most. So Gain VS Investement is not obvious if your project is not huge. Remember test pyramid.
Connected tests / UI tests on device are a great may to automate screenshot generation.
One day at my daily job, we were facing native Android crash impossible to reproduce on our main app: a DJ app. The SoudSystem, the part generating the audio 🔊, was crashing. The crash rate was becoming important and we had to find a solution.
The "solution" was to take 20 devices, connect them to the same computer and run the same "connected test" with "end to end" scenarios. The goal was to try to reproduce and make stats about crashes to find a determinist way to reproduce the issue.
The project was important, multiple devices (heads) connected to one computer (body), we named the project internally: Hydra's project!
We created a cron to make the tests run every night. We made fully automated crash reports by pulling result from devices with adb. We had a screenshot system to monitor step by step senarios. Lot of Android API, resolutions and ROMs tested. We even made a html / css / javascript dashboard to monitor our progression.
We put a lot of time and effort. And finally, we succeed to reproduce the crash. Yes, finally. But the crash was still random. And impossible to determine the reason of the crash. All our efforts were not giving us the solution to fix the cause of the crash.
We gave up the project. Only few weeks later, we managed to understand the Why? and that was not thanks to Connected tests. We spot the cause via a dedicated sample that was testing our SoundSystem under conditions that the app was not able to reproduce in a determinist manner.
To sum up the "Hydra Project". Even if the SoundSystem was not fix with Hydra's results, the "connected tests" with determinist scenarios can be use to stress test your product. Create samples for your libraries to reproduce "clients" issues.
From my past experiences, I'm using some principles when dealing with tests:
-
- Always write testable code
-
- Do not write too much test. Rigidity. Stay pragmatic with a good balance: COST vs GAIN.
-
- Write almost only UnitTest
-
- Split your project in order to have sample for critical feature (for example, the audio on a DJ application)
Links:
- Uncle Bob series of YouTube videos
- Article Testim
To go further, you can do prevention to avoid issues. One way is with the "Fail fast" approach.
Other articles and projects on Mercandj.