Create a gist now

Instantly share code, notes, and snippets.

@jeluard / Secret
Last active Aug 14, 2017

What would you like to do?

This document outlines options to improve our testing strategy.

Reason to test: To provide feedback on the quality state of the Status builds as quickly as possible in a reproducible manner. (important: feedback, quickly, reproducible)

Quality state (attributes/dimensions) Functionality: Does it work? Does it work on specific devices/OS? If it works then other quality attributes: e.g. Performance; Usability, etc.

Ways to check and what can be covered Unit tests (what is already working will not be broken by changes) Manual tests, (check new functionality in PRs, regression and integration/system tests of development build with limited device coverage, review new UI designs before implementation) Automated UI tests to cover basic functionality (quickly get smoke coverage, measure performance, battery/memory/network consumption, run tests on real devices in the cloud platforms) Community manual tests (extra device/os/language coverage, testing development builds, feedback on perf/memory/network usage, review new ui?) Prevent bugs: review new stories/functionality, ui design and describe tests -> before implementing developer knows what will be checked.

Unit tests

Short and fast tests that validate pure functions. Can be run often. Based on cljs.test. When appropriate test.check is a great tool to find non obvious issues.

Some higher level tests specific to re-frame can be written using re-frame-test.

Every PR should have tests for new/updated functions. High coverage is not the target but basic functionality and limit cases.

Integration tests

At the other end of the spectrum the shippable application is tested with more complex user scenario. Those tests will be written in ClojureScript too to leverage the existing codebase. Appium provides a nodejs layer that we can leverage.

Non-regression tests

It might be useful to introduce performance and upgrade tests to make sure new releases do not break customer expectations.

Worth investigating

tiabc commented Jul 26, 2017

Unit tests goals:

  1. provide fast feedback to a developer that everything is alright.
  2. Guard the source code from accidental changes.
    Unit tests should test a single unit of application, usually a function. It's not necessary that the function is pure, it can be with side-effects. In this case, side-effects should be incapsulated in a separate dependent component which should be mocked to test the original function.
    Unit tests must be fast and run within, say, 10 seconds and provide a decent degree of guarantee for the developer that the new code is alright. For example, one should be able to add a fast pre-commit hook to ensure everything works fine.
    Unit test should also unambiguously show where the error is.

However, many functions are often inconvenient to test with unit tests. For example, functions with little business logic and some database operations can be tested with functional tests without loss in verbosity.
Functional tests are also the place to test some internal workflows and may not be fast.
Functional tests must only run in an isolated environment as close to production as possible.

  1. Check dependencies inter-operation.
  2. Check internal flows.
  3. Be a supplement to unit tests where the functional counterpart is more convenient that a unit test.

Finally, there are integration tests which should cover user flows. Goals:

  1. Check multiple internal flows in simultaneous operation.
  2. Ensure most important (up to all) user scenarios properly work.
  3. Other goals you've mentioned: the system works properly in all environments.

rasom commented Jul 26, 2017

roma [08:07]
@tiabc i always had problems with this terminology, like what you call integration tests here i would call functional tests and vice versa. So as far as i understand integration tests are about black-box testing of some set of modules together, not necessarily all modules so their env is still not complete and equal to prod. When functional tests are about testing of the whole app as black box, so here env should be really similar to prod. From lower level it looks like unit->integration->functional. So i would say that unit tests, integration tests (in case of RN app re-frame-test stuff can be considered as such) are under developers’ responsibility, when functional tests should be done by both devs and QA and should cover “Ensure most important (up to all) user scenarios properly work.“.

Based on “system level, element” to be tested: Unit, integration of units in 1 sub-system, sub-system, integration of sub-systems, system test

Based on “our question to the system”: functional (does it work?), performance (how fast?), platform compatibility (does it work on this device/OS?), interoperability (does it interact OK with external system?) etc.

Based on our knowledge of internal logic -> white-box (we see the code and know what happens inside), grey-box (some knowledge about internal stuff is there but not all), black-box (we have no idea what’s inside, we can just see inputs and outputs)

On what “public” app interface do we operate and run tests: UI, API, mixed, "non-public"

Do we automate? Manual, automated

There are other types that I will not touch now (e.g regression, smoke, acceptance etc)

Types are complementary, so we test element X to find answer to question Y, e.g automated white-box integration functional tests. Some types of tests are usually done on specific levels with specific question in mind: e.g. automated white-box functional unit test. Usually, only 1 type out of all is mentioned because it’s kind of “obvious in that culture”, so in some company all functional tests are about functional system tests and all unit test is all types we automate.

It’s also important to define what are our units, sub-systems, system and external systems. Definition of units and their integration is up to developers. Sub-systems can be all “separate components” delivered by our 2 projects. System is the integration of these components: app that user would install. External systems would be all that we can’t control: DApps, services like Twilio for status-react.

In our case it looks like we will have:

  1. Unit functional tests
  2. Integration of units functional tests
  3. Sub-system functional tests (for 1 sub-system, e.g. status-react)
  4. Integration of sub-systems (integration between status-go and status-react)
  5. System tests (complete app)
  6. System interoperability tests (how our app interacts with external systems we are using)

If you are fine with proposed types structure, let’s use it for further discussions. I like both Ivan’s and Roma’s vision, we are really talking about same things using different terminology, but understanding of these things is quite similar 🙂 I will write a bit more on goals for each 6 types and move it to doc if it makes sense to you.

A few notes regarding status-react automated tests:

Current changes

I've made a small change that removes the need to manually add every new test namespace within the runner (issue #1574 and PR #1606).

In the previous test code, each test namespace had to be explicitly required and included in the test runner, which was slightly tedious but offered great flexibility in terms of what tests may or may not be included in the test run. Specifically, protocol tests have a separate runner and contacts tests are excluded at the moment. This brings us to a question of how do we allow similar flexibility, but preferably in a simple way. My initial idea was to put some keyword tags in test namespace metadata, however I've found a much simpler (i.e. dumber) solution in the mean time.

Test hierarchy proposal

We can specify sets of namespace path prefixes and call them say test suites. Any test namespaces not in a suite will then be considered members of the 'default' unnamed suite. Furthermore, we can define one special suite called 'ignored' that will be, well... ignored in every test run.

In our current example, we'd have three main suites:

  • the one with protocol tests, say we call it protocol
  • the one with contacts tests, that's ignore
  • the default one with everything else.

The behavior I'm proposing would be something like this:

  • when we do lein test-cljs the runner only runs default test suite
  • when we do something like lein test-cljs SUITE e.g. lein test-cljs protocol, the runner only runs a specific suite
  • we may run more than one suite, e.g. lein test-cljs default,protocol
  • or force it to run all suites, e.g. lein test-cljs all (this should still skip the ignored ones)

Ideally, every functional unit test that checks a pure function (or an impure one with some dependencies mocked) should be in the default test suite and as should every test that checks integration of units, but does not depend on anything external.

The main valid reason for a test namespace to belong to a special suite is if it is not a functional test, not a unit test or both. Some examples would be:

  • test that rely on external dependencies (network resources) that may not be available on all machines that are supposed to run it, or have limited availability or possibly rely on resources that have to be manually started.
  • performance/load tests or really any tests that take a lot of time to complete. We want to run automated tests as often as possible, but not if it breaks the dev flow too much - only reasonably fast tests may be in the default suite.
  • tests that are trying to replicate some architecture specific behavior that will not be available on all the environments, e.g. something that may or may not pass on a Mac, but will always fail on a Linux or Win.

That said, the hierarchy of cljs tests in status-react would be something like this:

All tests > Suite (or default) > Namespace > Test function > Assertion

We may or may not do a similar thing for Appium tests, in any case that is beyond the scope of this proposal at least.

Increasing test coverage proposal

While high test coverage should not be a goal in its own right and for its own sake, there are several other goals that largely overlap with the necessity for automated tests, namely documentation and refactoring. IMHO, test cases are probably the most useful internal documentation a piece of code can have. In fact, that's pretty much why I, as a newcomer, picked this up in the first place - just wanted to find out more about the codebase and naturally gravitated towards tests. Regarding refactoring, there really is no such a thing without automated tests, as many as possible at least.

The whole idea how to increase the coverage is really very simple:

  • for any new code, also submit the test cases (this can tie in nicely with another proposal I saw around - to state acceptance criteria in issues)
  • for any old code, retroactively write the tests during the refactoring phase of that unit. Since we're refactoring it, it's along the way anyway and it may decrease the likelihood that the refactoring itself is the source of new bugs.
  • if the old code is tricky to test, that may be a warning that it actually should be refactored, e.g. broken down into smaller functions or whatever (within reason).
  • it also makes little sense to test some things, e.g. thin wrappers around dependency functions, unit tests for code that does nothing except integrate other functions, code that is so dependent of environment that nothing less than an appium test or more likely a manual test session will properly replicate, etc.

To conclude, and strictly for humorous effect, a somewhat relevant link:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment