Cordova testing improvements

Cordova_testing_improvements.md

Cordova testing strategy

There are a few areas, we're testing in Cordova:

1. Unit tests
2. Integration tests
3. e2e tests.

and there are four main components which quality we want to be confident in:

1. CLI
2. LIB
3. Platforms
4. Plugins

Current problems with testing.

Unit tests:

1. Current tests coverage is low:

A few numbers:

 Area  | Coverage, %

:-------:|:------------ CLI | ~20 LIB | ~80 Android | ~13 (only 'create' functionality is covered) iOS | 0 Windows | ~50

2. The code is quite untestable. Most of the logic is written in solid manner and cannot be split into small parts for testing. A perfect example is cordova/platform.js file, where all platform addition/update logic is incorporated into one single function. This became less actual for platforms after implementing PlatormApi.

3. Because of previous point a lot of unit tests is actually looks and works like kind of e2e. They covers a great pieces of functionality, involving FS and network operations in one spec instead of testing small, granular things.

Also this causes unit tests to take a long time to pass (over 6 minutes on my machine for cordova-lib)

Integration tests:

The integration tests are actually missing. mobilespec alows to run some basic scenario, which may be considered as integration test.

e2e tests:

There are some basic e2e tests inside of platforms that covers only partial functionality:

Platform	Covered scenarios
iOS	'create', 'build', 'list' (devices/emulators)
Android	'create', 'build' (different variations of path/project name), 'add plugin', 'update' (from 2 different platform versions)
Windows	'create', 'build' (w/ different cpu architectures)

Though e2e tests existas in platforms, they still covers only a part of functionality and doesn't cover a most used options/cli flags for commands. Also the set of e2e tests is not standardized.

The ideas on improvements.

1. Refactor tests/logic to test so the tests would became really 'unit' - most actual for CLI/LIB, less for platforms
2. Introduce code coverage measurement and track it (using public services: https://coveralls.io, https://codeclimate.com) - encourage devs to write unit tests for their code
3. Improve code coverage for CLI, LIB and platforms
4. Implement a unified reference set of unit (mostly for PlatformApi) and e2e tests for platforms. Technically this may look like a separate pluggable package (node module) - similar to plugin-test-framework - which could be installed into platform and should contain both the set of tests, unified for all platforms, and a test runner, which will grab and execute both platform-agnostic and platform-specific tests.