Testing strategy after GA

gistfile1.txt

Goal

1. Improve test coverage
2. Improve turnaround

Approaches

1. Prompt unit tests and their code coverage
2. Measure testing quality frequently and reguarly.
3. Monitor the changes of the product percisely and reguarly.
4. Simplify integration tests, prompt live tests.

Actions

1. Add test utilites to run unit test fast and easy.
2. Implement automation mechanism to run unit test for pull request.
3. Implement automation mechanism to publish unit test code coverage.
4. Implement automation mechanism to compare commands surfaces build by build.
5. Implement automation mechanism to compare commands help document build by build.
6. Gradually fade out VCR tests by stopping adding new VCR tests and convert the rest of the test base.
7. Add test utilities to author integration tests easier.
8. Improve integration test debugging experience.
9. Revamp Travis CI task to focus on unit test and selected integration tests only.
10. Implement automation mechanism to report command coverage, parameter coverage for integration tests.
11. Run automation against both pinned dependencies and latest SDKs.
12. Run integration tests regularly.