-
Radically reduce the per-pull request build time
- Eliminate agent "spin up" time
- Don't waste resources executing tests for things that don't need testing, such as documentation updates and README fixes.
- Don't compile the entire project for each step in the build process.
- If possible, only execute dependent tests - not all tests (incremental testing.)
-
Reduce unit test flakiness
- Harden existing unit tests that are known to be problematic.
- Skip or delete the ones that aren't.
-
Reduce performance test flakiness
- Modernize NBench system to use dynamic ranges of accepted values. (part of a much bigger project)
- Parameterize tests on a hardware-specific basis.
-
Harden MNTR specs
- Migrate MNTR to run as a
dotnet
global tool, so it's easier to use - Improve output formatting so it's easier to debug - use distributed tracing tools to visualize output.
- Migrate MNTR to run as a
-
Migrate to Azure Pipelines
- No spin-up time.
- Can use build and test plans to copy output binaries from one node to another (very, very difficult to do well in TeamCity.)
- Can still run multiple tests types in parallel.
-
Create incremental build tool that actually works
- Stand-alone tool that can compute the diff between current pull request and target branch, determine all files that have been affected.
- Use Roslyn Analyzer or even a simple project-dependency analysis to determine which tests are affected by which code changes.
- Only run the narrowly selected test suite.
-
NBench 2.0
- Support for test fixtures
- Parameterize one performance test with many outside variables and combinations
- Can customize performance tolerances for specific pieces of hardware
- bencher.io support
- Ability to memorize and update performance values for specific projects, branches, and hardware configurations.
- Can dynamically determine whether or not something is / isn't within previously accepted tolerances.
- Support for test fixtures
-
Better MNTR tooling
- Create a sidecar version of MNTR that uses Phobos' tracing and correlation system internally.
- Emit data to stand-alone Zipkin or Jaeger instance that belongs to the Akka.NET project.
- Correlate each
ActorSystem
name to the ID of the test run.
Note: StreamRefs support made it into v1.4.0 already.