Markus Schirp On Mutation Operators

mbj [10:59 PM] 
@alexdenisov Just a quick write down on mutation operators, conceptually I see the biggest separation into the following four classes that have different usefulness / roles for a mutation testing tool:

* Sematic reductions - These are almost always guaranteed to be "good" mutations the user definitively wants to kill as they expose unused semantics. Most trivial exampe is `a; b` (where both perform a not dependent side effect) to: `b`. Simple line and branch coverage would in case side effect `b` is measured give a 100% covered result, where the mutation would expose `a` is not covered in case the mutation is alive. Note that semantic reductions come in various alternative forms, many of them are language / intermediate specific. The thing to IMO recall on these is: They will as they expose untested semantics almost never equivalents as: If the semantics are unneded, they should be removed, and if they are untested a test should be added. Another good canonical example for semantic reduction is `a(b)` -> `b`. And transitively all `n1(n2(x))` -> `n1(x)` / `n2(x)`. I assume these pass a type check. I suspect 80 - 90% of traditional mutations for a useful tool fall into this class.
* Orthogonal replacement - These are normally "good" mutations, but can lead to equivalents. A good example for a useful orthogonal replacement is `a < b` to `a > b`. It cannot be argued which one has less semantics, but it can be argued that an alive mutation for these show significant uncoverage that should be fixed.
* Neutral mutations - These are useful to discover integration errors. The most trivial neutral mutation is `a` -> `a` (noop). Mutant uses this mutation to test if the insertion mechanism by itself already causes test failures in the environment (so a future semantic reduction may not be detectable as the insertion process itself has a bug causing test errors, not the mutation effect causing errors). I've seen horrible integraiton errors that lead to a high number of false "good coverage" without these tests. Other neutral mutations are normally language / convention specific. Example is that for many languages `a += b` op assign, should just be syntax sugar for `a = a + b`, where for others the verbose form should not be equivalent. Mutating to semantic equivalent forms is useful to test if structure laws are still obeyed by having the tests still green. Summary: Neutral mutations typically have a test green expectation and are useful to test mutation testing tool errors and language misuse.
* Semantic additions - These are usually not mutations that make sense to use in the traditional sense. Mutating `a` to `a; b` only makes sense if you have tests that can guarantee that "not more than specified" is implemented, and outside of systems with formal proofs (or mutation testings with reduction) usually tests cannot guarantee the absence of unspecified semantics. Nevertheless semantic additions can be really useful to add temporal instrumentation to code to gather more data to use to emit interesting mutations for the classes above. Example is that there is a private version of mutant that does "Runtime type detection" so I can use static type data gathered from runs under "type profiling operators" to emit type directed mutations. Summary I think semantic additions are most useful for tooling.

[11:01]  
@alexdenisov Sorry for this wall of text, I've much more detailed thoughts on each one and always wanted to write it down in a more digestable form but never had the time to do so. Hence you just got he unstructured brain dump I just serialized from lots of mostly private discussions in the mutant community.

[11:02]  
Most mutation testing systems only do semantic reductions and orthogonal replacements. And IMO the focus should be on good language specific semantic reductions as most (coverage) ground is to be gained there.

[11:02]  
`a(b(c))` -> `a(c)` and `b(c)` is very useful for discovering value filters that do not have a test case that filter etc.

[11:03]  
Mutating a way default arguments in method signatures is also a good sematic reduction.

[11:03]  
Mutest only has a small number of operators, if you wanted I could add git issues for the semantic reductions we have in mutant, there are many that can likely be applied for mutest also.

======

In a nutshell these are IMO the most useful, ordered descending to usefullness as observed by me:

* Statement deletion: `a; b` -> `a`, `b`
* Argument propagation: `a(b)` -> `b`, and all transitive forms `a(b(c))` -> `a(c)`
* Branch promotion: `if a; b; end` -> `a` etc, also for switch / case and others
* Argument removal `foo(a, b)` -> `foo(a)`, `foo(b)`
* Operator argument promotion: `a + b` -> `a`, `b` (for all binary operators)
* Operator removal `!a` -> `a` (for all unary operators)
* Scalar boundary mutations `5` -> `-5`, `0`, `1` , `4` , `6` / `true` -> `false` etc.
* Operator inversion `*` -> `/` (edited)