Skip to content

Instantly share code, notes, and snippets.

@hanshoglund
Last active September 15, 2020 21:56
Show Gist options
  • Save hanshoglund/d6f85b78e8874ce266d6906111758f27 to your computer and use it in GitHub Desktop.
Save hanshoglund/d6f85b78e8874ce266d6906111758f27 to your computer and use it in GitHub Desktop.
Golden Tests.md

In the following, we'll assume a setting with a Git repo, master branch and pull requests (e.g. Github). We'll also assume that some tests are run on each pull request to verify correctness of a commit which is a candidate to be the next state of the master branch, of which is a direct descendant (whether this candidate is authored by a user or the result of an auto-merge between a user-authored commit and the last master branch doesn't matter).

What is a "normal test"?

A predicate run on the candidate commit x. P :: Commit -> Bool.

What is a "golden test"?

In principle a function Commit -> A (e.g. running a compiler on some source code) and some predicate A -> A -> Bool (e.g. ==).

When the predicate returns false, we either have an error (e.g. the compiler now produces incorrect outputs for the source code), or the change is acceptable (e.g. the output changed in a way that is behaviorally equivalent). In the latter case we can override the breakage, which is known as "accepting" the test.

In practice a golden test is implemented by storing the A from the last known accept along with each commit. We can think of the master branch as Tree (A, Commit).

Merges can only be accepted if the A stored in the two branches are equal. (TODO why?)

If there is a conflict a rebase followed by an accept should be performed instead.

Enforcing orderings

With a Commit -> A (e.g. some performance measurement) and some predicate A -> A -> Bool (e.g. >=, meaning performance must not regress). The >= could of course be any partial order.

Here we could similarly "accept" breakages explicitly.

TODO could we allow merges in non-equal conditions here?

Protobuf

This is an example of a partial order. If A and B are protobuf messages, A <= B iff the keys of A is a subset of the keys in B. By checking this for all Protobuf messages in our repo, we assure that the never remove a field rule is not violated.

TODO relation to vector clocks (and CRDTs)

TODO relation to database migrations

TODO application in API versioning

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment