on integration testing

integration.md

when people say "integration testing", the feeling I get is that the definition that most people are using is "unit tests that happen to perform i/o". is this the definition that most people are using?

there's another definition, which is the definition I learned when I first learned about unit testing, which I have never seen anyone actually use: a unit test is an individual unit of testing, and "integration testing" is when you sequence the unit tests to create an integrated suite of tests. that is ... integration testing is when you integrate your unit tests, not when you test how your system integrates with another system. Those are distinct concepts! My suggestion here is not that the latter concept isn't valuable, it is valuable, it's just distinct, and rarely do I see the first concept being executed well.

For example, let's say you were testing some CRUD API and you wanted to test two things: the create and the update. The strategy that I most commonly witness is as follows:

• create a unit test for your create action. Start with a fixed, known state (let's call it c0), then run the create. The system is in some new state c1. Check the response to the create routine, as well as check that c1 is the value of the state that you expect.
• independently, create a unit test for your update action. Start with some known-good state (let's call it u0), a state of an existing object in a database. Creating this state is itself work: it's new work to create this platonic starting state. Run your update against this platonic state (u0), producing some new state (u1). Check the response to your update action and check that u1 is the new state value that you expect.

That's all well and good, but you've now created a handful of new problems:

• how do you define the success criteria of the create action (that is, the verification p such that p(c1) indicates that the test for create passes) that is not in terms of the read action, in order to guarantee isolation of the things under test? What value is provided by testing the create action alone? Does this not create a new hazard where the verification logic of the create test can diverge from the actual logic of the read action?
• how do you define the initial state for the update test? (in this example, u0.) Is that not simply the result of the create action? The update action is now being tested off of a platonic starting state. How do you know that this platonic starting state is reachable by your system? Is it not the case that c1, the output of the create test and u0, the input of the update test, should always be equal or your tests are invalid? If that state is reachable now, how do you ensure that it continues to be reachable as your system changes?
• if your update is tested off of a platonic starting state that is not the exact output of the create action, you now have two problems: your update is not testing the state reached by the create routine, and you've created a new, false requirement that the update action be usable against a state that is not reachable by your system. You had to go through all of the trouble of creating this state, which is new work, when the create action ... literally does that work. The value provided by the isolation has to be significantly greater than the cost of having created that state, otherwise you're just creating busywork.

anyway, this comes up a lot for me since my primary project is a stateful multiplayer server whose only job is to contain and communicate the state of a game. integration testing this thing is ... hard. curious what people do for integration testing from a conceptual level, not from like a tools/language/library level. do other people also face the problem I'm facing, or are people finding testing against platonic states relatively unproblematic and it sounds more like I'm doing it wrong?

write tests as an n-ary tree, and each path to a leaf gets executed separately, but you've only had to define each node once for the tree instead of once per path.

yes, exactly. It's called "tea" because you're "reading the tea leaves" to tell your fortune.

I also love that failure at a node can short circuit the rest of the path so I don't see an enormous amount of failure output when I break something fundamental.

yeah one of the shortcomings of using Go's sub-tests natively is that if your test fails and exits early, the sub-tests are never created. So a pass might be like "300 tests passed", and a failure might be like "150 tests passed, and 1 test failed", when the reality is what happened was "150 tests passed, 1 test failed, and 149 tests were skipped". Although the Go sub-test allows you to mark tests as skipped explicitly, the ergonomics of doing so means that it's very easy to mess that up. tea handles that for you automatically.

I can also see such a tree getting unwieldy to manage and hard to follow. It reads like "small functions" code because each step is broken up spatially and you have to follow its descendents much less naturally.

this is a massive problem we have now with tea. Writing tests is super easy, looking back at the tree and adding tests to a large tree that already exists is nightmarishly confusing. I have to do some work to improve the ergonomics of larger test graphs.

I've never been a cucumber/convey fan

yeah so one of the problems that convey has is that because a test accesses the side-effects of its ancestors via closures, and stack frames always have a single parent, a given test can only appear along a single path. Since tea uses structs and struct fields to persist the runtime environment from test to test instead of stack frames and closures, tea has no such constraint. For example:

func TestCreateBook(t *testing.T) {
    // since you're creating a test -plan- and executing it in separate phases,
    // you can manipulate the plan arbitrarily -before- running it. This function takes
    // a node in a test plan as its input, and adds to it a bunch of children.
    addChildren := func(root *tea.Tree) *tea.Tree {
        root.Child(&getBook{title: "the giving tree", expectError: ErrNotFound})

        withBook := root.Child(&createBook{title: "the giving tree"})
        withBook.Child(&getBook{title: "the giving tree"})
    }

    sql := tea.New(&startSQLServer{})
    mem := tea.New(&startMemServer{})

    addChildren(sql)
    addChildren(mem)

    tea.Run(t, sql)
    tea.Run(t, mem)
}

You can do that now and it works. The entire tree is a data structure that can be manipulated arbitrarily, so you can adopt tea easily into projects that use table-driven tests.

Also any two equal test values are equivalent. I think this makes them "referentially transparent" but I've never done FP so I dunno. This:

a := tea.New(&A{})
a.Child(&B{X: 1}).Child(&C{Z: 10})
a.Child(&B{X: 2}).Child(&C{Z: 10})

Is exactly the same thing as this:

a := tea.New(&A{})
c := &C{Z: 10}
a.Child(&B{X: 1}).Child(c)
a.Child(&B{X: 2}).Child(c)

It doesn't matter that they're pointers to the same struct because tea doesn't actually use that struct: that value is only treated as a template, it's copied before it's ever used. That value is never actually mutated; we create a new value to mutate to ensure isolation. This works presently and I rely on it.

I've been working on making it so that you can combine nodes in the plan to treat it as a DAG instead of a tree. So long as there are no cycles you can always break the DAG apart into its component paths. E.g., that prior example would hypothetically turn into this:

func TestCreateBook(t *testing.T) {
    sql := tea.New(&startSQLServer{})
    mem := tea.New(&startMemServer{})

    allDBs := sql.And(mem)
    allDBs.Child(&getBook{title: "the giving tree", expectError: ErrNotFound})
    withBook := allDBs.Child(&createBook{title: "the giving tree"})
    withBook.Child(&getBook{title: "the giving tree"})

    tea.Run(t, allDBs)
}

(that's not implemented yet though.)

things like Quickcheck and Hypothesis, which are probably the closest I know of, don't really tell you how to approach this.

oh cool these weren't on my radar, thanks for mentioning them, they'll be good prior art to look at.

There's also this consistent push from people from various angles (infra, FP advocates, etc) for everything to be "stateless" which.. yeah, fine stateless and immutable things are easier to reason about and preferable where possible, but also the world is stateful and everything in computing built on top of state, so we need to be honest and admit that we're offloading those problems and they're still important to solve instead of just avoid.

yeahhhh I encounter this -a lot-. My project serves only a single purpose: to handle the state management so that other systems don't have to. So much conventional wisdom is poorly suited to projects of this nature because usually people are just using a database, but what I'm making has many similarities to databases.

anyway thanks for the feedback, it sounds like making a library like this isn't raising all sorts of red flags to you.