Skip to content

Instantly share code, notes, and snippets.

@delbetu
Created January 6, 2023 17:09
Show Gist options
  • Save delbetu/922064ce4e69c965d1da36b5f36014b0 to your computer and use it in GitHub Desktop.
Save delbetu/922064ce4e69c965d1da36b5f36014b0 to your computer and use it in GitHub Desktop.
Rails Flaky Tests Guide

Here are some of the things I've identified that can cause flaky tests.

Race conditions

A common type of race condition in tests is this: the test submits a form and then clicks a link on the subsequent page. BUT sometimes the page loads a little more slowly than at other times, and so the test driver clicks the link before the link actually shows up on the page. It's a race between the link showing up and the test trying to click the link. In the cases where the click "wins" the race, the test will fail because the link isn't available yet. Other kinds of race conditions exist too but that's a common case.

Leaked state

Let's imagine there's a test suite with just two tests in it, test A and test B. Both tests create a user in the database with the email address "test@example.com". But the difference between the two tests is that test A deletes the user at the end of the test, while test B leaves the user lying around in the database. If test A runs before test B then there's no problem because test A cleans up after itself. But if test B runs before test A (which will happen sometimes because test runners tend to randomize the test run order) then there's a problem, because test A will try to create user "test@example.com", but "test@example.com" already exists. If there's a unique constraint on the user's email address, then the test will fail.

Network dependency

If there are tests in your test suite that depend on external network calls, then those tests may pass most of the time, but then fail when, for example, a) the network goes down or b) your tests hit a service repeatedly and get rate-limited for the service you're calling.

Randomness

To use a very obvious and contrived example, assert_equal(1, rand(2)) will fail 50% of the time. A more realistic example is the order of database records. If you simply call .first rather than specifying a specific database record, you're liable to get different database records on different runs because the order isn't guaranteed to always be the same.

Time dependency

If you aren't careful with your timezones and such, you can write tests that pass at certain times of day and not others. This happened to me once when I was working late. I wondered why my tests were suddenly failing "for no reason", but then I discovered that I had made a mistake with regard to timing.

So, how do you avoid writing tests that flake due to one of the above problems? That's a topic for another day, but I think simply being familiar with the different ways a test can become flaky is helpful because it gives you a list of "usual suspects" to think about when doing detective work on flaky tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment