Skip to content

Instantly share code, notes, and snippets.

@darkofabijan
Forked from pyromaniackeca/flaky-test-tips.md
Created March 30, 2017 15:50
Show Gist options
  • Save darkofabijan/a7a12c06dcfcc2e5a936de27df460677 to your computer and use it in GitHub Desktop.
Save darkofabijan/a7a12c06dcfcc2e5a936de27df460677 to your computer and use it in GitHub Desktop.

Biggest sources of flaky tests (from our experience)

FactoryGirl

When using FactoryGirl to create records in tests, it is necessary to be aware that you are usually creating a lot more records than is obvious in the code. While this is usually harmless aside from slowing down tests, it is capable of creating some mind boggling flakiness. Some scenarios where this is probable are tests that count the number of records, tests that are expecting a certain record and assume that only it exists. Improperly set model level validations can also contribute to this.

Timestamps

When working with records in a way that implicitly sets timestamps with the intention of comparing them it is possible to create a test which is dependent on code running faster or slower than a certain threshold. This is troublesome since background processes can have an influence on execution time and running your tests on different machines (i.e. local, ci) can also produce different execution times. This can be avoided with either explicitly setting time stamps or using Timecop and understanding the difference between #travel and #freeze.

Outside requests

Any request hitting and outside service is inherently more probably to be flaky. Any time the service in question is experiencing issues, is unavailable, or there are network issues in general the test will fail. Worst of all is that you as a developer have no control over this which can result in long periods of time where it is impossible to test your code properly. These kinds of requests need to be properly subbed either with vcr or direct method stubbing.

Asynchronous code

While we don't have any experience with async flakiness in RSpec, it is certainly something to keep in mind when investigating a flaky test. If there is any async code running in the flaky test it is a prime candidate for the source of flakiness.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment