Scientific control
A scientific control is an experiment or observation designed to minimize the effects of variables other than the independent variable. This increases the reliability of the results, often through a comparison between control measurements and the other measurements.
~Wikipedia
When we're trying to reproduce behaviors in our app, the closest we can get to recreating the exact conditions by which it occurred, the more likely we are to reproduce it.
For example, when a bug occurs...
- In a new feature branch, but not in the master branch.
- In the staging database, but not the local database.
- On CircleCI, but not locally.
Any of the above differences could hide a behavior, or create a new behavior.
So if we find ourselves saying, "it works for me," we should go through a checklist to make sure we sync our environments as best as possible.
Here's a checklist for recreating a behavior:
- Are we using the same git branch?
- Are we using the exact same database?
- Are we running it in the same app environment? (dev/test/production)
And here's a list of approaches I've taken to make this easier:
If a bug occurs only in production, write a script that will...
- Export the remote data*
- Import it locally to a separate database (without overriding your local)
- Configure your database to point to the imported database
- Depending on your data handling policies, this might also require that database is anonymized first before it ever leaves the server.
Having this process automated can be really handy so that any time a bug occurs, you can react quickly and rapidly recreate the environment locally.
The git bisect
lets you run a command for every version of your code between two different times.
For example, you can find a version of your code from 3 months ago that works, then find a version of your code that's broken, and it will run your tests on every revision that led up to that code so you can find which commits the breaking changes were introduced.
I won't get into the How To of that: you can start here: https://git-scm.com/docs/git-bisect.