Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
How to report flaky tests
## Determine if it's a flaky test
If the test *consistently* fails locally it's not a flaky test and you should not commit to master. Period.
If the test *consistently* fails only on our CI build, you must ssh on the CI box:
ssh yourCIserver
sudo su jenkins
cd jobs/appName/workspace/appName/
and investigate. You can run the same commands you run on your workstation.
* start from running the failing test
* if it doesn't fail run the component test using the rspec seed (
* if it doesn't fail run the entire build
* if it doesn't fail consistently... read on
## Build fails but rebuilding passes. Flaky test :(
### Guideline for reporting
#### Mark the build notes
* you should go to "edit build information" and title "flaky test" and description the github issue that tracks the flaky test. ie.
#### Check for existing
* Check if the failing test is already in our issue list on:
* if it is comment with the build number ie.
#### Add new one
* one failing test per github issue
* the title should look like: `ERRORING_FILE_NAME_AND_LINE || RUBY EXCEPTION` for example: `spec/features/preview/article_mobile_spec.rb:24 || Capybara::Poltergeist::TimeoutError`
* mark the issue with the tag "flaky-test"
* use the ``` to paste the code
### Guideline for updating flaky test
If you have a possible work around, test it locally and make sure it doesn't break anything else, then:
* in the commit message refer to its issue number
* update the issue with the commit number if doesn't happen automatically
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment