Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
How to report flaky tests
## Determine if it's a flaky test
If the test *consistently* fails locally it's not a flaky test and you should not commit to master. Period.
If the test *consistently* fails only on our CI build, you must ssh on the CI box:
```
ssh yourCIserver
```
then:
```
sudo su jenkins
cd jobs/appName/workspace/appName/
```
and investigate. You can run the same commands you run on your workstation.
* start from running the failing test
* if it doesn't fail run the component test using the rspec seed (https://www.relishapp.com/rspec/rspec-core/docs/configuration/set-the-order-and-or-seed)
* if it doesn't fail run the entire build
* if it doesn't fail consistently... read on
## Build fails but rebuilding passes. Flaky test :(
### Guideline for reporting
#### Mark the build notes
* you should go to "edit build information" and title "flaky test" and description the github issue that tracks the flaky test. ie. http://yourCIserver.com/job/appName/472/
#### Check for existing
* Check if the failing test is already in our issue list on: https://github.com/org/appName/issues?labels=flaky-test&page=1&state=open
* if it is comment with the build number ie. http://yourCIserver.com/job/appName/472/
#### Add new one
* one failing test per github issue
* the title should look like: `ERRORING_FILE_NAME_AND_LINE || RUBY EXCEPTION` for example: `spec/features/preview/article_mobile_spec.rb:24 || Capybara::Poltergeist::TimeoutError`
* mark the issue with the tag "flaky-test"
* use the ``` to paste the code
### Guideline for updating flaky test
If you have a possible work around, test it locally and make sure it doesn't break anything else, then:
* in the commit message refer to its issue number
* update the issue with the commit number if doesn't happen automatically
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment