How to report flaky tests

how_to_report_flaky_test

## Determine if it's a flaky test

If the test *consistently* fails locally it's not a flaky test and you should not commit to master. Period.

If the test *consistently* fails only on our CI build, you must ssh on the CI box:
```
ssh yourCIserver
```

then:
```
sudo su jenkins
cd jobs/appName/workspace/appName/
```
and investigate. You can run the same commands you run on your workstation.
* start from running the failing test
* if it doesn't fail run the component test using the rspec seed (https://www.relishapp.com/rspec/rspec-core/docs/configuration/set-the-order-and-or-seed)
* if it doesn't fail run the entire build
* if it doesn't fail consistently... read on


## Build fails but rebuilding passes. Flaky test :(

### Guideline for reporting

#### Mark the build notes
* you should go to "edit build information" and title "flaky test" and description the github issue that tracks the flaky test. ie. http://yourCIserver.com/job/appName/472/

#### Check for existing
* Check if the failing test is already in our issue list on: https://github.com/org/appName/issues?labels=flaky-test&page=1&state=open
* if it is comment with the build number ie. http://yourCIserver.com/job/appName/472/

#### Add new one
* one failing test per github issue
* the title should look like: `ERRORING_FILE_NAME_AND_LINE || RUBY EXCEPTION` for example: `spec/features/preview/article_mobile_spec.rb:24 || Capybara::Poltergeist::TimeoutError`
* mark the issue with the tag "flaky-test"
* use the ``` to paste the code


### Guideline for updating flaky test
If you have a possible work around, test it locally and make sure it doesn't break anything else, then:
* in the commit message refer to its issue number
* update the issue with the commit number if doesn't happen automatically