http://leveluprails.herokuapp.com/test.html
Unit tests in Rails are meant to test the models - the individual pieces of business logic. In our overlord application, this was our RSpec tests - exercising the underlying Bomb class outside of the web framework. This tests things like, "can I activate a bomb with this code" or "make sure this incorrect code won't activate the bomb". These tests are meant to test small pieces of the code in a vacuum - there is very little context or pre/post setup; state should not be maintained between tests because the tests are independent.
Functional Tests in Rails are meant to test a single controller - this might represent one step of your Rails application. In our overlord project, this was our cucumber test, generally speaking. This includes the logic from the unit test above, as well as the routing behavior of the "controller", along with the web framework. The functional tests are very similar to the Unit tests in their narrow scope (like "activate the bomb with this code"), however, they include much more of the "wrapping" around them (like HTTP request, response, POST, response checking). They typically don't have extensive setup/teardown, but can for certain cases. Like unit tests, state should not be maintained between tests because the tests are independent.
Integration tests. These tests in Rails represent testing many controllers interacting. This is generally all of the pieces of the application working together in a flow that would represent a complete user interaction with the application from start to finish. State is very important in these tests, and will change throughout a test, and need to be maintained between each action in a single Integration test. In our Overlord application, this would really involve chaining together a bunch of functional tests to represent a single application use - start, activate, deactivate. Or even start, activate, deactivate, start again, deactivate attempt, explode - in sequence without clearing the state. In a real application, Integration tests would interact with all of the parts of an application, and would not use mocks or doubles for any part of it (they still use test data).
We use Capybara for functional and integrations tests. Its strengths are that it can integrate easily with our stack (Rails/Sinatra, RSpec, Cucumber). It can fully exercise a webpage with Selenium (including javascript interactions), or more quickly interact directly with a local webserver in headless mode for performance. It is more difficult to execute the javascript interactions (the selenium or webkit drivers must be used), which can be challenging in a CI environment. Also, the headless tests only work locally, which is another challenge for CI.
The drivers are what actually interacts with the webserver that runs your application. This is how capybara translats the DSL into commands executed against the web application. The default driver is Rack::Test, which works with any Rack based web app (Rails, Sinatra both run on Rack). The advantage of this is that it is fast - both in setup and execution. This executes tests "headlessly", which means without a browser, so that improves performance. Furthermore, because our stack is natively supported, it's easy for us to setup and maintain our tests.
Name a case where a driver other than the default would be necessary, then go implement it with your overlord code.
Another driver is necessary when you want to test something not running on your own local server, or some javascript interaction.
First, tell me what a flapping test is. Then, explain some ways that flapping tests are more destructive than even failing or non-existent tests.
A flapping test is on that fails apparently randomly. Flapping tests reduce the confidence in the entire test suite. Under normal circumstances the build "failing" is an all hands on deck moment (or can be). With flapping tests, when the build fails, people will just say, "it's random" or "run it again". When a real issue comes along, we start to ignore those because we can't distinguish them from random issues.
Name some gotchas when writing tests that can cause them to become fragile, and how to fix those problems.
In a web environment, depending on asynchronous calls to complete in a timely manner or specific order can be troublesome. There is lots of variability in a browser, as well as within service response times. This can be avoided in a test scenario by explicitly waiting for asynchronous calls to complete before continuing with a sequence of events.
In a web environment, making changes to the database directly in test code during a test that also exercises the web interface. The two systems (web, test) may end up on different connections to the database, and may not see the same transactional states. To avoid this, setup all of the data in fixtures before the tests are run, so that the all database changes are going through either the web or the test code, but not both.