Used properly, the waiting strategy that certain Capybara matchers employ can solve most of those pesky intermittently failing full-stack/browser tests. Pretty much all of them† are due to race conditions between the processes running your tests, the app you're testing, and the browser (which includes both the rendering of the dom, your javascript running there, and if you're calling outside services, The Internet.) The race condition may resolve with 100% consistency on your local dev machine, but when your CI server runs the code, sometimes one process wins, sometimes another. The result is an intermittently failing test that you can only repro on a remote CI machine.
If your developent environment is significantly different than the CI environment, the barriers to effectively troubleshooting the problem may be too high. What I've seen a lot of developers do is just "punt" and insert a sleep(2) in their tests to increase the probabilty that the other processes finish first. And then write the test assuming this will be the case every time. This has a couple of negative effects:
-
It slows your tests down by guaranteeing that this one test will take at least 2 seconds, even if everything is running fast and the delay, this time, is unneccesary.
-
Produces a false failure by guaranteeing that this line in the test will fail after 2 seconds--even if the system was going to do the right thing, albeit it slowly.
Capybara's built-in waiting matchers solve this with the following behavior: If an assertion fails, it will loop running the test over and over again. When it passes, it moves on. If it doesn't pass within 2 seconds (or whatever you have your Capybara.max_default_wait_time set to) then, and only then, does it fail.
This solves point 1. above. But only slightly improves the situation for 2. by being a little more explicit about the timeout setting.
I've heard advice given for these situations: "To make your test wait for an AJAX call to complete, make it so that a successfull AJAX response changes something on the page. And then assert that that thing is changed with a matcher that has the waiting behavior." This is good advice a lot of the time.
Let's say that we want to see if an update on one page updates content on the "My Account" page.
click_button("Update via AJAX")
expect(page).to have_css('#status', text: 'Updated')
visit("My Account")
expect(page).to have_content('New Data')
The second line will wait until it sees the updated browser element, and then the rest of the test can run.
A case I saw recently where this advice won't work, was a shopping cart checkout form that used Stripe for payment processing. The flow there was:
- User enters their billing information including credit card number.
- User hits 'submit'
- Stripe.js takes that credit card info, makes a call to Stripe.com to validate and store it, and returns a token/identifier for the card. The token is appended to the form as a hidden input.
- The entire form is then submitted to the Rails app to proceed with the checkout.
My test might look like:
fill_in(:last_form_field, with: 'last bit of data')
click_button('Submit')
expect(page).to have_css('input[type=hidden][name=stripe_token]')
do other stuff...
In this case, something on the page (the hidden input) is being modified when the AJAX call to stripe returns. However, immediately after that, javascript POSTs the form to our app. We've created a race condition between the looping matcher and the browser. Capybara may or may not catch the change before the form is submitted and the page goes away.
The solution here is to use a different matcher for the last assertion that has nothing to do with content on the page. There is a matcher for current_path that waits:
expect(page).to have_current_path('/order_confirmation')
The negative version works too:
expect(page).not_to have_current_path('/order_form')
If your path has a query string that may change, there is option to ignore it:
expect(page).to have_current_path('/order_confirmation', only_path: true)
Using one of these, we can ensure that the test will move on only after Stripe returns its payload and the form is successfully submitted.
† In the days of old, before hashes were ordered, the second-most common reason for tests that passed on some machines and not others was an inadvertant dependency on the order of a hash. It would typically be consistent per machine and thus similarly hard to debug.