This written in response to http://bit.ly/2dJEXSW. While your test suite might look like the provided pyramid based on number of tests, I think it accidentally sends the wrong impression that unit tests are the most valuable in your codebase. I disagree.
At Gowalla we had a developer obsessed with unit tests. At the time unit tests were in vouge and they meant fast, integration tests meant slow. So they spent a ton of time writing them.
This caused two problems. Having enough unit tests to cover everything that would be exerciesed in an integration test meant a HUGE amount of unit tests. This caused the suite to be very brittle. Trivial changes would break the test suite, and things that would break the site, would not break the test suite. Once we had our signup page broken for 3 days and no one realized because it wasn't tested. The second problem was the suite was SLOW, like really slow. So no one wrote new tests, no one ran the suite. It was a fight to get a CI server. We eventually threw away a large number of unit tests during a re-write and wrote integration tests instead.
In this scenario the unit tests were basically one person's design tool to get feedback on a specific set of components. I support unit tests when they're layered on top of integration tests, because I need to be able to throw them away and still know if my overall application is breaking or failing.
I aspire to break more components of the Heroku buildpack into unit-tests, because it does make it easier to iterate on. It provides immediate feedback, but without a base of integration tests to confirm that my unit tests aren't giving me a false sense of security.
My problems with advocating for unit tests over integration, is that people listen. There are plenty of codebases with little or no tests, unlike platformatec which is a consultancy and tests, in one Rubyist's famous words, "all the fucking time". Starting with unit tests is a mistake, and advocating for such is setting up people who haven't worked with extensive integration tests for failure.
I spoke about this topic (kindof) at length in "Testing the Untestable" https://www.youtube.com/watch?v=QHMKIHkY1nM. In which i talk about how we integration test the Heroku Ruby Buildpack by deploying 70-ish apps to Heroku and it takes under 10 minutes to run on Travis.
My basic premise in the talk was that you MUST test the things that hurt when they fail. Any other tests after that are gravy. I support good unit tests. I think this article isn't very pragmatic and will accidentaly harm the majority of its readers.
Hi @schnemms, I agree with you that that unit tests can "leave you with things that would break the site, would not break the test suite." Especially unit tests that are so isolated with stubbing and mocks. These tests lead to something that nothing real is done. Then, of course, it's not useful. In a Rails application, we almost never have a "pure unit test." Most of the model tests, for example, can hit the database at some point. There's some jargon in the community (or maybe just in Brazil?) that people still call them unit tests. Even if we use the views tests, models, helpers we call them "unit." Maybe it was the first point of confusion.
The main point of the article is avoiding only using Capybara tests(acceptance tests or UI tests) to test your application and explore more the other folders(controllers, requests, helpers, views, models), even if some of the acceptance overlaps with integration or unit tests. It's good because the integration and unit tests help you have better feedback. The post wasn't about which type of test has more value(it depends on the application and team context), it's was more about what we are missing if we ignore the integration and unit tests.
I thought it was evident in conclusion with these points:
Maybe I should make it clearer in the blog post. I'll think about it. Thanks for you feedback.