Skip to content

Instantly share code, notes, and snippets.

@HazelGrant

HazelGrant/test_thoughts.md

Last active September 9, 2016 18:45
Show Gist options
  • Save HazelGrant/a9c18f9d436c04931723822389fd79fb to your computer and use it in GitHub Desktop.
Save HazelGrant/a9c18f9d436c04931723822389fd79fb to your computer and use it in GitHub Desktop.
Download ZIP
thoughts on testing for work
Raw
test_thoughts.md
title
Testing Thoughts

Internal Guidelines

Basics

Generally we use Minitest. We still need some familiarity with Rspec because we inherit some legacy projects which use it. Matt, on why we use Minitest:

I like to use MiniTest for writing tests because of its simplicity and it provides a complete set of testing facilities without the noise. Best of all it's shipped with Ruby [...].

Matt's post from 2013 remains relevant and quite helpful.

Coverage

Our agreed-upon baseline goal for every project is 80% test coverage.

In order to monitor the coverage, we use SimpleCov. The default setup for this is in test/test_helper.rb and MUST come before any other code in the file (or the coverage will not be calculated in a sane/predictable manner).

Gemfile

group :test do
  gem 'simplecov', require: false
end

test/test_helper.rb

require 'simplecov'
SimpleCov.start

ENV['RAILS_ENV'] ||= 'test'
require File.expand_path('../../config/environment', __FILE__)

etc.

See SimpleCov Documentation for more information.

TODO

Test & investigate:

SimpleCov must be running in the process that you want the code coverage analysis to happen on. When testing a server process (i.e. a JSON API endpoint) via a separate test process (i.e. when using Selenium) where you want to see all code executed by the rails server, and not just code executed in your actual test files, you'll want to add something like this to the top of script/rails (or bin/rails for Rails 4):

if ENV['RAILS_ENV'] == 'test'
  require 'simplecov'
  SimpleCov.start 'rails'
  puts "required SimpleCov"
end

Managing the time to test (Potential blog post)

On as many projects as possible, incorporate testing into estimates for all features. Consider the time it takes to test to be part and parcel with the time it takes to deliver. Use test-driven development where it makes sense, especially if it helps you to break down the problem. When and/or if you find that test-driven development is hindering your ability to explore the problem, explore in code until you have a better understanding and then return to testing.

On fixing bugs, always write a test for the bug fix. This provides us with a robust regression test suite and helps prevent repeat bugs. It also documents potential problems with the application code and let's people reviewing the test suite understand the major concerns of the application.

On projects where budget/time is limited, highest priority is to ship, still absolutely slow down and take the time to write tests. Taking ten or fifteen minutes to write a test for the code you're working on is inexpensive now and profitable in the long-run. A code base with 20% test coverage and a limited budget isn't going to get to 80% coverage over night or even over the course of a week or a month. But testing should always be lurking in the back of your mind, and whenever you see the opportunity to increase coverage (especially if it's in a part of the code base that is relied upon in multiple places and/or is prone to breaking), take it.

Increasing test coverage from 20% to 21% is improvement. Prioritize regular incremental improvement over sweeping changes. (Avoid sweeping changes.)

One way to keep testing constantly in mind is to run your test suite before you commit anything. Get into the habit. Map bundle exec rake to br (or some equal short, memorable alias).

~/.bash_profile

alias br='bundle exec rake'

For multiple reasons, the advice in this section may not always be practical.

Reasons it might not be practical:

Advice that's not practical: Running tests before every commit

Reasons that might not be practical:

  • Tests are sloooooooooow
  • No, really, like, painfully, mind-bogglingly, hideously, I'm-gonna-be-here-all-frakking-day slow
  • Or they're just slow enough to break context and waste ten minutes that could have been spent doing something useful
  • Tests are flaky and inconsistent
  • Tests are slow, flaky, and inconsistent, and make you want to stab your eyes out
  • When tests fail, you spend more time rewriting the tests than fixing anything that might actually be wrong with the code base

How to address those reasons practically:

  • Manually test things while working with crappy code bases
  • Separate rake tasks for 'good' tests versus 'bad' tests
  • Spend time speeding up tests - identify causes for slow tests and address them one at a time
  • Run super slow tests in automated build process and run faster tests before committing
  • Run tests that directly address the area you've been working with
  • Address flakiness and inconsistencies in test failures
  • Make up personal, app-specific strategies for improving the code base
  • Run gigantic test suites in parallel

Advise that's not practical: Taking ten or fifteen minutes to write a test

Reasons that might not be practical:

  • They're integration tests, and it takes ten or fifteen minutes just to figure out what syntax to use that won't explode every third run
  • Figuring out how to separate a bunch of junk from what needs to be tested takes way too long
  • It's really just gonna take hours and hours to build any sort of test suite that doesn't completely suck and the client doesn't have the budget
  • All of the code is way too tightly coupled to write reasonable tests

How to address those reasons practically:

  • Separate test files that "work"
  • Focus on high-level integration tests for tightly-coupled code that test expected/desired behavior
  • etc.

TODO

  • Write section on test speed
  • Write section on damage control for flaky test suites
  • Write section on using semaphore to test/build with acceptance/integration and using unit tests in dev
  • Prefer unit over integration
  • Seriously consider: is an integration test here necessary?

Tools

  • Minitest
  • Capybara
  • Fabrication
  • SimpleCov
  • DatabaseCleaner
  • Mocha
  • Webkit Capybara
  • Selenium Webkit
  • Pry
  • Fakeweb
  • Launchy
  • Capybara Screenshot
  • Awesome Print
  • Minitest Spec Syntax
  • Poltergeist - prefer for newer projects? It's faster than selenium-webkit, but incompatible with it

Best Practices

  • Integration tests that deal with Javascript
  • Integration tests that don't deal with Javascript
  • Model tests
  • Other unit tests

Coping with data between tests:

Using the let syntax provides data which is only created in the tests that calls for it. This makes tests faster and also helsp keep the database clean between tests.

require 'test_helper'

describe ThingTest do
  include TestHelper

  let(:some_data) { Fabricate(:data) }

  it 'exists in this test' do
    assert some_data.exists?
  end

  it 'does not exist in this test' do
    assert true
  end
end

vs.

require 'test_helper'

describe ThingTest do
  include TestHelper

  before do
    @some_data = Fabricate(:data)
  end

  it 'exists here' do
    assert @some_data.exists?
  end

  it 'also exists in this test' do
    assert true
  end
end

This can lead to some nasty bugs, especially in integration/system tests where the connection to the database might be different from the rest of the test suite. Let's say object Data has a name with validates :name, presence: true, uniqueness: true, and we're doing some integration testing where we have to deal with Data:

require 'system_helper'

describe 'ThingIntegration' do
  include SystemHelper

  before do
    @some_data = Fabricate(:data, name: 'Some Unique Name')
  end

  it 'exists here' do
    visit data_path(@some_data)
    assert true
  end

  it 'will blow up unexpectedly' do
    visit data_path(@some_data)
    assert true
  end
end

One of these tests will pass - the first one to run (test order should be random). The other will blow up because @some_data will be created, while the previous test's @some_data has already been created and is still floating around in the database. DatabaseCleaner didn't clean it up because we're using a driver that causes these tests to run with a different database connection than our unit tests.

Generating random names with a gem like Faker can cause even more problems - because Faker can't guarantee that it'll generate a name it hasn't generated before, and therefore sometimes your tests will pass and sometimes they won't.

Using let(some_data) { Fabricate(:data, name: 'Some Unique Name') } should prevent this? --> TEST ME

A quick gotcha with let -> the object doesn't exist until you call it. Sometimes you need to make an assertion based on the object already existing. In this case, it may be faster to just add assert some_data to the top of the test, or:

require 'system_helper'

describe 'ThingIntegration' do
  include SystemHelper

  before do
    @some_data ||= Fabricate(:data)
  end
end

Something to look into (why aren't we using this?):

Monkey patch that forces all threads to share the same connection, which works on Capybara because it starts the web server in a thread:

class ActiveRecord::Base
  mattr_accessor :shared_connection
  @@shared_connection = nil

  def self.connection
    @@shared_connection || retrieve_connection
  end
end

ActiveRecord::Base.shared_connection = ActiveRecord::Base.connection

I have this blog post somewhere.

Project Defaults what they are, why they are, where to find them, what to cut/use depending on the project

stubbing user/admin auth: ApplicationController.any_instance.stubs(:authenticate_admin).returns(nil) ApplicationController.any_instance.stubs(:authenticate_user).returns(nil)

  • fill_trix
  • scroll_to
  • attempt_login_with
  • etc.

Selenium vs. Webkit

Bucket of useful system/test helper methods that can be transferred between projects

  • wait_for_ajax
  • handle_js_confirm
  • t(&args)
  • post_json/get_json/put_json/delete_json/etc.

Useful tactics for writing about tests

  • For clients: metaphors, simple language, understanding what they are looking for, being able to relate what we're doing to what you're looking for, put aside developer sensibilities and aesthetics, let's get pragmatic about this, you have a business to run, you have research to do, you have users to gather
  • For new developers: identify ALL ASSUMPTIONS. if you are making an assumption, point it out. let new developer know that if they see an assumption we're making without being aware of it, they should point it out. question. do not be afraid to say "I don't know" or "I was wrong"
  • For team: examples, data, code snippets, rationale, building habits, code reviews, pull requests, questioning, concise...

Simplest possible way to explain these things. Consistent test_helper file to start with.

test/test_helper.rb

require 'simplecov'
SimpleCov.start

require 'rubygems'
require 'database_cleaner'

ENV["RAILS_ENV"] ||= "test"
require File.expand_path("../../config/environment", __FILE__)

require "fabrication"
Fabrication.configure do |config|
  fabricator_path = "test/fabricators"
end

module TestHelper
  def setup
    DatabaseCleaner.strategy = :truncation
    DatabaseCleaner.clean
  end

  def teardown
  end
end

test/system_helper.rb

require 'test_helper'
require 'rack/test'
require 'mocha/setup'
require 'capybara/dsl'
require 'capybara/rails'
require 'capybara/webkit'

module SystemHelper
  include Capybara::DSL
  include Rack::Test::methods
  include TestHelper
  include Rails.application.routes.url_helpers

  Capybara::Webkit.configure do |config|
    config.block_unknown_urls
  end

  def setup_capybara
    Capybara.javascript_driver = :webkit
    Capybara.current_driver = Capybara.javascript_driver
    Capybara.default_max_wait_time = 3
  end

  def teardown_capybara
    Capybara.reset_sessions!
    Capybara.use_default_driver
  end
end

Concepts

  • Integration/Acceptance
    • Capybara
    • Coping with Javascript-heavy applications
  • Data management
    • Keeping the database clean between tests
    • Using only the data needed to get the point across
    • Performing only the actions necessary to get the point across (don't save when an unsaved object will provide the information you need)
  • Keeping track of coverage
    • Lines covered - at least %80/application is what we're aiming for
    • Keep track of any lines of code that get hit by tests dozens/hundreds/even thousands of times - in a perfect world, each line of code is hit once
    • Do tests pass/fail when they need to - are you actually testing what you think you're testing
  • Shared assumptions/"the way things are done" & explanations
    • "the way things are done" enables trust and consistency - reduces cognitive overhead, AT THE SAME TIME, comes with risks of complacency and inability to adapt to changing requirements/new tools, so question, but do not change unless you can convince everyone that the change will genuinely make things better
    • minitest - simple, it's "just ruby", no expansive DSL to keep in mind - it's the language of the application, shares roughly the same context
    • capybara - slower, but useful in integration/system tests due to ease of navigating the browser
  • Finding the time to test
  • Deciding WHAT to test
  • Balancing client budget with testing needs
  • Dealing with untested code
  • Give in to the way the code base does it, introduce change gradually, try to naturally fit into what's already present - if what's present sucks, then make it suck less incrementally
  • Write tests for bugs before making bug fixes, as often as possible

Resources

  • Matt's post from 2013

Levels

  • Explain testing to clients
  • Explain testing to new developers
  • Talk about testing (consistently) internally
    • What can we agree we need to trust from our tests?

Random thoughts in no particular order

  • Keep CSS selectors out of the code - prefer to test the things that are not going to change
  • This is never going to be perfectly possible - big changes will happen, and they will break tests - bear this in mind and make maintaining the test suite as important as getting the features right
  • We use Minitest
  • We use spec syntax
  • Do not nest describe blocks more than two levels deep (too much context)
  • If it's hard to set up a test, there's too much going on
  • When you fix specific bugs, use TDD
  • When writing a feature where TDD gets in the way instead of helping, test after the fact and use a "tdd"-ish process, where you comment out segments of code, write a test that should fail, ensure the test fails, uncomment code, ensure test passes
  • Prefer let(:variable) { SOMETHING } over before { variable = SOMETHING } - keep the database clean between tests!
  • Use Capybara for integration/acceptance/feature specs (do we have any agreed upon way of doing this? Accept that each client may be different, and have a preference for greenfields?)
  • Use headless driver when possible - it's faster, but not as easy to debug. We could keep a non-headless driver around for debugging but there are inconsistencies between the two.
  • Tests that pass locally often fail in Semaphore, inconsistently. We have a few projects with tests that consistently fail to be consistent when they hit Semaphore - put some sort of bounty on figuring these things out?
  • Prefer unit tests over system/integration specs - they're faster, and the easier they are to write, the more modular the code is
  • Keep structure of language in tests consistent
  • Keep similar tests similar to each other - let the differences stand out
  • Avoid Mocks & Stubs for the most part - use Mocha where absolutely necessary
  • Avoid saving or creating objects in the database when instantiating will work just as well (speed)
  • Do not create more objects that you need
  • Do not let the data confuse the intent of the test - use data that is obviously junk so that it's obvious what does and does not matter (example: instead of 'company_name: AT&T' use 'company_name: Some Random Company Name' or 'company_name: zzzzz')
  • TestHelper module for unit test helper methhods
  • AcceptanceHelper/SystemHelper for acceptance/system test helper methods (generally with capybara)
  • For System tests, it's faster to stub out authentication instead of manually having the test user sign in before each and every test - unless specifically testing sign in/sign out process
  • If you port code over from other projects, make sure to bring along associated tests and double check for context-specific code that needs to be removed/changed
  • Use DatabaseCleaner to ensure that sweet clean database
  • Use SimpleCov to track progress
  • Things to look for: large swatches of code that are missed, or particular segments of code that are hit by the test swuite an unreasonable amount of times
  • Put SimpleCov.start AT THE VERY TOP OF THE TEST HELPER FILE or things will get missed

Points for clients:

  • Yeah, we spend time on it - is worth it. Code bases quickly grow complex and we developers need to make up for the fact that our brains are not giant super computers by putting the context we know when we know it into formal terms, so that the knowledge of that context still exists in the code base even after we've moved on to a different context. That is probably not the way to explain to clients.
  • Try again.
  • Money. Time now saves time later. Time now keeps the project from being dead later. Time now makes your code base healthier. It's healthier because of bugs. Let's try to explain this to a non-developer.

Think about your calendar. You have a lot to do on any given day and it can be difficult to keep that all in your head. So you use technological extensions that exist outside of yourself in order to store what you know now about what you need to do tomorrow. Now you don't need to worry about whether you remember that knowledge tomorrow, because your calendar is going to ping you. You save a lot of time and energy not having to think about all of the things you have to do all of the time. If you had to sit down every few minutes and think to yourself, "Do I have anything I need to do in the next hour?" and you had to recall all of the things that you have to do in the next few days, or weeks, or months, and think about when and where they happen and whether or not any of those things happen in the next hour, you would lose a lot of time just trying to remember everything. But you don't often have to think to yourself about what you're saving by using the calendar. And yes, it requires some maintenance to keep the calendar. Tests are the same way.

Tests encode intention. They keep the context even when we lose it, because we write the context into the tests when it's fresh in our minds. It's not that we don't work hard to think about what we might be effecting in the code base when we make changes - just like you try to be aware of when your next meeting is so you're mentally prepared even if you know the calendar is going to buzz you and you don't actually NEED to know. It's a layer of security. It's a layer of security that becomes more and more important as the code base gets bigger and capable of doing more and more things. However, if we don't take care of the test suite while the code base is small, then getting the test suite up to snuff when the code base is big enough for it to REALLY matter is a huge and laborous feat, requiring us to remember all of the context. It's too easy to miss everything.

You keep your appointments more consistently with a calendar and we keep our code base bug free more consistently with a good test suite. And the better that test suite is designed and managed, the easier the entire code base is to design and manage - this saves time. Think about it - take my word for it - blah - it's much easier to put the context as it's supposed to be down in a test when the context is fresh in your mind. Yes it adds some time to the process of developing. But when we find a bug in untested code, it takes even MORE time, because we have to reliably reproduce the bug, figure out where it's coming from, make a fix, and then somehow guarantee that the fix we made didn't just break or touch a dozen other things.

Relate what clients care about to testing - how does testing lead to more users? How does testing lead to happier users? How does testing lead to users more willing to part with their cash? >_> Blah blah blah.

Points for new developers:

  • Assumptions we make and things we do because consistency is important: minitest over rspec, spec syntax, keep to basic assertions (assert, refute) and try to avoid more complicated ones (assert_equal, however, gives a better error message than assert this == that)
  • Consider taking the time to write the test to be AS IMPORTANT, AN INTEGRAL PART OF writing the code that needs to get written
  • Run the test suite as often as possible - ESPECIALLY if the code base doesn't go through Semaphore (it really should go through Semaphore though)
  • If the test suite is unbearably slow, take some time to speed it up
  • Also research more about testing and help us get some language together to explain to clients why this is worth our time/their money
  • Guiding principles - a lot of them as seen in "thoughts in no particular order"

How we talk about testing internally:

  • Do you run the test suite before commiting? If not, why? What can we do to make this easier for you to do and remember?
  • Do you write tests when you submit a change?

THE MORE JS-INTENSIVE AN APP IS, THE WORSE THE TEST SUITE IS GOING TO BE when testing js, do not test specific js functionality - test the IMPORTANT side effects

acceptance tests that need to work with JS and require a JS-friendly driver need special care they are prone to fits and spats of crappiness - lots of waiting for ajax requests and whatnot, lots of reloading the browser again, and again, and again in order to get all of the context rolling for each and every individual test

Do not forget nil values. Do not forget that the data IS GOING TO COME IN WONKY AT SOME POINT.

Be careful with tests that loop over data and make an assertion per loop - your messages are going to suck and it'll be harder to debug

In tests, it's important to balance the desire to make code DRY with the understanding that the tests need to be thorough (we look for > 80% per project), fast, readable, easy to debug, etc.

We could keep a list of projects in need of testing love. Of course we also have to bear in mind that some clients do not have the budget to get crazy with testing. But taking ten or fifteen minutes here or there to bump a test suite that's at 30% to 33% is worth the time and effort - especially if it's a consistent thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment