Fail Rails system tests if there are any JavaScript errors

README.md

We're on the cusp of 2022 and most software is now written in JavaScript, but it is still virtually impossible to fail integration tests on a JavaScript error and get a useful stack trace from the browser.

This gist includes my description of the problem and sample code from my attempts to solve it.

What I want:

• Integration test should fail (preferabbly immediately) on any uncaught JavaScript error (window.onerror, unhandledrejection, etc)
• The stack trace should use source maps and be close to what browsers show in DevTools

The tools I'm currently using:

• Ruby on Rails system tests
• which uses Capybara
• driven by Selenium WebDriver using headless chrome

What I've tried:

• assert_no_console_messages defined below asserts that there is nothing in the browser console after each test. This is insufficient because:
  1. It only includes the error message and not the stack trace
  2. It's only checked at the end of the tests (or manually by calling the assertion after each step), so it's very difficult to figure out which step in the test triggered the error or where in the source code it is coming from.
• A custom error tracker in the app to capture errors (see errors.js), which can be accessed from the tests. assert_no_js_errors defined below is an improvement on checking for console messages, but is still insufficient because:
  1. error.stack (in all browsers) does not use sourcemaps, so the stack trace is virtual useless in a modern app
  2. Errors are not persisted, so assert_no_js_errors must be called on every page after asynchronous actions, but before navigating to a different page (calling the assertion after one of the other Capybara assertions like assert_content helps becasue it will wait until the expected behavior is done. Errors could be sent to the server or persisted in localStorage, which would be an improvement.
  3. Assertions are still only performed manually or at the end of the test instead of when the error occurs

Opportunities for improvement:

• stacktrace.js looks promising for improving the stack traces. I tried TraceKit, but it is very outdated and does not support source maps.

Any other ideas?

application.html.erb

<!-- app/layouts/application.html.erb -->

<%= javascript_include_tag "errors" if Rails.env.test? %>

application_system_test_case.rb

# test/application_system_test_case.rb
require "test_helper"

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  driven_by :selenium, using: :headless_chrome, screen_size: [1400, 1400]

  teardown do
    assert_no_js_errors verify: false
    assert_no_console_messages
  end

  def assert_no_js_errors(verify: true)
    errors = JSON.parse(evaluate_script("JSON.stringify(window.errorTracker?.errors || false)"))

    if verify_tracking
      assert errors.is_a?(Array), "Error tracking not working. Make sure `errors.js` is loaded in tests"
    end
    assert errors.empty?, errors.map { |error| error["stack"] }.join("\n\n") if errors
  end

  def assert_no_console_messages(level = nil) # "SEVERE"
    logs = page.driver.browser.logs.get(:browser)
    logs = logs.select { |log| log.level == level } if level
    assert logs.empty?, "Unexpected console messages:\n" + logs.map(&:message).join("\n\n")
  end
end

errors.js

// app/assets/javascript/errors.js

class ErrorTracker {
  constructor () {
    this.errors = []
  }

  capture (error) {
    // Missing stack trace for some reason. Try to generate one
    if (!error.stack) {
      error.stack = (new Error(error.message || error)).stack
    }

    this.errors.push({
      name: error.name, // e.g. ReferenceError
      message: error.message || error, // e.g. x is undefined
      url: document.location.href,
      stack: error.stack
    })
  }
}

const errorTracker = new ErrorTracker()

window.onerror = function (msg, url, lineNo, columnNo, error) {
  errorTracker.capture(error)
}

window.addEventListener('unhandledrejection', event => {
  errorTracker.capture(event.reason)
})

// Make available to tests
window.errorTracker = errorTracker