Skip to content

Instantly share code, notes, and snippets.

@bkeepers
Last active October 24, 2022 20:02
Show Gist options
  • Save bkeepers/6b1796482363d0f22cf412168954b87e to your computer and use it in GitHub Desktop.
Save bkeepers/6b1796482363d0f22cf412168954b87e to your computer and use it in GitHub Desktop.
Fail Rails system tests if there are any JavaScript errors

We're on the cusp of 2022 and most software is now written in JavaScript, but it is still virtually impossible to fail integration tests on a JavaScript error and get a useful stack trace from the browser.

This gist includes my description of the problem and sample code from my attempts to solve it.

What I want:

  • Integration test should fail (preferabbly immediately) on any uncaught JavaScript error (window.onerror, unhandledrejection, etc)
  • The stack trace should use source maps and be close to what browsers show in DevTools

The tools I'm currently using:

  • Ruby on Rails system tests
  • which uses Capybara
  • driven by Selenium WebDriver using headless chrome

What I've tried:

  • assert_no_console_messages defined below asserts that there is nothing in the browser console after each test. This is insufficient because:
    1. It only includes the error message and not the stack trace
    2. It's only checked at the end of the tests (or manually by calling the assertion after each step), so it's very difficult to figure out which step in the test triggered the error or where in the source code it is coming from.
  • A custom error tracker in the app to capture errors (see errors.js), which can be accessed from the tests. assert_no_js_errors defined below is an improvement on checking for console messages, but is still insufficient because:
    1. error.stack (in all browsers) does not use sourcemaps, so the stack trace is virtual useless in a modern app
    2. Errors are not persisted, so assert_no_js_errors must be called on every page after asynchronous actions, but before navigating to a different page (calling the assertion after one of the other Capybara assertions like assert_content helps becasue it will wait until the expected behavior is done. Errors could be sent to the server or persisted in localStorage, which would be an improvement.
    3. Assertions are still only performed manually or at the end of the test instead of when the error occurs

Opportunities for improvement:

  • stacktrace.js looks promising for improving the stack traces. I tried TraceKit, but it is very outdated and does not support source maps.

Any other ideas?

<!-- app/layouts/application.html.erb -->
<%= javascript_include_tag "errors" if Rails.env.test? %>
# test/application_system_test_case.rb
require "test_helper"
class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
driven_by :selenium, using: :headless_chrome, screen_size: [1400, 1400]
teardown do
assert_no_js_errors verify: false
assert_no_console_messages
end
def assert_no_js_errors(verify: true)
errors = JSON.parse(evaluate_script("JSON.stringify(window.errorTracker?.errors || false)"))
if verify_tracking
assert errors.is_a?(Array), "Error tracking not working. Make sure `errors.js` is loaded in tests"
end
assert errors.empty?, errors.map { |error| error["stack"] }.join("\n\n") if errors
end
def assert_no_console_messages(level = nil) # "SEVERE"
logs = page.driver.browser.logs.get(:browser)
logs = logs.select { |log| log.level == level } if level
assert logs.empty?, "Unexpected console messages:\n" + logs.map(&:message).join("\n\n")
end
end
// app/assets/javascript/errors.js
class ErrorTracker {
constructor () {
this.errors = []
}
capture (error) {
// Missing stack trace for some reason. Try to generate one
if (!error.stack) {
error.stack = (new Error(error.message || error)).stack
}
this.errors.push({
name: error.name, // e.g. ReferenceError
message: error.message || error, // e.g. x is undefined
url: document.location.href,
stack: error.stack
})
}
}
const errorTracker = new ErrorTracker()
window.onerror = function (msg, url, lineNo, columnNo, error) {
errorTracker.capture(error)
}
window.addEventListener('unhandledrejection', event => {
errorTracker.capture(event.reason)
})
// Make available to tests
window.errorTracker = errorTracker
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment