Configuring CircleCI with Jest and test splitting

Configuring CircleCI with Jest.md

Configuring CircleCI with Jest and test splitting

I've found plenty of documentation online for how to run Jest on CircleCI with test splitting, but none of them have solved two problems that I've run into, with a relatively basic (albeit large) React + Next.js project. And I bet others have these problems, too. Namely:

• Jest on CircleCI won't run tests with [ or ] in the filename. (That's a particular problem when using NextJS.)
• CircleCI won't split by timings when using Jest with jest-junit

I've found solutions for both of these problems, but it wasn't easy. I can't be the first person to encounter these problems, right?

The dream config

If you're just here for the spoilers, I've highlighted two lines here that you probably want in your CircleCI config if you're using Jest, and want to split your test runs across multiple runners.

jobs:
  unitTests:
    docker:
      - image: cimg/node:16.13.2
    parallelism: 4
    steps:
      - checkout
      - run:
          name: Install Yarn packages
          command: yarn
      - run:
          name: Run Jest tests using test splitting
          command: |
            TESTFILES=$(circleci tests glob "src/**/*.test.js" | circleci tests split --split-by=timings)
            # Per https://circleci.com/docs/2.0/collect-test-data/#jest, uses --runInBand to avoid slowing down Circle
            yarn test --ci --runInBand --reporters=default --reporters=jest-junit --runTestsByPath $TESTFILES  # You need --runTestsByPath to make sure all the test suites actually run
          environment:
            JEST_JUNIT_OUTPUT_DIR: './reports/junit'
            JEST_JUNIT_ADD_FILE_ATTRIBUTE: 'true'  # You need this to make --split-by=timings work. Once added, it'll work from the second run onwards.
      - store_test_results:
          path: ./reports
      - store_artifacts:
          path: ./reports

Problems and solutions

Using brackets in filenames with Jest, CircleCI, and test splitting

Simple solution: pass --runTestsByPath to Jest.

Why? Here's how test splitting works:

• CircleCI, using circleci tests glob finds all files matching the pattern you provide; say, src/**/*.test.js.
  • If your naming pattern matches ours, then that means a test suite for [id].js would be named [id].test.js and reside in __tests__. (We name some files like that so we can use Next.js Dynamic Routes).
• Then, using circleci tests split, CircleCI will filter that list of files, so you're only left with the ones your runnner needs to run.
• You pass the result of that into jest.
• Suppose [id].test.js is in the set of files for your runner. Jest will interpret that filename as a RegEx pattern, which... is a problem, because that means [id].test.js doesn't mean "the file named [id].test.js; rather, it means "any file named i.test.js or d.test.js.
• As a result, the tests in [id].test.js get ignored entirely.
• But if you pass --runTestsByPath to jest, it won't interpret the passed arguments as patterns; instead, it'll interpret them as direct filename references. Which is good, because that's what circleci tests split gives you. We don't need pattern matching here.

If you think this problem might affect you, check the number of tests you run on CircleCI, and compare it to running jest locally. If the numbers aren't the same, you're missing something.

Splitting by timings doesn't work

Simple solution: if you're using jest and jest-junit, you need to set JEST_JUNIT_ADD_FILE_ATTRIBUTE='true' in your environment.

I've seen a load of sample CircleCI configs online using jest, which pass --split-by=timings to circleci tests split, but don't explain how it works.

And if you follow those configs verbatim, you'll see this easily-ignored message in your CircleCI output:

Error autodetecting timing type, falling back to weighting by name. Autodetect no matching filename or classname.  If file names are used, double check paths for absolute vs relative.
Example input file: "src/__tests__/_error.test.js"
Example file from timings: ""

Why? By default, jest-junit doesn't add filenames to the report it produces. You can look at the report for yourself: not a filename to be found.

But CircleCI is looking for filenames. After all, both circleci tests glob and circleci tests split return filenames. So when it tries to split by timings, circleci tests split will ask CircleCI for timings from previous test runs, by filename, and CircleCI will respond with a "uh... I don't have that. None of these reports have filenames in them. I only have test names."

If you look at the README for jest-junit, though, you'll see a bit of a hidden gem in the config options:

JEST_JUNIT_ADD_FILE_ATTRIBUTE: Add file attribute to the output. This config is primarily for Circle CI. This setting provides richer details but may break on other CI platforms. Must be a string.

This has existed in jest-junit for several major version releases. But I haven't seen it mentioned anywhere, other than on its README. I definitely haven't seen it, or its friend addFileAttribute, mentioned in any of the articles I've seen explaining how to split Jest tests on Circle.

If you turn it on, then every test result in the junit.xml that gets produced will have a file attribute, denoting which test file the test was in. Now, Circle has enough data to figure out how long each test file takes to run.