Overview of how we're incorporating the RSpec runtime log data (using TeamCity as our CI service):

incorporating-runtime-data-on-ci.md

Overview of how we're incorporating the runtime log data (using TeamCity as our CI service):

1. After a TeamCity build is completed, the runtime log file is stored in the Artifacts section for each build.
2. Each new build runs a process that issues an API call to TeamCity. This API call looks for the latest, successful build and downloads the log file. That request something like this: HTTParty.get("#{team-city-uri}/builds/#{parameters-that-identify-a-completed-successful-build}/artifacts/content/parallel_rspec_runtime.log")
3. After fetching the file, we read the file, remove any uwanted noise caused by stdout, and write the results to a new file.
4. When calling parallel_tests, we pass the new file(1). Example: parallel_rspec --group-by runtime --runtime-log /logs/parallel_rspec_runtime.log -- path_to_spec_files

(1) Word of caution, Parallel Tests states that "when a runtime log is filled" the default group_by strategy is runtime. However, after digging into the source of the gem, we found that when a flag for an unrelated feature was passed, if the gem was running in default mode, the strategy would be set to “filesize”. More details here: https://github.com/grosser/parallel_tests/blob/9bc92338e2668ca4c2df81ba79a38759fcee2300/lib/parallel_tests/cli.rb#L305

To prevent any surprises and ensure you're using the runtime strategy, it's best to be explicit and include --group-by runtime.

Splitting the test files across workers:

The parallel_test flag --only-group supports splitting your test across different machines. Currently, our build process runs the RSpec test suite on one machine that has ~89 CPUs. We use the --only-group flag to assign a set of specs to each CPU. Example below:

parallel_rspec
  --only-group ENV['CPU_NUMBER']
  --group-by runtime
  --runtime-log /logs/parallel_rspec_runtime.log
  -- path_to_spec_files

Using parallel_tests to split the test files is deterministic, so we can tell each CPU (or in the future, each machine, perhaps in your case, each "worker") which "group" of tests to run.

💛 Hope that helps!

Thanks for this. I use Travis and have been able to "share" the runtime log by using the shared cache mechanism. The issue with that though is that each worker gets a copy of the cache, updates it, then pushes it back to storage, so in essence, the last worker wins and it's cache is the last to save, overwriting updates to the runtime log of any of the other workers.

I'm looking into a new "workspaces" feature of travis where it's more like a shared drive between all current workers in a build, but it's still an alpha feature.