Skip to content

Instantly share code, notes, and snippets.

@nikomatsakis

nikomatsakis/parallel-rustc.md Secret

Last active May 2, 2019
Embed
What would you like to do?

I'm putting out a call. Perhaps you read my recent blog post about how rustc needs more than coders? Well, right now we are in the midst of trying to ship a parallel rustc, which has the promise of making a big impact on compilation times, but if we're going to push this over the finish line, it's going to need someone to help keep the effort organized. Perhaps this could be you! (Or maybe more than one of you.)

Where we are

Up until now, Zoxc has been pursuing the engineering aspects of this mostly on their own, with michaelwoerister doing a lot of reviewing. Currently, support for parallel compilation is in master, but it is disabled by default on nightly builds. This is because it adds some overhead that wouldn't otherwise be needed (it requires the compiler to use locks, for example, on some internal data structures). Zoxc is presently experimenting with ways to reduce that overhead.

Our general plan is to deploy parallel rustc in three phases:

  • Phase 1: experimental, off by default. In this mode, we would distribute nightly builds that can support parallelism but which don't by default. This implies a certain amount of overhead for the default user, but hopefully that overhead will be quite low.
    • During this time, we will be asking people to opt-in to parallelism, likely via an environment variable. We will ask them to use various bits of tooling and submit performance measurements and bug reports.
    • This will last for one release. If all goes well, this "off by default" parallel support can ride the trains as normal. If we encounter a major problem, or the sequential overhead is higher than anticipated, we'll disable parallel support before branching to beta and start over.
  • Phase 2: experimental, on by default. Presuming all goes well in the previous phase, we'll enable parallel builds by default (in nightly builds).
    • During this time, we will still ask people to send us data and experience reports. We'll keep an extra eye out for parallelism related problems.
  • Phase 3: shipped. Presuming that no "show stopper" bugs are found in phase 2, we will let the "on by default" support ride the trains into stable, and rustc will be parallel!

For a more detailed look at our plan, you can check out this gist.

Where we need help

At present, the place where we need the most help is in the organizational aspects of things. Specifically, I am looking for a person (or set of people) to help guide this overall plan forward. This would involve dividing up the concrete plan into smaller, concrete steps, like the following:

  • Writing the overall announcements (and/or maybe blog posts)
  • Documenting the exact steps people should take to give us feedback during the experimental periods
    • Ideally, we would make this very easy for people, and have shell scripts or other tooling that dumps all the relevant information we might want
  • When we are running experiments, monitoring the internals threads to track and summarize the feedback

Some of those tasks you might do yourself, but for others, you might try to recruit people to help out -- if you are going to do so, then the more direction you can give the better! And of course you can always draw on folks like me to help where needed.

Think you'd like to help out?

Drop onto the #t-compiler/wg-parallel-rustc Zulip channel and introduce yourself!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.