I'm putting out a call. Perhaps you read my recent blog post about how rustc needs more than coders? Well, right now we are in the midst of trying to ship a parallel rustc, which has the promise of making a big impact on compilation times, but if we're going to push this over the finish line, it's going to need someone to help keep the effort organized. Perhaps this could be you! (Or maybe more than one of you.)
Up until now, Zoxc has been pursuing the engineering aspects of this mostly on their own, with michaelwoerister doing a lot of reviewing. Currently, support for parallel compilation is in master, but it is disabled by default on nightly builds. This is because it adds some overhead that wouldn't otherwise be needed (it requires the compiler to use locks, for example, on some internal data structures). Zoxc is presently experimenting with ways to reduce that overhead.
Our general plan is to deploy parallel rustc in three phases:
- Phase 1: experimental, off by default. In this mode, we would
distribute nightly builds that can support parallelism but which
don't by default. This implies a certain amount of overhead for the
default user, but hopefully that overhead will be quite low.
- During this time, we will be asking people to opt-in to parallelism, likely via an environment variable. We will ask them to use various bits of tooling and submit performance measurements and bug reports.
- This will last for one release. If all goes well, this "off by default" parallel support can ride the trains as normal. If we encounter a major problem, or the sequential overhead is higher than anticipated, we'll disable parallel support before branching to beta and start over.
- Phase 2: experimental, on by default. Presuming all goes well in
the previous phase, we'll enable parallel builds by default (in nightly builds).
- During this time, we will still ask people to send us data and experience reports. We'll keep an extra eye out for parallelism related problems.
- Phase 3: shipped. Presuming that no "show stopper" bugs are found in phase 2, we will let the "on by default" support ride the trains into stable, and rustc will be parallel!
For a more detailed look at our plan, you can check out this gist.
At present, the place where we need the most help is in the organizational aspects of things. Specifically, I am looking for a person (or set of people) to help guide this overall plan forward. This would involve dividing up the concrete plan into smaller, concrete steps, like the following:
- Writing the overall announcements (and/or maybe blog posts)
- Documenting the exact steps people should take to give us feedback
during the experimental periods
- Ideally, we would make this very easy for people, and have shell scripts or other tooling that dumps all the relevant information we might want
- When we are running experiments, monitoring the internals threads to track and summarize the feedback
Some of those tasks you might do yourself, but for others, you might try to recruit people to help out -- if you are going to do so, then the more direction you can give the better! And of course you can always draw on folks like me to help where needed.
Drop onto the #t-compiler/wg-parallel-rustc Zulip channel and introduce yourself!