As described in detail here, julia can take really excessive amounts of time to execute the first
@everywhere statement on many processes — around 1 hour for thousands of processes — even if the actual code being executed everywhere is trivial. Basically, the
Distributed functions need to be precompiled to make this happen quickly.
This gist provides a simple way to do so — at least on Slurm clusters (though the same principles should apply elsewhere). Just submit
precompile.jl as a batch job (adjusting the
SBATCH directives as needed), and it should create a sysimage that you can use to run future batch jobs. Check end of the log of the Slurm job to see exactly how to use the sysimage.
Note that both the original julia process and all processes created with
addprocs should use the
--sysimage=/path/to/sys_everywhere.so argument. Doing so reduces the time taken to execute the first
@everywhere statement by a factor of ~20 for ~100 processes, and possibly more for a greater number of processes.