Julia 1.9 launch benchmark.md

Benchmarks

These are benchmarks about https://discourse.julialang.org/t/julia-1-9-0-beta2-precompilation-taking-almost-twice-as-long/92498 and fonsp/Pluto.jl#2453 . These benchmarks measure both precompilation time and TTFX by manually clearing the .julia/compiled folder before each sample, to simulate the experience of first-time Julia users.

Notebooks:

• Data science notebook: https://gist.github.com/fonsp/87a236e25b8b5ee5adc87a0dd279cb2f
• DifferentialEquations notebook: https://gist.github.com/fonsp/b01448ec8df5cfce125adeaa9a2d1fbd
• Images notebook: https://computationalthinking.mit.edu/Fall22/images_abstractions/images/

Context: two timings

The new caching system affects new Pluto users in two ways:

• The time to launch the Pluto server (precompilation of Pluto.jl)
• The time to run a notebook (precompilation of packages used in the notebook)

In Julia 1.9, both have become slower because new Julia/Pluto users are compiling packages with an empty cache.

Result

These are averages of three runs:

Julia 1.8.5

starting pluto the first time: 25.4 sec

• precompile: 15.1 sec
• import Pluto: 1.1 sec
• Pluto.run() until main menu loaded: 9.2 sec

Data science notebook

running your first notebook: 121.7 sec

• precompile: 77.0 sec
• import All, The, Packages: 6.8 sec
• running all other cells: 21.8 sec

running the same notebook again: 45.0 sec

• precompile: 0.4 sec
• import All, The, Packages: 6.8 sec
• running all other cells: 21.1 sec

DifferentialEquations notebook

running your first notebook: 386 sec

• precompile: 105 sec
• import All, The, Packages: 249 sec
• running all other cells: 27.3 sec

Julia 1.9.0-beta3

starting pluto the first time: 57.8 sec

• precompile: 50.0 sec
• import Pluto: 1.4 sec
• Pluto.run() until main menu loaded: 6.4 sec

Data science notebook

running your first notebook: 195.7 sec

• precompile: 141.0 sec
• import All, The, Packages: 8.5 sec
• running all other cells: 29.8 sec

running the same notebook again: 59.0 sec

• precompile: 0.4 sec
• import All, The, Packages: 7.2 sec
• running all other cells: 32.2 sec

DifferentialEquations notebook

running your first notebook: 512 sec

• precompile: 418 sec
• import All, The, Packages: 29.7 sec
• running all other cells: 30.5 sec

Experiment

In this experiment, I deleted the ~/.julia/compiled folder to simulate the experience of a new Julia user: they just installed Julia and Pluto, and are now running notebooks for the first time. I ran this experiment three times per Julia version.

To reproduce:

SYSTEM

I used an M1 16GB Macbook Air with energy saving mode off. Both Julia executables are the MacOS ARM build from julialang.org.

SETUP

Create a Pkg environment (folder), and add Pluto 0.19.20 (latest version). Go through the sample process once.

EACH SAMPLE

1. Stop all Julia processes, and delete the .julia/compiled/ folder.
2. Run julia with this as the active env.
3. import Pkg and @time Pkg.precompile()
4. @time import Pluto
5. With a stopwatch in your hand, Pluto.run(). Stop the clock when you see your recent notebooks listed.
6. Paste the notebook URL in the Open File box, press enter once. Press enter twice and start the stopwatch. Stop the stopwatch when all cells have finished.
7. Use the Process Tab to break down the notebook run times. Packages > instantiate is precompilation. Focus on the cell that has using Plots, ... to see its runtime. Subtract import runtime from total Evaluating cells time.
8. To measure the second time, click the Pluto logo to go back to the main menu. Click the X to shut down the notebook, and go back to step 6.
9. Close the tab, stop Pluto, stop Julia.

Raw results

Julia 1.8.5






precompile

15.0 sec
15.2 sec
15.1 sec

import Pluto

1.1 sec
1.1 sec
1.2 sec

start Pluto

9.0 sec
9.7 sec
8.9 sec


### Data science notebook

run notebook

118 sec (of which 75.0 sec on `Pkg.instantiate()`, 6.5 sec on `import AllThePackages`, 21.6 sec on running other cells)
122 sec (of which 76.7 sec on `Pkg.instantiate()`, 7.0 sec on `import AllThePackages`, 21.5 sec on running other cells)
125 sec (of which 79.2 sec on `Pkg.instantiate()`, 7.0 sec on `import AllThePackages`, 22.4 sec on running other cells)

second run, same session:

44 sec (of which 0.3 sec on `Pkg.instantiate()`, 7.2 sec on `import AllThePackages`, 19.6 sec on running other cells)
45 sec (of which 0.5 sec on `Pkg.instantiate()`, 7.1 sec on `import AllThePackages`, 22.0 sec on running other cells)
46 sec (of which 0.3 sec on `Pkg.instantiate()`, 7.0 sec on `import AllThePackages`, 21.6 sec on running other cells)

### DifferentialEquations notebook
(I started the sample from scratch for this second notebook, but I did not write down the timings of the previous steps because I already had the 3 samples from the first notebooks.)

run notebook

393 sec (of which 107 sec on `Pkg.instantiate()`, 252 sec on `import AllThePackages`, 27.3 sec on running other cells)
379 sec (of which 102 sec on `Pkg.instantiate()`, 245 sec on `import AllThePackages`, 27.3 sec on running other cells)




Julia 1.9.0-beta3


precompile

49.9 sec
52.1 sec
50.8 sec

import Pluto

1.4 sec
1.5 sec
1.4 sec

start Pluto

6.9 sec
6.2 sec
6.1 sec

### Data science notebook

run notebook

191 sec (of which 135 sec on `Pkg.instantiate()`, 8.5 sec on `import AllThePackages`, 30.1 sec on running other cells)
199 sec (of which 143 sec on `Pkg.instantiate()`, 9.0 sec on `import AllThePackages`, 30.1 sec on running other cells)
197 sec (of which 145 sec on `Pkg.instantiate()`, 8.1 sec on `import AllThePackages`, 29.2 sec on running other cells)


second run, same session:

60 sec (of which 0.4 sec on `Pkg.instantiate()`, 7.1 sec on `import AllThePackages`, 33.2 sec on running other cells)
60 sec (of which 0.5 sec on `Pkg.instantiate()`, 8.3 sec on `import AllThePackages`, 30.3 sec on running other cells)
57 sec (of which 0.4 sec on `Pkg.instantiate()`, 6.1 sec on `import AllThePackages`, 33.1 sec on running other cells)

### DifferentialEquations notebook
(I started the sample from scratch for this second notebook, but I did not write down the timings of the previous steps because I already had the 3 samples from the first notebooks.)

run notebook

506 sec (of which 408 sec on `Pkg.instantiate()`, 30.9 sec on `import AllThePackages`, 29.4 sec on running other cells)
522 sec (of which 423 sec on `Pkg.instantiate()`, 31.2 sec on `import AllThePackages`, 30.6 sec on running other cells)
508 sec (of which 416 sec on `Pkg.instantiate()`, 27.1 sec on `import AllThePackages`, 31.5 sec on running other cells)