Package managers comparison from the CI time perspective

benchmark-package-managers.md

Benchmark

TLDR

On a (cached) github action. pnpm7 ±50s (244Mb store cache) / yarn4-nm ±75s (190Mb yarn install cache)

Intro

Tests made with pnpm 7+ and yarn 4.0.0rc (node-modules/pnp) based on https://github.com/belgattitude/nextjs-monorepo-example. Tests have been made in a way to evaluate CI performance. There's other perf bench: pnpm or yarn. Conclusions are pretty similar. This benchmark helped to create this yarn install composite action

With cache.

• Yarn (pnp-loose): local: 7s / github action: wip)
• Yarn (node-modules) local: 21s / github action: ±70s, post: +5s (190MB)
• Pnpm local: 10s / github action: ±40s: post: +10s (244MB)

Without cache.

• Yarn local: 71s (github action: ±2mins)
• Pnpm local: 34s (github action: ±1m20s)

Some observations:

• Yarn is able to rotate the downloaded cache archives. That detail is important on CI, cause it's possible keep the "warm cache" even if the yarn.lock has been changed (only download changed deps). This also reduce cache invalidation on the ci. Example @action/cache with node-module-linker. Works in pnp and node_modules linkers. For example if you have things like renovatebot... pnpm will start with cold cache, yarn not completely. This affects monthly ci time... but carbon emissions too.
• Yarn in pnp opt-out from nextjs standalone optimization / @vercel/nft important for nextjs/lambdas/serverless/vercel or whatever (to be tested !)
• Yarn in pnp is not yet totally supported by tools like nx or turbo (to be tested !)
• Yarn in pnp is the clear winner but it's sometimes difficult to make it work with some deps (it's improving fast).
• Yarn in pnp will also improve performance/memory of the node apps (runtime / startup time)
• Pnpm tackles the doppelgangers. Need to see with yarn.
• Pnpm have some niceties to work with docker in monorepos
• Yarn performs a bit better with cache compressionLevel: 0 but it affects the lock and at the end action/cache have more work to do (see notes)

Results

Running yarn install --immutable --inline-builds with hyperfine (5 runs).

Scenario	Time (seconds)	Github Action	action/cache
Yarn cold cache (nm/links-local)	71.410 ± 1.835	±2m
Yarn warm (nm/links-local)	22.765 ± 0.417	±1m10s	~190Mb (post: ±5s)
Yarn warm (nm/links-local/install-state)	21.576 ± 0.390	±1m10s	~190 + 3Mb (post: ±5s)
Yarn warm (nm/links-local/install-state/nocomp)	19.182 ± 0.323	±50s	~?Mb + 3Mb (post: >20s)
Yarn warm (nm/links-global/install-state)	21.657 ± 0.393	±1m10s
Yarn warm (pnp-loose)	6.822 ± 0.247
Pnpm 7.13.3 cold cache	34.696 ± 1.413	±1m20s
Pnpm 7.13.3 warm cache	10.780 ± 0.881	±40s	~244Mb (post: ±10s)

Note

• all tests assume there's a lock file.
• the "On Github Action" is not a proper bench, just an estimate. The action/cache(+++) represent an estimate of the action/cache post restore operation. In other words sometimes it's possible to gain on install time with more agressive cache, but at the end the cache will have to be (un-)compressed by the action wich takes additional time. As far as I tried, the best is just to keep the local .yarn/cache (compression of hardlinks-global store or node_modules don't give better results).
• pnpm benchmark was done with strict-peer-dependencies=false in .npmrc.

Local test machine:

  System:
    OS: Linux 5.15 Ubuntu 22.04.1 LTS 22.04.1 LTS (Jammy Jellyfish)
    CPU: (12) x64 Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz
    Memory: 6.02 GB / 15.29 GB
  Binaries:
    Node: 16.17.0 - ~/.nvm/versions/node/v16.17.0/bin/node
    Yarn: 4.0.0-rc.22 - ~/.nvm/versions/node/v16.17.0/bin/yarn
    npm: 8.15.0 - ~/.nvm/versions/node/v16.17.0/bin/npm

Run the benches

Install hyperfine:

wget https://github.com/sharkdp/hyperfine/releases/download/v1.15.0/hyperfine_1.15.0_amd64.deb
sudo dpkg -i hyperfine_1.15.0_amd64.deb

Clone the example repository:

cd /tmp
git clone git@github.com:belgattitude/nextjs-monorepo-example.git
cd /tmp/nextjs-monorepo-example
mkdir .yarn/bench
npm i --global rimraf

Yarn cold cache (nm/hardlinks-local)

Linker	Mode	Cache	Install state	Compress
node-modules	hardlinks-local	no	no	mixed

hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/nm_hardlinks-local_no-cache_no-state_comp-mixed.md" \
          --prepare "npx --yes rimraf '**/node_modules' .yarn/install-state.gz .yarn/cache .yarn/global" \
          "YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds"

Yarn warm cache (nm/hardlinks-local)

Linker	Mode	Cache	Install state	Compress	Time
node-modules	hardlinks-local	yes	no	mixed

hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/nm_hardlinks-local_cache_no-state_comp-mixed.md" \
          --prepare "npx --yes rimraf '**/node_modules' .yarn/install-state.gz .yarn/global" \
          "YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds"

Command	Mean [s]	Min [s]	Max [s]	Relative
YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds	22.765 ± 0.417	22.064	23.115	1.00

Yarn warm cache (nm/hardlinks-local/install-state)

Linker	Mode	Cache	Install state	Compress	Time
node-modules	hardlinks-local	yes	yes	mixed

hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/nm_hardlinks-local_cache_state_comp-mixed.md" \
          --prepare "npx --yes rimraf '**/node_modules' .yarn/global" \
          "YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds"

Command	Mean [s]	Min [s]	Max [s]	Relative
YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds	21.576 ± 0.390	21.102	21.948	1.00

Yarn warm cache (nm/hardlinks-local/install-state/no-comp)

Linker	Mode	Cache	Install state	Compress
node-modules	hardlinks-local	yes	yes	0 - off

hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/nm_hardlinks-local_cache_state_no-comp.md" \
          --prepare "npx --yes rimraf '**/node_modules' .yarn/global" \
          "YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=0 PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds"

Command	Mean [s]	Min [s]	Max [s]	Relative
YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=0 PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds	19.182 ± 0.323	18.806	19.482	1.00

Warm cache (nm/hardlinks-global/install-state)

Linker	Mode	Cache	Install state	Compress
node-modules	hardlinks-global	yes	yes	mixed

hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/nm_hardlinks-global_cache_state_comp.md" \
          --prepare "npx --yes rimraf '**/node_modules' .yarn/global" \
          "YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds"

Command	Mean [s]	Min [s]	Max [s]	Relative
YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds	21.657 ± 0.393	21.115	22.016	1.00

Yarn PNP Warm cache (loose)

hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/yarn_pnp_loose_cache_comp-mixed.md" \
          --prepare "npx --yes rimraf '.yarn/unplugged' .yarn/global" \
         "YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NODE_LINKER=pnp YARN_PNP_MODE=loose YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds"

Command	Mean [s]	Min [s]	Max [s]	Relative
YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NODE_LINKER=pnp YARN_PNP_MODE=loose YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds	6.822 ± 0.247	6.490	7.175	1.00

PNPM 7 Cold cache

# Content of .npmrc
strict-peer-dependencies=false
store-dir=.yarn/pnpm/global-cache

hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/pnpm_no-cache.md" \
          --prepare "npx --yes rimraf '**/node_modules' .yarn/pnpm/global-cache; pnpm prune store" \
          "PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 pnpm i"

Command	Mean [s]	Min [s]	Max [s]	Relative
PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 pnpm i	34.696 ± 1.413	33.841	37.210	1.00

PNPM 7 Warm cache

hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/pnpm_cache.md" \
          --prepare "npx --yes rimraf '**/node_modules'; pnpm prune store" \
          "PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 pnpm i"

Command	Mean [s]	Min [s]	Max [s]	Relative
PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 pnpm i	10.780 ± 0.881	10.190	12.341	1.00

Notes

CI

Tests have been done locally. On the CI we generally want to account for action/cache compression perf as well.

nmMode

nmMode: hardlinks-local

Compression level

Disabling cache compression might give a small speed-up on some CI (there's trade-offs). It affect the lock file. In other words you can't have it set differently for local and on the CI (at least if using yarn install --immutable) Best is to keep YARN_COMPRESSION_LEVEL at 'mixed' (auto)

With mixed compression level

Without compression

Why not caching node_modules

The action/cache has more to do in term of compression/decompression