Benchmark for c90b46e

benchmark-for-c90b46e.md

tldr;

The metrics of benchmarks executed in sequence within the same job in GitHub Action have too much fluctuation, making the metrics unreliable.

Detail

Here's the five benches run for vitejs/vite#12723. Runs agained the commit in PR (c90b46e) and the one (3f4d109) before it. As patak described

We are seeing in sapphi-red/performance-compare, a reduction from ~1650ms to ~1350ms on M1

For this PR, the start time should get performance gain and the build speed remain unchanged. Both commit metrics in each table are sequentially executed within the same container.

I read some articles that also pointed out the performance is not consistent.

• Is GitHub Actions suitable for running benchmarks? | Quansight Labs
• CI for performance: Reliable benchmarking in noisy environments
• ▲ Donny (강동윤) on Twitter: "Is there a way to reliably benchmark as a CI process? GitHub Actions public runners have too much noise." / Twitter

The fallback could make vite-benchmark be a CLI tool in users' physical machine to reduce the effort to run benchmark for consistent performance 😅.

Repoisotry:

https://github.com/fi3ework/vite-benchmark

Cases explaining:

• perf 1 case is https://github.com/fi3ework/vite-benchmark/tree/main/cases/perf-1, this classic vite 2.7 slow case.
• perf 2 is https://github.com/fi3ework/vite-benchmark/tree/main/cases/perf-2, from https://github.com/sapphi-red/performance-compare.

Metrics explaining:

• Dev cold/hot start cost: extracted from "ready in X ms" (cold/hot start)
• Dev cold/hot bundle cost: extaracted from "bundled in Xms" (cold/hot start)
• Build cost: extracted from "built in Xs"

Five times result

1

not recorred.

2

3

4

5

6

latest on site: https://fi3ework.github.io/vite-benchmark/compare/?compares=vitejs%2Fvite%403f4d109...vitejs%2Fvite%40c90b46e