Lighthouse Scoring Changes 5.6.0 -> 6.0

README.md

Results

name	url	mean	mean Δ
performance	https://www.partypoker.com/	33.8 -> 48.6	14.8
performance	https://www.vevo.com/	17.6 -> 30.8	13.2
performance	https://www.ebs.in/IPS/	12.4 -> 25.4	13
performance	https://www.ifeng.com/	37.2 -> 49.4	12.2
performance	https://www.thefreedictionary.com/	29.2 -> 38.2	9
performance	https://www.tianya.cn/m/	29.8 -> 38.4	8.6
performance	https://depositfiles.com/	38.8 -> 47	8.2
performance	https://www.imageshack.us/login	69.8 -> 78	8.2
performance	https://www.att.com/	36 -> 43.2	7.2
performance	https://www.cnet.com/	25 -> 32.2	7.2
performance	https://www.ebay.com/	44.2 -> 51	6.8
performance	https://www.56.com/	35.4 -> 41.8	6.4
performance	https://noclip.website/	24 -> 30.2	6.2
performance	https://www.mlb.com/	18.8 -> 23.4	4.6
performance	https://www.foxnews.com/	18 -> 22.2	4.2
performance	https://en.softonic.com/	30.8 -> 34.8	4
performance	https://www.rakuten.co.jp/	17 -> 20.6	3.6
performance	https://www.codewars.com/	76.4 -> 78.6	2.2
performance	https://www.verizonwireless.com/	32 -> 33.8	1.8
performance	https://sfbay.craigslist.org/	96.6 -> 98	1.4
performance	https://www.bing.com/	98.6 -> 99	0.4
performance	https://m.huffpost.com/	18.2 -> 18.2	0
performance	https://www.linkedin.com/	96.2 -> 96.2	0
performance	https://www.twitpic.com/	100 -> 100	0
performance	https://mail.ru/	15.8 -> 15	-0.8
performance	https://www.4shared.com/	56 -> 55.2	-0.8
performance	https://www.wikipedia.org/	100 -> 99.2	-0.8
performance	https://www.blogger.com/about/	99 -> 97.8	-1.2
performance	https://www.dawn.com/	28 -> 26.4	-1.6
performance	https://www8.hp.com/us/en/home.html	14.4 -> 12.6	-1.8
performance	https://m.facebook.com/	85.8 -> 83.8	-2
performance	https://weather.com/	26.2 -> 24.2	-2
performance	https://m.youdao.com/	99.2 -> 97	-2.2
performance	https://www.espn.com/	17.4 -> 15	-2.4
performance	https://www.thestar.com.my/	9.6 -> 6.8	-2.8
performance	https://www.netflix.com/	52 -> 48.6	-3.4
performance	https://m.hexun.com/	44 -> 40.2	-3.8
performance	https://www.hulu.com/welcome	39.4 -> 35.2	-4.2
performance	https://www.msn.com/	57.2 -> 52.6	-4.6
performance	http://www.zol.com.cn/	65 -> 59.4	-5.6
performance	https://www.so-net.ne.jp/m/	22.2 -> 16	-6.2
performance	https://www.addthis.com/	62.6 -> 56.2	-6.4
performance	https://www.orange.fr/portail	24 -> 17.6	-6.4
performance	https://www.java.com/en/	96 -> 89.2	-6.8
performance	https://gm.58.com/glsanfrancisco-sl/	83.4 -> 76.4	-7
performance	https://www.onet.pl/	46.6 -> 39	-7.6
performance	https://birdsarentreal.com/	27.4 -> 19.2	-8.2
performance	https://m.sogou.com/	62.4 -> 54.2	-8.2
performance	https://www.deviantart.com/	37.2 -> 29	-8.2
performance	https://www.hatena.ne.jp/	38.6 -> 30.4	-8.2
performance	https://m.mop.com/	68.2 -> 59.2	-9
performance	https://wap.sogou.com/	62.8 -> 53.8	-9
performance	https://www.nih.gov/	61.6 -> 52.6	-9
performance	https://www.metacafe.com/	52 -> 42.2	-9.8
performance	https://mobile.twitter.com/	38.2 -> 28	-10.2
performance	https://www.gmx.net/	50 -> 39.2	-10.8
performance	https://www.flipkart.com/	36.2 -> 25.2	-11
performance	https://www.ocn.ne.jp/	41 -> 29.8	-11.2
performance	https://www.nokia.com/	56.2 -> 44.2	-12
performance	https://www.reddit.com/	37.2 -> 25	-12.2
performance	https://www.scribd.com/	56.4 -> 43	-13.4
performance	https://stripe.com/docs	54.4 -> 40.6	-13.8
performance	https://www.domaintools.com/	37 -> 23	-14
performance	https://www.alexa.com/	52 -> 37.4	-14.6
performance	https://www.tumblr.com/	62.4 -> 47.4	-15
performance	https://www.mozilla.org/en-US/	92.6 -> 76.6	-16
performance	https://www.amazon.co.jp/	51 -> 34	-17
performance	https://www.skype.com/en/	77.8 -> 60.8	-17
performance	https://www.ning.com/	52.2 -> 32.6	-19.6
performance	https://www.typepad.com/	80.6 -> 60.2	-20.4
performance	https://www.mgid.com/ru	56.8 -> 35.6	-21.2
performance	https://www.shopping.com/	69.4 -> 46.4	-23
performance	https://en-maktoob.yahoo.com/?p=xa	70.6 -> 46	-24.6
performance	https://www.irs.gov/	68.6 -> 44	-24.6
performance	https://www.instagram.com/	80 -> 51.4	-28.6

Steps to Reproduce

We run Lighthouse 5 times on a set of URLs and compare the median runs to each other. Lighthouse 6.0 is used to collect artifacts, and those artifacts are re-used with 5.6.0, so there is no page variance. Only scoring changes are captured.

1. Run Lighthouse on GCP (it takes too long locally and my machine will fall asleep without my attention) [1]
2. Copy results to timings-data/urls-6.0
3. Copy the same results to timings-data/urls-5.6.0, but delete all the LHR json files. [2]
4. Run the "collect" command for the 5.6.0 dataset. The artifacts are present, but since we deleted the LHRs it will re-run the auditting step of Lighthouse (fast) [3]
5. run the comparison command

[1]

# Run Lighthouse on GCP
# I tweaked an existing script we have to do a bulk run: https://github.com/GoogleChrome/lighthouse/blob/collect-gcp/lighthouse-core/scripts/lantern/collect/gcp-create-and-run.sh
git checkout collect-gcp # branched off dc5bb65
sh lighthouse-core/scripts/lantern/collect/gcp-create-and-run.sh

[2]

mkdir timings-data; cd timings-data
unzip ~/lh-6-data.zip # See Resources to download the same data
cp -r urls-6.0 urls-5.6.0
find urls-5.6.0 -name '*lhr*' -delete
cd .. # return to LH root

[3]

git checkout v5.6.0
git checkout origin/collect-gcp -- lighthouse-core/scripts/compare-runs.js lighthouse-core/scripts/lantern/collect/common.js lighthouse-core/scripts/lantern/collect/urls.js
yarn
urls=(`node -e 'console.log(require("./lighthouse-core/scripts/lantern/collect/urls.js").join("\n"))'`)
# Go comment out `throw new Error('Cannot change settings between gathering and auditing');` in runner.js
# Go comment out `lhr.i18n.icuMessagePaths =` in runner.js
node --max-old-space-size=4096 lighthouse-core/scripts/compare-runs.js \
  --name urls-5.6.0 \
  --collect \
  -n 5 \
  --lh-flags='--only-categories=performance' \
  --urls "${urls[@]}"

[4]

# Compare data.
node lighthouse-core/scripts/compare-runs.js \
  --name urls-5.6.0 \
  --name urls-6.0 \
  --compare \
  --filter='performance$' \
  --report-exclude 'key|min|max|stdev|cv|^n$' \
  --delta-property-sort mean

Resources

Data (must request access but will be given freely)

Similar analysis for 11ty sites

Scatterplot:

small correction to the script but other than that worked beautifully 👍

step 3

node --max-old-space-size=4096 lighthouse-core/scripts/compare-runs.js \
  --name urls-5.6.0 \
  --collect \ # <-- this should be --audit
  -n 5 \
  --lh-flags='--only-categories=performance' \
  --urls "${urls[@]}"