Skip to content

Instantly share code, notes, and snippets.

@connorjclark connorjclark/README.md
Last active May 15, 2020

Embed
What would you like to do?
Lighthouse Scoring Changes 5.6.0 -> 6.0

Results

name url mean mean Δ
performance https://www.partypoker.com/ 33.8 -> 48.6 14.8
performance https://www.vevo.com/ 17.6 -> 30.8 13.2
performance https://www.ebs.in/IPS/ 12.4 -> 25.4 13
performance https://www.ifeng.com/ 37.2 -> 49.4 12.2
performance https://www.thefreedictionary.com/ 29.2 -> 38.2 9
performance https://www.tianya.cn/m/ 29.8 -> 38.4 8.6
performance https://depositfiles.com/ 38.8 -> 47 8.2
performance https://www.imageshack.us/login 69.8 -> 78 8.2
performance https://www.att.com/ 36 -> 43.2 7.2
performance https://www.cnet.com/ 25 -> 32.2 7.2
performance https://www.ebay.com/ 44.2 -> 51 6.8
performance https://www.56.com/ 35.4 -> 41.8 6.4
performance https://noclip.website/ 24 -> 30.2 6.2
performance https://www.mlb.com/ 18.8 -> 23.4 4.6
performance https://www.foxnews.com/ 18 -> 22.2 4.2
performance https://en.softonic.com/ 30.8 -> 34.8 4
performance https://www.rakuten.co.jp/ 17 -> 20.6 3.6
performance https://www.codewars.com/ 76.4 -> 78.6 2.2
performance https://www.verizonwireless.com/ 32 -> 33.8 1.8
performance https://sfbay.craigslist.org/ 96.6 -> 98 1.4
performance https://www.bing.com/ 98.6 -> 99 0.4
performance https://m.huffpost.com/ 18.2 -> 18.2 0
performance https://www.linkedin.com/ 96.2 -> 96.2 0
performance https://www.twitpic.com/ 100 -> 100 0
performance https://mail.ru/ 15.8 -> 15 -0.8
performance https://www.4shared.com/ 56 -> 55.2 -0.8
performance https://www.wikipedia.org/ 100 -> 99.2 -0.8
performance https://www.blogger.com/about/ 99 -> 97.8 -1.2
performance https://www.dawn.com/ 28 -> 26.4 -1.6
performance https://www8.hp.com/us/en/home.html 14.4 -> 12.6 -1.8
performance https://m.facebook.com/ 85.8 -> 83.8 -2
performance https://weather.com/ 26.2 -> 24.2 -2
performance https://m.youdao.com/ 99.2 -> 97 -2.2
performance https://www.espn.com/ 17.4 -> 15 -2.4
performance https://www.thestar.com.my/ 9.6 -> 6.8 -2.8
performance https://www.netflix.com/ 52 -> 48.6 -3.4
performance https://m.hexun.com/ 44 -> 40.2 -3.8
performance https://www.hulu.com/welcome 39.4 -> 35.2 -4.2
performance https://www.msn.com/ 57.2 -> 52.6 -4.6
performance http://www.zol.com.cn/ 65 -> 59.4 -5.6
performance https://www.so-net.ne.jp/m/ 22.2 -> 16 -6.2
performance https://www.addthis.com/ 62.6 -> 56.2 -6.4
performance https://www.orange.fr/portail 24 -> 17.6 -6.4
performance https://www.java.com/en/ 96 -> 89.2 -6.8
performance https://gm.58.com/glsanfrancisco-sl/ 83.4 -> 76.4 -7
performance https://www.onet.pl/ 46.6 -> 39 -7.6
performance https://birdsarentreal.com/ 27.4 -> 19.2 -8.2
performance https://m.sogou.com/ 62.4 -> 54.2 -8.2
performance https://www.deviantart.com/ 37.2 -> 29 -8.2
performance https://www.hatena.ne.jp/ 38.6 -> 30.4 -8.2
performance https://m.mop.com/ 68.2 -> 59.2 -9
performance https://wap.sogou.com/ 62.8 -> 53.8 -9
performance https://www.nih.gov/ 61.6 -> 52.6 -9
performance https://www.metacafe.com/ 52 -> 42.2 -9.8
performance https://mobile.twitter.com/ 38.2 -> 28 -10.2
performance https://www.gmx.net/ 50 -> 39.2 -10.8
performance https://www.flipkart.com/ 36.2 -> 25.2 -11
performance https://www.ocn.ne.jp/ 41 -> 29.8 -11.2
performance https://www.nokia.com/ 56.2 -> 44.2 -12
performance https://www.reddit.com/ 37.2 -> 25 -12.2
performance https://www.scribd.com/ 56.4 -> 43 -13.4
performance https://stripe.com/docs 54.4 -> 40.6 -13.8
performance https://www.domaintools.com/ 37 -> 23 -14
performance https://www.alexa.com/ 52 -> 37.4 -14.6
performance https://www.tumblr.com/ 62.4 -> 47.4 -15
performance https://www.mozilla.org/en-US/ 92.6 -> 76.6 -16
performance https://www.amazon.co.jp/ 51 -> 34 -17
performance https://www.skype.com/en/ 77.8 -> 60.8 -17
performance https://www.ning.com/ 52.2 -> 32.6 -19.6
performance https://www.typepad.com/ 80.6 -> 60.2 -20.4
performance https://www.mgid.com/ru 56.8 -> 35.6 -21.2
performance https://www.shopping.com/ 69.4 -> 46.4 -23
performance https://en-maktoob.yahoo.com/?p=xa 70.6 -> 46 -24.6
performance https://www.irs.gov/ 68.6 -> 44 -24.6
performance https://www.instagram.com/ 80 -> 51.4 -28.6

Steps to Reproduce

We run Lighthouse 5 times on a set of URLs and compare the median runs to each other. Lighthouse 6.0 is used to collect artifacts, and those artifacts are re-used with 5.6.0, so there is no page variance. Only scoring changes are captured.

  1. Run Lighthouse on GCP (it takes too long locally and my machine will fall asleep without my attention) [1]
  2. Copy results to timings-data/urls-6.0
  3. Copy the same results to timings-data/urls-5.6.0, but delete all the LHR json files. [2]
  4. Run the "collect" command for the 5.6.0 dataset. The artifacts are present, but since we deleted the LHRs it will re-run the auditting step of Lighthouse (fast) [3]
  5. run the comparison command

[1]

# Run Lighthouse on GCP
# I tweaked an existing script we have to do a bulk run: https://github.com/GoogleChrome/lighthouse/blob/collect-gcp/lighthouse-core/scripts/lantern/collect/gcp-create-and-run.sh
git checkout collect-gcp # branched off dc5bb65
sh lighthouse-core/scripts/lantern/collect/gcp-create-and-run.sh

[2]

mkdir timings-data; cd timings-data
unzip ~/lh-6-data.zip # See Resources to download the same data
cp -r urls-6.0 urls-5.6.0
find urls-5.6.0 -name '*lhr*' -delete
cd .. # return to LH root

[3]

git checkout v5.6.0
git checkout origin/collect-gcp -- lighthouse-core/scripts/compare-runs.js lighthouse-core/scripts/lantern/collect/common.js lighthouse-core/scripts/lantern/collect/urls.js
yarn
urls=(`node -e 'console.log(require("./lighthouse-core/scripts/lantern/collect/urls.js").join("\n"))'`)
# Go comment out `throw new Error('Cannot change settings between gathering and auditing');` in runner.js
# Go comment out `lhr.i18n.icuMessagePaths =` in runner.js
node --max-old-space-size=4096 lighthouse-core/scripts/compare-runs.js \
  --name urls-5.6.0 \
  --collect \
  -n 5 \
  --lh-flags='--only-categories=performance' \
  --urls "${urls[@]}"

[4]

# Compare data.
node lighthouse-core/scripts/compare-runs.js \
  --name urls-5.6.0 \
  --name urls-6.0 \
  --compare \
  --filter='performance$' \
  --report-exclude 'key|min|max|stdev|cv|^n$' \
  --delta-property-sort mean

Resources

Data (must request access but will be given freely)

Similar analysis for 11ty sites

@paulirish

This comment has been minimized.

Copy link

paulirish commented May 14, 2020

Scatterplot:

image

@patrickhulce

This comment has been minimized.

Copy link

patrickhulce commented May 15, 2020

small correction to the script but other than that worked beautifully 👍

step 3

node --max-old-space-size=4096 lighthouse-core/scripts/compare-runs.js \
  --name urls-5.6.0 \
  --collect \ # <-- this should be --audit
  -n 5 \
  --lh-flags='--only-categories=performance' \
  --urls "${urls[@]}"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.