Skip to content

Instantly share code, notes, and snippets.

# Parses JSON results from WPT e.g. http://www.webpagetest.org/result/140506_W2_c46551bac76ab45d87e7e2f32c3159e6/?f=json
# Relies on JSON to be locally downloaded first (avoid's timeouts when results need to be retrieved from archive)
# Produces a data.frame with selected values from all test results
# url in code is really a path to a file, all.urls is a data frame with a list of files
require(RCurl)
require(stringr)
require(rjson)
1. Install nginx
sudo apt-get install nginx
configure nginx.conf
2. Install PHP
# Fast socket version
@andydavies
andydavies / resourcesperpage.sql
Created April 18, 2012 09:17
Extracts data to plot a histogram of resources vs count of pages from HTTP Archive data
-- Extracts data to plot histogram of resources vs count of pages from HTTP Archive data
-- For bigger groupings change ROUND(resources, 0): -1 = 10, -2 = 100
SELECT ROUND(resources, 0) AS ResourcesPerPage,
COUNT(*) AS Pages
FROM (select count(*) as resources from requests group by pageid) as temp
GROUP BY ResourcesPerPage;
@andydavies
andydavies / gist:2517046
Created April 28, 2012 08:09
Find icons in HTTPArchive.ord data
-- Find the favicon in HTTPArchive.org data
-- TODO: filter just of favicon.ico (still need mimetype?)
select * from requests where mimeType = 'image/x-icon'
@andydavies
andydavies / gist:2a82839e30bff61dcffb
Created December 9, 2015 12:15
Bookmarklet for securityheaders.io
Save this script as a bookmark, then use it to test the current page
javascript:(function(){hostname=document.location.hostname;window.open('https://securityheaders.io/?q=' + hostname);})();
@andydavies
andydavies / gist:4358912
Created December 22, 2012 13:38
View-Source bookmarklet for webkit (only ?) browsers works on iOS Safari e.g. iPad too Based on @weisjohn's bookmarklet with background colour forced to white (http://weisjohn.github.com/widgets/viewsource_bookmarklet/)
javascript:(function(){i=document.createElement('iframe');i.setAttribute('viewsource');i.setAttribute('width', '100%');i.src=location.href;document.body.insertBefore(i, document.body.childNodes[0]);i.style.backgroundColor='#fff';})();
@andydavies
andydavies / gist:3cdcca42bd9f4289424c
Created December 10, 2015 22:16
Query for HTTPArchive BigQuery data to list all font files for sites x top sites
select pages.url, pages.rank, req.url
from [httparchive:runs.latest_requests] as req
join (
select url, rank, pageid, reqFont from [httparchive:runs.latest_pages]
) as pages on pages.pageid = req.pageid
where lower(req.mimetype) contains "font"
and pages.rank <= 100 and pages.reqFont > 0
order by pages.rank
@andydavies
andydavies / gist:5489399
Created April 30, 2013 15:17
Quova API example - when tested it didn't support IPv6 and cellular gateways were always reported as line_speed:low regardless of end user connection
{
"ipinfo":{
"ip_address":"212.183.128.252",
"ip_type":"Mapped",
"Network":{
"organization":"vodafone limited",
"carrier":"vodafone limited",
"asn":25135,
"connection_type":"mobile wireless",
"line_speed":"low",
@courtneynash
@estellevw
@tameverts
@bbinto
@denisejacobs
@daianacheng
@kaelifa
@anna_debenham
@gerbille
@viviancromwell
@andydavies
andydavies / gist:6006984
Created July 16, 2013 08:45
DNS records for t.co publish 199.16.156.11 and 199.16.156.75 but connections to 199.16.156.11 fail so effectively waiting for TCP timeout before resolution
Andy-Daviess-MacBook-Pro:tmp ad$ wget https://t.co/tdI2DYxUZh
--2013-07-16 09:33:56-- https://t.co/tdI2DYxUZh
Resolving t.co... 199.16.156.11, 199.16.156.75
Connecting to t.co|199.16.156.11|:443... failed: Operation timed out.
Connecting to t.co|199.16.156.75|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://itunes.apple.com/gb/app/ia-writer/id439623248?mt=12 [following]
--2013-07-16 09:35:13-- https://itunes.apple.com/gb/app/ia-writer/id439623248?mt=12
Resolving itunes.apple.com... 95.100.130.217
Connecting to itunes.apple.com|95.100.130.217|:443... connected.