[TIx 7] Scraping the mobile version of a site with Ruby

scrape_mobile_site_with_ruby.md

I found myself in a situation where I wanted to examine the layout of the mobile version of a particular website. I tend to use Ruby's OpenURI module and the Nokogiri gem for my webscraping needs, and it turns out it's really easy to get a mobile version of the site with a bit more effort:

require 'open-uri'
require 'nokogiri'

# let's look at my GitHub profile as an example
url = 'https://github.com/O-I'

# this opens the URL and parses it as XML 
xml = Nokogiri::XML(open(url))

# from a desktop, this gives us the standard web view
# but what if we want to parse the site as served to a mobile device?

# spoof the user agent — in this case, an iPhone
user_agent = { "User-Agent" => "Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1C25 Safari/419.3" }

# then we can pass it in as optional argument to `open`
# and get back parsed XML as served to a mobile device
mobile_xml = Nokogiri::XML(open(url, user_agent))