Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
[TIx 7] Scraping the mobile version of a site with Ruby

I found myself in a situation where I wanted to examine the layout of the mobile version of a particular website. I tend to use Ruby's OpenURI module and the Nokogiri gem for my webscraping needs, and it turns out it's really easy to get a mobile version of the site with a bit more effort:

require 'open-uri'
require 'nokogiri'

# let's look at my GitHub profile as an example
url = 'https://github.com/O-I'

# this opens the URL and parses it as XML 
xml = Nokogiri::XML(open(url))

# from a desktop, this gives us the standard web view
# but what if we want to parse the site as served to a mobile device?

# spoof the user agent — in this case, an iPhone
user_agent = { "User-Agent" => "Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1C25 Safari/419.3" }

# then we can pass it in as optional argument to `open`
# and get back parsed XML as served to a mobile device
mobile_xml = Nokogiri::XML(open(url, user_agent))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment