Skip to content

Instantly share code, notes, and snippets.

@bhelx bhelx/fetch.rb
Last active May 9, 2017

Embed
What would you like to do?
Recursively fetch Craigslist apartments using their map's json api
REGION = 'neworleans'
BASE_URL = "https://#{REGION}.craigslist.org"
parse_listings = lambda do |url|
results = JSON.parse Net::HTTP.get(URI(url))
results.first.each do |item|
if item.key? 'GeoCluster'
parse_listings.call("#{BASE_URL}#{item['url']}")
else
post_id = item['PostingID'].to_i
puts "Listing: #{post_id}"
attrs = {
title: item['PostingTitle'],
price: item['Ask'].to_i,
uploaded: DateTime.strptime(item['PostedDate'], '%s'),
lat: item['Latitude'],
lng: item['Longitude'],
bedrooms: item['Bedrooms'].to_i,
post_id: post_id
}
if listing = Listing.unscoped.where(post_id: post_id).first
listing.update attrs
else
Listing.create attrs
end
end
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.