Skip to content

Instantly share code, notes, and snippets.

@siers
Created Jul 30, 2019
Embed
What would you like to do?
dumb, simple ss.com scraper for displaying notifications on diffs
#!/usr/bin/env ruby
script = <<-EOF
ccurl() {
curl -b cookie.txt -c cookie.txt "$@"
}
ccurl 'https://www.ss.com/lv/real-estate/flats/riga/agenskalns/filter/' -H 'authority: www.ss.com' -H 'cache-control: max-age=0' -H 'origin: https://www.ss.com' -H 'upgrade-insecure-requests: 1' -H 'content-type: application/x-www-form-urlencoded' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3' -H 'referer: https://www.ss.com/lv/real-estate/flats/riga/agenskalns/' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' &> /dev/null || exit
ccurl 'https://www.ss.com/lv/real-estate/flats/riga/agenskalns/filter/fDgSeF4belI=.html' -H 'authority: www.ss.com' -H 'cache-control: max-age=0' -H 'origin: https://www.ss.com' -H 'upgrade-insecure-requests: 1' -H 'content-type: application/x-www-form-urlencoded' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3' -H 'referer: https://www.ss.com/lv/real-estate/flats/riga/agenskalns/' -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' --data 'topt%5B8%5D%5Bmin%5D=150&topt%5B8%5D%5Bmax%5D=450&topt%5B1%5D%5Bmin%5D=&topt%5B1%5D%5Bmax%5D=&topt%5B3%5D%5Bmin%5D=&topt%5B3%5D%5Bmax%5D=&topt%5B4%5D%5Bmin%5D=&topt%5B4%5D%5Bmax%5D=&opt%5B6%5D=&sid=%2Flv%2Freal-estate%2Fflats%2Friga%2Fagenskalns%2F&opt%5B11%5D=' -L
EOF
require 'nokogiri'
require 'open3'
content =
if false
File.read('y')
else
content_, _, status = Open3.capture3('bash', stdin_data: script)
raise status.inspect if status.to_i != 0
File.write('y', content)
content_
end
html = Nokogiri::HTML.parse(content)
main = html.css('#page_main table')[2]
rows = main.css('tr').drop(1)
out = rows.map do |row|
[
row.css('td:nth-child(3)').text, # slud
row.css('td:nth-child(4)').text, # iela
row.css('td:nth-child(10)').text, # €
].inspect
end
puts(*out)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment