Skip to content

Instantly share code, notes, and snippets.

@danicuki
Created March 18, 2011 01:15
Show Gist options
  • Save danicuki/875466 to your computer and use it in GitHub Desktop.
Save danicuki/875466 to your computer and use it in GitHub Desktop.
Use many HTTP proxies to get pages on the web. Util for increasing count hits
require 'rubygems'
require 'restclient'
require 'nokogiri'
require 'json'
def call_url(url)
Thread.new do
puts "Chamando url #{url} no proxy #{RestClient.proxy}"
RestClient.get(url) rescue nil
end
end
#get proxies servers
while (true)
(1..9).each do |time|
RestClient.proxy = nil #"http://61.164.40.151:8080"
puts "\n\npagina #{time}"
# response = RestClient.get("http://www.ip-adress.com/proxy_list", "User-Agent" => "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13")
response = RestClient.get("http://www.samair.ru/proxy/time-0#{time}.htm") rescue nil
html_doc = Nokogiri::HTML(response)
lines = html_doc.xpath("//table[@class='tablelist']").children
lines.shift
eval "abcdefghijklmnopqrstuvxz".split("").map {|ex|"#{ex}=0"}.join(";")
script = html_doc.xpath("//script")[1]
if script
eval script.text
lines.each do |line|
unless line.children[0].children[1].nil?
px = line.children[0].children[1].text.split(")")[0].split("+")
px.shift
port = px.map {|c| eval(c)}.join("")
server = line.children[0].children[0].text
RestClient.proxy = "http://#{server}:#{port}"
# puts "proxy: #{RestClient.proxy}"
call_url("put your url here")
end
end
end
sleep(60)
end
sleep(1800)
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment