Created
May 3, 2011 19:42
-
-
Save joerussbowman/954058 to your computer and use it in GitHub Desktop.
My first ruby script, checks vanity url redirects
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
require 'net/http' | |
require 'uri' | |
if ARGV.length < 2 | |
puts "Syntax: urltest.rb [ redirects file ] [host ] (optional 1 for external redirects)" | |
puts "example for local redirects: urltest.rb vanity_urls.txt www.example.com" | |
exit | |
end | |
if File.exists?(ARGV[0]) | |
redirects_file = ARGV[0] | |
else | |
puts "#{ARGV[0]} does not exist." | |
exit | |
end | |
$max_redirects = 10 | |
host = ARGV[1] | |
max_threads = 5 # 4+1 for the main thread | |
threads = [] | |
def fetch(url, info, limit = $max_redirects) | |
Net::HTTP.get_response(URI.parse(url)) do |response| | |
case response | |
when Net::HTTPSuccess # just do nothing | |
when Net::HTTPRedirection then fetch(response['location'], info, limit - 1) | |
else | |
File.open("output.log", "a") do |errors| | |
errors.puts "\nError: REDIRECT INFO: #{info} REAL DESTINATION: #{url} [ #{response.code} ]\n" | |
end | |
puts "\nError: #{info}: #{response.code}\n" | |
end | |
end | |
end | |
File.open(redirects_file).each { |line| | |
if Thread.list.length < max_threads | |
# format of redirect info is: /prettyurl /redirects/to (or external.domain/redirects/to) | |
# as we're following redirects, it's not necessary to parse the final destination. | |
if line.length > 0 | |
url = "http://#{host}#{line.split(/\s/)[0]}" | |
x = Thread.new { | |
puts "Threads: #{Thread.list.length} - #{url}" | |
fetch(url, line) | |
} | |
threads << x | |
end | |
else | |
redo | |
end | |
} | |
threads.each_with_index { |name, i| threads[i].join } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
ok this is the last edit, as far as I'm going to get in my environment. I'm stuck at Ruby 1.8.5 and there's policy reasons to not compile 1.9.2. The catch is I need to set a user agent because I believe some of the external redirects are blocking the default. get_response and other methods in 1.8.5 don't appear to have support for more than 1 argument so I'm not able to pass a headers hash to it. 1.9.2 appears to support it though. Going to switch to Python and get the script done and move on to other tasks.