Skip to content

Instantly share code, notes, and snippets.

@mazuhl

mazuhl/url-grabber.rb

Last active December 23, 2015 23:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mazuhl/6711562 to your computer and use it in GitHub Desktop.
Save mazuhl/6711562 to your computer and use it in GitHub Desktop.
Download ZIP
Scan website and save all URLs to text file
Raw
url-grabber.rb
require 'rubygems'
require 'spidr'
links = []
wesite = "http://www.website.com"
puts "Scanning #{website} for links"
puts ""
f = File.open("urls.txt","w");
Spidr.site(website) do |spider|
spider.every_url {|url|
f.puts url
puts url
}
end
f.flush
f.close
puts ""
puts "Completed"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment