Skip to content

Instantly share code, notes, and snippets.

@jasonmadigan
Created August 24, 2012 09:55
Show Gist options
  • Star 3 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save jasonmadigan/3448470 to your computer and use it in GitHub Desktop.
Save jasonmadigan/3448470 to your computer and use it in GitHub Desktop.
404 Finder - Simple Ruby site crawler, finds 404s
#!/usr/bin/env ruby
# Simple 404 finder
require 'rubygems'
require 'anemone'
url = ARGV[0]
if url.nil?
puts "Usage: 404_finder.rb http://example.com"
exit
end
Anemone.crawl(url) do |anemone|
anemone.on_every_page do |page|
if !page.code.nil? and page.code > 302 and !page.url.to_s.include?('%23')
puts "[#{page.code}] #{page.url} - Referrer: #{page.referer}"
end
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment