Lekhika Dugtal CleverFool77
👋
palianytsia
/ Crawler.java
Created
December 9, 2012 19:40
Demo of a simple web crawler, that fetches the content for a given start URL, extracts the links from the content, goes on to crawl the extracted links (back to step 1) and stops after 1000 found URLs.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import java.io.BufferedReader; | |
| import java.io.IOException; | |
| import java.io.InputStreamReader; | |
| import java.net.MalformedURLException; | |
| import java.net.URL; | |
| import java.util.ArrayList; | |
| import java.util.HashSet; | |
| import java.util.Iterator; | |
| import java.util.LinkedHashSet; | |
| import java.util.List; |