Skip to content

Instantly share code, notes, and snippets.

@rdn32 rdn32/gist:5554769
Last active Dec 17, 2015

Embed
What would you like to do?
My solution to the webcrawler exercise in "A Tour of Go" (http://tour.golang.org/#70)The sample solution involved explicit locking, whereas my solution is purely based on communication (although the way it achieves this does seem a little bit awkward...)The assumption is that fetching things is what will take the time, so the contention on the c…
// (I've ommitted the code supplied as part of the exercise)
type crawl_result struct {
url string
depth int
body string
urls []string
err error
}
// Crawl uses fetcher to recursively crawl
// pages starting with url, to a maximum of depth.
func Crawl(url string, depth int, fetcher Fetcher) {
ch := make(chan *crawl_result)
fetch := func(url string, depth int) {
res := new(crawl_result)
res.depth = depth
res.url = url
res.body, res.urls, res.err = fetcher.Fetch(url)
ch <- res
}
current := 1
go fetch(url, depth)
seen := make(map[string]bool)
for current > 0 {
res := <- ch;
current--;
if res.err != nil {
fmt.Println(res.err)
} else {
fmt.Printf("found: %s %q\n", res.url, res.body)
depth := res.depth - 1
for _, u := range res.urls {
if depth > 0 && !seen[u] {
seen[u] = true
current++
go fetch(u, depth)
}
}
}
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.