public
Last active

async fetching of urls using goroutines and channels

  • Download Gist
gistfile1.go
Go
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56
package main
 
import (
"fmt"
"net/http"
"time"
)
 
var urls = []string{
"http://pulsoconf.co/",
"http://golang.org/",
"http://matt.aimonetti.net/",
}
 
type HttpResponse struct {
url string
response *http.Response
err error
}
 
func asyncHttpGets(urls []string) []*HttpResponse {
ch := make(chan *HttpResponse, len(urls)) // buffered
responses := []*HttpResponse{}
for _, url := range urls {
go func(url string) {
fmt.Printf("Fetching %s \n", url)
resp, err := http.Get(url)
resp.Body.Close()
ch <- &HttpResponse{url, resp, err}
}(url)
}
 
for {
select {
case r := <-ch:
fmt.Printf("%s was fetched\n", r.url)
responses = append(responses, r)
if len(responses) == len(urls) {
return responses
}
case <-time.After(50 * time.Millisecond):
fmt.Printf(".")
}
}
 
return responses
 
}
 
func main() {
results := asyncHttpGets(urls)
for _, result := range results {
fmt.Printf("%s status: %s\n", result.url,
result.response.Status)
}
}

Nice!

Your main() was slightly broken, see: https://gist.github.com/3804361

I'm trying to understand why you've got that Sleep() in there. Is it because buffered channels are non-blocking? Wouldn't it be more efficient to use a non-buffered channel so your select() gets responses as soon as they're available?

  • (I am just starting with Go, sorry if the answer is obvious!)

The sleep isn't required, I added it so the loops won't be too tight since I know that fetching a url takes some time.

You should be careful to close your response bodies when you're finished with them. Doing so releases the TCP connection to be reused for future requests.

Obviously it doesn't matter for this simple program but it would cause a leak if this was long running or looping.

Hi! It's nice to see more Go examples such as this one showing up in slide decks. :)

Couple of comments on this gist:

As @kr pointed out, a nicer version that doesn't require returning a slice is available here: https://gist.github.com/4013851/199db5624032fe503e9518b77375111ffb6ba54f

Please sign in to comment on this gist.

Something went wrong with that request. Please try again.