Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
async fetching of urls using goroutines and channels
package main
import (
"fmt"
"net/http"
"time"
)
var urls = []string{
"https://splice.com/",
"https://golang.org/",
"https://matt.aimonetti.net/",
}
type HttpResponse struct {
url string
response *http.Response
err error
}
func asyncHttpGets(urls []string) []*HttpResponse {
ch := make(chan *HttpResponse, len(urls)) // buffered
responses := []*HttpResponse{}
for _, url := range urls {
go func(url string) {
fmt.Printf("Fetching %s \n", url)
resp, err := http.Get(url)
if err == nil {
resp.Body.Close()
}
ch <- &HttpResponse{url, resp, err}
}(url)
}
for {
select {
case r := <-ch:
fmt.Printf("%s was fetched\n", r.url)
responses = append(responses, r)
if len(responses) == len(urls) {
return responses
}
case <-time.After(50 * time.Millisecond):
fmt.Printf(".")
}
}
return responses
}
func main() {
results := asyncHttpGets(urls)
for _, result := range results {
if result.err != nil {
fmt.Printf("%s error: %v\n", result.url,
result.err)
continue
}
fmt.Printf("%s status: %s\n", result.url,
result.response.Status)
}
}
@ismasan

This comment has been minimized.

Copy link

@ismasan ismasan commented Sep 29, 2012

Nice!

Your main() was slightly broken, see: https://gist.github.com/3804361

@ismasan

This comment has been minimized.

Copy link

@ismasan ismasan commented Sep 29, 2012

I'm trying to understand why you've got that Sleep() in there. Is it because buffered channels are non-blocking? Wouldn't it be more efficient to use a non-buffered channel so your select() gets responses as soon as they're available?

  • (I am just starting with Go, sorry if the answer is obvious!)
@mattetti

This comment has been minimized.

Copy link
Owner Author

@mattetti mattetti commented Sep 29, 2012

The sleep isn't required, I added it so the loops won't be too tight since I know that fetching a url takes some time.

@bgentry

This comment has been minimized.

Copy link

@bgentry bgentry commented Sep 29, 2012

You should be careful to close your response bodies when you're finished with them. Doing so releases the TCP connection to be reused for future requests.

Obviously it doesn't matter for this simple program but it would cause a leak if this was long running or looping.

@kr

This comment has been minimized.

Copy link

@kr kr commented Nov 4, 2012

Hi! It's nice to see more Go examples such as this one showing up in slide decks. :)

Couple of comments on this gist:

@mattetti

This comment has been minimized.

Copy link
Owner Author

@mattetti mattetti commented Nov 28, 2012

As @kr pointed out, a nicer version that doesn't require returning a slice is available here: https://gist.github.com/4013851/199db5624032fe503e9518b77375111ffb6ba54f

@pranjal5215

This comment has been minimized.

Copy link

@pranjal5215 pranjal5215 commented Jul 29, 2014

How about fetching http requests with a timeout .
This one does not work; gives a runtime error .

https://gist.github.com/pranjal5215/18f95fa506d59db9c740

What I m basically trying to do is fetch URLs with a timeout (have kept very small timeout; want to timeout deliberately), but on timeout throws runtime.

[I am New to go]

@chrispmontoya

This comment has been minimized.

Copy link

@chrispmontoya chrispmontoya commented Dec 22, 2018

Thank you!

@DudeFactory

This comment has been minimized.

Copy link

@DudeFactory DudeFactory commented Feb 8, 2020

Hello from the future.
The code doesn't work.
Error is:

./prog.go:46:2: unreachable code
Go vet exited.

Fetching http://pulsoconf.co/
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0xffffffff addr=0x0 pc=0x2b79aa]

goroutine 6 [running]:
main.asyncHttpGets.func1(0x832180, 0x328cb4, 0x14, 0x864ed8)
/tmp/sandbox137476626/prog.go:28 +0xea
created by main.asyncHttpGets
/tmp/sandbox137476626/prog.go:25 +0xa0

I'm new in GO, I tried to understand the issue, but without result. Could you please help?

@mattetti

This comment has been minimized.

Copy link
Owner Author

@mattetti mattetti commented Feb 8, 2020

@DudeFactory the problem was that one of the domains wasn't active anymore and I wasn't checking the error when fetching the URL.
I fixed the domain and added an error check after resp, err := http.Get(url) see: https://gist.github.com/mattetti/3798173#file-gistfile1-go-L28

Hopefully that helps clarifying your confusion.

@DudeFactory

This comment has been minimized.

Copy link

@DudeFactory DudeFactory commented Feb 9, 2020

@DudeFactory the problem was that one of the domains wasn't active anymore and I wasn't checking the error when fetching the URL.
I fixed the domain and added an error check after resp, err := http.Get(url) see: https://gist.github.com/mattetti/3798173#file-gistfile1-go-L28

Hopefully that helps clarifying your confusion.

Thanks. I tried only check resp is nil

@DudeFactory

This comment has been minimized.

Copy link

@DudeFactory DudeFactory commented Feb 10, 2020

Still doesn't work for me. For example, I added new domains:

	"http://www.webmagnat.ro",
	"http://nickelfreesolutions.com",
	"http://scheepvaarttelefoongids.nl",
	"http://tursan.net",
	"http://plannersanonymous.com",
	"http://saltstack.com",
	"http://deconsquad.com",
	"http://migom.com",
	"http://tjprc.org",
	"http://worklife.dk",
	"http://food-hub.org",

So it is stuck and some one suggest me use "Wait Group construct", but why it is working for you?

Stackoverflow question: https://stackoverflow.com/questions/60148016/how-send-n-get-requests-where-n-10-urls

@mattetti

This comment has been minimized.

Copy link
Owner Author

@mattetti mattetti commented Feb 10, 2020

@DudeFactory I fixed the example, try with your domains now. The problem was that my quick fix was skipping writing to the channel in case of error so I didn't have to check if there was an error. The fix was to check for errors when reading from the channel and print the proper statement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.