Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Golang http close body blocking problem
package main
import (
"crypto/tls"
"net"
"net/http"
"time"
"fmt"
"errors"
)
func redirectPolicy(req *http.Request, via []*http.Request) error {
if len(via) >= 3 {
return errors.New("stopped after 3 redirects")
}
return nil
}
func main() {
transport := &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
Dial: func(netw, addr string) (net.Conn, error) {
// we want to wait a maximum of 1.75 seconds...
// since we're specifying a 1 second connect timeout and deadline
// (read/write timeout) is specified in absolute time we want to
// calculate that time first (before connecting)
deadline := time.Now().Add(800 * time.Millisecond)
c, err := net.DialTimeout(netw, addr, time.Second)
if err != nil {
return nil, err
}
c.SetDeadline(deadline)
return c, nil
}}
httpclient := &http.Client{Transport: transport, CheckRedirect: redirectPolicy}
req, err := http.NewRequest("GET", "http://bukk.it/boss.gif", nil)
req.Header.Add("User-Agent", "Bitly ImgGtr - Saving your bandwith and our time")
resp, err := httpclient.Do(req)
if err != nil {
return
}
start := time.Now().UnixNano()
resp.Body.Close()
fmt.Println("Close took: ", time.Now().UnixNano() - start)
}
@blakesmith

This comment has been minimized.

Copy link

commented Jul 15, 2012

Thanks a ton for this gist. I'm trying to make my http requests have a timeout and was a little confused about how to do it. This helps me understand the API a bit better, especially since I'm new to Go. Cheers!

@mynameisfiber

This comment has been minimized.

Copy link
Owner Author

commented Jul 15, 2012

@blakesmith be careful though, this code outlines a problem that you cannot terminate a request before the whole body has been read. A solution to this is to make the actual connection object (c in the Dial anonymous function) global. If you do this, you can close the connection, then the body, and essentially abort the connection whenever you want (regardless of timeout).

@blakesmith

This comment has been minimized.

Copy link

commented Jul 16, 2012

That is something I'll have to deal with, so thanks a ton for the heads up.

Out of curiosity, are you doing anything interesting or fun with Go?

@blakesmith

This comment has been minimized.

Copy link

commented Jul 16, 2012

@mynameisfiber So if my SetDeadline is set at 3 seconds, and a slow responder is only halfway through the response body by the time3 seconds has elapsed, the SetDeadline timeout won't fire?

Did you solve this by wrapping the entire request in another timeout that can catch slow responders? You seemed to allude to this with your 'make c global' approach, but not the actual mechanism you used to kill slow responses.

@mynameisfiber

This comment has been minimized.

Copy link
Owner Author

commented Jul 16, 2012

@blakesmith so, that's not completely right. If the SetDeadline triggers, then you will get a non-nil error if doing a ReadAll. Basically, whatever buffer you are using to read the data from will no longer get filled up and you will get some sort of indication that the connection has been terminated.

What I was talking about was manually terminating the connection before all data has been read and before the deadline. I was running into this because I only wanted to read the first 64 bits from a connection and then terminate (the goal was to only read image headers and determine the image resolution without having to download the entire file). In order to do this, I had to make the connection object global from within the dial function and then terminate the connection first (infact, the code above is a snippet from that).

@mynameisfiber

This comment has been minimized.

Copy link
Owner Author

commented Jul 16, 2012

Also, we're doing a bunch of fun things with go! We are making a new version of bitly/simplequeue that is native go with many more advanced features and message guarantees. We also have a sweet json library for saner json manipulation and I have a slightly more complete redis library in my repo in addition to an "image fetcher" that determines the resolution of an image based on the first several bytes! What are you working on?

@blakesmith

This comment has been minimized.

Copy link

commented Jul 17, 2012

Thanks for the info! At second glance, I shouldn't hit the same snags with my use case, since I should always be trying to fetch the entire body.

simplequeue seems like a cool idea; definitely something ripe for a Go port. What made you guys want to use an HTTP queueing system over something like Kestrel queues? Let me know if/when you guys open source it, I'd love to give it a whirl.

I just learned Go a couple weeks ago. First thing I did was port a toy service I built out for converting images into ascii art called skeeter. Now I'm playing around building out an http pinger and analytics gatherer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.