Golang http close body blocking problem

gistfile1.go

package main

import (
	"crypto/tls"
    "net"
    "net/http"
    "time"
    "fmt"
    "errors"
)

func redirectPolicy(req *http.Request, via []*http.Request) error {
	if len(via) >= 3 {
		return errors.New("stopped after 3 redirects")
	}
	return nil
}

func main() {
	transport := &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
		Dial: func(netw, addr string) (net.Conn, error) {
			// we want to wait a maximum of 1.75 seconds...
			// since we're specifying a 1 second connect timeout and deadline 
			// (read/write timeout) is specified in absolute time we want to 
			// calculate that time first (before connecting)
			deadline := time.Now().Add(800 * time.Millisecond)
			c, err := net.DialTimeout(netw, addr, time.Second)
			if err != nil {
				return nil, err
			}
			c.SetDeadline(deadline)
			return c, nil
		}}
	httpclient := &http.Client{Transport: transport, CheckRedirect: redirectPolicy}

    req, err := http.NewRequest("GET", "http://bukk.it/boss.gif", nil)
	req.Header.Add("User-Agent", "Bitly ImgGtr - Saving your bandwith and our time")

	resp, err := httpclient.Do(req)
	if err != nil {
		return 
	}

    start := time.Now().UnixNano()
    resp.Body.Close()
    fmt.Println("Close took: ", time.Now().UnixNano() - start)
}

Thanks a ton for this gist. I'm trying to make my http requests have a timeout and was a little confused about how to do it. This helps me understand the API a bit better, especially since I'm new to Go. Cheers!

@blakesmith be careful though, this code outlines a problem that you cannot terminate a request before the whole body has been read. A solution to this is to make the actual connection object (c in the Dial anonymous function) global. If you do this, you can close the connection, then the body, and essentially abort the connection whenever you want (regardless of timeout).

That is something I'll have to deal with, so thanks a ton for the heads up.

Out of curiosity, are you doing anything interesting or fun with Go?

@mynameisfiber So if my SetDeadline is set at 3 seconds, and a slow responder is only halfway through the response body by the time3 seconds has elapsed, the SetDeadline timeout won't fire?

Did you solve this by wrapping the entire request in another timeout that can catch slow responders? You seemed to allude to this with your 'make c global' approach, but not the actual mechanism you used to kill slow responses.

@blakesmith so, that's not completely right. If the SetDeadline triggers, then you will get a non-nil error if doing a ReadAll. Basically, whatever buffer you are using to read the data from will no longer get filled up and you will get some sort of indication that the connection has been terminated.

What I was talking about was manually terminating the connection before all data has been read and before the deadline. I was running into this because I only wanted to read the first 64 bits from a connection and then terminate (the goal was to only read image headers and determine the image resolution without having to download the entire file). In order to do this, I had to make the connection object global from within the dial function and then terminate the connection first (infact, the code above is a snippet from that).

Also, we're doing a bunch of fun things with go! We are making a new version of bitly/simplequeue that is native go with many more advanced features and message guarantees. We also have a sweet json library for saner json manipulation and I have a slightly more complete redis library in my repo in addition to an "image fetcher" that determines the resolution of an image based on the first several bytes! What are you working on?

Thanks for the info! At second glance, I shouldn't hit the same snags with my use case, since I should always be trying to fetch the entire body.

simplequeue seems like a cool idea; definitely something ripe for a Go port. What made you guys want to use an HTTP queueing system over something like Kestrel queues? Let me know if/when you guys open source it, I'd love to give it a whirl.

I just learned Go a couple weeks ago. First thing I did was port a toy service I built out for converting images into ascii art called skeeter. Now I'm playing around building out an http pinger and analytics gatherer.