Last active
January 5, 2017 17:02
-
-
Save vegitron/645cb77725194f865b9b9d3af6723ed5 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Using python's httplib, there is no way to set a timeout for the full request cycle. A timeout can be given, which will apply to the connection attempt and each blocking socket operation. | |
However, a malicious (or poorly performing) server can keep a connection open for an indefinite amount of time by slowly sending data. Consider this server process: | |
https://gist.github.com/vegitron/bc883ddc88fe9253adc3e0bccea6445e | |
and this client: | |
https://gist.github.com/vegitron/4ee269b6492ff80d350e108363689d5c | |
With a timeout of 0.5, the client takes 0.501363039017 seconds. With a timeout of 2.5, it takes 10.0041370392 seconds. | |
This is explained in the documentation, but it's a problem. A commonly suggested solution is to use SIGALRM to set a timeout, but that doesn't work in a multi-threaded environment. Moving to multi-process introduces big slow downs as I can't use connection pools, and data needs to be serialized and deserialized for the parent process. | |
I would like to propose an addition to httplib that would add a hook to httplib.HTTPResponse._read_chunked (python 2) and http.client.HTTPResponse._readall_chunked (python 3) that would either: | |
1) support an overall read timeout, by tracking a per-response start time and raising a timeout exception if that chunk read finishes after the given timeout | |
2) support a per-chunk callback function, so a client of httplib/http.client could define that logic for themselves. | |
Current possible timeouts, and where they can happen: | |
connect | read chunk | read chunk | read chunk | |
[ timeout ] [ timeout ] [ timeout ] [ timeout ] | |
Proposed addition: | |
connect | read chunk | read chunk | read chunk | |
[ timeout ] [ timeout ] [ timeout ] [ timeout ] | |
[ total read time out ] |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment