Currently WSGI servers or WSGI middlewares cannot accept chunked requests or perform request filtering because a WSGI application needs to depend on the content length as this is the only thing that the specification currently guarantees will work.
Essentially this means that each WSGI setup is currently in violation with HTTP/1.1 which requires chunked requests to be supported.
Different WSGI servers have different solutions for this problem, so this specification proposes two pre-defined setups that will support chunked requests. One is the safe solution that is the de-facto of what PEP 333 and PEP 3333 provide, the other one is a new usage mode that is the de-facto implementation that is already available in most WSGI servers to date.
A two step proposal to fix the situation with different behaviors of input streams in WSGI.
WSGI servers have two options to providing the input stream:
wsgi.inputas socket file unchanged. This means that
wsgi.input_terminatedis set to False or not added to the WSGI environ at all. In that case the WSGI application is required to look at the
CONTENT_LENGTHand only read up to that point. Applications in that case are required to count up the bytes received and comparing it against the content length. If less than
CONTENT_LENGTHis received the application must respond to this in order to not accidentally assume a regular end of the stream has been reached.
wsgi.inputas an end-of-file terminated stream. In that case
wsgi.input_terminatedis set to True and an app is required to read to the end of the file and disregard
In addition to that the server is also required to raise an IOError if the client disconnects without transmitting all the data as it would otherwise not be possible for the client to reliably detect a disconnect.
This is already what many WSGI servers do, but for an application there has not been a way to detect that behavior yet.
Pseudocode for a WSGI implementation:
def get_input_stream(environ): stream = environ['wsgi.input'] # This part is new if environ.get('wsgi.input_terminated'): return stream # This part was necessary before anyways to not accidentally # read past the length of the stream. return wrap_stream(environ['wsgi.input'], environ['CONTENT_LENGTH'])
The only thing that needs to be changed in the WSGI server is either
nothing (for instance wsgiref or any other simple WSGI server that just
puts the socket through does nothing) or a server like mod_wsgi or
gunicorn that terminate the input stream set the flag
wsgi.input_terminated to True when making the WSGI environ.
Some common questions that came up doing this propsal.
I never wrapped my input streams before, why/how do I do that?
This is actually quite tricky due to a bunch of ugly warts about the
stream system in Python. There is not much you can do about that than
carefully wrapping the input stream. This implementation is similar to
what Werkzeug does. A lot of WSGI implementations currently do not
properly wrap the input stream and already assume that
wsgi.input_terminated is actually set.
In addition to just wrapping the stream for not reading past the content length it's also crucial to detect disconnected clients which is marked by less data then the content lenght being received.
Here is how Werkzeug wraps the stream:
def wrap_stream(input_stream, content_length): return LimitedStream(input_stream, max(0, int(content_length or 0))) class LimitedStream(object): def __init__(self, stream, limit): self._read = stream.read self._readline = stream.readline self._pos = 0 self._limit = limit def read(self, size=None): if self._pos >= self._limit: return b'' if size is None or size < 0: size = self._limit to_read = min(self._limit - self._pos, size) read = self._read(to_read) if to_read and len(read) != to_read: raise IOError('The client went away') self._pos += len(read) return read def readline(self, size=None): if self._pos >= self._limit: return b'' if size is None: size = self._limit - self._pos else: size = min(size, self._limit - self._pos) line = self._readline(size) if size and not line: raise IOError('The client went away') self._pos += len(line) return line def readlines(self, size=None): result =  if size is not None: end = min(self._limit, last_pos + size) else: end = self._limit while 1: if size is not None: size -= last_pos - self._pos if self._pos >= end: break result.append(self.readline(size)) if size is not None: last_pos = self._pos return result def __next__(self): line = self.readline() if not line: raise StopIteration() return line def __iter__(self): return self # Python 2 support next = __next__
How do I use this in a middleware?
Generally the answer is you don't. Filterings should be done on the webserver and that's the whole point of this proposal as it makes filtering on the server possible.
However it does now also allow you to do it in a middleware if you absolutely need to do that. This can for instance be useful if you need to do demasking or custom decryption on the input data stream.
Here is an example of how you can implement unzipping in a middleware:
import zlib class UnzippingStream(object): def __init__(self, stream): self._stream = stream self._decomp = zlib.decompressobj() self._buf_out = bytearray() self._hit_eof = False def _feed(self, read_all=False): if self._hit_eof: return True if read_all: input_data = self._stream.read() else: input_data = self._stream.read(2048) new_in = self._decomp.unconsumed_tail + input_data self._buf_out += self._decomp.decompress(bytes(new_in)) if not input_data: self._buf_out += self._decomp.flush() self._hit_eof = not input_data return self._hit_eof def _fetch(self, length=None): rv = self._buf_out[:length] del self._buf_out[:length] return bytes(rv) def read(self, size=None): if size is None: self._feed(True) was_empty = False while 1: if size is None: return self._fetch() elif was_empty or len(self._buf_out) >= size: return self._fetch(size) was_empty = not self._feed() def readline(self, size=None): while 1: was_empty = self._feed() rv = self._buf_out.splitlines(True) if rv and (was_empty or rv.endswith(b'\r\n') or \ len(rv) > 1 and rv.endswith((b'\r', b'\n'))): return self._fetch(len(rv)) if was_empty: return b'' def readlines(self): result =  while 1: rv = self.readline() if not rv: break result.append(rv) return rv class UnzippingMiddleware(object): def __init__(self, app): self.app = app def __call__(self, environ, start_response): # Fetch the input stream. See function definition above for # more information. input_stream = get_input_stream(environ) # Wrap the stream. Since this changes the content length we # now need to inform the application that the input is # terminated and the content length becomes hit only. wsgi['wsgi.input'] = UnzippingStream(input_stream) wsgi['wsgi.input_terminated'] = True # Invoke the application. return self.app(environ, start_response)
Why not a separate stream?
Why would you want a boolean marker instead of a separate stream object? Primarily because it keeps the WSGI environment nice and tidy and middlewares just need to flip the input terminated flag when they wrap it. It means the proposal is fully backwards compatible and that flag just indicates the edge cases. Input stream handling is areadly pretty complicated for both server and client and this proposal is just fixing the edge cases more consistently.
Hopefully going into the future most WSGI servers are going to set the flag to True as this removes complexity from the WSGI client application. Until that however is the case clients will need to support the worst case which is that flag not being set in which case they are required to work within the vague wording of the original specification.
In addition to that a boolean flag can be injected very easily from the outside, so mod_wsgi for instance already can support this proposal by just setting it from the apache config.
What if my middleware depends on
This proposal adds new functionality: namely that a server can finally accept chunked requests and do request content filtering. Your middleware will need to be fixed to support that. That said, this proposal is not removing functionality. Previously if you sent a chunked request to a WSGI server it would just have timed out most likely or failed in some other form. If your middleware continues to rely on a content length that might be incorrect then the same behavior is restored: your application will not respond properly to this request.
Yes: such middlewares will indeed need to now honor the termination flag, but on the other hand this is not a regression or a new problem. If anything this just means that unless your whole stack upgrades you won't benefit from this new flag.