gistfile1.txt

<http-connection-pools xmlns="http://docs.openrepose.org/repose/http-connection-pool/v1.0">

    <!-- Configuration for the default pool.  Any users of the service will by default, retrieve HTTP connections
        using this default pool configuration.
    -->
    <pool id="default"
          default="true"
          chunked-encoding="false"
          http.conn-manager.max-total="500"
          http.conn-manager.max-per-route="500"
          http.socket.timeout="600000"
          http.socket.buffer-size="8192"
          http.connection.timeout="10000"
          http.connection.max-line-length="8192"
          http.connection.max-header-count="100"
          http.connection.max-status-line-garbage="200"
          http.tcp.nodelay="true"
          keepalive.timeout="60"/>

</http-connection-pools>

• something before haproxy disconnects abruptly (TBD what though)
• this affects the client (such as an in-progress download is stopped abruptly - like us-texas-3)
• also haproxy sees a client disconnect

if repose is not present, we see the error in glance api logs and system recovers

if repose is present

• repose gets an exception when trying to flush the output stream to the client, but fails to close the input stream to glance
• because stream is still open and connection still present, glance then is still sending content, and so we see huge queues on the download path between glance and repose
• glance for each active download spawns a set of connections to ceph, 1 per drive, 2 threads per connection (100-300 drives)
• memory shoots up due to allocation of stackspace for the threads supporting the connections that support the download
• if too many of these errors, we see out of memory at the box level