Stream a tar file with flask.

flask_tar.py

import os
from cStringIO import StringIO
import tarfile
from flask import Flask, Response


app = Flask(__name__)


def get_string_io_len(s):
    pos = s.tell()
    s.seek(0, os.SEEK_END)
    length = s.tell()
    s.seek(pos)
    return length    

class CrazyBuffer(object):
    def __init__(self):
        self._buffer = StringIO()
    def read(self, nbytes=None):
        pass
    def write(self, data):
        self._buffer.write(data)
    def close(self):
        pass
    def tell(self):
        return 0
    def seek(self, offset, whence=None):
        return 0
    def get_value(self):
        return self._buffer.getvalue()
    def reset(self):
        self._buffer.close()
        self._buffer = StringIO()

def streamed_tar_response(files):
    crazy_buffer = CrazyBuffer()
    tar = tarfile.open("test.tar", "w", crazy_buffer)
    for i, f in enumerate(files):
        info = tarfile.TarInfo(name="File_%s.txt" % i)
        #stringIO specific here
        info.size = get_string_io_len(f)
        tar.addfile(info, f)
        yield crazy_buffer.get_value()
        crazy_buffer.reset()
    # Making shit up at this point in case there
    # is someting in the buffer after the loop.
    # Not sure if actually needed!
    tar.close()
    yield crazy_buffer.get_value()

@app.route("/tar", methods=["GET"])
def get_tar():
    files = [
        StringIO("I am file 1."),
        StringIO("I am file 2.")
    ]
    return Response(streamed_tar_response(files), mimetype="application/x-tar")

if __name__ == "__main__":
    app.run()

Terrible. This reads the entire tar file into memory and sends it all at once to the client. The only reason you call it streaming is cause there's a generator involved.

¯_(ツ)_/¯

Oh wait my bad. It's not as bad as I thought. As each file is added, it's read into memory and then sent to the client as a whole. Fine if all your files are small, but not what I'm looking for.

Oh wait my bad. It's not as bad as I thought. As each file is added, it's read into memory and then sent to the client as a whole. Fine if all your files are small, but not what I'm looking for.

I had the same problem. What's your solution?

https://github.com/ioistired/dirserver in there you'll find tarfile_stream, which is a port of tarfile from the stdlib to generators. see https://github.com/ioistired/dirserver/blob/00af943bd9cadd6c670f060b99f3e7b925d32e0f/app.py#L207 for usage instructions keep in mind the license of both files. they're both permissive but have sightly different licenses. let me know if you need help. alternatively to porting, you can write your own WSGI server (instead of using Flask's werkzeug), and pass a file object to the response. that may be better but it's been a while since I knew how to do that.

…

On April 3, 2023 8:43:38 AM UTC, Gu-f ***@***.***> wrote: @Gu-f commented on this gist: > Oh wait my bad. It's not as bad as I thought. As each file is added, it's read into memory and then sent to the client as a whole. Fine if all your files are small, but not what I'm looking for. I had the same problem. What's your solution? -- Reply to this email directly or view it on GitHub: https://gist.github.com/boris317/3693796#gistcomment-4524779 Unsubscribe from this thread: https://github.com/notifications/unsubscribe-auth/AAEDNYBJNNDBXLCFL5SE2LTW7KETXBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFQKSXMYLMOVS2I5DSOVS2I3TBNVS3W5DIOJSWCZC7OBQXE5DJMNUXAYLOORPWCY3UNF3GS5DZVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFU4ZTMOJTG44TNJ3UOJUWOZ3FOKTGG4TFMF2GK You are receiving this because you commented on the thread Message ID: ***@***.***>