Created
September 10, 2012 20:57
-
-
Save boris317/3693796 to your computer and use it in GitHub Desktop.
Stream a tar file with flask.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import os | |
from cStringIO import StringIO | |
import tarfile | |
from flask import Flask, Response | |
app = Flask(__name__) | |
def get_string_io_len(s): | |
pos = s.tell() | |
s.seek(0, os.SEEK_END) | |
length = s.tell() | |
s.seek(pos) | |
return length | |
class CrazyBuffer(object): | |
def __init__(self): | |
self._buffer = StringIO() | |
def read(self, nbytes=None): | |
pass | |
def write(self, data): | |
self._buffer.write(data) | |
def close(self): | |
pass | |
def tell(self): | |
return 0 | |
def seek(self, offset, whence=None): | |
return 0 | |
def get_value(self): | |
return self._buffer.getvalue() | |
def reset(self): | |
self._buffer.close() | |
self._buffer = StringIO() | |
def streamed_tar_response(files): | |
crazy_buffer = CrazyBuffer() | |
tar = tarfile.open("test.tar", "w", crazy_buffer) | |
for i, f in enumerate(files): | |
info = tarfile.TarInfo(name="File_%s.txt" % i) | |
#stringIO specific here | |
info.size = get_string_io_len(f) | |
tar.addfile(info, f) | |
yield crazy_buffer.get_value() | |
crazy_buffer.reset() | |
# Making shit up at this point in case there | |
# is someting in the buffer after the loop. | |
# Not sure if actually needed! | |
tar.close() | |
yield crazy_buffer.get_value() | |
@app.route("/tar", methods=["GET"]) | |
def get_tar(): | |
files = [ | |
StringIO("I am file 1."), | |
StringIO("I am file 2.") | |
] | |
return Response(streamed_tar_response(files), mimetype="application/x-tar") | |
if __name__ == "__main__": | |
app.run() |
¯_(ツ)_/¯
Oh wait my bad. It's not as bad as I thought. As each file is added, it's read into memory and then sent to the client as a whole. Fine if all your files are small, but not what I'm looking for.
Oh wait my bad. It's not as bad as I thought. As each file is added, it's read into memory and then sent to the client as a whole. Fine if all your files are small, but not what I'm looking for.
I had the same problem. What's your solution?
https://github.com/ioistired/dirserver
in there you'll find tarfile_stream, which is a port of tarfile from the stdlib to generators.
see https://github.com/ioistired/dirserver/blob/00af943bd9cadd6c670f060b99f3e7b925d32e0f/app.py#L207 for usage instructions
keep in mind the license of both files. they're both permissive but have sightly different licenses. let me know if you need help.
alternatively to porting, you can write your own WSGI server (instead of using Flask's werkzeug), and pass a file object to the response. that may be better but it's been a while since I knew how to do that.
…On April 3, 2023 8:43:38 AM UTC, Gu-f ***@***.***> wrote:
@Gu-f commented on this gist:
> Oh wait my bad. It's not as bad as I thought. As each file is added, it's read into memory and then sent to the client as a whole. Fine if all your files are small, but not what I'm looking for.
I had the same problem. What's your solution?
--
Reply to this email directly or view it on GitHub: https://gist.github.com/boris317/3693796#gistcomment-4524779
Unsubscribe from this thread: https://github.com/notifications/unsubscribe-auth/AAEDNYBJNNDBXLCFL5SE2LTW7KETXBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFQKSXMYLMOVS2I5DSOVS2I3TBNVS3W5DIOJSWCZC7OBQXE5DJMNUXAYLOORPWCY3UNF3GS5DZVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFU4ZTMOJTG44TNJ3UOJUWOZ3FOKTGG4TFMF2GK
You are receiving this because you commented on the thread
Message ID: ***@***.***>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Terrible. This reads the entire tar file into memory and sends it all at once to the client. The only reason you call it streaming is cause there's a generator involved.