Skip to content

Instantly share code, notes, and snippets.

@nbari
Last active July 16, 2023 18:04
Show Gist options
  • Star 22 You must be signed in to star a gist
  • Fork 14 You must be signed in to fork a gist
  • Save nbari/7335384 to your computer and use it in GitHub Desktop.
Save nbari/7335384 to your computer and use it in GitHub Desktop.
python chunk upload files
#!/usr/bin/env python
import os
import requests
import uuid
from random import randint
from uuid import uuid4
def read_in_chunks(file_object, chunk_size=65536):
while True:
data = file_object.read(chunk_size)
if not data:
break
yield data
def main(file, url):
content_name = str(file)
content_path = os.path.abspath(file)
content_size = os.stat(content_path).st_size
print content_name, content_path, content_size
f = open(content_path)
index = 0
offset = 0
headers = {}
for chunk in read_in_chunks(f):
offset = index + len(chunk)
headers['Content-Type'] = 'application/octet-stream'
headers['Content-length'] = content_size
headers['Content-Range'] = 'bytes %s-%s/%s' % (index, offset, content_size)
index = offset
try:
r = requests.put(url, data=chunk, headers=headers)
print "r: %s, Content-Range: %s" % (r, headers['Content-Range'])
except Exception, e:
print e
if __name__ == '__main__':
url = 'http://localhost:8080/test_upload/%s' % str(uuid4())
main('images/test_image_%d.jpg' % randint(1,3), url)
@bfrakes
Copy link

bfrakes commented Apr 7, 2018

Thanks!! Very helpful

@mfatkhutdinova
Copy link

Hello! Thank you)
But I have problem: Exception: Error: Requested URL Not Found:
400 Client Error: Bad Request for url
Without headers['Content-Range'], I can upload my file but only last piece. Maybe you know, how I can solve this problem?

@pghole
Copy link

pghole commented Jun 29, 2021

This will cause multiple requests to the '/test_upload' api. Is there any way where you can read in chunks and send in the same request (1 request). Pls. suggest me if you know any of the approach. My question - psf/requests#5849

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment