Skip to content

Instantly share code, notes, and snippets.

@batok
Created June 11, 2010 04:38
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 4 You must be signed in to fork a gist
  • Save batok/434053 to your computer and use it in GitHub Desktop.
Save batok/434053 to your computer and use it in GitHub Desktop.
A python script to do asynchronous upload to google storage using boto and eventlet libraries
# this program uploads to google storage using boto and eventlet all the jpg files of a selected folder
import eventlet
bcon = eventlet.import_patched("boto.gs.connection")
import glob
FOLDER = "/Users/myself/Documents/" # replace this with your chosen folder
BUCKET_NAME = "whateveryourbucketname" # replace this with your bucket name
def upload(myfile):
c = bcon.GSConnection()
bucket = c.get_bucket(BUCKET_NAME)
key = bucket.new_key(myfile)
with open("{0}{1}".format(FOLDER , myfile), "rb") as f:
print "begin upload of ", myfile
key.set_contents_from_file(f, policy="private")
print "uploaded", myfile
return myfile
if __name__ == "__main__":
pool = eventlet.GreenPool(200)
for fname in pool.imap(upload, [x.split("/")[-1] for x in glob.glob("{0}*jpg".format(FOLDER)) ]):
print "file ", fname , " at google storage"
@batok
Copy link
Author

batok commented Jun 11, 2010

This python script works with python 2.6. I think there are some issues with using eventlet in python 2.7. I tried with python 2.7rc1 but failed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment