public
Last active

parallelpip demo

  • Download Gist
core.py
Python
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57
#! -*- Coding: utf-8 -*-
from gevent import monkey
monkey.patch_all()
 
import gevent
import time
from envoy import run
from sys import exit, argv
import subprocess
import pip
 
def syntax():
print "Usage:python core.py normal|parallel [libraries]"
 
def normal_download(lib = None):
try:
if lib:
start = time.time()
print "normal download started"
for l in lib:
print "Trying to install %s"%l
run("pip install %s"%l)
return time.time() - start
else:
syntax()
exit()
except:
print "Unhandled exception"
exit()
 
def parallel_download(lib = None):
try:
if lib:
print "spawning using gevent"
jobs = [gevent.spawn(pip.call_subprocess, ["pip","install",l]) \
for l in lib]
start = time.time()
print "joined all gevent, d/l started"
gevent.joinall(jobs)
for job in jobs:
print job.value
return time.time() - start
else:
syntax()
exit()
except Exception, e:
print "unhandled exception", e
exit()
 
if __name__ == "__main__":
if argv[1] == 'parallel':
print(parallel_download(argv[2:])," seconds for parallel d/l")
elif argv[1] == 'normal':
print(normal_download(argv[2:]), "seconds for normal d/l")
else:
syntax()
exit()
how to test
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111
1. Create two virtualenv named normalpip and parallelpip.
2. Install envoy, gevent in both the env(pip install envoy gevent)
3. . normalpip/bin/activate
4.clone the code and name as core.py.
5.python core.py normal flask requests
6. . parallelpip/bin/activate
7.python core.py parallel flask requests
sample output
=============
(normalpip)kracekumar@python-lover:~/codes/python/asyncpip$ python core.py normal flask requests
normal download started
Trying to install flask
Trying to install requests
(146.34172201156616, 'seconds for normal d/l')
 
(testparallelpip)kracekumar@python-lover:~/codes/python/asyncpip$ python core.py parallel flask requests
spawning using gevent
joined all gevent, d/l started
Downloading/unpacking flask
Downloading Flask-0.8.tar.gz (494Kb): 494Kb downloaded
Running setup.py egg_info for package flask
warning: no files found matching '*' under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'examples'
warning: no previously-included files matching '*.pyo' found under directory 'examples'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes/.git'
Downloading/unpacking Werkzeug>=0.6.1 (from flask)
Downloading Werkzeug-0.8.3.tar.gz (1.1Mb): 1.1Mb downloaded
Running setup.py egg_info for package Werkzeug
warning: no files found matching '*' under directory 'werkzeug/debug/templates'
warning: no files found matching '*' under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'examples'
warning: no previously-included files matching '*.pyo' found under directory 'examples'
no previously-included directories found matching 'docs/_build'
Downloading/unpacking Jinja2>=2.4 (from flask)
Downloading Jinja2-2.6.tar.gz (389Kb): 389Kb downloaded
Running setup.py egg_info for package Jinja2
warning: no previously-included files matching '*' found under directory 'docs/_build'
warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'jinja2'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
Installing collected packages: flask, Werkzeug, Jinja2
Running setup.py install for flask
warning: no files found matching '*' under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'examples'
warning: no previously-included files matching '*.pyo' found under directory 'examples'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes/.git'
Running setup.py install for Werkzeug
warning: no files found matching '*' under directory 'werkzeug/debug/templates'
warning: no files found matching '*' under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'examples'
warning: no previously-included files matching '*.pyo' found under directory 'examples'
no previously-included directories found matching 'docs/_build'
Running setup.py install for Jinja2
warning: no previously-included files matching '*' found under directory 'docs/_build'
warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'jinja2'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
Successfully installed flask Werkzeug Jinja2
Cleaning up...
Downloading/unpacking requests
Downloading requests-0.10.6.tar.gz (61Kb): 61Kb downloaded
Running setup.py egg_info for package requests
warning: no files found matching 'test_requests.py'
Downloading/unpacking certifi>=0.0.7 (from requests)
Downloading certifi-0.0.8.tar.gz (118Kb): 118Kb downloaded
Running setup.py egg_info for package certifi
Downloading/unpacking chardet>=1.0.0 (from requests)
Downloading chardet-1.0.1.tar.gz (156Kb): 156Kb downloaded
Running setup.py egg_info for package chardet
Installing collected packages: requests, certifi, chardet
Running setup.py install for requests
warning: no files found matching 'test_requests.py'
Running setup.py install for certifi
Running setup.py install for chardet
Successfully installed requests certifi chardet
Cleaning up...
None
None
(83.12853598594666, ' seconds for parallel d/l')

Please sign in to comment on this gist.

Something went wrong with that request. Please try again.