kracekumar / core.py
Created

Embed URL

HTTPS clone URL

SSH clone URL

You can clone with HTTPS or SSH.

Download Gist

parallelpip demo

View core.py
Raw
File suppressed. Click to show.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 
#! -*- Coding: utf-8 -*-

from gevent import monkey

monkey.patch_all()

 

import gevent

import time 

from envoy import run

from sys import exit, argv

import subprocess

import pip

 

def syntax():

    print "Usage:python core.py normal|parallel [libraries]"

 

def normal_download(lib = None):

    try:

        if lib:

            start = time.time()

            print "normal download started"

            for l in lib:

                print "Trying to install %s"%l

                run("pip install %s"%l)

            return time.time() - start

        else:

            syntax()

            exit()

    except:

        print "Unhandled exception"

        exit()

 

def parallel_download(lib = None):

    try:

        if lib:

            print "spawning using gevent"

            jobs = [gevent.spawn(pip.call_subprocess, ["pip","install",l]) \

                                                      for l in lib]

            start = time.time()

            print "joined all gevent, d/l started"

            gevent.joinall(jobs)

            for job in jobs:

                print job.value

            return time.time() - start

        else:

            syntax()

            exit()

    except Exception, e:

        print "unhandled exception", e

        exit()

 

if __name__ == "__main__":

    if argv[1] == 'parallel':

        print(parallel_download(argv[2:])," seconds for parallel d/l")

    elif argv[1] == 'normal':

        print(normal_download(argv[2:]), "seconds for normal d/l")

    else:

        syntax()

        exit()

 
View core.py
Raw
File suppressed. Click to show.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 
1. Create two virtualenv named normalpip and parallelpip.

2. Install envoy, gevent in both the env(pip install envoy gevent)

3. . normalpip/bin/activate

4.clone the code and name as core.py.

5.python core.py normal flask requests

6. . parallelpip/bin/activate

7.python core.py parallel flask requests

sample output

=============

(normalpip)kracekumar@python-lover:~/codes/python/asyncpip$ python core.py normal flask requests

normal download started

Trying to install flask

Trying to install requests

(146.34172201156616, 'seconds for normal d/l')

 

(testparallelpip)kracekumar@python-lover:~/codes/python/asyncpip$ python core.py parallel flask requests

spawning using gevent

joined all gevent, d/l started

Downloading/unpacking flask

  Downloading Flask-0.8.tar.gz (494Kb): 494Kb downloaded

  Running setup.py egg_info for package flask

    

    warning: no files found matching '*' under directory 'tests'

    warning: no previously-included files matching '*.pyc' found under directory 'docs'

    warning: no previously-included files matching '*.pyo' found under directory 'docs'

    warning: no previously-included files matching '*.pyc' found under directory 'tests'

    warning: no previously-included files matching '*.pyo' found under directory 'tests'

    warning: no previously-included files matching '*.pyc' found under directory 'examples'

    warning: no previously-included files matching '*.pyo' found under directory 'examples'

    no previously-included directories found matching 'docs/_build'

    no previously-included directories found matching 'docs/_themes/.git'

Downloading/unpacking Werkzeug>=0.6.1 (from flask)

  Downloading Werkzeug-0.8.3.tar.gz (1.1Mb): 1.1Mb downloaded

  Running setup.py egg_info for package Werkzeug

    

    warning: no files found matching '*' under directory 'werkzeug/debug/templates'

    warning: no files found matching '*' under directory 'tests'

    warning: no previously-included files matching '*.pyc' found under directory 'docs'

    warning: no previously-included files matching '*.pyo' found under directory 'docs'

    warning: no previously-included files matching '*.pyc' found under directory 'tests'

    warning: no previously-included files matching '*.pyo' found under directory 'tests'

    warning: no previously-included files matching '*.pyc' found under directory 'examples'

    warning: no previously-included files matching '*.pyo' found under directory 'examples'

    no previously-included directories found matching 'docs/_build'

Downloading/unpacking Jinja2>=2.4 (from flask)

  Downloading Jinja2-2.6.tar.gz (389Kb): 389Kb downloaded

  Running setup.py egg_info for package Jinja2

    

    warning: no previously-included files matching '*' found under directory 'docs/_build'

    warning: no previously-included files matching '*.pyc' found under directory 'jinja2'

    warning: no previously-included files matching '*.pyc' found under directory 'docs'

    warning: no previously-included files matching '*.pyo' found under directory 'jinja2'

    warning: no previously-included files matching '*.pyo' found under directory 'docs'

Installing collected packages: flask, Werkzeug, Jinja2

  Running setup.py install for flask

    

    warning: no files found matching '*' under directory 'tests'

    warning: no previously-included files matching '*.pyc' found under directory 'docs'

    warning: no previously-included files matching '*.pyo' found under directory 'docs'

    warning: no previously-included files matching '*.pyc' found under directory 'tests'

    warning: no previously-included files matching '*.pyo' found under directory 'tests'

    warning: no previously-included files matching '*.pyc' found under directory 'examples'

    warning: no previously-included files matching '*.pyo' found under directory 'examples'

    no previously-included directories found matching 'docs/_build'

    no previously-included directories found matching 'docs/_themes/.git'

  Running setup.py install for Werkzeug

    

    warning: no files found matching '*' under directory 'werkzeug/debug/templates'

    warning: no files found matching '*' under directory 'tests'

    warning: no previously-included files matching '*.pyc' found under directory 'docs'

    warning: no previously-included files matching '*.pyo' found under directory 'docs'

    warning: no previously-included files matching '*.pyc' found under directory 'tests'

    warning: no previously-included files matching '*.pyo' found under directory 'tests'

    warning: no previously-included files matching '*.pyc' found under directory 'examples'

    warning: no previously-included files matching '*.pyo' found under directory 'examples'

    no previously-included directories found matching 'docs/_build'

  Running setup.py install for Jinja2

    

    warning: no previously-included files matching '*' found under directory 'docs/_build'

    warning: no previously-included files matching '*.pyc' found under directory 'jinja2'

    warning: no previously-included files matching '*.pyc' found under directory 'docs'

    warning: no previously-included files matching '*.pyo' found under directory 'jinja2'

    warning: no previously-included files matching '*.pyo' found under directory 'docs'

Successfully installed flask Werkzeug Jinja2

Cleaning up...

Downloading/unpacking requests

  Downloading requests-0.10.6.tar.gz (61Kb): 61Kb downloaded

  Running setup.py egg_info for package requests

    

    warning: no files found matching 'test_requests.py'

Downloading/unpacking certifi>=0.0.7 (from requests)

  Downloading certifi-0.0.8.tar.gz (118Kb): 118Kb downloaded

  Running setup.py egg_info for package certifi

    

Downloading/unpacking chardet>=1.0.0 (from requests)

  Downloading chardet-1.0.1.tar.gz (156Kb): 156Kb downloaded

  Running setup.py egg_info for package chardet

    

Installing collected packages: requests, certifi, chardet

  Running setup.py install for requests

    

    warning: no files found matching 'test_requests.py'

  Running setup.py install for certifi

    

  Running setup.py install for chardet

    

Successfully installed requests certifi chardet

Cleaning up...

None

None

(83.12853598594666, ' seconds for parallel d/l')
sholsapp commented

The parallel version gives me errors when I run with many dependencies that have a common set of transitive dependencies. The error pops up when multiple projects try to simultaneously installed a transitive dependency. Any ideas how to alleviate/fix that problem?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.