parallelpip demo

core.py

#! -*- Coding: utf-8 -*-

"""
Before running the create a virtualenv (I used version 1.7, so no-site-packages is default), activate virtualenv and install gevent, envoy and try running
"""
from gevent import monkey
monkey.patch_all()

import gevent
import time 
from envoy import run
from sys import exit, argv
import subprocess
import pip

def syntax():
    print "Usage:python core.py normal|parallel [libraries]"

def normal_download(lib = None):
    try:
        if lib:
            start = time.time()
            print "normal download started"
            for l in lib:
                print "Trying to install %s"%l
                run("pip install %s"%l)
            return time.time() - start
        else:
            syntax()
            exit()
    except:
        print "Unhandled exception"
        exit()

def parallel_download(lib = None):
    try:
        if lib:
            print "spawning using gevent"
            jobs = [gevent.spawn(pip.call_subprocess, ["pip","install",l]) \
                                                      for l in lib]
            start = time.time()
            print "joined all gevent, d/l started"
            gevent.joinall(jobs)
            for job in jobs:
                print job.value
            return time.time() - start
        else:
            syntax()
            exit()
    except:
        print "unhandled exception"
        exit()

if __name__ == "__main__":
    if argv[1] == 'parallel':
        print(parallel_download(argv[2:])," seconds for parallel d/l")
    elif argv[1] == 'normal':
        print(normal_download(argv[2:]), "seconds for normal d/l")
    else:
        syntax()
        exit()

(normalpip)kracekumar@python-lover:~/codes/python/asyncpip$ python core.py normal flask requests
normal download started
Trying to install flask
Trying to install requests
(146.34172201156616, 'seconds for normal d/l')

(testparallelpip)kracekumar@python-lover:~/codes/python/asyncpip$ python core.py parallel flask requests
spawning using gevent
joined all gevent, d/l started

Successfully installed requests certifi chardet
Cleaning up...
None
None
(83.12853598594666, ' seconds for parallel d/l')

Sweet!

Hmm there was an error in the snippet. I am trying it with a really huge list of packages and found that snippet didn't import pip. Updating it now.

This is a cool idea, but what about the dependency graph? e.g., if I have two packages that require something like pycurl- won't this install pycurl twice?

@robbyt: You need to find dependency tree and sort them else, you will get requirement already satisfied.

I am curious how much insight pip, as a python module instead of subprocess, can give us. Pip seems to do a good job determining these things if you're installing a requirements file.

pip install -r file.txt, reads line by line and tries to find package url and installs them one by one, So looking into https://github.com/pypa/pip/blob/develop/pip/init.py , https://github.com/pypa/pip/blob/develop/pip/commands/install.py, https://github.com/pypa/pip/blob/develop/pip/req.py, in same order I was able to find how it works, so we need to trace how this works, we will able to get an idea how to proceed.

Nowaday crate.io and vaincheese.herokuapp.com provides details about packages, I have a clone of vaincheese and fixed a major bug(https://github.com/kracekumar/vaincheese), so in this if we can request or create an api to return an link to project source code url or setup.py file, we are almost done.

How to avoid infinite depth install

Example: ppip install flask jinja blaze

Here flask requires jinja, werkzeug
blaze requires jina, flask, pyyaml,redis etc.

To solve the problem we need to think how to solve all possible circular library installation like classical graph problem.