IPython startup script to detect and inject VIRTUAL_ENV's site-packages dirs.

00-detect-virtualenv-sitepackages.py

"""IPython startup script to detect and inject VIRTUAL_ENV's site-packages dirs.

IPython can detect virtualenv's path and injects it's site-packages dirs into sys.path.
But it can go wrong if IPython's python version differs from VIRTUAL_ENV's.

This module fixes it looking for the actual directories. We use only old stdlib
resources so it can work with as many Python versions as possible.

References:
http://stackoverflow.com/a/30650831/443564
http://stackoverflow.com/questions/122327/how-do-i-find-the-location-of-my-python-site-packages-directory
https://github.com/ipython/ipython/blob/master/IPython/core/interactiveshell.py#L676

Author: Henrique Bastos <henrique@bastos.net>
License: BSD
"""
import os
import sys
from warnings import warn


virtualenv = os.environ.get('VIRTUAL_ENV')

if virtualenv:

    version = os.listdir(os.path.join(virtualenv, 'lib'))[0]
    site_packages = os.path.join(virtualenv, 'lib', version, 'site-packages')
    lib_dynload = os.path.join(virtualenv, 'lib', version, 'lib-dynload')

    if not (os.path.exists(site_packages) and os.path.exists(lib_dynload)):
        msg = 'Virtualenv site-packages discovery went wrong for %r' % repr([site_packages, lib_dynload])
        warn(msg)

    try:
        i = sys.path.index("") + 1
    except ValueError:
        i = 0
    sys.path.insert(i, site_packages)
    sys.path.insert(i+1, lib_dynload)

I don't know if there's a way to send pull requests for a gist. Anyway, there's a suggestion below.

Keeping the default Python behavior to search for modules in current directory first, change lines 34 and 35 to this:

    try:
        i = sys.path.index("") + 1
    except ValueError:
        i = 0
    sys.path.insert(i, site_packages)
    sys.path.insert(i+1, lib_dynload)

It allows local custom modules override their version even inside the virtualenv.

Good work. Keep walking.

Nice! Done! I'm trusting you've tested it #lol

hahaha

Yes. I'm using it for a few days and it's working.

I was following your Medium post to get something setup on my machine. I started running into some issues with packages that used get_distribution from pkg_resources (in setup tools) in their __init__.py files. The error would look like:

Python 3.6.1 (default, Oct  6 2017, 10:41:43) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.2.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]:  from pkg_resources import get_distribution

In [2]: __version__ = get_distribution('numpy').version
---------------------------------------------------------------------------
DistributionNotFound                      Traceback (most recent call last)
<ipython-input-2-3b18f3abdfeb> in <module>()
----> 1 __version__ = get_distribution('numpy').version

~/.pyenv/versions/3.6.1/envs/jupyter3_install/lib/python3.6/site-packages/pkg_resources/__init__.py in get_distribution(dist)
    560         dist = Requirement.parse(dist)
    561     if isinstance(dist, Requirement):
--> 562         dist = get_provider(dist)
    563     if not isinstance(dist, Distribution):
    564         raise TypeError("Expected string, Requirement, or Distribution", dist)

~/.pyenv/versions/3.6.1/envs/jupyter3_install/lib/python3.6/site-packages/pkg_resources/__init__.py in get_provider(moduleOrReq)
    434     """Return an IResourceProvider for the named module or requirement"""
    435     if isinstance(moduleOrReq, Requirement):
--> 436         return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
    437     try:
    438         module = sys.modules[moduleOrReq]

~/.pyenv/versions/3.6.1/envs/jupyter3_install/lib/python3.6/site-packages/pkg_resources/__init__.py in require(self, *requirements)
    979         included, even if they were already activated in this working set.
    980         """
--> 981         needed = self.resolve(parse_requirements(requirements))
    982 
    983         for dist in needed:

~/.pyenv/versions/3.6.1/envs/jupyter3_install/lib/python3.6/site-packages/pkg_resources/__init__.py in resolve(self, requirements, env, installer, replace_conflicting, extras)
    865                     if dist is None:
    866                         requirers = required_by.get(req, None)
--> 867                         raise DistributionNotFound(req, requirers)
    868                 to_activate.append(dist)
    869             if dist not in req:

DistributionNotFound: The 'numpy' distribution was not found and is required by the application

It turned out that by the time that the script in this gist was called, pkg_resources had already been imported (at least once) before.
Its documentation states that:

add_entry(entry)
Add a path item to the entries, finding any distributions on it. You should use this when you add additional items to sys.path and you want the global working_set to reflect the change. This method is also called by the WorkingSet() constructor during initialization.

This method uses find_distributions(entry,True) to find distributions corresponding to the path entry, and then add() them. entry is always appended to the entries attribute, even if it is already present, however. (This is because sys.path can contain the same value more than once, and the entries attribute should be able to reflect this.)

Thus, the solution that I found (not sure how "correct" it is...) is to:
Add import pkg_resources at the top, then add:

pkg_resources.working_set.add_entry(site_package)
pkg_resources.working_set.add_entry(lib_dynload)

along with the sys.path.insert statements at the end.

Hi,
When using this script, I found that it ignored packages that were installed in 'develop' mode via python setup.py develop or pip install -e .. It seems like when something is installed in develop mode, an egg-link is added to the site-packages which points to the active package directory. I'm not sure how this egg-link is resolved normally in python, but at some point that directory is directly added to sys.path. To mimic this behavior, I added the following anywhere after the variable site_packages is defined:

egg_links = [i for i in os.listdir(site_packages) if 'egg-link' in i]
     for egg in egg_links:
         with open(os.path.join(site_packages,egg), 'r') as fin:
            sys.path.insert(0, fin.readline().rstrip('\n'))

This simply looks for anything that has 'egg-link' in the name, then opens the file and adds the first line to the path. No idea if you could have an edge case in which there could be multiple paths or lines in the egg-link, but this works for my needs.