make-local-package-index.py

#!/usr/bin/python
"""
Build a local index of all the packages which pip has previously downloaded and
cached so you can install any of them without requiring network access.

Might be useful for creating new Virtualenvs while developing on a plane.

Usage:

    * Set the environment var PIP_DOWNLOAD_CACHE (e.g., in your `.profile`)
      (This is a sensible thing to do anyway.)
    * Install basketweaver: `pip install basketweaver`
    * Carry on pip installing stuff as normal
    ...
    * Lose your Internet connection
    * Run: ./make-local-package-index.py
    * pip install -i "file:///$PIP_DOWNLOAD_CACHE/index" somepackage
    * Prophet!

Pip has a download cache which you can enable by setting the environment
variable `PIP_DOWNLOAD_CACHE`. (`~/.pip/cache` would be a sensible choice.)
This causes pip to save a copy of any tarballs it downloads which it will then
use in future instead of re-downloading. However, pip still needs access to
PyPI in order to know which is the latest version of each package and where it
should be fetched from.

Basketweaver is a handy tool which can take a directory full of eggs or
tarballs and build an index of them (i.e., a set of html files). Unfortunately,
you can't build a local index directly out of your download cache because the
cached files have URL-encoded names and so pip rejects them as incorrectly
named. In addition, you can't just symlink the files under different names
because pip resolves symlinks before checking the names (grrr). Hence this
little script which just copies the cached files into a new directory, renames
them, and then runs `basketweaver.makeindex` on the result.
"""

import errno
import os
import urlparse
import shutil

from basketweaver import makeindex

PACKAGE_DIR = 'package_copies'
try:
    pip_cache = os.environ['PIP_DOWNLOAD_CACHE']
except KeyError:
    raise EnvironmentError("You need to have enabled pip's download cache "
                           "before you can use this script.\n")
os.chdir(pip_cache)
try:
    os.mkdir(PACKAGE_DIR)
except OSError as e:
    if e.errno != errno.EEXIST:
        raise
packages = []
for fname in os.listdir('.'):
    if fname.endswith('.content-type') or os.path.isdir(fname):
        continue
    package_name = os.path.basename(urlparse.unquote(fname))
    package_file = os.path.join(PACKAGE_DIR, package_name)
    if not os.path.exists(package_file):
        shutil.copy(fname, package_file)
    packages.append(package_file)
makeindex.main(packages)