Skip to content

Instantly share code, notes, and snippets.

@nmukerje
Last active May 10, 2018 22:17
Show Gist options
  • Save nmukerje/3e216e03edb82c1b69d41aa4ec46cedd to your computer and use it in GitHub Desktop.
Save nmukerje/3e216e03edb82c1b69d41aa4ec46cedd to your computer and use it in GitHub Desktop.
Python Dependency Zip creation
# Install pip and virtualend if not already installed
$>sudo easy_install pip
$>sudo pip install virtualen
$>sudo pip install virtualenvwrapper
Add to .bash/profile:
# set where virutal environments will live
export WORKON_HOME=$HOME/.virtualenvs
# ensure all new environments are isolated from the site-packages directory
export VIRTUALENVWRAPPER_VIRTUALENV_ARGS='--no-site-packages'
# use the same directory for virtualenvs as virtualenvwrapper
export PIP_VIRTUALENV_BASE=$WORKON_HOME
# makes pip detect an active virtualenv and install to it
export PIP_RESPECT_VIRTUALENV=true
if [[ -r /usr/local/bin/virtualenvwrapper.sh ]]; then
source /usr/local/bin/virtualenvwrapper.sh
else
echo "WARNING: Can't find virtualenvwrapper.sh"
fi
$>mkvirtualenv testenv
(testenv)$>pip install requests
(testenv)$>pip freeze > requirements.txt
(testenv)$>deactivate testenv
(testenv)$>workon testenv
$>zip -r ~/deps.zip ~/.virtualenvs/testenv/lib/python2.7/dist-packages
$>pyspark --py-files deps.zip file.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment