Geoyi/python_environment_setup.md

## python_environment_setup.md

      
    Raw
  

              python_environment_setup.md
            
          
    Overview

When you're working on multiple coding projects, you might want a couple different version of Python and/or modules installed. That way you can keep each project in its own sandbox instead of trying to juggle multiple projects (each with different dependencies) on your system's version of Python. This intermediate guide covers one way to handle multiple Python versions and Python environments on your own (i.e., without a package manager like conda). See the Using the workflow section to view the end result.
Use cases


Working on 2+ projects that each have their own dependencies; e.g., a Python 2.7 project and a Python 3.6 project, or developing a module that needs to work across multiple versions of Python. It's not reasonable to uninstall/reinstall modules every time you want to switch environments.
If you want to execute code on the cloud, you can set up a Python environment that mirrors the relevant cloud instance. For example, one of Amazon's main EC2 deep learning instances runs Python 3.4, and you may hit obstacles if you use code developed on your machine with Python 3.6.
You might have some working Python code and want to make sure everything stays frozen so that it'll still work in the future. Without virtual environments, upgrading Python modules could unintentionally break that year-old project. Having to go back and determine the correct version for each dependency would be a huge pain.

This guide shows how to solve these issues with pyenv and virtualenv (along with virtualenvwrapper). It shows how to obtain lower-level control of your development environment (compared to Anaconda/conda, for example) without too much technical complication. This is intended for MacOS, but all the tools work on Unix-like systems -- you'll just have to make use of apt-get instead of brew and detour through the original installation guides in some spots.
For comparison to Anaconda, see note below
Overview of the tools and setup


pyenv: Short for "Python environment." This manages which version of Python is visible to your computer (and temporarily hides any others). With pyenv, you can install multiple versions of Python and quickly switch between the "activated" version (i.e., the version your computer will use to execute code). Here, we'll switch the version of Python we're using with virtualenvwrapper's workon command (as described later).
Installation/use: brew install pyenv on Mac. Also install pyenv-virtualenv like brew install pyenv-virtualenv, which we'll need later. See the docs for installation via git clone on other other systems. Then you can install new Python versions, list installed versions, and set the Python version in the terminal like:
pyenv install 3.6.3  # Install Python version
pyenv install 3.4.0
pyenv versions       # List Python versions
pyenv global 3.6.3   # Set your system's Python version

Technical details: Every time you execute a Python script or use pip, pyenv works in the background to intercept that command and send it to the Python environment that is currently activated. It does this using shims on the PATH environment variable, which allow Python-related commands to be dynamically rerouted.


virtualenv: Short for "virtual environment." This manages separate directories for the modules you install (e.g., with pip). That way, each virtual environment can have it's own set of installed modules  that is walled off from every other virtual environment so they don't interfere with each other. Like with pyenv, we'll  switch virtual environments with virtualenvwrapper's workon command (as described later).
Installation:
pip install virtualenv in your terminal
Use: It's possible to use virtualenv directly as (as described here), but we'll use virtualenvwrapper instead.
Technical details: virtualenv keeps each environment (and its installed modules) in separate folders; therefore, each is like a silo that doesn't interact with any other virtual environment. Usually, the exact file location is defined by the user, but we can use virtualenvwrapper to instead handle this for us.


virtualenvwrapper. This helps pyenv and virtualenv gel like PB&J. With it, you can effectively switch between a single environment that has both the Python version and virtual environment wrapped in one bundle. Make sure pyenv and virtualenv are installed before you install this wrapper.
Installation: pip install virtualenvwrapper and then brew install pyenv-virtualenvwrapper to extend pyenv. Then you'll need to do some one-time setup; in your .bashrc/.bash_profile, add the following:
# Setup virtualenv home
export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh

# Tell pyenv-virtualenvwrapper to use pyenv when creating new Python environments
export PYENV_VIRTUALENVWRAPPER_PREFER_PYVENV="true"

Make sure that the directory you define for WORKON_HOME actually exists, and check that tthen either restart your terminal or source the relevant configuration (i.e., source ~/.bashrc or source ~/.bash_profile). You can find the full virtualenvwrapper installation instructions here.


Using the workflow

We're all ready to use this in the terminal! First, we'll set the Python environment with pyenv, and then make a couple virtual environments with virtualenvwrapper. Then we'll use the workon command to switch between them.
pyenv global 3.6.3           # Set your system's Python version with pyenv
mkvirtualenv my_project_py3  # Create a new virtual environment using virtualenvwrapper; it'll be tied to Python 3.6.3
pip install numpy scipy      # Install the packages you want in this environment

pyenv global 2.7.0       # Set your system's Python version with pyenv
mkvirtualenv webdev_py2  # Create/switch to a new virtual environment; it'll use Python 2.7.0
pip install flask boto

workon                 # List the environments available
workon my_project_py3  # Use virtualenvwrapper to switch back to my_project_py3

Troubleshooting


If you're on MacOS and have issues with pyenv like:
zipimport.ZipImportError: can't decompress data; zlib not available
make: *** [install] Error 1
pyenv: version `3.5.0' is not installed

Make sure you have newest version of XCode CLI installed by running: xcode-select --install


If you have file not found issues with pyenv's virtualenvwrapper.sh, you should be able to check where it lives with pyenv which virtualenvwrapper.sh


Other notes


Anaconda does have functionality to handle some of the problems outlined above. Generally, it provides a lower bar for entry to Python development because the Anaconda software distribution contains both the conda package manager as well as many useful python modules, which is great for new Python users. Anaconda is also a good choice for Windows users as getting all Python packages to play nicely together is a challenge on Windows. However, there are some downsides to Anaconda/conda:

Any package that can't be installed via the conda package manager must be installed with pip or some other method; at that point, you're managing two install streams. For more advanced development, this can get messy.
You aren't always going to be using the most up-to-date version of modules because Continuum must repackage each one into their own system before calling conda update will actually provide the newest version of that module's code.
Different versions of Python will not always have the latest module updates -- Continuum focuses its resources on Python 2.7 and 3.6 right now, and you're relying on their team to incorporate all package updates to those version as well as the less-popular versions (like 3.4 and 3.5).
Because the Anaconda software distribution is a large self-contained install, it'll install many packages that you might not need. Miniconda solves this to some degree as it only contains the conda package manager and its dependencies.

That aside, a good discussion of Anaconda's benefits, and some counter-arguements are here.


Virtual environment prefix on source prompt
If you want your command prompt to show the virtual environment you're currently working with, add this to you .bashrc/.bash_profile:
# Prefix source prompt with virtualenvwrapper environment
if which pyenv > /dev/null; then eval "$(pyenv init -)"; fi
eval "$(pyenv virtualenv-init -)"

Your terminal command prompt will now look something like (my_project_py3) Mark@Marks-MBP:~/Builds/ $


Directory scheme
This is my own personal preference, but when setting up my Python environment, I also tend to store modules I'm developing in a Builds directory (i.e., /Users/wronk/Builds). Similarly, I put data in /Users/wronk/Data. Then, I'll define an environment variable in my .bashrc/.bash_profile (e.g., named BUILDS_DIR and DATA_DIR) so that writing scripts/Python code is more agnostic of the exact machine I'm using.
For example, any shell scripts can traverse directories from the BUILDS_DIR environment variable instead of a hard-coded path, and I'll use something like my_data_dir = os.environ['DATA_DIR'] in Python code so it'll work on any machine that mirrors my directory scheme. That tends to look cleaner in code and is easier for getting the same code running locally and on the cloud (or another computer).