There are different tools for managing virtual Python environments (see this answer on Stack Overflow for an overview). Here we give some instructions on how to get started using virtual environments on the JULIA system using venv (Python 3 support only) and virtualenvwrapper (an extension of the popular virtualenv, which can be used with both Python 2 and 3).
venv is a package automatically shipped with Python 3. This method is quite straightforward, but it does not support Python 2 (for that, you can look at the second method using virtualenvwrapper which works for both Python 2 and 3).
To begin, we find the Python installation that we want to use by doing
$ module avail python
Then we can load (for instance) Python 3.6 with
$ module load python/3.6.1
Now, while python will still point to /usr/bin/python, python3 points to /cm/shared/global/opt/python/3.6.1/bin/python3 (which can be seen by doing which python3).
We can create an environment and activate as follows:
$(which python3) -m venv myenv
$ source myenv/bin/activate
Additionally, it can be a good idea to create environments with the --system-site-packages flag, in this case
$(which python3) -m venv --system-site-packages myenv
With this flag, the newly created Python environment will inherit packages that are available in the system site-packages directory (of to the Python module we loaded). This can be useful since system-level available Python packages are are in some cases compiled with the best parameters for the system, which makes them faster. For example, this is the case for numpy, which is linked against Intel's MKL library.
Once we have created the environment, we can check that python and python3 now (and whenever the environment is activated) point to independent Python executables in our environment directory copied from the system Python that we loaded at the beginning.
(myenv)$ which python
~/myenv/bin/python
(myenv)$ which python3
~/myenv/bin/python3
(myenv)$ python --version
Python 3.6.1
(myenv)$ python3 --version
Python 3.6.1
(myenv)$ which pip
~/myenv/bin/pip
(myenv)$ which pip3
~/myenv/bin/pip3
We can now start installing packages using pip install.
Again we find the Python installation by doing
$ module avail python
Then we can load Python 3.6 with
$ module load python/3.6.1
Now, while python will still point to /usr/bin/python, python3 points to /cm/shared/global/opt/python/3.6.1/bin/python3 (which can be seen by doing which python3). The same goes for pip, so to install virtualenvwrapper we need to use
$ pip3 install virtualenvwrapper
It may be the case that you need to specify the --user option:
$ pip3 install --user virtualenvwrapper
virtualenvwrapper, as the name suggests, is actually just a set of extensions of the popular virtualenv tool, so under the hood virtualenvwrapper is going to call virtualenv. It is a good idea to check whether virtualenv actually points to the virtualenv installation corresponding to the Python module we loaded. So which virtualenv should return
/cm/shared/global/opt/python/modules/3.6/virtualenv/15.1.0/bin/virtualenv
If that is not the case, loading the virtualenv module with module load virtualenv should fix the problem.
Now we can follow the setup instructions on the virtualenvwrapper docs. So we can do
$ export WORKON_HOME=~/Envs
$ mkdir -p $WORKON_HOME
Then, we need to source virtualenvwrapper.sh. But we need to find it first, as the installation will probably have put it in some other place than /usr/local/bin. The quickest way I found to figure out where it is is to do again
$ pip3 install virtualenvwrapper
The first line of the output will say something like
Requirement already satisfied: virtualenvwrapper in /gpfs/homeb/pcp0/pcp0135/.local/lib/python3.6/site-packages
Then I know that virtualenvwrapper.sh will be in /gpfs/homeb/pcp0/pcp0135/.local/bin
(of course pcp0135 will be substituted by a different user name). You can also use find but by default it doesn't look in hidden directories, so you have to be aware of that. Now we can run
$ source /gpfs/homeb/pcp0/pcp0135/.local/bin/virtualenvwrapper.sh
There's a good chance you will get the error message
virtualenvwrapper.sh: There was a problem running the initialization hooks.
If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenvwrapper has been installed for
VIRTUALENVWRAPPER_PYTHON=/usr/bin/python and that PATH is
set properly.
That is because the variable VIRTUALENVWRAPPER_PYTHON points to /usr/bin/python, whereas we actually installed virtualenvwrapper for a different Python. To fix this, it should be sufficient to do
export VIRTUALENVWRAPPER_PYTHON=$(which python3)
And then source again with
$ source /gpfs/homeb/pcp0/pcp0135/.local/bin/virtualenvwrapper.sh
Now we should be able to start creating environments as
$ mkvirtualenv myenv
Once again, for reasons explained in the first part, it may be a good idea to allow our environments to inherit packages from the global site-packages directory, by using the --system-site-packages:
$ mkvirtualenv --system-site-packages myenv
We can check that everything is in order:
$ workon myenv
(myenv)$ which python
~/Envs/myenv/bin/python
(myenv)$ which python3
~/Envs/myenv/bin/python3
(myenv)$ python --version
Python 3.6.1
(myenv)$ python3 --version
Python 3.6.1
(myenv)$ which pip
~/Envs/myenv/bin/pip
(myenv)$ which pip3
~/Envs/myenv/bin/pip3
If everything looks good, we can start installing packages with pip!
Now, to avoid having to go through these configuration steps every time we log in, we can add a few lines to our login configurations, so that we can start creating and editing environments as soon as we log in. Since we are on an interactive login shell when we ssh into the system, we can add some commands to our ~/.profile file.
echo 'module load python/3.6.1' >> ~/.profile
echo 'module load virtualenv' >> ~/.profile
echo 'export WORKON_HOME=~/Envs' >> ~/.profile
echo 'export VIRTUALENVWRAPPER_PYTHON=$(which python3)' >> ~/.profile
echo 'source /gpfs/homeb/pcp0/pcp0135/.local/bin/virtualenvwrapper.sh' >> ~/.profile