pietromarchesi/PythonJULIA2.md

## PythonJULIA2.md

      
    Raw
  

              PythonJULIA2.md
            
          
    Python environments on the JULIA system

There are different tools for managing virtual Python environments (see this answer on Stack Overflow for an overview). Here we give some instructions on how to get started using virtual environments on the JULIA system using venv (Python 3 support only) and virtualenvwrapper (an extension of the popular virtualenv, which can be used with both Python 2 and 3).
Recommended: managing environments using venv (Python 3 only)

venv is a package automatically shipped with Python 3.  This method is quite straightforward, but it does not support Python 2 (for that, you can look at the second method using virtualenvwrapper which works for both Python 2 and 3).
To begin, we find the Python installation that we want to use by doing
$ module avail python

Then we can load (for instance) Python 3.6 with
$ module load python/3.6.1

Now, while python will still point to /usr/bin/python,  python3 points to /cm/shared/global/opt/python/3.6.1/bin/python3 (which can be seen by doing which python3).
We can create an environment and activate as follows:
$(which python3) -m venv basic36
$ source basic361/bin/activate

We can check that python and python3 now (and whenever the environment is activated) point to independent Python executables copied from the system Python that we loaded at the beginning.
(basic36)$ which python
~/basic361/bin/python
(basic36)$ which python3
~/basic361/bin/python3
(basic36)$ python --version
Python 3.6.1
(basic36)$ python3 --version
Python 3.6.1
(basic36)$ which pip
~/basic361/bin/pip
(basic36)$ which pip3
~/basic361/bin/pip3

We can now start installing packages using pip install.
Managing environments with virtualenvwrapper

Again we find the Python installation by doing
$ module avail python

Then we can load Python 3.6 with
$ module load python/3.6.1

Now, python will still point to /usr/bin/python,  python3 points to /cm/shared/global/opt/python/3.6.1/bin/python3 (which can be seen by doing which python3). The same goes for pip, so to install virtualenvwrapper we need to use
$ pip3 install virtualenvwrapper

It may be the case that you need to specify the --user option:
$ pip3 install --user virtualenvwrapper

virtualenvwrapper, as the name suggests, is actually just a set of extensions of the popular virtualenv tool, so under the hood virtualenvwrapper is going to call virtualenv. It is a good idea to check whether virtualenv actually points to the virtualenv installation corresponding to the Python module we loaded. So which virtualenv should return
/cm/shared/global/opt/python/modules/3.6/virtualenv/15.1.0/bin/virtualenv

If that is not the case, loading the virtualenv module with module load virtualenv should fix the problem.
Now we can follow the instructions on the virtualenvwrapper docs. So we can do
$ export WORKON_HOME=~/Envs
$ mkdir -p $WORKON_HOME

Then, we need to source virtualenvwrapper.sh. But we need to find it first, as the installation will probably have put it in some other place than /usr/local/bin. The quickest way I found to figure out where it is is to do again
$ pip3 install virtualenvwrapper

The first line of the output will say something like
Requirement already satisfied: virtualenvwrapper in /gpfs/homeb/pcp0/pcp0135/.local/lib/python3.6/site-packages

Then I know that virtualenvwrapper.sh will be in /gpfs/homeb/pcp0/pcp0135/.local/bin
(of course pcp0135 will be substituted by a different user name). You can also use find but by default it doesn't look in hidden directories, so you have to be aware of that.  Now we can run
$ source /gpfs/homeb/pcp0/pcp0135/.local/bin/virtualenvwrapper.sh 

There's a good chance you will get the error message
virtualenvwrapper.sh: There was a problem running the initialization hooks.

If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenvwrapper has been installed for
VIRTUALENVWRAPPER_PYTHON=/usr/bin/python and that PATH is
set properly.

That is because the variable VIRTUALENVWRAPPER_PYTHON points to /usr/bin/python, whereas we actually installed virtualenvwrapper for a different Python. To fix this, it should be sufficient to do
export VIRTUALENVWRAPPER_PYTHON=$(which python3)

And then source again with
$ source /gpfs/homeb/pcp0/pcp0135/.local/bin/virtualenvwrapper.sh

Now we should be able to start creating environments as
$ mkvirtualenv basic36

We can check that everything is in order:
$ workon basic36
(basic36)$ which python
~/Envs/basic36/bin/python
(basic36)$ which python3
~/Envs/basic36/bin/python3
(basic36)$ python --version
Python 3.6.1
(basic36)$ python3 --version
Python 3.6.1
(basic36)$ which pip
~/Envs/basic36/bin/pip
(basic36)$ which pip3
~/Envs/basic36/bin/pip3

If everything looks good, we can start installing packages with pip!
Now, to avoid having to go through these configuration steps every time we log in, we can add a few lines to our login configurations, so that we can start creating and editing environments as soon as we log in. Since we are on an interactive login shell when we ssh into the system, we can add some commands to our ~/.profile file.
echo 'module load python/3.6.1' >> ~/.profile
echo 'module load virtualenv' >> ~/.profile
echo 'export WORKON_HOME=~/Envs' >> ~/.profile 
echo 'export VIRTUALENVWRAPPER_PYTHON=$(which python3)' >> ~/.profile 
echo 'source /gpfs/homeb/pcp0/pcp0135/.local/bin/virtualenvwrapper.sh' >> ~/.profile