PythonJULIA.md

Setting up a Python environment on JULIA with virtualenvwrapper

I find the system Python I want to use by doing

$ module avail python

Then we can load Python 3.6 with

$ module load python/3.6.1

Now, while python will still point to /usr/bin/python, python3 points to /cm/shared/global/opt/python/3.6.1/bin/python3 (which can be seen by doing which python3).

The same goes for pip, so to install virtualenvwrapper we need to use

$ pip3 install virtualenvwrapper

Now we can follow the instructions on the virtualenvwrapper docs. So we can do

$ export WORKON_HOME=~/Envs
$ mkdir -p $WORKON_HOME

Then, we need to source virtualenvwrapper.sh. But we need to find it first, as the installation will probably have put it in some other place than /usr/local/bin. The quickest way I found to figure out where it is is to do again

$ pip3 install virtualenvwrapper

The first line of the output will say something like

Requirement already satisfied: virtualenvwrapper in /gpfs/homeb/pcp0/pcp0135/.local/lib/python3.6/site-packages

Then I know that virtualenvwrapper.sh will be in /gpfs/homeb/pcp0/pcp0135/.local/bin (of course pcp0135 will be substituted by a different user name). You can also use find but by default it doesn't look in hidden directories, so you have to be aware of that. Now we can run

$ source /gpfs/homeb/pcp0/pcp0135/.local/bin/virtualenvwrapper.sh 

There's a good chance you will get the error message

virtualenvwrapper.sh: There was a problem running the initialization hooks.

If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenvwrapper has been installed for
VIRTUALENVWRAPPER_PYTHON=/usr/bin/python and that PATH is
set properly.

That is because the variable VIRTUALENVWRAPPER_PYTHON points to /usr/bin/python, whereas we actually installed virtualenvwrapper for a different Python. To fix this, it should be sufficient to do

export VIRTUALENVWRAPPER_PYTHON=$(which python3)

And then source again with

$ source /gpfs/homeb/pcp0/pcp0135/.local/bin/virtualenvwrapper.sh

Now we should be able to start creating environments as

$ mkvirtualenv myenvironment

Now, to avoid having to go through these steps every time we log in, we can add a few lines to our login configurations, so that we can start creating and editing environments as soon as we log in. Since we are on an interactive login shell when we ssh into the system, we can add some commands to our ~/.profile file.

echo 'module load python/3.6.1' >> ~/.profile
echo 'export WORKON_HOME=~/Envs' >> ~/.profile 
echo 'export VIRTUALENVWRAPPER_PYTHON=$(which python3)' >> ~/.profile 
echo 'source /gpfs/homeb/pcp0/pcp0135/.local/bin/virtualenvwrapper.sh' >> ~/.profile

Up to this point I thought everything was working fine but if now I check

$ workon basic36
(basic36)$ which python
~/Envs/basic36/bin/python
(basic36)$ which python3
/cm/shared/global/opt/python/3.6.1/bin/python3
(basic36)$ which pip
~/Envs/basic36/bin/pip
(basic36)$ which pip3
/cm/shared/global/opt/python/3.6.1/bin/pip3

Basically, instead of having created a new Python executable in ~/Envs/basic36, our python3 and pip3 still point to the system Python, which means that we won't be able to install anything there as we do not have permission. I tried specifying the Python directly during the generation of the environment,

$ which python3
/cm/shared/global/opt/python/3.6.1/bin/python3
$ mkvirtualenv basic361 --python=$(which python3)
Running virtualenv with interpreter /cm/shared/global/opt/python/3.6.1/bin/python3
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 8, in <module>
    import base64
  File "/cm/shared/global/opt/python/3.6.1/lib/python3.6/base64.py", line 9, in <module>
    import re
  File "/cm/shared/global/opt/python/3.6.1/lib/python3.6/re.py", line 142, in <module>
    class RegexFlag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'

Or directly with virtualenv, as virtualenv ./basic361 --python=$(which python3), but that gives the same error. Dead end! (for now..)

Alternative solution using venv (Python 3 only)

venv is a package automatically shipped with Python 3 (see this answer).

We can create an environment and activate as follows:

$(which python3) -m venv basic361
$ source basic361/bin/activate

Then to make sure that everything is in order:

(basic361)$ which pip
~/basic361/bin/pip
(basic361)$ which pip3
~/basic361/bin/pip3
(basic361)$ which python
~/basic361/bin/python
(basic361)$ which python3
~/basic361/bin/python3
(basic361)$ python --version
Python 3.6.1
(basic361)$ python3 --version
Python 3.6.1

We can now start installing packages using pip install.

---