Skip to content

Instantly share code, notes, and snippets.

@ax3l
Last active Oct 5, 2016
Embed
What would you like to do?
King of my own Python

How to be a King of your own Python

Besides best intentions, pre-installed and not correctly (cross-)compiled modules are painful when living on the cutting-dev-edge. Let's start from scratch! :)

The following intro is necessary for a cluster environment where every binary needs to be cross-compiled and $HOME is not mounted on the compute nodes. Furthermore, cross-dependencies and ABI incompatibilities force you to build the full software stack with the same compiler.

Environment Vars

# just a reminder for myself to load specific compilers, you can skip that
. ~/picongpu.profile

# a minimal python that is working on compute nodes, else compile one
module load python

# --user writes there
export PYTHONUSERBASE=$MEMBERWORK/$proj

# a weird cache that usually sits in $HOME
export PYTHON_EGG_CACHE=$PYTHONUSERBASE/.python-eggs
# where the actual binaries (and some messy pre-installed libs) sit
# -> must be mounted on the compute nodes!
#export PYTHONHOME=$(dirname $(which python))/..
export PYTHONHOME=/lustre/atlas/sw/xk6/python/2.7.9/sles11.3_gnu4.3.4/
# where our packages reside
export PYTHONPATH=$PYTHONUSERBASE/lib/python2.7/site-packages:$PYTHONPATH
# the place where ipython profiles are placed (default is ~/.ipython)
export IPYTHONDIR=$PYTHONUSERBASE
# find our tools
export PATH=$PYTHONUSERBASE/bin:$PATH

Own Setup Tools

Run only once:

## once
# $PYTHONUSERBASE/lib/python2.7/site-packages
#
wget https://bootstrap.pypa.io/ez_setup.py
python ez_setup.py --user

# in case that this does not work:
wget  https://bootstrap.pypa.io/get-pip.py
python get-pip.py --user
# you can still run
easy_install --user pip

Install Packages

Packages might need actual cross-compilation and also might depend on already cross-compiled libraries and tools.

So run this in an interactive session or submit script:

aprun -N1 `which python` `which easy_install` --user -U numpy
aprun -N1 `which python` `which easy_install` --user -U adios

Note: need to re-try with pip.

Test Packages

python -c "import numpy; print(numpy.__file__)"
aprun -N1 `which python` -c "import numpy; print(numpy.__file__)"
python -c "import adios; print(adios.__file__)"
aprun -N1 `which python` -c "import adios; print(adios.__file__)"

Execute Scripts

aprun -N1 `which python` ./checkCheckpoint.py
@ax3l

This comment has been minimized.

Copy link
Owner Author

@ax3l ax3l commented Jun 10, 2015

should add something about python virtual environments: http://docs.python-guide.org/en/latest/dev/virtualenvs/

@ax3l

This comment has been minimized.

Copy link
Owner Author

@ax3l ax3l commented Oct 5, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment