Skip to content

Instantly share code, notes, and snippets.

Last active Oct 5, 2016
What would you like to do?
King of my own Python

How to be a King of your own Python

Besides best intentions, pre-installed and not correctly (cross-)compiled modules are painful when living on the cutting-dev-edge. Let's start from scratch! :)

The following intro is necessary for a cluster environment where every binary needs to be cross-compiled and $HOME is not mounted on the compute nodes. Furthermore, cross-dependencies and ABI incompatibilities force you to build the full software stack with the same compiler.

Environment Vars

# just a reminder for myself to load specific compilers, you can skip that
. ~/picongpu.profile

# a minimal python that is working on compute nodes, else compile one
module load python

# --user writes there

# a weird cache that usually sits in $HOME
# where the actual binaries (and some messy pre-installed libs) sit
# -> must be mounted on the compute nodes!
#export PYTHONHOME=$(dirname $(which python))/..
export PYTHONHOME=/lustre/atlas/sw/xk6/python/2.7.9/sles11.3_gnu4.3.4/
# where our packages reside
export PYTHONPATH=$PYTHONUSERBASE/lib/python2.7/site-packages:$PYTHONPATH
# the place where ipython profiles are placed (default is ~/.ipython)
# find our tools

Own Setup Tools

Run only once:

## once
# $PYTHONUSERBASE/lib/python2.7/site-packages
python --user

# in case that this does not work:
python --user
# you can still run
easy_install --user pip

Install Packages

Packages might need actual cross-compilation and also might depend on already cross-compiled libraries and tools.

So run this in an interactive session or submit script:

aprun -N1 `which python` `which easy_install` --user -U numpy
aprun -N1 `which python` `which easy_install` --user -U adios

Note: need to re-try with pip.

Test Packages

python -c "import numpy; print(numpy.__file__)"
aprun -N1 `which python` -c "import numpy; print(numpy.__file__)"
python -c "import adios; print(adios.__file__)"
aprun -N1 `which python` -c "import adios; print(adios.__file__)"

Execute Scripts

aprun -N1 `which python` ./

This comment has been minimized.

Copy link
Owner Author

@ax3l ax3l commented Jun 10, 2015

should add something about python virtual environments:


This comment has been minimized.

Copy link
Owner Author

@ax3l ax3l commented Oct 5, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment