ax3l/KomoP.md

## KomoP.md

      
    Raw
  

              KomoP.md
            
          
    How to be a King of your own Python

Besides best intentions, pre-installed and not correctly (cross-)compiled modules are painful when living on the cutting-dev-edge.
Let's start from scratch! :)
The following intro is necessary for a cluster environment where every binary needs to be cross-compiled and $HOME is not mounted on the compute nodes. Furthermore, cross-dependencies and ABI incompatibilities force you to build the full software stack with the same compiler.
Environment Vars

# just a reminder for myself to load specific compilers, you can skip that
. ~/picongpu.profile

# a minimal python that is working on compute nodes, else compile one
module load python

# --user writes there
export PYTHONUSERBASE=$MEMBERWORK/$proj

# a weird cache that usually sits in $HOME
export PYTHON_EGG_CACHE=$PYTHONUSERBASE/.python-eggs
# where the actual binaries (and some messy pre-installed libs) sit
# -> must be mounted on the compute nodes!
#export PYTHONHOME=$(dirname $(which python))/..
export PYTHONHOME=/lustre/atlas/sw/xk6/python/2.7.9/sles11.3_gnu4.3.4/
# where our packages reside
export PYTHONPATH=$PYTHONUSERBASE/lib/python2.7/site-packages:$PYTHONPATH
# the place where ipython profiles are placed (default is ~/.ipython)
export IPYTHONDIR=$PYTHONUSERBASE
# find our tools
export PATH=$PYTHONUSERBASE/bin:$PATH
Own Setup Tools

Run only once:
## once
# $PYTHONUSERBASE/lib/python2.7/site-packages
#
wget https://bootstrap.pypa.io/ez_setup.py
python ez_setup.py --user

# in case that this does not work:
wget  https://bootstrap.pypa.io/get-pip.py
python get-pip.py --user
# you can still run
easy_install --user pip
Install Packages

Packages might need actual cross-compilation and also might depend on already cross-compiled libraries and tools.
So run this in an interactive session or submit script:
aprun -N1 `which python` `which easy_install` --user -U numpy
aprun -N1 `which python` `which easy_install` --user -U adios
Note: need to re-try with pip.
Test Packages

python -c "import numpy; print(numpy.__file__)"
aprun -N1 `which python` -c "import numpy; print(numpy.__file__)"
python -c "import adios; print(adios.__file__)"
aprun -N1 `which python` -c "import adios; print(adios.__file__)"
Execute Scripts

aprun -N1 `which python` ./checkCheckpoint.py