alchem0x2A/gpaw_config_euler.md

## gpaw_config_euler.md

      
    Raw
  

              gpaw_config_euler.md
            
          
    Preparation


Compiler: python/3.1.1 (shipped) gcc-6.3.0, open_mpi/3.0.0, libxc/4.3.0, fftw/3.3.3, openblas/0.2.13_seq, scalapack
tar for numpy scipy matplotlib

Working on virtualenv

python -m venv gpaw
source ~/.virtualenv/gpaw/bin/activate
Alternatively, if you want to use the shipped =numpy= and =scipy= versions,
then install the virtualenv with system-package enabled like
python -m venv --system-site-packages gpaw
Now we are working on a clean virtualenv with all packages to be installed
Compile numpy with openblas

/Skip these steps if you choose to use the virtualenv with system-site-packages/
Download the numpy source from git repo or from the PyPI:
pip download --no-binary=:all: numpy
Create site.cfg to cope with compiler and library information
cd 'path/to/numpy'
cp site.cfg.example site.cfg
Uncomment and modify the section with openblas
[openblas]                                                                                                                   
libraries = openblas
library_dirs = /cluster/apps/openblas/0.2.13_seq/x86_64/gcc_5.2.0/lib
include_dirs = /cluster/apps/openblas/0.2.13_seq/x86_64/gcc_5.2.0/include
runtime_library_dirs = /cluster/apps/openblas/0.2.13_seq/x86_64/gcc_5.2.0/lib

install numpy
python setup.py build && python setupy.py install
check numpy configuration
python -c "import numpy; numpy.__config__.show()"
Check if openblas_info and openblas_lapack_info are correct.
Compile scipy

More straightforward than numpy. The OpenBLAS environment can be automatically detected if numpy corrected compiled.
cd 'path/to/numpy'
python setup.py build
python setup.py install
check numpy configuration
python -c "import scipy; scipy.__config__.show()"
Alternative install with pip

Define the flag OPENBLAS before pip install.
In some cases one or more packages are not compiled against OpenBLAS so you may with recompile manually.
OPENBLAS=$GOTOBLAS2/lib python -m pip install --upgrade --no-binary numpy
OPENBLAS=$GOTOBLAS2/lib python -m pip install --upgrade --no-binary scipy
OPENBLAS=$GOTOBLAS2/lib python -m pip install --upgrade matplotlib
Other packages

matplotlib ase
(Alternative) compile libxc to a recent version

Follow the instructions on  http://www.tddft.org/programs/libxc/installation/ for the installation.
Optionally install the python bindings to the libxc (from the root dir of libxc)
python -m pip install .
Install libvdwxc

Follow the instructions on the website and specify the root path.
MPI compiler is compulsory in some cases as the build system cannot detect the mpicc automatically
./configure --with-mpi CC="mpicc" FC="mpif90" --prefix="$VIRTUAL_ENV/dependency/libvdwxc"
make && make install
Install ELPA

Download the ELPA package (such as http://elpa.mpcdf.mpg.de/html/Releases/2019.05.001/elpa-2019.05.001.tar.gz)
and check the =INSTALL.md= inside.
For successfull installation of ELPA, use both GNU version of =CC= and =FC=, and disable =AVX2= support:
configure FC=mpif90 CC=mpicc --disable-avx2 --prefix="$VIRTUAL_ENV/dependency/elpa"
make && make install
Test installations

Requires nose and pytest
python -c "import numpy; numpy.test()"
python -c "import scipy; scipy.test()"
ase test
Install GPAW

Download the newest version of GPAW (currently 19.8.1)
Modify the customize.py, enable the nessary parts (=openblas=, =libxc=, =libvdwxc=, =elpa=)
Alternatively you may need to add the lib64 with libpython3.xm.so into LIBRARY_PATH (not LD_LIBRARY_PATH)
A sample configuration file looks like:
import os, os.path
from os.path import expandvars
libraries = ["openblas"]
library_dirs += [expandvars("$MODULE_openblas_ROOT_DIR/lib")]
#library_dirs = [""]

# flake8: noqa

compiler = 'gcc'
mpicompiler = 'mpicc'  # use None if you don't want to build a gpaw-python
mpilinker = 'mpicc'
platform_id = 'euler-centos'

# FFTW3:
fftw = True
if fftw:
    libraries += ['fftw3']
    library_dirs += [expandvars("$FFTW3_ROOT_DIR/lib64")]

# ScaLAPACK (version 2.0.1+ required):
scalapack = True
if scalapack:
    libraries += ['scalapack']
    library_dirs += ["$MODULE_scalapack_ROOT_DIR/lib"]
    define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
    define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]

# Build MPI-interface into _gpaw.so:
parallel_python = True
if not parallel_python:
    compiler = 'mpicc'
    define_macros += [('PARALLEL', '1')]
    mpicompiler = None

# Use Elpa (requires ScaLAPACK and Elpa API 20171201):
if 1:
    elpa = True
    elpadir = expandvars("$VIRTUAL_ENV/dependency/elpa")
    libraries += ['elpa']
    library_dirs += ['{}/lib'.format(elpadir)]
    extra_link_args += ['-Wl,-rpath={}/lib'.format(elpadir)]
    include_dirs += ['{}/include/elpa-2019.05.001'.format(elpadir)]

# LibXC:
# In order to link libxc installed in a non-standard location
# (e.g.: configure --prefix=/home/user/libxc-2.0.1-1), use:

# - dynamic linking (requires rpath or setting LD_LIBRARY_PATH at runtime):
if 1:
    xc = expandvars('$MODULE_libxc_ROOT_DIR')
    include_dirs += [xc + 'include']
    library_dirs += [xc + 'lib']
    # You can use rpath to avoid changing LD_LIBRARY_PATH:
    extra_link_args += ['-Wl,-rpath={xc}/lib'.format(xc=xc)]
    if 'xc' not in libraries:
        libraries.append('xc')


# libvdwxc:
if 1:
    libvdwxc = True
    path = expandvars('$VIRTUAL_ENV/dependency/libvdwxc-build')
    extra_link_args += ['-Wl,-rpath=%s/lib' % path]
    library_dirs += ['%s/lib' % path]
    include_dirs += ['%s/include' % path]
    libraries += ['vdwxc']
Then install with normal install procedure
python setup build && python setup install
If installation succeeded but some errors occur, use the method below to delete the files generated (pip uninstall incapable)
https://gist.github.com/lovaulonze/4e31d83091a9d9543c0cfd297a1ded07
Test GPAW

Run the tests like on the website
gpaw info #test basic settings, only scalapack flag is no
gpaw test -j 8 #depends on the system
gpaw-python -m gpaw info #scalapack flag should also be yes
mpiexec -n 24 gpaw-python -m gpaw test #may need to submit to the job queue
Make the module file

mkdir -p ~/modules/gpaw/
emacs -nw ~/modules/gpaw/gcc_5.2.0
Then edit the contents in the "gcc_5.2.0" modulefile:
#%Module1.0#####################################################################
#
#STATUS
#stat:moddep:gcc/5.2.0 open_mpi/1.10.0 openblas/0.2.13_seq scalapack/2.0.2 fftw/3.3.4 libxc/3.0.0
#stat:modauto:
#END STATUS

proc ModulesHelp { } {
    global helpmsg
    puts stderr "\t$helpmsg\n"
}

set version gcc_5.2.0
set curmod  [module-info name]

set topdir  /cluster/home/ttian/.virtualenvs/gpaw/bin/

# check if requirements are met
set envguess  [exec /cluster/apps/scripts/guess_compiler.pl --path=compiler=__CC__\ open_mpi=__MPI__]
set need_cb   "gcc"
set need_cc   "5.2.0"
set need_ompi "1.10.0"


#   if { [string compare $envguess "compiler=${need_cb}_$need_cc open_mpi=$need_ompi"] != 0 } {
#     puts stderr "$curmod error: this module requires $need_cb/$need_cc and open_mpi/$need_ompi to be loaded."
#      puts stderr "hint: run 'module purge ; module load $need_cb/$need_cc open_mpi/$need_ompi $curmod' to resolve this problem."
#      break
#   }
if { [string compare $envguess "compiler=${need_cb}_$need_cc open_mpi=$need_ompi"] != 0 } {
        module load ${need_cb}/$need_cc
        module load open_mpi/$need_ompi
	module load openblas/0.2.13_seq
	module load fftw/3.3.4
	module load scalapack/2.0.2
	module load libxc/3.0.0
	module load python/3.6.0
}

# virtualenv
if { [module-info mode load] || [module-info mode switch2] } {
    puts stdout "source /cluster/home/ttian/.virtualenvs/gpaw/bin/activate;"
} elseif { [module-info mode remove] && ![module-info mode switch3] } {
    puts stdout "deactivate;"
}


# not reached on error

Finally add the following line in .bashrc or .bash_profile to enable local modules:
module use $HOME/modules