michaelchughes/README.md

## README.md

      
    Raw
  

              README.md
            
          
    Goal

Install working tensorflow or pytorch via standard conda environment workflow.
Basic Setup : Install pytorch in a fresh conda environment

The recommended conda-based install process works smoothly:
$ # Create a fresh environment
$ conda create --name py37_torch python=3.7 --yes

$ # Activate new environment
$ source activate py37_torch

$ # Install tensorflow
$ conda install tensorflow --yes

$ # Install pytorch 
$ conda install pytorch-cpu torchvision-cpu -c pytorch --yes
Roadblock

The gotcha is that when we try to then use the package we just installed, we get an GLIBC error like this:
$ python -c "import torch"

ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by .../site-packages/torch/lib/libshm.so)

Badness! Clearly, the current computing system doesn't have a recent-enough GLIBC. However, if this is a cluster computing system, you often don't have root access and can't easily upgrade the GLIBC.
Step 1: Install recent copies of glibc and libc++ in userspace

Credit: StackOverflow answer by Theo T.
Step 1a: (NEW FOR PYTHON 3.7) Download and unpack some pre-compiled GLIBC shared libraries

This is for Python 3.7 (works for 3.6 too!)
(See an older list for Python 2.7 at bottom of this doc).
$ # Make a folder within the environment to hold useful things
$ mkdir -p /path/to/conda/envs/py37_torch1.0/custom_libs/
$ cd /path/to/conda/envs/py37_torch1.0/custom_libs/

$ # Get libc files (URL verified by MCH on 2019/08/21)
$ wget http://mirrors.kernel.org/ubuntu/pool/main/g/glibc/libc6_2.23-0ubuntu10_amd64.deb
$ wget http://mirrors.kernel.org/ubuntu/pool/main/g/glibc/libc6-dev_2.23-0ubuntu10_amd64.deb

$ # Unpack files into current directory (will create usr/ and lib/ and lib64/ folders)
$ ar p libc6_2.23-0ubuntu10_amd64.deb data.tar.xz | tar xvJ
$ ar p libc6-dev_2.23-0ubuntu10_amd64.deb data.tar.xz | tar xvJ
What have we accomplished? You should have some new folders in your current directory, labeld usr/ and lib/ and lib64/.
We can verify that before, we had an OLD libc, and now we have a shiny new one!
Check the OLD location of libc.so.6

$ strings /lib/libc.so.6 | grep GLIBC_2. | tail -n3
GLIBC_2.10
GLIBC_2.11
GLIBC_2.12
NEW version of libc.so.6 in working directory

$ strings lib/x86_64-linux-gnu/libc.so.6 | grep GLIBC_2 | tail -n3
GLIBC_2.18
GLIBC_2.22
GLIBC_2.23
Step 1b: Download and unpack some pre-compiled LIBSTDC++ shared libraries

# Get libstdc++ (URL verified by MCH on 2019/02/18)
wget ftp://195.220.108.108/linux/mageia/distrib/4/x86_64/media/core/updates/libstdc++6-4.8.2-3.2.mga4.x86_64.rpm

# Alternative URL:
# wget http://ftp.riken.jp/Linux/scientific/6.0/x86_64/os/Packages/libstdc++-4.4.4-13.el6.x86_64.rpm

# Unpack into current directory (will add content to lib/ and lib64/ folders)
rpm2cpio libstdc++6-4.8.2-3.2.mga4.x86_64.rpm | cpio -idmv
Step 2: Use patchelf to make your python install use these userspace libraries instead of the system defaults

Credit: Stackoverview answer by Evalds Urtans
Step 2a: Install patchelf into current conda env

# Be sure correct environment is active
$ source activate py37_torch

# Install patchelf
(py37_torch) $ conda install patchelf -c conda-forge --yes

Step 2b: Use attached script to alter the conda env's python executable to use the custom GLIBC libraries

(py37_torch) $ bash rewrite_python_exe_glibc_with_patchelf.sh

-- DEPRECATED --

Step 1a: (OLD FOR PYTHON 2.7) Download and unpack some pre-compiled GLIBC shared libraries

$ # Make a folder within the environment to hold useful things
$ mkdir -p /path/to/conda/envs/py27_torch1.0/custom_libs/
$ cd /path/to/conda/envs/py27_torch1.0/custom_libs/

$ # Get libc files (URL verified by MCH on 2019/02/18)
$ wget https://launchpadlibrarian.net/137699828/libc6_2.17-0ubuntu5_amd64.deb
$ wget https://launchpadlibrarian.net/137699829/libc6-dev_2.17-0ubuntu5_amd64.deb

$ # Unpack files into current directory (will create usr/ and lib/ and lib64/ folders)
$ ar p libc6_2.17-0ubuntu5_amd64.deb data.tar.gz | tar zx
$ ar p libc6-dev_2.17-0ubuntu5_amd64.deb data.tar.gz | tar zx

  
## rewrite_python_exe_glibc_with_patchelf.sh
#!/usr/env bash

# TODO edit this line to specify location of new glibc
export GLIBC_PATH=/cluster/tufts/hugheslab/miniconda2/envs/ape/custom_libs/
export GLIBC_LD_PATH=$GLIBC_PATH/lib/x86_64-linux-gnu/ld-2.23.so

if [[ ! -f $GLIBC_LD_PATH ]]; then
    echo "ERROR: Provided GLIBC_LD_PATH not valid"
    exit
fi

echo "OVERWRITING PYTHON EXECUTABLE:"
python_exe=`which python`
echo $python_exe

IS_CONDA_ENV=`python -c "print('$python_exe'.count('/envs/') > 0)"`
echo "IS_CONDA_ENV: $IS_CONDA_ENV"

if [[ $IS_CONDA_ENV -ne 'True' ]]; then
    echo "ERROR: Current python executable not in conda env. Will not alter to avoid problems."
    exit
fi

CONDA_ENV_LIB=`python -c "print('$python_exe'.replace('/bin/python', '/lib'))"`

echo "CREATING BACKUP PYTHON"
python_tmp_exe=`python -c "print('$python_exe'.replace('python', 'python_backup'))"`
cp $python_exe $python_tmp_exe
echo "$python_tmp_exe"

rpath=$GLIBC_PATH/lib/x86_64-linux-gnu:$CONDA_ENV_LIB:/usr/lib64:/lib64:/lib

echo "CALLING PATCHELF on 'python' binary"
patchelf --set-interpreter $GLIBC_LD_PATH --set-rpath $rpath $python_exe
echo "DONE! patchelf complete"
	#!/usr/env bash

	# TODO edit this line to specify location of new glibc
	export GLIBC_PATH=/cluster/tufts/hugheslab/miniconda2/envs/ape/custom_libs/
	export GLIBC_LD_PATH=$GLIBC_PATH/lib/x86_64-linux-gnu/ld-2.23.so

	if [[ ! -f $GLIBC_LD_PATH ]]; then
	echo "ERROR: Provided GLIBC_LD_PATH not valid"
	exit
	fi

	echo "OVERWRITING PYTHON EXECUTABLE:"
	python_exe=`which python`
	echo $python_exe

	IS_CONDA_ENV=`python -c "print('$python_exe'.count('/envs/') > 0)"`
	echo "IS_CONDA_ENV: $IS_CONDA_ENV"

	if [[ $IS_CONDA_ENV -ne 'True' ]]; then
	echo "ERROR: Current python executable not in conda env. Will not alter to avoid problems."
	exit
	fi

	CONDA_ENV_LIB=`python -c "print('$python_exe'.replace('/bin/python', '/lib'))"`

	echo "CREATING BACKUP PYTHON"
	python_tmp_exe=`python -c "print('$python_exe'.replace('python', 'python_backup'))"`
	cp $python_exe $python_tmp_exe
	echo "$python_tmp_exe"

	rpath=$GLIBC_PATH/lib/x86_64-linux-gnu:$CONDA_ENV_LIB:/usr/lib64:/lib64:/lib

	echo "CALLING PATCHELF on 'python' binary"
	patchelf --set-interpreter $GLIBC_LD_PATH --set-rpath $rpath $python_exe
	echo "DONE! patchelf complete"