update version script

README.md

Goal

Install working tensorflow or pytorch via standard conda environment workflow.

Basic Setup : Install pytorch in a fresh conda environment

The recommended conda-based install process works smoothly:

$ # Create a fresh environment
$ conda create --name py37_torch python=3.7 --yes

$ # Activate new environment
$ source activate py37_torch

$ # Install tensorflow
$ conda install tensorflow --yes

$ # Install pytorch 
$ conda install pytorch-cpu torchvision-cpu -c pytorch --yes

Roadblock

The gotcha is that when we try to then use the package we just installed, we get an GLIBC error like this:

$ python -c "import torch"

ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by .../site-packages/torch/lib/libshm.so)

Badness! Clearly, the current computing system doesn't have a recent-enough GLIBC. However, if this is a cluster computing system, you often don't have root access and can't easily upgrade the GLIBC.

Step 1: Install recent copies of glibc and libc++ in userspace

Credit: StackOverflow answer by Theo T.

Step 1a: (NEW FOR PYTHON 3.7) Download and unpack some pre-compiled GLIBC shared libraries

$ # Make a folder within the environment to hold useful things
$ mkdir -p /path/to/conda/envs/py37_torch/custom_libs/
$ cd /path/to/conda/envs/py27_torch/custom_libs/

$ # Get libc files (URL verified by MCH on 2019/08/21)
$ wget http://mirrors.kernel.org/ubuntu/pool/main/g/glibc/libc6_2.23-0ubuntu10_amd64.deb
$ wget http://mirrors.kernel.org/ubuntu/pool/main/g/glibc/libc6-dev_2.23-0ubuntu10_amd64.deb

$ # Unpack files into current directory (will create usr/ and lib/ and lib64/ folders)
$ ar p libc6_2.23-0ubuntu10_amd64.deb data.tar.xz | tar xvJ
$ ar p libc6-dev_2.23-0ubuntu10_amd64.deb data.tar.xz | tar xvJ

(See an older list for Python 2.7 at bottom of this doc).

What have we accomplished? You should have some new folders in your current directory, labeld usr/ and lib/ and lib64/.

We can verify that before, we had an OLD libc, and now we have a shiny new one!

Check the OLD location of libc.so.6

$ strings /lib/libc.so.6 | grep GLIBC_2. | tail -n3
GLIBC_2.10
GLIBC_2.11
GLIBC_2.12

NEW version of libc.so.6 in working directory

$ strings lib/x86_64-linux-gnu/libc.so.6 | grep GLIBC_2 | tail -n3
GLIBC_2.18
GLIBC_2.22
GLIBC_2.23

Step 1b: Download and unpack some pre-compiled LIBSTDC++ shared libraries

# Get libstdc++ (URL verified by MCH on 2019/02/18)
wget ftp://195.220.108.108/linux/mageia/distrib/4/x86_64/media/core/updates/libstdc++6-4.8.2-3.2.mga4.x86_64.rpm

# Alternative URL:
# wget http://ftp.riken.jp/Linux/scientific/6.0/x86_64/os/Packages/libstdc++-4.4.4-13.el6.x86_64.rpm

# Unpack into current directory (will add content to lib/ and lib64/ folders)
rpm2cpio libstdc++6-4.8.2-3.2.mga4.x86_64.rpm | cpio -idmv

Step 2: Use patchelf to make your python install use these userspace libraries instead of the system defaults

Credit: Stackoverview answer by Evalds Urtans

Step 2a: Install patchelf into current conda env

# Be sure correct environment is active
$ source activate py37_torch

# Install patchelf
(py37_torch) $ conda install patchelf -c conda-forge --yes

Step 2b: Use attached script to alter the conda env's python executable to use the custom GLIBC libraries

(py37_torch) $ bash rewrite_python_exe_glibc_with_patchelf.sh

-- DEPRECATED --

Step 1a: (OLD FOR PYTHON 2.7) Download and unpack some pre-compiled GLIBC shared libraries

$ # Make a folder within the environment to hold useful things
$ mkdir -p /path/to/conda/envs/py27_torch1.0/custom_libs/
$ cd /path/to/conda/envs/py27_torch1.0/custom_libs/

$ # Get libc files (URL verified by MCH on 2019/02/18)
$ wget https://launchpadlibrarian.net/137699828/libc6_2.17-0ubuntu5_amd64.deb
$ wget https://launchpadlibrarian.net/137699829/libc6-dev_2.17-0ubuntu5_amd64.deb

$ # Unpack files into current directory (will create usr/ and lib/ and lib64/ folders)
$ ar p libc6_2.17-0ubuntu5_amd64.deb data.tar.gz | tar zx
$ ar p libc6-dev_2.17-0ubuntu5_amd64.deb data.tar.gz | tar zx

rewrite_python_exe_glibc_with_patchelf.sh

#!/usr/env bash

# TODO edit this line to specify location of new glibc 
export GLIBC_PATH=path/to/conda/envs/py27_torch1.0/custom_libs/

if [[ ! -f $GLIBC_PATH/lib/x86_64-linux-gnu/ld-2.23.so ]]; then
    echo "ERROR: Provided GLIBC_PATH not valid"
    exit
fi

echo "OVERWRITING PYTHON EXECUTABLE:"
python_exe=`which python`
echo $python_exe

IS_CONDA_ENV=`python -c "print('$python_exe'.count('/envs/') > 0)"`
echo "IS_CONDA_ENV: $IS_CONDA_ENV"

if [[ $IS_CONDA_ENV -ne 'True' ]]; then
    echo "ERROR: Current python executable not in conda env. Will not alter to avoid problems."
    exit
fi

CONDA_ENV_LIB=`python -c "print('$python_exe'.replace('/bin/python', '/lib'))"`

echo "CREATING BACKUP PYTHON"
python_tmp_exe=`python -c "print('$python_exe'.replace('python', 'python_backup'))"`
cp $python_exe $python_tmp_exe
echo "$python_tmp_exe"

rpath=$GLIBC_PATH/lib/x86_64-linux-gnu:$CONDA_ENV_LIB:/usr/lib64:/lib64:/lib
interp_path=$GLIBC_PATH/lib/x86_64-linux-gnu/ld-2.23.so

echo "CALLING PATCHELF on 'python' binary"
patchelf --set-interpreter $interp_path --set-rpath $rpath $python_exe
echo "DONE! patchelf complete"