Skip to content

Instantly share code, notes, and snippets.

@venik venik/build_tf.sh
Last active Apr 23, 2019

Embed
What would you like to do?
Bash script for local building TensorFlow on Mac/Linux with all CPU optimizations (default pip package has only SSE)
#!/usr/bin/env bash
# Author: Sasha Nikiforov
# source of inspiration
# https://stackoverflow.com/questions/41293077/how-to-compile-tensorflow-with-sse4-2-and-avx-instructions
# Detect platform
if [ "$(uname)" == "Darwin" ]; then
# MacOS
raw_cpu_flags=`sysctl -a | grep machdep.cpu.features | cut -d ":" -f 2 | tr '[:upper:]' '[:lower:]'`
elif [ "$(uname)" == "Linux" ]; then
# GNU/Linux
raw_cpu_flags=`grep flags -m1 /proc/cpuinfo | cut -d ":" -f 2 | tr '[:upper:]' '[:lower:]'`
else
echo "Unknown plaform: $(uname)"
exit -1
fi
# check if VirtuelEnv activated
if [ -z "$VIRTUAL_ENV" ]; then
echo "VirtualEnv is not activated"
exit -1
fi
VENV_BIN=$VIRTUAL_ENV/bin
VENV_LIB=$VIRTUAL_ENV/lib
# bazel tf needs these env vars
export PYTHON_BIN_PATH=$VENV_BIN/python
export PYTHON_LIB_PATH=$VENV_LIB/`ls $VENV_LIB | grep python`
COPT="--copt=-march=native"
for cpu_feature in $raw_cpu_flags
do
case "$cpu_feature" in
"sse4.1" | "sse4.2" | "ssse3" | "fma" | "cx16" | "popcnt" | "maes")
COPT+=" --copt=-m$cpu_feature"
;;
"avx1.0")
COPT+=" --copt=-mavx"
;;
*)
# noop
;;
esac
done
bazel clean
./configure
bazel build -c opt $COPT -k //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
pip install --upgrade /tmp/tensorflow_pkg/`ls /tmp/tensorflow_pkg/ | grep tensorflow`
@phobrain

This comment has been minimized.

Copy link

phobrain commented Aug 26, 2017

What/where is the ./configure script?

@rogeriochaves

This comment has been minimized.

Copy link

rogeriochaves commented Aug 27, 2017

@phobrain you have to clone the tensorflow repo and run this script inside it, more details: https://www.tensorflow.org/install/install_sources

@phobrain

This comment has been minimized.

Copy link

phobrain commented Aug 29, 2017

Thanks! I had a tree checked out or likely unpacked according to some instruction page, but it didn't have the configure script. Following the install_sources link led to a successful compile. Had to get latest keras from github with 5-day-old fix for compatibility. Looks like it's cranking a lot faster now. Even without that, it's worth it to see the warnings gone. :-)

@phobrain

This comment has been minimized.

Copy link

phobrain commented Aug 29, 2017

For training a truncated Inceptionv3 (pics 299x299) siamese-style in keras, batch_size=32, train/validate sizes=40/40,

virtualenv version epochs (seconds): 153, 131, 128, 130, 128 (total runtime 12m 31s)
compiled version: 114, 86, 86, 85, 87 (total 9m 1s)

Macbook Pro 2012.

@venik

This comment has been minimized.

Copy link
Owner Author

venik commented Aug 30, 2017

@rogeriochaves thx :)

@phobrain great, thx for the input

@greis

This comment has been minimized.

Copy link

greis commented Jan 20, 2018

Tensorflow 1.4.1 did not work with bazel 0.9.0 for me. Had to downgrade bazel to 0.8.1

@venik

This comment has been minimized.

Copy link
Owner Author

venik commented May 8, 2018

works with tag v1.11.0, commit c19e293 python 3, I wasnt able to build with python 2, but didnt spend much time to do it Note: python 3.7 wont work on MacOS, you need python 3.6 or mb 3.5. More details below
tensorflow/tensorflow#21441

Due to some problem with virtualenv small patch is needed to build TF on virtualenv.

diff --git a/configure.py b/configure.py
index 7edab53964..5987c083cc 100644
--- a/configure.py
+++ b/configure.py
@@ -155,7 +155,7 @@ def get_python_path(environ_cp, python_bin_path):
   try:
     library_paths = run_shell(
         [python_bin_path, '-c',
-         'import site; print("\\n".join(site.getsitepackages()))']).split('\n')
+         'from distutils.sysconfig import get_python_lib; print(get_python_lib())']).split('\n')
   except subprocess.CalledProcessError:
     library_paths = [run_shell(
         [python_bin_path, '-c',
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.