Largely based on the Tensorflow 1.6 gist, and Tensorflow 1.7 gist for xcode, this should hopefully simplify things a bit.
- NVIDIA Web-Drivers 387.10.10.10.30.103 for 10.13.4
- CUDA-Drivers 387.178
- CUDA 9.1 Toolkit
- cuDNN 7.0.5 (latest release for mac os)
- Python 2.7
- Command Line Tool 8.3.2
- bazel 0.11.1
- Tensorflow 1.7 Source Code
If you don't know how to setup eGPU on Mac checkout these step. Make sure you have eGPU working before installation. (You sould see your specific graphic card name in Apple > About this Mac > System Report ... > Graphics/Displays)
The rest steps are the same as normal GPU setup.
If you are like me using MacBook Pro (15-inch, 2016) runing 10.13.4 (17E199) and eGPU: NVIDIA GeForce GTX 1080 Ti 11 GiB (or any 6.1 compatible version in nvidia page).
You could, at your own risk, skip the Prepare and Compile steps below,
download .whl from here and install it:
$ pip install tensorflow-1.7.0-cp36-cp36m-macosx_10_7_x86_64.whlAnd be sure to test after installation. But remember this is not safe.
For package management, ignore if you have your own python, wget or you want to download manually.
$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew install wgetDownload and install from http://www.nvidia.com/download/driverResults.aspx/130460/en-us
Download and install from http://www.nvidia.com/object/macosx-cuda-387.178-driver.html
Download and install from CommandLineToolsforXcode8.3.2.dmg
Or Find Command Line Tools for Xcode 8.3.2 on https://developer.apple.com/download/more/
If you have Homebrew installed
$ brew install bazel
or Download the binary here
$ chmod 755 bazel-0.10.0-installer-darwin-x86_64.sh
$ ./bazel-0.10.0-installer-darwin-x86_64.sh
It should be something along the lines of cuda_9.1.128_mac.dmg
Edit ~/.bash_profile and add the following:
export CUDA_HOME=/usr/local/cuda
export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:/usr/local/cuda/extras/CUPTI/lib
export LD_LIBRARY_PATH=$DYLD_LIBRARY_PATH
export PATH=$DYLD_LIBRARY_PATH:$PATH:/Developer/NVIDIA/CUDA-9.1/bin
We want to compile some CUDA sample to check if the GPU is correctly recognized and supported.
$ cd /Developer/NVIDIA/CUDA-9.1/samples
$ chown -R $(whoami) *
$ make -C 1_Utilities/deviceQuery
$ ./bin/x86_64/darwin/release/deviceQuery
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "GeForce GTX 1080 Ti"
CUDA Driver Version / Runtime Version 9.1 / 9.1
CUDA Capability Major/Minor version number: 6.1
Total amount of global memory: 11264 MBytes (11810963456 bytes)
(28) Multiprocessors, (128) CUDA Cores/MP: 3584 CUDA Cores
GPU Max Clock rate: 1645 MHz (1.64 GHz)
Memory Clock rate: 5505 Mhz
Memory Bus Width: 352-bit
L2 Cache Size: 2883584 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: Yes
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Supports Cooperative Kernel Launch: Yes
Supports MultiDevice Co-op Kernel Launch: No
Device PCI Domain ID / Bus ID / location ID: 0 / 196 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 9.1, CUDA Runtime Version = 9.1, NumDevs = 1
Result = PASS
If not already done, register at https://developer.nvidia.com/cudnn Download cuDNN 7.0.5
Change into your download directory and follow the post installation steps.
$ tar -xzvf cudnn-9.1-osx-x64-v7-ga.tgz
$ sudo cp cuda/include/cudnn.h /usr/local/cuda/include
$ sudo cp cuda/lib/libcudnn* /usr/local/cuda/lib
$ sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib/libcudnn*Skip if you have your own idea of which python/pip to use:
$ which python
/usr/local/anaconda3/bin/python
$ which pip
/usr/local/anaconda3/bin/pipOr Download get-pip and run it in python. More info here
python get-pip.py
If I remeber correctly, pip will automatically install the tensorflow dependencies (wheel, six etc)
$ cd /tmp
$ git clone https://github.com/tensorflow/tensorflow
$ cd tensorflow
$ git checkout v1.7.0
Apply the following patch to fix a couple build issues:
$ wget https://gist.githubusercontent.com/Willian-Zhang/088e017774536880bd425178b46b8c17/raw/xtensorflow17macos.patch
$ git apply xtensorflow17macos.patch
Except CUDA support, CUDA SDK version and Cuda compute capabilities, I left the other settings untouched.
Pay attension to Cuda compute capabilities, you might want to find your own according to guide.
$ ./configure
You have bazel 0.10.0 installed.
Please specify the location of python. [Default is /usr/bin/python]:
Found possible Python library paths:
/Library/Python/2.7/site-packages
Please input the desired Python library path to use. Default is [/Library/Python/2.7/site-packages]
Do you wish to build TensorFlow with Google Cloud Platform support? [Y/n]:
No Google Cloud Platform support will be enabled for TensorFlow.
Do you wish to build TensorFlow with Hadoop File System support? [Y/n]:
No Hadoop File System support will be enabled for TensorFlow.
Do you wish to build TensorFlow with Amazon S3 File System support? [Y/n]:
No Amazon S3 File System support will be enabled for TensorFlow.
Do you wish to build TensorFlow with Apache Kafka Platform support? [y/N]:
No Apache Kafka Platform support will be enabled for TensorFlow.
Do you wish to build TensorFlow with XLA JIT support? [y/N]:
No XLA JIT support will be enabled for TensorFlow.
Do you wish to build TensorFlow with GDR support? [y/N]:
No GDR support will be enabled for TensorFlow.
Do you wish to build TensorFlow with VERBS support? [y/N]:
No VERBS support will be enabled for TensorFlow.
Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]:
No OpenCL SYCL support will be enabled for TensorFlow.
Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.
Please specify the CUDA SDK version you want to use, e.g. 7.0. [Leave empty to default to CUDA 9.0]: 9.1
Please specify the location where CUDA 9.1 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:
Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7.0]:
Please specify the location where cuDNN 7 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size. [Default is: 3.5,5.2] (type your own, check on https://developer.nvidia.com/cuda-gpus, mine is 6.1 for GTX 1080 Ti)
Do you want to use clang as CUDA compiler? [y/N]:
nvcc will be used as CUDA compiler.
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
Do you wish to build TensorFlow with MPI support? [y/N]:
No MPI support will be enabled for TensorFlow.
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]:
Not configuring the WORKSPACE for Android builds.
Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See tools/bazel.rc for more details.
--config=mkl # Build with MKL support.
--config=monolithic # Config for mostly static monolithic build.
Configuration finished
Takes about 47 minutes on my machine.
$ bazel build --config=cuda --config=opt --action_env PATH --action_env LD_LIBRARY_PATH --action_env DYLD_LIBRARY_PATH //tensorflow/tools/pip_package:build_pip_package$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
$ ls ls /tmp/tensorflow_pkg
tensorflow-1.7.0-cp36-cp36m-macosx_10_7_x86_64.whlIf you want to use virtualenv or something, now is the time. Or just:
$ pip install /tmp/tensorflow_pkg/tensorflow-1.7.0-cp36-cp36m-macosx_10_7_x86_64.whlFiles in /tmp would be cleaned after reboot.
cp /tmp/tensorflow_pkg/*.whl ~/
It's useful to leave the .whl file lying around in case you want to install it for another environment.
See if everything got linked correctly
$ cd ~
$ python
>>> import tensorflow as tf
>>> tf.Session()
2018-04-08 03:25:15.740635: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:859] OS X does not support NUMA - returning NUMA node zero
2018-04-08 03:25:15.741260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.645
pciBusID: 0000:c4:00.0
totalMemory: 11.00GiB freeMemory: 10.18GiB
2018-04-08 03:25:15.741288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-04-08 03:25:16.157590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-04-08 03:25:16.157614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-04-08 03:25:16.157620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-04-08 03:25:16.157753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9849 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:c4:00.0, compute capability: 6.1)
<tensorflow.python.client.session.Session object at 0x10968ef60>$ pythonimport tensorflow as tf
tf.enable_eager_execution()
tf.executing_eagerly() # => True
x = [[2.]]
m = tf.matmul(x, x)
print("hello, {}".format(m)) # => "hello, [[4.]]"$ pip install keras
$ wget https://gist.githubusercontent.com/Willian-Zhang/290dceb96679c8f413e42491c92722b0/raw/mnist-cnn.py
$ python mnist_cnn.py
/usr/local/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
x_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples
Train on 60000 samples, validate on 10000 samples
Epoch 1/12
2018-04-08 03:29:00.155517: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:859] OS X does not support NUMA - returning NUMA node zero
2018-04-08 03:29:00.155661: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.645
pciBusID: 0000:c4:00.0
totalMemory: 11.00GiB freeMemory: 10.11GiB
2018-04-08 03:29:00.155677: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-04-08 03:29:00.562343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-04-08 03:29:00.562373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-04-08 03:29:00.562403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-04-08 03:29:00.562536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9781 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:c4:00.0, compute capability: 6.1)
2018-04-08 03:29:00.563022: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 9.55G (10256140800 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-04-08 03:29:00.868307: E tensorflow/core/grappler/clusters/utils.cc:127] Not found: TF GPU device with id 0 was not registered
2018-04-08 03:29:00.906005: E tensorflow/core/grappler/clusters/utils.cc:127] Not found: TF GPU device with id 0 was not registered
2018-04-08 03:29:00.973462: E tensorflow/core/grappler/clusters/utils.cc:127] Not found: TF GPU device with id 0 was not registered
59904/60000 [============================>.] - ETA: 0s - loss: 0.2624 - acc: 0.92022018-04-08 03:29:07.381067: E tensorflow/core/grappler/clusters/utils.cc:127] Not found: TF GPU device with id 0 was not registered
60000/60000 [==============================] - 8s 129us/step - loss: 0.2620 - acc: 0.9203 - val_loss: 0.0587 - val_acc: 0.9825
Epoch 2/12
60000/60000 [==============================] - 4s 66us/step - loss: 0.0891 - acc: 0.9733 - val_loss: 0.0437 - val_acc: 0.9850
Epoch 3/12
60000/60000 [==============================] - 4s 66us/step - loss: 0.0681 - acc: 0.9789 - val_loss: 0.0341 - val_acc: 0.9881
Epoch 4/12
60000/60000 [==============================] - 4s 67us/step - loss: 0.0569 - acc: 0.9829 - val_loss: 0.0398 - val_acc: 0.9859
Epoch 5/12
60000/60000 [==============================] - 4s 70us/step - loss: 0.0480 - acc: 0.9856 - val_loss: 0.0303 - val_acc: 0.9898
Epoch 6/12
60000/60000 [==============================] - 4s 66us/step - loss: 0.0438 - acc: 0.9869 - val_loss: 0.0288 - val_acc: 0.9897
Epoch 7/12
60000/60000 [==============================] - 4s 66us/step - loss: 0.0379 - acc: 0.9881 - val_loss: 0.0287 - val_acc: 0.9905
Epoch 8/12
60000/60000 [==============================] - 4s 66us/step - loss: 0.0357 - acc: 0.9892 - val_loss: 0.0277 - val_acc: 0.9915
Epoch 9/12
60000/60000 [==============================] - 4s 65us/step - loss: 0.0329 - acc: 0.9898 - val_loss: 0.0268 - val_acc: 0.9906
Epoch 10/12
60000/60000 [==============================] - 4s 66us/step - loss: 0.0312 - acc: 0.9903 - val_loss: 0.0295 - val_acc: 0.9911
Epoch 11/12
60000/60000 [==============================] - 4s 66us/step - loss: 0.0281 - acc: 0.9908 - val_loss: 0.0292 - val_acc: 0.9908
Epoch 12/12
60000/60000 [==============================] - 4s 65us/step - loss: 0.0277 - acc: 0.9917 - val_loss: 0.0260 - val_acc: 0.9919
Test loss: 0.02598250026818114
Test accuracy: 0.9919
You can use cuda-smi to watch the GPU memory usages. In case the of the mnist example in keras, you should see the free memory drop down to maybe 2% and the fans spin up. Not quite sure what the grappler/clusters/utils.cc:127 warning is, however.
$ ./cuda-smi.dms
Device 0 [PCIe 0:196:0.0]: GeForce GTX 1080 Ti (CC 6.1): 10350 of 11264 MB (i.e. 91.9%) Free
# when GPU
$ ./cuda-smi.dms
Device 0 [PCIe 0:196:0.0]: GeForce GTX 1080 Ti (CC 6.1): 1181.1 of 11264 MB (i.e. 10.5%) Free
Tested on a MacBook Pro (15-inch, 2016) 10.13.4 (17E199) 2.7 GHz Intel Core i7 and NVIDIA GeForce GTX 1080 Ti 11 GiB
Thanks for the recipe William!
Just want to add that installing bazel with brew led to "error: unknown target CPU 'armv7-a'", the way to solve this problem was to install bazel directly from https://github.com/bazelbuild/bazel/releases. Hope this helps to anyone else having this issue.