Skip to content

Instantly share code, notes, and snippets.

@kylemcdonald
Last active March 26, 2024 05:52
Show Gist options
  • Save kylemcdonald/0698c7749e483cd43a0e to your computer and use it in GitHub Desktop.
Save kylemcdonald/0698c7749e483cd43a0e to your computer and use it in GitHub Desktop.
How to build Caffe for OS X.

Theory of Building Caffe on OS X

Introduction

Our goal is to run python -c "import caffe" without crashing. For anyone who doesn't spend most of their time with build systems, getting to this point can be extremely difficult on OS X. Instead of providing a list of steps to follow, I'll try to explain why each step happens.

This page has OS X specific install instructions.

I assume:

  • You are using OS X 10.9+ (I'm using 10.10.5)
  • You have downloaded Caffe (this was written against commit 4c561fdd7e69ba67e891166c9d700ec6bf46ef74)
  • You have Xcode installed. Run xcode-select --install to verify that the command line tools are installed.
  • You have Homebrew installed.

Dependencies

The first dependency is downloading and installing CUDA and CuDNN. CUDA installs itself in /Developer/NVIDIA/ and symlinks to /usr/local/cuda/. I move the CuDNN 7.0 r3 folder cudnn to /Developer/NVIDIA/ and rename it to CuDNN-7.0-r3. Usually this would be placed somewhere else, like /usr/local/cudnn but I decided to do something similar to the CUDA installer.

Next, we will install Python with homebrew: brew install python. Caffe recommends using Anaconda, but as far as I can tell it only helps by bundling a few modules (numpy, scipy, sklearn) that we can just as easily install with pip at the end.

When homebrew asks you to run pip install --upgrade pip setuptools only run pip install --upgrade pip. Otherwise you will have conflicts between a Python 2 and Python 3 installation.

Then we run the following as stated in the document, which will download and install a bunch of prerequisites for Caffe. The installation instructions assume you have numpy and protobuf installed. If not, first run:

brew install numpy protobuf

Then continue with the recommended steps:

brew install -vd snappy leveldb gflags glog szip lmdb
brew tap homebrew/science
brew install hdf5 opencv

boost and boost-python need to be installed from source. However version 1.58 is incompatible. To fix this we will modify the Homebrew formulas. Run brew edit boost, replace the contents with this formula and brew edit boost-python and use this formula. If you're having trouble with the default brew editor, run export HOMEBREW_EDITOR='open -t' in Terminal before running brew edit to change the editor. When you are done with this tutorial, replace the formula files with their original formulas by opening /usr/local/Library/Formula/ and running git checkout -- ..

For now, continue as recommended:

brew install --build-from-source --with-python -vd protobuf
brew install --build-from-source -vd boost boost-python

Sanity Checks

At this point we need to make sure that two things are the case:

  1. All shared libraries that reference a C++ standard library are using libc++ and not libstdc++
  2. boost-python is linked against the Homebrew-installed Python, not the system python (because we will also link Caffe against Homebrew's Python, and because pip is using Homebrew's Python).

We can check this with otool -L:

$ brew info boost
...
$ otool -L /usr/local/Cellar/boost/1.57.0/lib/libboost_system.dylib # this is one of the libs that caffe links against
...
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
...
$ brew info boost-python
...
$ otool -L /usr/local/Cellar/boost-python/1.57.0/lib/libboost_python.dylib
...
/usr/local/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)
...

It looks like Boost is the right version (1.57.0) and it's linking against the right libraries. If it's not, you will get Sigabrt and other strange errors from Python later when you try to import caffe.

If we were running an older versions of OS X, it would use libstdc++ by default for compiling Caffe, which means all the Homebrew formulas would have to be modified to match. Since we're using a newer version we don't have to do this.

Compiling Caffe

Next, we adjust the Makefile.config:

  1. Uncomment USE_CUDNN := 1 because we want to use it.
  2. Edit the PYTHON_INCLUDE. The example values use the system path for Python. We modify them to use the Homebrew path: PYTHON_INCLUDE := /usr/local/lib/python2.7/site-packages/numpy/core/include/ /usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/include/python2.7.
  3. For the same reason as the previous step, edit PYTHON_LIB: PYTHON_LIB := /usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/
  4. Uncomment WITH_PYTHON_LAYER := 1.
  5. Change the INCLUDE_DIRS to add CuDNN: INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /Developer/NVIDIA/CuDNN-7.0-r3/include
  6. Change the LIBRARY_DIRS to add CuDNN: LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /Developer/NVIDIA/CuDNN-7.0-r3/lib. It's very important that $(PYTHON_LIB) comes first here, otherwise make will try to use the system python that it finds in /usr/lib.
  7. Do not change BLAS := atlas. By default this will cause Caffe to link with Accelerate, which is a fast BLAS implementation provided by Apple with OS X.

Run make all from the main Caffe directory. If you have a quad-core processor or better you can type make all -j4 to speed up compilation a bit. Here are some errors you might run into with make:

  • If it complains about a specific library, you could have missed one of the prerequisites. Check with brew info x, where x is the name of the library it's complaining about, that it's correctly installed.
  • If it complains that something is not found for architecture x86_64, this is libstc++ vs libc++ mistmatch and you'll need to double-check all the prerequisites installed by Homebrew that they don't say libstdc++ in the otool -L output.
  • If it complains about pyconfig.h, make sure you are passing the right Python path to PYTHON_INCLUDE. For example, open /usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/include/python2.7 to see that pyconfig.h is in there. If not, you need to install Python with brew, or the path is different for your installation (e.g., a different version number like 2.7.11).

Once make all finishes, run make pycaffe.

Next, make sure some system variables are configured correctly for Python. Open your ~/.bash_profile in a text editor (e.g., nano ~/.bash_profile). If you're using TextEdit to edit ~/.bash_profile make sure smart quotes are turned off. Add these three lines:

export DYLD_LIBRARY_PATH='/Developer/NVIDIA/CuDNN-7.0-r3/lib':$DYLD_LIBRARY_PATH
export DYLD_LIBRARY_PATH='/usr/local/cuda/lib/':$DYLD_LIBRARY_PATH
export PYTHONPATH=~/Documents/caffe/python:$PYTHONPATH

The first two tell Python where to load the dylibs for CUDA and CuDNN from. The third line tells Python where to load pycaffe from. Customize the third line so it matches where you have caffe installed.

Now, if we try to test with python -c "import caffe" we will get an error No module named skimage.io. So we run pip install scikit-image. If we test again, No module named scipy so we pip install scipy. Finally, testing one more time:

$ python
>>> import caffe
>>> 

Success.

Conclusion

If you want to install Caffe and use it with the system Python, or even with Anaconda Python, this whole process might be easier (or slightly better documented), but I haven't tried that since I started with a Homebrew-installed version of Python. But if you use another version of Python, I recommend there is only one Python on your system, and you shoud leave all the values for Python paths with their defaults.

@GloriaIce
Copy link

Hi,
I have problem when running brew install --build-from-source -vd boost boost-python, it generates warnings and seems not to be ended. I think it might be the boost and boost-python document. Could you explain how do you know how to modify the two documents?
Thanks

@SnowmanTackler
Copy link

Thanks for the guide!

@damilarelana
Copy link

damilarelana commented Jun 11, 2016

Hi @kylemcdonald thanks for the guide.

You are correct that using import caffe throwing up errors (bumpy version related) which was messy to correct.
Was able to follow your steps.
make runtest -j4 is throwing the following errors:

.build_release/tools/caffe dyld: Library not loaded: @rpath/libcudnn.5.dylib Referenced from: /Users/Lana/Downloads/Personal/ML/Caffe/.build_release/tools/caffe Reason: image not found make: *** [runtest] Trace/BPT trap: 5

@Apple-Of-Eden
Copy link

Hey Kylemcdonald, thanks a bunch for this. I found this tons more useful than Berkley caffe installation page.

@xyzzhangfan
Copy link

Hi, thanks for your guide but when I follow this guide I meet a problem.
After run follow command I didn't get the python path.
otool -L /usr/local/opt/boost-python/lib/libboost_python.dylib

I only got these:
/usr/local/opt/boost-python/lib/libboost_python-mt.dylib:
/usr/local/opt/boost-python/lib/libboost_python-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
I don't know how to fix this can you give me any hits?

Thank you very much!

@tpanum
Copy link

tpanum commented Jun 26, 2016

I got the same issue as @xyzzhangfan

@doctorpangloss
Copy link

DYLD_LIBRARY_PATH is ignored in 10.11 because of system integrity protection, so this guide is as "wrong" as nearly every other guide you'll find.

@doctorpangloss
Copy link

@Beritra
Copy link

Beritra commented Jun 19, 2018

Thank you very much! This tutorial has helped me a lot!

@gokalper
Copy link

Thanks for the guide, it looks like
brew tap homebrew/science is deprecated .... what packages do we need in its place ?

@notebookdata
Copy link

Hi, @gokalper @Beritra what is the Mac machine requirements to install caffe2?

I have Mac mini with 500GB storage, 4GB RAM, and Intel Core i5 processor. is it possible to install caffe2 on my Mac mini?

OS: MacOS Mojave 10.14.2
Processor: Intel Core i5 2.5 GHz
Graphics: Intel HD Graphics 4000 1536 MB
RAM: 4GB DDR3
Storage: 500GB SATA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment