Compile and run Caffe on Ubuntu 15.10

Caffe Ubuntu 15.10.md

Ubuntu 15.10 have been released for a couple of days. It is a bleeding-edge system coming with Linux kernel 4.2 and GCC 5. However, compiling and running Caffe on this new system is no longer as smooth as on earlier versions. I have done some research related to this issue and finally find a way out. I summarize it here in this short tutorial and I hope more people and enjoy this new system without breaking their works.

Install NVIDIA Driver

The latest NVIDIA driver is officially included in Ubuntu 15.10 repositories. One can install it directly via apt-get.

sudo apt-get install nvidia-352-updates nvidia-modprobe

The nvidia-modprobe utility is used to load NVIDIA kernel modules and create NVIDIA character device files automatically everytime your machine boots up.

Reboot your machine and verify everything works by issuing nvidia-smi or running deviceQuery in CUDA samples.

References

1. How to do it manually
2. Understanding character special file

More infomation related to NVIDIA driver and CUDA installation can be found here.

Install CUDA 7.5

CUDA 7.5 is provided by neither Ubuntu official nor CUDA repositories for Ubuntu 15.10. But we can still install it using .run file.

cd ~
wget http://developer.download.nvidia.com/compute/cuda/7.5/Prod/local_installers/cuda_7.5.18_linux.run
chmod +x cuda_7.5.18_linux.run
./cuda_7.5.18_linux.run --extract=/home/$USER
sudo ./cuda-linux64-rel-7.5.18-19867135.run

Follow the instructions there and configure runtime library after the installation finishes.

sudo bash -c "echo /usr/local/cuda/lib64/ > /etc/ld.so.conf.d/cuda.conf"
sudo ldconfig

Install all other dependencies

Follow instructions here

Download Caffe

cd ~
git clone https://github.com/BVLC/caffe.git

By now you should be unable to compile it due to serveral incompatibility problems. Let's fix them one by one.

Hack CUDA to support GCC 5

By default, CUDA 7.5 does not support compling with GCC 4.10 or later versions. It will report the following errors when compiling Caffe.

In file included from /usr/local/cuda/include/cuda_runtime.h:76:0,
                 from <command-line>:0:
/usr/local/cuda/include/host_config.h:115:2: error: #error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
 #error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
  ^

You can try install gcc-4.9 and g++-4.9 to solve this problem temporarily. But the linker will report undefined references errors later because all other libraries (protobuf,leveldb,opencv) in Ubuntu 15.10 are compiled with GCC version 5.2. They are not compatible.

To solve this problem we have to "hack" the CUDA toolkit by editing /usr/local/cuda/include/host_config.h. Comment line 115.

--- #error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
+++ //#error -- unsupported GNU version! gcc versions later than 4.9 are not supported!

Fix hdf5 naming problem

Your machine may report the following error when compiling Caffe even though libhdf5-serial-dev package has been already installed in it.

./include/caffe/util/hdf5.hpp:6:18: fatal error: hdf5.h: No such file or directory

This is because of change of default path and name of hdf5 head files and libraries in Ubuntu 15.10. To solve this problem, we can simply modify Makefile files.

Append /usr/include/hdf5/serial/ to INCLUDE_DIRS at line 85 in Makefile.config.

--- INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
+++ INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial/

Modify hdf5_hl and hdf5 to hdf5_serial_hl and hdf5_serial at line 173 in Makefile

--- LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_hl hdf5
+++ LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_serial_hl hdf5_serial

Compile and Test Caffe

By now, all incompatibility issues should have been solved. We can start to compile and test Caffe.

cd ~/caffe
make -j8
make test -j8
make runtest -j8

Hopefully all tests pass.

Notes

1. Running sudo make runtest once may help you fix a lot of runtime errors until next reboot, although it is not a recommended way.
2. You might need to append LC_ALL="en_US.UTF-8" to /etc/environment and reboot your machine if there are runtime errors related to locale::facet::_S_create_c_locale name not valid.

I made an account just to comment here. Man, you were a lifesaver! When I tried to install Caffe, i also ran into the hdf5 problem: "

fatal error: hdf5.h: No such file or directory

". I found some solutions on Google, but another problem then came up:

/usr/bin/ld: cannot find -lhdf5_hl
/usr/bin/ld: cannot find -lhdf5
collect2: error: ld returned 1 exit status
Makefile:545: recipe for target '.build_release/lib/libcaffe.so' failed
make: *** [.build_release/lib/libcaffe.so] Error

I spent days on end trying to find a solution but it was futile! Some posts suggested creating a symlink at usr/lib/x86_64-linux-gnu but the directory doesn't exist! . Luckily I found your post, hopefully other people won't have to go through all this anymore.
On a sidenote, i am still puzzled as to why Google didn't come up with this place sooner.......

Awesome, thanks !

You sir, are a life saver.

Thanks!

Perfect! Thanks!

Very useful note!

Thank you, you save my life.

getting the following errors:

.build_release/lib/libcaffe.so: undefined reference to google::base::CheckOpMessageBuilder::NewString()' .build_release/lib/libcaffe.so: undefined reference togoogle::protobuf::internal::NameOfEnum(google::protobuf::EnumDescriptor const_, int)'
.build_release/lib/libcaffe.so: undefined reference to caffe::BlockingQueuecaffe::Batch<double_>::pop(std::string const&)' .build_release/lib/libcaffe.so: undefined reference to caffe::BlockingQueuecaffe::Batch<float_>::pop(std::string const&)'
collect2: error: ld returned 1 exit status
Makefile:616: recipe for target '.build_release/tools/convert_imageset.bin' failed
make: *_* [.build_release/tools/convert_imageset.bin] Error 1

I'm trying to build OpenCV on 16.04 and getting the GNU GCC version error because 16.04 has GCC 5.3.1. I tried your hacking CUDA to support GCC 5, which got rid of that particular error, but now I'm getting:

/usr/include/string.h: In function 'void* mempcpy_inline(void, const void, size_t)':
/usr/include/string.h:652:42: error: 'memcpy' was not declared in this scope
return (char *) memcpy (__dest, __src, __n) + __n;

This page seems to have the answer to my question:
https://groups.google.com/forum/#!searchin/caffe-users/15.10/caffe-users/a6TDx89IByY/6Po32WqzCwAJ

Thank you very much for your advice.
It went well.

thanks, it works greatly

I'm trying to build OpenCV on 16.04 and getting the GNU GCC version error because 16.04 has GCC 5.3.1. I tried your hacking CUDA to support GCC 5, which got rid of that particular error, but now I'm getting:

/usr/include/string.h: In function 'void* __mempcpy_inline(void, const void, size_t)':
/usr/include/string.h:652:42: error: 'memcpy' was not declared in this scope
return (char *) memcpy (__dest, __src, __n) + __n;

Hi! I was struggling the same problem while building Tensorflow under Ubuntu 16.04 with CUDA and the solution is (in case with Caffe) to add the following line in your Makefile.config:
CUSTOM_CXX := g++ -D_FORCE_INLINES
It worked for me and I finished building this hell under Ubuntu 16.04

Works like a charm. Thanks a lot!

i wish you much good fortune upon your head, kind sir

Hey, is this hack really feasible? To me it more sounds like we are ignoring a real error:
--- #error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
+++ //#error -- unsupported GNU version! gcc versions later than 4.9 are not supported!

Has worked so far for me but I am curious to know how ignoring the error would make it feasible?

Many thanks,
Now it is working! ^^

Thank you!

Thanks for your post.
If anyone else is having problems with the locale::facet::_S_create_c_locale name not valid error and lacks root privileges needed to add something to /etc/environment as I did: You can run the tests by

env LC_ALL="en_US.UTF-8" make runtests

and this error should not occur. (Note: this does not set the variable outside of the used command)

I'm getting this with make runtest
Cuda number of devices: 9406365
Setting to use device 0
Current device id: 0
Current device name:
F0126 13:15:38.632215 35566 gpu_memory.cpp:137] Check failed: error == cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version

Details:
g++ (Ubuntu 5.4.1-2ubuntu1~16.04) 5.4.1 20160904
MKL 2017 Update 1
Linux Description: Linux Mint 18.1 Serena
CUDA:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 378.09 Driver Version: 378.09 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 960 Off | 0000:04:00.0 On | N/A |
| 0% 25C P8 7W / 160W | 94MiB / 4032MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 760 Off | 0000:83:00.0 N/A | N/A |
| 42% 33C P8 N/A / N/A | 1MiB / 4036MiB | N/A Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2427 G /usr/lib/xorg/Xorg 72MiB |
| 0 2937 G cinnamon 20MiB |
| 1 Not Supported |
+-----------------------------------------------------------------------------+

Section on hd5 really helps. Thanks!

Definitely a life saver. Got both the hdf5 and gcc errors and while I managed to fix the hdf5 error, the gcc error was fixed thanks to you.

Got caffe working on Ubuntu 16.10 finally..

Edit: I also had to change the links to gcc and g++

sudo rm /usr/bin/gcc
sudo rm /usr/bin/g++

ln -s /usr/bin/gcc-5 /usr/bin/gcc
ln -s /usr/bin/g++-5 /usr/bin/g++

Thanks a lot for this post !!

You deserve a star!

I wish I could give you 5 stars, not just 1, clearly saved me a lot of time.

I am still getting build error. Why my ubuntu 14.04 cannot locate the libary correctly?

CXX examples/cifar10/convert_cifar_data.cpp
CXX examples/mnist/convert_mnist_data.cpp
CXX .build_release/src/caffe/proto/caffe.pb.cc
AR -o .build_release/lib/libcaffe.a
LD -o .build_release/lib/libcaffe.so.1.0.0-rc3
/usr/bin/ld: cannot find -lhdf5_serial_hl
/usr/bin/ld: cannot find -lhdf5_serial
/usr/bin/ld: cannot find -lopenblas
collect2: error: ld returned 1 exit status
make: *** [.build_release/lib/libcaffe.so.1.0.0-rc3] Error 1

Nice post. It help me a lot with hdf5 problem. However I am still facing this problem.

.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::WriteBytes(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, google::protobuf::io::CodedOutputStream*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::DescriptorPool::FindFileByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'

Anybody know how to fix this?
-Thank you-

This guild helps me solve the hdf5 naming problem! Thanks a lot!

thank you!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Thank you a lot guy you save my day