Create a gist now

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Compile and run Caffe on Ubuntu 15.10

Ubuntu 15.10 have been released for a couple of days. It is a bleeding-edge system coming with Linux kernel 4.2 and GCC 5. However, compiling and running Caffe on this new system is no longer as smooth as on earlier versions. I have done some research related to this issue and finally find a way out. I summarize it here in this short tutorial and I hope more people and enjoy this new system without breaking their works.

Install NVIDIA Driver

The latest NVIDIA driver is officially included in Ubuntu 15.10 repositories. One can install it directly via apt-get.

sudo apt-get install nvidia-352-updates nvidia-modprobe

The nvidia-modprobe utility is used to load NVIDIA kernel modules and create NVIDIA character device files automatically everytime your machine boots up.

Reboot your machine and verify everything works by issuing nvidia-smi or running deviceQuery in CUDA samples.

References
  1. How to do it manually
  2. Understanding character special file

More infomation related to NVIDIA driver and CUDA installation can be found here.

Install CUDA 7.5

CUDA 7.5 is provided by neither Ubuntu official nor CUDA repositories for Ubuntu 15.10. But we can still install it using .run file.

cd ~
wget http://developer.download.nvidia.com/compute/cuda/7.5/Prod/local_installers/cuda_7.5.18_linux.run
chmod +x cuda_7.5.18_linux.run
./cuda_7.5.18_linux.run --extract=/home/$USER
sudo ./cuda-linux64-rel-7.5.18-19867135.run

Follow the instructions there and configure runtime library after the installation finishes.

sudo bash -c "echo /usr/local/cuda/lib64/ > /etc/ld.so.conf.d/cuda.conf"
sudo ldconfig

Install all other dependencies

Follow instructions here

Download Caffe

cd ~
git clone https://github.com/BVLC/caffe.git

By now you should be unable to compile it due to serveral incompatibility problems. Let's fix them one by one.

Hack CUDA to support GCC 5

By default, CUDA 7.5 does not support compling with GCC 4.10 or later versions. It will report the following errors when compiling Caffe.

In file included from /usr/local/cuda/include/cuda_runtime.h:76:0,
                 from <command-line>:0:
/usr/local/cuda/include/host_config.h:115:2: error: #error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
 #error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
  ^

You can try install gcc-4.9 and g++-4.9 to solve this problem temporarily. But the linker will report undefined references errors later because all other libraries (protobuf,leveldb,opencv) in Ubuntu 15.10 are compiled with GCC version 5.2. They are not compatible.

To solve this problem we have to "hack" the CUDA toolkit by editing /usr/local/cuda/include/host_config.h. Comment line 115.

--- #error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
+++ //#error -- unsupported GNU version! gcc versions later than 4.9 are not supported!

Fix hdf5 naming problem

Your machine may report the following error when compiling Caffe even though libhdf5-serial-dev package has been already installed in it.

./include/caffe/util/hdf5.hpp:6:18: fatal error: hdf5.h: No such file or directory

This is because of change of default path and name of hdf5 head files and libraries in Ubuntu 15.10. To solve this problem, we can simply modify Makefile files.

Append /usr/include/hdf5/serial/ to INCLUDE_DIRS at line 85 in Makefile.config.

--- INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
+++ INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial/

Modify hdf5_hl and hdf5 to hdf5_serial_hl and hdf5_serial at line 173 in Makefile

--- LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_hl hdf5
+++ LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_serial_hl hdf5_serial

Compile and Test Caffe

By now, all incompatibility issues should have been solved. We can start to compile and test Caffe.

cd ~/caffe
make -j8
make test -j8
make runtest -j8

Hopefully all tests pass.

Notes

  1. Running sudo make runtest once may help you fix a lot of runtime errors until next reboot, although it is not a recommended way.
  2. You might need to append LC_ALL="en_US.UTF-8" to /etc/environment and reboot your machine if there are runtime errors related to locale::facet::_S_create_c_locale name not valid.
@catfishking

This comment has been minimized.

Show comment
Hide comment
@catfishking

catfishking Dec 19, 2015

This guide helps me a lot. It solves the undefined references error messages which I struggled for a whole night.
Thanks!

This guide helps me a lot. It solves the undefined references error messages which I struggled for a whole night.
Thanks!

@yysijie

This comment has been minimized.

Show comment
Hide comment
@yysijie

yysijie Dec 31, 2015

This guide helps me a lot. It solves the undefined references error messages which I struggled for a whole day.
Thanks!

yysijie commented Dec 31, 2015

This guide helps me a lot. It solves the undefined references error messages which I struggled for a whole day.
Thanks!

@zhouxidong

This comment has been minimized.

Show comment
Hide comment
@zhouxidong

zhouxidong Jan 11, 2016

tks a lot especially for the undefined references problem

tks a lot especially for the undefined references problem

@en4bz

This comment has been minimized.

Show comment
Hide comment
@en4bz

en4bz Jan 31, 2016

Works! Thanks a lot!

en4bz commented Jan 31, 2016

Works! Thanks a lot!

@rosenfeldamir

This comment has been minimized.

Show comment
Hide comment
@rosenfeldamir

rosenfeldamir Feb 5, 2016

Thanks a million !!!!
I've wasted too many hours on trying to figure the compilation errors caused by the gcc version. Instead of "hacking" cuda to support gcc5, I used gcc 4.8, which of course failed....
Now it finally works!

Thanks a million !!!!
I've wasted too many hours on trying to figure the compilation errors caused by the gcc version. Instead of "hacking" cuda to support gcc5, I used gcc 4.8, which of course failed....
Now it finally works!

@dangmanhtruong

This comment has been minimized.

Show comment
Hide comment
@dangmanhtruong

dangmanhtruong Feb 11, 2016

I made an account just to comment here. Man, you were a lifesaver! When I tried to install Caffe, i also ran into the hdf5 problem: "

fatal error: hdf5.h: No such file or directory

". I found some solutions on Google, but another problem then came up:

/usr/bin/ld: cannot find -lhdf5_hl
/usr/bin/ld: cannot find -lhdf5
collect2: error: ld returned 1 exit status
Makefile:545: recipe for target '.build_release/lib/libcaffe.so' failed
make: *** [.build_release/lib/libcaffe.so] Error

I spent days on end trying to find a solution but it was futile! Some posts suggested creating a symlink at usr/lib/x86_64-linux-gnu but the directory doesn't exist! . Luckily I found your post, hopefully other people won't have to go through all this anymore.
On a sidenote, i am still puzzled as to why Google didn't come up with this place sooner.......

I made an account just to comment here. Man, you were a lifesaver! When I tried to install Caffe, i also ran into the hdf5 problem: "

fatal error: hdf5.h: No such file or directory

". I found some solutions on Google, but another problem then came up:

/usr/bin/ld: cannot find -lhdf5_hl
/usr/bin/ld: cannot find -lhdf5
collect2: error: ld returned 1 exit status
Makefile:545: recipe for target '.build_release/lib/libcaffe.so' failed
make: *** [.build_release/lib/libcaffe.so] Error

I spent days on end trying to find a solution but it was futile! Some posts suggested creating a symlink at usr/lib/x86_64-linux-gnu but the directory doesn't exist! . Luckily I found your post, hopefully other people won't have to go through all this anymore.
On a sidenote, i am still puzzled as to why Google didn't come up with this place sooner.......

@matthieudelaro

This comment has been minimized.

Show comment
Hide comment
@matthieudelaro

matthieudelaro Mar 7, 2016

Awesome, thanks !

Awesome, thanks !

@navaneethkk

This comment has been minimized.

Show comment
Hide comment
@navaneethkk

navaneethkk Mar 8, 2016

You sir, are a life saver.

You sir, are a life saver.

@ternaus

This comment has been minimized.

Show comment
Hide comment

ternaus commented Mar 20, 2016

Thanks!

@yarenty

This comment has been minimized.

Show comment
Hide comment
@yarenty

yarenty Mar 24, 2016

Perfect! Thanks!

yarenty commented Mar 24, 2016

Perfect! Thanks!

@vodp

This comment has been minimized.

Show comment
Hide comment
@vodp

vodp Apr 4, 2016

Very useful note!

vodp commented Apr 4, 2016

Very useful note!

@neouyghur

This comment has been minimized.

Show comment
Hide comment
@neouyghur

neouyghur Apr 25, 2016

Thank you, you save my life.

Thank you, you save my life.

@donaldsmith2060

This comment has been minimized.

Show comment
Hide comment
@donaldsmith2060

donaldsmith2060 May 23, 2016

getting the following errors:

.build_release/lib/libcaffe.so: undefined reference to google::base::CheckOpMessageBuilder::NewString()' .build_release/lib/libcaffe.so: undefined reference togoogle::protobuf::internal::NameOfEnum(google::protobuf::EnumDescriptor const_, int)'
.build_release/lib/libcaffe.so: undefined reference to caffe::BlockingQueuecaffe::Batch<double_>::pop(std::string const&)' .build_release/lib/libcaffe.so: undefined reference tocaffe::BlockingQueuecaffe::Batch<float_>::pop(std::string const&)'
collect2: error: ld returned 1 exit status
Makefile:616: recipe for target '.build_release/tools/convert_imageset.bin' failed
make: *_* [.build_release/tools/convert_imageset.bin] Error 1

getting the following errors:

.build_release/lib/libcaffe.so: undefined reference to google::base::CheckOpMessageBuilder::NewString()' .build_release/lib/libcaffe.so: undefined reference togoogle::protobuf::internal::NameOfEnum(google::protobuf::EnumDescriptor const_, int)'
.build_release/lib/libcaffe.so: undefined reference to caffe::BlockingQueuecaffe::Batch<double_>::pop(std::string const&)' .build_release/lib/libcaffe.so: undefined reference tocaffe::BlockingQueuecaffe::Batch<float_>::pop(std::string const&)'
collect2: error: ld returned 1 exit status
Makefile:616: recipe for target '.build_release/tools/convert_imageset.bin' failed
make: *_* [.build_release/tools/convert_imageset.bin] Error 1

@rayiproov

This comment has been minimized.

Show comment
Hide comment
@rayiproov

rayiproov May 24, 2016

I'm trying to build OpenCV on 16.04 and getting the GNU GCC version error because 16.04 has GCC 5.3.1. I tried your hacking CUDA to support GCC 5, which got rid of that particular error, but now I'm getting:

/usr/include/string.h: In function 'void* mempcpy_inline(void, const void, size_t)':
/usr/include/string.h:652:42: error: 'memcpy' was not declared in this scope
return (char *) memcpy (__dest, __src, __n) + __n;

I'm trying to build OpenCV on 16.04 and getting the GNU GCC version error because 16.04 has GCC 5.3.1. I tried your hacking CUDA to support GCC 5, which got rid of that particular error, but now I'm getting:

/usr/include/string.h: In function 'void* mempcpy_inline(void, const void, size_t)':
/usr/include/string.h:652:42: error: 'memcpy' was not declared in this scope
return (char *) memcpy (__dest, __src, __n) + __n;

@donaldsmith2060

This comment has been minimized.

Show comment
Hide comment
@hiromotomasaki

This comment has been minimized.

Show comment
Hide comment
@hiromotomasaki

hiromotomasaki Jul 4, 2016

Thank you very much for your advice.
It went well.

Thank you very much for your advice.
It went well.

@tldrafael

This comment has been minimized.

Show comment
Hide comment
@tldrafael

tldrafael Jul 4, 2016

thanks, it works greatly

thanks, it works greatly

@naughtyfox

This comment has been minimized.

Show comment
Hide comment
@naughtyfox

naughtyfox Jul 15, 2016

I'm trying to build OpenCV on 16.04 and getting the GNU GCC version error because 16.04 has GCC 5.3.1. I tried your hacking CUDA to support GCC 5, which got rid of that particular error, but now I'm getting:

/usr/include/string.h: In function 'void* __mempcpy_inline(void, const void, size_t)':
/usr/include/string.h:652:42: error: 'memcpy' was not declared in this scope
return (char *) memcpy (__dest, __src, __n) + __n;

Hi! I was struggling the same problem while building Tensorflow under Ubuntu 16.04 with CUDA and the solution is (in case with Caffe) to add the following line in your Makefile.config:
CUSTOM_CXX := g++ -D_FORCE_INLINES
It worked for me and I finished building this hell under Ubuntu 16.04

I'm trying to build OpenCV on 16.04 and getting the GNU GCC version error because 16.04 has GCC 5.3.1. I tried your hacking CUDA to support GCC 5, which got rid of that particular error, but now I'm getting:

/usr/include/string.h: In function 'void* __mempcpy_inline(void, const void, size_t)':
/usr/include/string.h:652:42: error: 'memcpy' was not declared in this scope
return (char *) memcpy (__dest, __src, __n) + __n;

Hi! I was struggling the same problem while building Tensorflow under Ubuntu 16.04 with CUDA and the solution is (in case with Caffe) to add the following line in your Makefile.config:
CUSTOM_CXX := g++ -D_FORCE_INLINES
It worked for me and I finished building this hell under Ubuntu 16.04

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Aug 10, 2016

Works like a charm. Thanks a lot!

ghost commented Aug 10, 2016

Works like a charm. Thanks a lot!

@jjdevbiz

This comment has been minimized.

Show comment
Hide comment
@jjdevbiz

jjdevbiz Aug 11, 2016

i wish you much good fortune upon your head, kind sir

i wish you much good fortune upon your head, kind sir

@monajalal

This comment has been minimized.

Show comment
Hide comment
@monajalal

monajalal Aug 23, 2016

Hey, is this hack really feasible? To me it more sounds like we are ignoring a real error:
--- #error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
+++ //#error -- unsupported GNU version! gcc versions later than 4.9 are not supported!

Has worked so far for me but I am curious to know how ignoring the error would make it feasible?

monajalal commented Aug 23, 2016

Hey, is this hack really feasible? To me it more sounds like we are ignoring a real error:
--- #error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
+++ //#error -- unsupported GNU version! gcc versions later than 4.9 are not supported!

Has worked so far for me but I am curious to know how ignoring the error would make it feasible?

@YounYC

This comment has been minimized.

Show comment
Hide comment
@YounYC

YounYC Oct 17, 2016

Many thanks,
Now it is working! ^^

YounYC commented Oct 17, 2016

Many thanks,
Now it is working! ^^

@yukitsuji

This comment has been minimized.

Show comment
Hide comment
@yukitsuji

yukitsuji Oct 26, 2016

Thank you!

Thank you!

@hollownike

This comment has been minimized.

Show comment
Hide comment
@hollownike

hollownike Nov 29, 2016

Thanks for your post.
If anyone else is having problems with the locale::facet::_S_create_c_locale name not valid error and lacks root privileges needed to add something to /etc/environment as I did: You can run the tests by

env LC_ALL="en_US.UTF-8" make runtests

and this error should not occur. (Note: this does not set the variable outside of the used command)

Thanks for your post.
If anyone else is having problems with the locale::facet::_S_create_c_locale name not valid error and lacks root privileges needed to add something to /etc/environment as I did: You can run the tests by

env LC_ALL="en_US.UTF-8" make runtests

and this error should not occur. (Note: this does not set the variable outside of the used command)

@svanschalkwyk

This comment has been minimized.

Show comment
Hide comment
@svanschalkwyk

svanschalkwyk Jan 26, 2017

I'm getting this with make runtest
Cuda number of devices: 9406365
Setting to use device 0
Current device id: 0
Current device name:
F0126 13:15:38.632215 35566 gpu_memory.cpp:137] Check failed: error == cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version

Details:
g++ (Ubuntu 5.4.1-2ubuntu1~16.04) 5.4.1 20160904
MKL 2017 Update 1
Linux Description: Linux Mint 18.1 Serena
CUDA:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 378.09 Driver Version: 378.09 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 960 Off | 0000:04:00.0 On | N/A |
| 0% 25C P8 7W / 160W | 94MiB / 4032MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 760 Off | 0000:83:00.0 N/A | N/A |
| 42% 33C P8 N/A / N/A | 1MiB / 4036MiB | N/A Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2427 G /usr/lib/xorg/Xorg 72MiB |
| 0 2937 G cinnamon 20MiB |
| 1 Not Supported |
+-----------------------------------------------------------------------------+

svanschalkwyk commented Jan 26, 2017

I'm getting this with make runtest
Cuda number of devices: 9406365
Setting to use device 0
Current device id: 0
Current device name:
F0126 13:15:38.632215 35566 gpu_memory.cpp:137] Check failed: error == cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version

Details:
g++ (Ubuntu 5.4.1-2ubuntu1~16.04) 5.4.1 20160904
MKL 2017 Update 1
Linux Description: Linux Mint 18.1 Serena
CUDA:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 378.09 Driver Version: 378.09 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 960 Off | 0000:04:00.0 On | N/A |
| 0% 25C P8 7W / 160W | 94MiB / 4032MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 760 Off | 0000:83:00.0 N/A | N/A |
| 42% 33C P8 N/A / N/A | 1MiB / 4036MiB | N/A Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2427 G /usr/lib/xorg/Xorg 72MiB |
| 0 2937 G cinnamon 20MiB |
| 1 Not Supported |
+-----------------------------------------------------------------------------+

@remoteblue

This comment has been minimized.

Show comment
Hide comment
@remoteblue

remoteblue Mar 13, 2017

Section on hd5 really helps. Thanks!

Section on hd5 really helps. Thanks!

@adyscorpius

This comment has been minimized.

Show comment
Hide comment
@adyscorpius

adyscorpius Mar 20, 2017

Definitely a life saver. Got both the hdf5 and gcc errors and while I managed to fix the hdf5 error, the gcc error was fixed thanks to you.

Got caffe working on Ubuntu 16.10 finally..

Edit: I also had to change the links to gcc and g++

sudo rm /usr/bin/gcc
sudo rm /usr/bin/g++

ln -s /usr/bin/gcc-5 /usr/bin/gcc
ln -s /usr/bin/g++-5 /usr/bin/g++

adyscorpius commented Mar 20, 2017

Definitely a life saver. Got both the hdf5 and gcc errors and while I managed to fix the hdf5 error, the gcc error was fixed thanks to you.

Got caffe working on Ubuntu 16.10 finally..

Edit: I also had to change the links to gcc and g++

sudo rm /usr/bin/gcc
sudo rm /usr/bin/g++

ln -s /usr/bin/gcc-5 /usr/bin/gcc
ln -s /usr/bin/g++-5 /usr/bin/g++
@SonakshiGrover

This comment has been minimized.

Show comment
Hide comment
@SonakshiGrover

SonakshiGrover Apr 5, 2017

Thanks a lot for this post !!

Thanks a lot for this post !!

@IcedDoggie

This comment has been minimized.

Show comment
Hide comment
@IcedDoggie

IcedDoggie Apr 5, 2017

You deserve a star!

You deserve a star!

@mzaradzki

This comment has been minimized.

Show comment
Hide comment
@mzaradzki

mzaradzki Apr 21, 2017

I wish I could give you 5 stars, not just 1, clearly saved me a lot of time.

I wish I could give you 5 stars, not just 1, clearly saved me a lot of time.

@kyuhyong

This comment has been minimized.

Show comment
Hide comment
@kyuhyong

kyuhyong Jun 10, 2017

I am still getting build error. Why my ubuntu 14.04 cannot locate the libary correctly?

CXX examples/cifar10/convert_cifar_data.cpp
CXX examples/mnist/convert_mnist_data.cpp
CXX .build_release/src/caffe/proto/caffe.pb.cc
AR -o .build_release/lib/libcaffe.a
LD -o .build_release/lib/libcaffe.so.1.0.0-rc3
/usr/bin/ld: cannot find -lhdf5_serial_hl
/usr/bin/ld: cannot find -lhdf5_serial
/usr/bin/ld: cannot find -lopenblas
collect2: error: ld returned 1 exit status
make: *** [.build_release/lib/libcaffe.so.1.0.0-rc3] Error 1

I am still getting build error. Why my ubuntu 14.04 cannot locate the libary correctly?

CXX examples/cifar10/convert_cifar_data.cpp
CXX examples/mnist/convert_mnist_data.cpp
CXX .build_release/src/caffe/proto/caffe.pb.cc
AR -o .build_release/lib/libcaffe.a
LD -o .build_release/lib/libcaffe.so.1.0.0-rc3
/usr/bin/ld: cannot find -lhdf5_serial_hl
/usr/bin/ld: cannot find -lhdf5_serial
/usr/bin/ld: cannot find -lopenblas
collect2: error: ld returned 1 exit status
make: *** [.build_release/lib/libcaffe.so.1.0.0-rc3] Error 1
@herleeyandi

This comment has been minimized.

Show comment
Hide comment
@herleeyandi

herleeyandi Sep 8, 2017

Nice post. It help me a lot with hdf5 problem. However I am still facing this problem.

.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::WriteBytes(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, google::protobuf::io::CodedOutputStream*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::DescriptorPool::FindFileByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'

Anybody know how to fix this?
-Thank you-

Nice post. It help me a lot with hdf5 problem. However I am still facing this problem.

.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::internal::WireFormatLite::WriteBytes(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, google::protobuf::io::CodedOutputStream*)'
.build_release/lib/libcaffe.so: undefined reference to `google::protobuf::DescriptorPool::FindFileByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'

Anybody know how to fix this?
-Thank you-

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment