Building Tensorflow from source on Ubuntu 16.04LTS for maximum performance:
forked from https://gist.github.com/Brainiarc7/6d6c3f23ea057775b72c52817759b25c, edited by CCJ.
TensorFlow is now distributed under an Apache v2 open source license on GitHub.
On Ubuntu 16.04LTS+:
Step 1. Install NVIDIA CUDA:
To use TensorFlow with NVIDIA GPUs, the first step is to install the CUDA Toolkit as shown:
wget -c -v -nc https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.2.88-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1604_9.2.88-1_amd64.deb
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
sudo apt-get update
sudo apt-get install cuda
Keep checking the NVIDIA CUDA webpage for new releases as applicable. This article is accurate as at the time of writing.
Ensure that you have the latest driver:
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update && sudo apt-get -y upgrade
On Ubuntu 18.04LTS, this should be enough for the device driver:
sudo apt-get install nvidia-kernel-source-396 nvidia-driver-396
Failure to do this will result in a broken driver installation.
When done, create a library configuration file for cupti:
/etc/ld.so.conf.d/cupti.conf
With the content:
/usr/local/cuda/extras/CUPTI/lib64
Confirm that the library configuration file for CUDA libraries also exists with the correct settings:
/etc/ld.so.conf.d/cuda.conf
The content should be:
/usr/local/cuda/lib64
When done, load the new configuration:
sudo ldconfig -vvvv
Useful environment variables for CUDA:
Edit the /etc/environment file and append the following:
CUDA_HOME=/usr/local/cuda
Now, append the PATH variable with the following:
/usr/local/cuda/bin:$HOME/bin
When done, remember to source the file:
source /etc/environment
You can also install CUDA manually. However, take care not to install its' bundled driver.
Step 2. Install NVIDIA cuDNN:
Once the CUDA Toolkit is installed, download the latest cuDNNN Library for Linux, based on the CUDA version you're using. In this case, we're on CUDA 9.1, so we will refer to the version name below (note that you will need to register for the Accelerated Computing Developer Program).
Once downloaded, uncompress the files and copy them into the CUDA Toolkit directory (assumed here to be in /usr/local/cuda/ for Ubuntu 16.04LTS):
$ sudo tar -xvf cudnn-9.1-* -C /usr/local
Step 3. Install and upgrade PIP:
TensorFlow itself can be installed using the pip package manager. First, make sure that your system has pip installed and updated:
$ sudo apt-get install python-pip python-dev
$ pip install --upgrade pip
Step 4. Install Bazel: Step 4.1 Method 1 to install Bazel:
To build TensorFlow from source, the Bazel build system (and the latest available openjdk) must first be installed as follows.
$ sudo apt-get install software-properties-common swig
$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer
$ echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
$ curl https://storage.googleapis.com/bazel-apt/doc/apt-key.pub.gpg | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install bazel
Step 4.2 Method 2 to install Bazel using binary installer (CCJ used)
see https://docs.bazel.build/versions/master/install-ubuntu.html
see https://www.tensorflow.org/install/source
- Step 1: Install required packages
sudo apt-get install pkg-config zip g++ zlib1g-dev unzip python- Step 2: Download Bazel. To download the Bazel binary installer named bazel--installer-linux-x86_64.sh from the Bazel releases page on GitHub. Here I used
bazel-0.19.0-installer-linux-x86_64.sh(i.e., version == 0.19.0). - Step 3: Run the installer:
chmod +x bazel-<version>-installer-linux-x86_64.sh
./bazel-<version>-installer-linux-x86_64.sh --userThe --user flag installs Bazel to the $HOME/bin directory on your system and sets the .bazel path to $HOME/.bazel. Use the --help command to see additional installation options.
- Step 4: Set up your environment. If you ran the Bazel installer with the
--userflag as above, the Bazel executable is installed in your$HOME/bindirectory. It’s a good idea to add this directory to your default paths, as follows:
export PATH="$PATH:$HOME/bin"You can also add this command to your ~/.bashrc file.
Step 5. Install TensorFlow
To obtain the best performance with TensorFlow we recommend building it from source.
First, clone the TensorFlow source code repository:
$ git clone https://github.com/tensorflow/tensorflow
$ # or clone a specific Git branch via `git clone -b <branch> <remote_repo>`
$ git clone -b r1.9 https://github.com/tensorflow/tensorflow
$ cd tensorflow
The last step is no longer needed:
~~$ git reset --hard a23f5d7 ~~
Then run the configure script as follows:
$ ./configure
Output:
Please specify the location of python. [Default is /usr/bin/python]: [enter]
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] n
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with GPU support? [y/N] y
GPU support will be enabled for TensorFlow
Please specify which gcc nvcc should use as the host compiler. [Default is /usr/bin/gcc]: [enter]
Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 8.0
Please specify the location where CUDA 8.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: [enter]
Please specify the Cudnn version you want to use. [Leave empty to use system default]: 5
Please specify the location where cuDNN 5 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: [enter]
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]: 5.2,6.1 [see https://developer.nvidia.com/cuda-gpus]
Setting up Cuda include
Setting up Cuda lib64
Setting up Cuda bin
Setting up Cuda nvvm
Setting up CUPTI include
Setting up CUPTI lib64
Configuration finished
Then call bazel to build the TensorFlow pip package:
bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2 --config=cuda //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
This will build the package with optimizations for FMA, AVX and SSE.
A stock build would be as such:
bazel build //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
You can use the stock build as shown above if you had passed the configuration flags (for optimization) directly to the configure script above. Use this string:
--copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2
Which will replace --march=native (the default).
If you're on Skylake to Coffee lake, this is what you need.
Build the pip package : Bazel build
CCJ just used the following command, since during run
./configuremost of the flags have been set.
To build from a release branch:
$ bazel build --config=opt //tensorflow/tools/pip_package:build_pip_packageI met this problem : Build from source -> build the pip package -> GPU support -> bazel build -> ERROR: Config value cuda is not defined in any .rc file. The solution which works for me is:
build tensorflow 1.9 I have do this,in source file add context
import $HOME/Download/tensorflow/tools/bazel.rcon top line of (hide file) "$HOME/Download/tensorflow/.bazelrc", instead of the file.tf_configure.bazelrcwhich was mentioned in some other solutions.
Build the pip package : Build the pip packag
The bazel build command creates an executable named build_pip_package — this is the program that builds the pip package. Run the executable as shown below to build a .whl package in the /tmp/tensorflow_pkg directory.
And finally install the TensorFlow pip package
For Python 2.7:
$ sudo pip install --upgrade /tmp/tensorflow_pkg/tensorflow-*.whl
Python 3.4:
$ sudo pip3 install --upgrade /tmp/tensorflow_pkg/tensorflow-*.whl
I want to install the TensorFlow in the virtual environment:
So, I did the following:
$ source /home/ccj/tensorflow/bin/activate
(tensorflow) $ pip install /tmp/tensorflow_gpu_r1.9_pkg/tensorflow-1.9.0-cp35-cp35m-linux_x86_64.whlshown as in the image in this gist.
Step 5. Upgrade protobuf:
Upgrade to the latest version of the protobuf package:
For Python 2.7:
$ sudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/protobuf-3.0.0b2.post2-cp27-none-linux_x86_64.whl
For Python 3.4:
$ sudo pip3 install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/protobuf-3.0.0b2.post2-cp34-none-linux_x86_64.whl
Step 6. Test your installation:
To test the installation, open an interactive Python shell and import the TensorFlow module:
$ cd
$ python
>>> import tensorflow as tf
tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally
tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so locally
tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally
tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally
With the TensorFlow module imported, the next step to test the installation is to create a TensorFlow Session, which will initialize the available computing devices and provide a means of executing computation graphs:
>>> sess = tf.Session()
This command will print out some information on the detected hardware configuration. For example, the output on a system containing a Tesla M40 GPU is:
>>> sess = tf.Session()
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: Tesla M40
major: 5 minor: 2 memoryClockRate (GHz) 1.112
pciBusID 0000:04:00.0
Total memory: 11.25GiB
Free memory: 11.09GiB
To manually control which devices are visible to TensorFlow, set the CUDA_VISIBLE_DEVICES environment variable when launching Python. For example, to force the use of only GPU 0:
$ CUDA_VISIBLE_DEVICES=0 python
You should now be able to run a Hello World application:
>>> hello_world = tf.constant("Hello, TensorFlow!")
>>> print sess.run(hello_world)
Hello, TensorFlow!
>>> print sess.run(tf.constant(123)*tf.constant(456))
56088
This is the result I test the TensorFLow installation:
$ source /home/ccj/tensorflow/bin/activate
(tensorflow) $ python
>> import tensorflow as tf
>> tf.VERSION
'1.9.0'
>>> sess = tf.Session()
2019-03-20 21:15:30.764726: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:03:00.0
totalMemory: 10.91GiB freeMemory: 10.75GiB
2019-03-20 21:15:30.764779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-03-20 21:15:31.124212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-20 21:15:31.124271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-03-20 21:15:31.124283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-03-20 21:15:31.124656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with
10403 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1)
>>> hello_world = tf.constant("Hello, TensorFlow!")
>>> print (sess.run(hello_world))
b'Hello, TensorFlow!'
>>> print (sess.run(tf.constant(123)*tf.constant(456)))
56088The corresponding screenshot is shown as here.
Tips:
To achieve similar results without building the packages, you can deploy nvidia-docker and install tensorflow from NVIDIA's NGC registry.
Use this to deploy nvidia-docker on Ubuntu: https://gist.github.com/Brainiarc7/a8ab5f89494d053003454efc3be2d2ef
Use the NGC to deploy the preconfigured containers. Optimized builds for Tensorflow, Caffe, Torch, etc are also available: https://www.nvidia.com/en-us/gpu-cloud/deep-learning-containers/
Also see the NGC panel: https://ngc.nvidia.com/registry
This article tells
Done! And enjoy!