Instantly share code, notes, and snippets.

Embed
What would you like to do?
Build Tensorflow from source, for better performance on Ubuntu.

Building Tensorflow from source on Ubuntu 16.04LTS for maximum performance:

TensorFlow is now distributed under an Apache v2 open source license on GitHub.

On Ubuntu 16.04LTS+:

Step 1. Install NVIDIA CUDA:

To use TensorFlow with NVIDIA GPUs, the first step is to install the CUDA Toolkit as shown:

wget -c -v -nc https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.2.88-1_amd64.deb

sudo dpkg -i cuda-repo-ubuntu1604_9.2.88-1_amd64.deb

sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub

sudo apt-get update

sudo apt-get install cuda

Keep checking the NVIDIA CUDA webpage for new releases as applicable. This article is accurate as at the time of writing.

Ensure that you have the latest driver:


sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update && sudo apt-get -y upgrade

On Ubuntu 18.04LTS, this should be enough for the device driver:

sudo apt-get install nvidia-kernel-source-396 nvidia-driver-396

Failure to do this will result in a broken driver installation.

When done, create a library configuration file for cupti:

/etc/ld.so.conf.d/cupti.conf

With the content:

/usr/local/cuda/extras/CUPTI/lib64 

Confirm that the library configuration file for CUDA libraries also exists with the correct settings:

/etc/ld.so.conf.d/cuda.conf

The content should be:

/usr/local/cuda/lib64

When done, load the new configuration:

sudo ldconfig -vvvv

Useful environment variables for CUDA:

Edit the /etc/environment file and append the following:

CUDA_HOME=/usr/local/cuda

Now, append the PATH variable with the following:

/usr/local/cuda/bin:$HOME/bin

When done, remember to source the file:

source /etc/environment

You can also install CUDA manually. However, take care not to install its' bundled driver.

Step 2. Install NVIDIA cuDNN:

Once the CUDA Toolkit is installed, download the latest cuDNNN Library for Linux, based on the CUDA version you're using. In this case, we're on CUDA 9.1, so we will refer to the version name below (note that you will need to register for the Accelerated Computing Developer Program).

Once downloaded, uncompress the files and copy them into the CUDA Toolkit directory (assumed here to be in /usr/local/cuda/ for Ubuntu 16.04LTS):

$ sudo tar -xvf cudnn-9.1-* -C /usr/local

Step 3. Install and upgrade PIP:

TensorFlow itself can be installed using the pip package manager. First, make sure that your system has pip installed and updated:

$ sudo apt-get install python-pip python-dev
$ pip install --upgrade pip

Step 4. Install Bazel:

To build TensorFlow from source, the Bazel build system (and the latest available openjdk) must first be installed as follows.

$ sudo apt-get install software-properties-common swig
$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer
$ echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
$ curl https://storage.googleapis.com/bazel-apt/doc/apt-key.pub.gpg | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install bazel

Step 5. Install TensorFlow

To obtain the best performance with TensorFlow we recommend building it from source.

First, clone the TensorFlow source code repository:

$ git clone https://github.com/tensorflow/tensorflow
$ cd tensorflow

The last step is no longer needed:

$ git reset --hard a23f5d7 

Then run the configure script as follows:

$ ./configure

Output:

Please specify the location of python. [Default is /usr/bin/python]: [enter]
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] n
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with GPU support? [y/N] y
GPU support will be enabled for TensorFlow
Please specify which gcc nvcc should use as the host compiler. [Default is /usr/bin/gcc]: [enter]
Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 8.0
Please specify the location where CUDA 8.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: [enter]
Please specify the Cudnn version you want to use. [Leave empty to use system default]: 5
Please specify the location where cuDNN 5 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: [enter]
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]: 5.2,6.1 [see https://developer.nvidia.com/cuda-gpus]
Setting up Cuda include
Setting up Cuda lib64
Setting up Cuda bin
Setting up Cuda nvvm
Setting up CUPTI include
Setting up CUPTI lib64
Configuration finished

Then call bazel to build the TensorFlow pip package:

bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2 --config=cuda //tensorflow/tools/pip_package:build_pip_package


bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

This will build the package with optimizations for FMA, AVX and SSE.

A stock build would be as such:

bazel build //tensorflow/tools/pip_package:build_pip_package

bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

You can use the stock build as shown above if you had passed the configuration flags (for optimization) directly to the configure script above. Use this string:

--copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2

Which will replace --march=native (the default).

If you're on Skylake to Coffee lake, this is what you need.

And finally install the TensorFlow pip package

For Python 2.7:

$ sudo pip install --upgrade /tmp/tensorflow_pkg/tensorflow-*.whl

Python 3.4:

$ sudo pip install --upgrade /tmp/tensorflow_pkg/tensorflow-*.whl

Step 5. Upgrade protobuf:

Upgrade to the latest version of the protobuf package:

For Python 2.7:

$ sudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/protobuf-3.0.0b2.post2-cp27-none-linux_x86_64.whl

For Python 3.4:

$ sudo pip3 install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/protobuf-3.0.0b2.post2-cp34-none-linux_x86_64.whl

Step 6. Test your installation:

To test the installation, open an interactive Python shell and import the TensorFlow module:

   $ cd
   $ python


>>> import tensorflow as tf
tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally
tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so locally
tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally
tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally

With the TensorFlow module imported, the next step to test the installation is to create a TensorFlow Session, which will initialize the available computing devices and provide a means of executing computation graphs:

>>> sess = tf.Session()

This command will print out some information on the detected hardware configuration. For example, the output on a system containing a Tesla M40 GPU is:

>>> sess = tf.Session()
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: Tesla M40
major: 5 minor: 2 memoryClockRate (GHz) 1.112
pciBusID 0000:04:00.0
Total memory: 11.25GiB
Free memory: 11.09GiB

To manually control which devices are visible to TensorFlow, set the CUDA_VISIBLE_DEVICES environment variable when launching Python. For example, to force the use of only GPU 0:

$ CUDA_VISIBLE_DEVICES=0 python

You should now be able to run a Hello World application:

    >>> hello_world = tf.constant("Hello, TensorFlow!")
    >>> print sess.run(hello_world)
    Hello, TensorFlow!
    >>> print sess.run(tf.constant(123)*tf.constant(456))
    56088
    

Tips:

To achieve similar results without building the packages, you can deploy nvidia-docker and install tensorflow from NVIDIA's NGC registry.

Use this to deploy nvidia-docker on Ubuntu: https://gist.github.com/Brainiarc7/a8ab5f89494d053003454efc3be2d2ef

Use the NGC to deploy the preconfigured containers. Optimized builds for Tensorflow, Caffe, Torch, etc are also available: https://www.nvidia.com/en-us/gpu-cloud/deep-learning-containers/

Also see the NGC panel: https://ngc.nvidia.com/registry

@Nachiket2605

This comment has been minimized.

Show comment
Hide comment
@Nachiket2605

Nachiket2605 Jun 8, 2017

Hi, thanks a lot for the steps. I used expunge_async for the configuration which speeds up the process quite a bit. Unfortunately post configuration, I am taking way too long for the bagel to perform the build. Any suggestions on that?
Thanks!

Nachiket2605 commented Jun 8, 2017

Hi, thanks a lot for the steps. I used expunge_async for the configuration which speeds up the process quite a bit. Unfortunately post configuration, I am taking way too long for the bagel to perform the build. Any suggestions on that?
Thanks!

@Brainiarc7

This comment has been minimized.

Show comment
Hide comment
@Brainiarc7

Brainiarc7 Jun 12, 2017

Hey there @Nachiket2605,

That's most likely dependent on your storage type (SSD vs HDD) and the I/O scheduler in use.

Also, what are the specifications for the build machine?

Owner

Brainiarc7 commented Jun 12, 2017

Hey there @Nachiket2605,

That's most likely dependent on your storage type (SSD vs HDD) and the I/O scheduler in use.

Also, what are the specifications for the build machine?

@dydokamil

This comment has been minimized.

Show comment
Hide comment
@dydokamil

dydokamil Jun 17, 2017

When a new version of Tensorflow/CUDA/cudnn comes out do you have to rebuild everything again?

dydokamil commented Jun 17, 2017

When a new version of Tensorflow/CUDA/cudnn comes out do you have to rebuild everything again?

@Brainiarc7

This comment has been minimized.

Show comment
Hide comment
@Brainiarc7

Brainiarc7 Jun 26, 2017

Hello there,

You'd have to rebuild from source again.

Remember that the binary versions are compiled with generic optimization flags, a case that produces unoptimized binaries.

So, if you;re on Skylake, for example, you won't get AVX - enabled builds, for example.

Owner

Brainiarc7 commented Jun 26, 2017

Hello there,

You'd have to rebuild from source again.

Remember that the binary versions are compiled with generic optimization flags, a case that produces unoptimized binaries.

So, if you;re on Skylake, for example, you won't get AVX - enabled builds, for example.

@rlegithub

This comment has been minimized.

Show comment
Hide comment
@rlegithub

rlegithub Aug 4, 2017

I ran the ./configure step and hit this error from bazel:
WARNING: Output base '/usr2/username/.cache/bazel/_bazel_username/920d9f5954f35059efc25c5cf6e13b64' is on NFS. This may lead to surprising failures and undetermined behavior.
Extracting Bazel installation...

Googled online pointed out that if I can pass the following option, it would solve the problem: --output_user_root=/local/mnt/workspace/tmp/bazel/

Do you know how could I pass the --output_user_root option to bazel when calling the ./configure?

Thanks
RLE

rlegithub commented Aug 4, 2017

I ran the ./configure step and hit this error from bazel:
WARNING: Output base '/usr2/username/.cache/bazel/_bazel_username/920d9f5954f35059efc25c5cf6e13b64' is on NFS. This may lead to surprising failures and undetermined behavior.
Extracting Bazel installation...

Googled online pointed out that if I can pass the following option, it would solve the problem: --output_user_root=/local/mnt/workspace/tmp/bazel/

Do you know how could I pass the --output_user_root option to bazel when calling the ./configure?

Thanks
RLE

@Brainiarc7

This comment has been minimized.

Show comment
Hide comment
@Brainiarc7

Brainiarc7 Aug 5, 2017

Hello there,

It'd be ideal to pass that to the Bazel build step, unless its' failing at the ./configure phase:

bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2 --config=cuda //tensorflow/tools/pip_package:build_pip_package --output_user_root={your_path}

Owner

Brainiarc7 commented Aug 5, 2017

Hello there,

It'd be ideal to pass that to the Bazel build step, unless its' failing at the ./configure phase:

bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2 --config=cuda //tensorflow/tools/pip_package:build_pip_package --output_user_root={your_path}

@joe-of-all-trades

This comment has been minimized.

Show comment
Hide comment
@joe-of-all-trades

joe-of-all-trades Sep 22, 2017

Hi, I'm wondering how much improvement over official Tensorflow release with GPU support you can get by doing this ? Why would this be better than the official release ?

joe-of-all-trades commented Sep 22, 2017

Hi, I'm wondering how much improvement over official Tensorflow release with GPU support you can get by doing this ? Why would this be better than the official release ?

@Brainiarc7

This comment has been minimized.

Show comment
Hide comment
@Brainiarc7

Brainiarc7 Oct 2, 2017

This build improves on supporting native processor instructions on your platform.

The default build does not, unless built from source.

Owner

Brainiarc7 commented Oct 2, 2017

This build improves on supporting native processor instructions on your platform.

The default build does not, unless built from source.

@hamadazahera

This comment has been minimized.

Show comment
Hide comment
@hamadazahera

hamadazahera Oct 12, 2017

Hi,

I got this error while running ./configure file :
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
........
ERROR:

/home/hamadazahera/.cache/bazel/_bazel_hamadazahera/1d40efb9cd64d5ba27f112122670e970/external/io_bazel_rules_closure/closure/private/defs.bzl:27:16: The set constructor for depsets is deprecated and will be removed. Please use the depset constructor instead. You can temporarily enable the deprecated set constructor by passing the flag --incompatible_disallow_set_constructor=false
ERROR: error loading package '': Extension file 'closure/private/defs.bzl' has errors
ERROR: error loading package '': Extension file 'closure/private/defs.bzl' has errors
Building: no action running


Could you please help me to fix this error ? I am using Bazel 0.6.1 on Ubuntu 17

hamadazahera commented Oct 12, 2017

Hi,

I got this error while running ./configure file :
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
........
ERROR:

/home/hamadazahera/.cache/bazel/_bazel_hamadazahera/1d40efb9cd64d5ba27f112122670e970/external/io_bazel_rules_closure/closure/private/defs.bzl:27:16: The set constructor for depsets is deprecated and will be removed. Please use the depset constructor instead. You can temporarily enable the deprecated set constructor by passing the flag --incompatible_disallow_set_constructor=false
ERROR: error loading package '': Extension file 'closure/private/defs.bzl' has errors
ERROR: error loading package '': Extension file 'closure/private/defs.bzl' has errors
Building: no action running


Could you please help me to fix this error ? I am using Bazel 0.6.1 on Ubuntu 17

@wedesoft

This comment has been minimized.

Show comment
Hide comment
@wedesoft

wedesoft Nov 3, 2017

Thanks a lot. That should be the Tensorflow main page.

wedesoft commented Nov 3, 2017

Thanks a lot. That should be the Tensorflow main page.

@Brainiarc7

This comment has been minimized.

Show comment
Hide comment
@Brainiarc7
Owner

Brainiarc7 commented Nov 20, 2017

You're welcome, @wedesoft

@Brainiarc7

This comment has been minimized.

Show comment
Hide comment
@Brainiarc7

Brainiarc7 Nov 20, 2017

@hamadazahera,

Consider re-fetching your source.

Owner

Brainiarc7 commented Nov 20, 2017

@hamadazahera,

Consider re-fetching your source.

@Brainiarc7

This comment has been minimized.

Show comment
Hide comment
@Brainiarc7
Owner

Brainiarc7 commented Nov 23, 2017

Thanks @wedesoft

@matheusportela

This comment has been minimized.

Show comment
Hide comment
@matheusportela

matheusportela Dec 4, 2017

Great tutorial, @Brainiarc7. Just take a look at the formatting in step 6 because it is a bit misconfigured.

matheusportela commented Dec 4, 2017

Great tutorial, @Brainiarc7. Just take a look at the formatting in step 6 because it is a bit misconfigured.

@RyanRosario

This comment has been minimized.

Show comment
Hide comment
@RyanRosario

RyanRosario Dec 15, 2017

Heads up for others. I had a lot of problems using bazel to build TensorFlow. No matter what I did, it seemed that bazel could not find libcublas.so.9.0.

I did two things at the same time and unfortunately I do not know which one solved it. Pretty sure it was:
sudo echo "/usr/local/cuda-9.0/lib64" > /etc/ld.so.conf.d/cuda.conf
sudo ldconfig

from https://devtalk.nvidia.com/default/topic/845363/libcublas-so-7-0-cannot-open-shared-object-file/

I also ran bazel with sudo. One of them made it work.

RyanRosario commented Dec 15, 2017

Heads up for others. I had a lot of problems using bazel to build TensorFlow. No matter what I did, it seemed that bazel could not find libcublas.so.9.0.

I did two things at the same time and unfortunately I do not know which one solved it. Pretty sure it was:
sudo echo "/usr/local/cuda-9.0/lib64" > /etc/ld.so.conf.d/cuda.conf
sudo ldconfig

from https://devtalk.nvidia.com/default/topic/845363/libcublas-so-7-0-cannot-open-shared-object-file/

I also ran bazel with sudo. One of them made it work.

@Brainiarc7

This comment has been minimized.

Show comment
Hide comment
@Brainiarc7

Brainiarc7 Dec 16, 2017

Hello @RyanRosario,

That step you mentioned is mandatory for all projects that require access to the CUDA shared libraries and runtimes, and should be a part of installing the CUDA SDK documentation.

On distributions where a repository based installation method is provided, that step may even be done for you automatically. Ubuntu, for one, does this. However, with manual installation methods, be it from the binary installer for a single user or for multiple users on a HPC cluster (for loading via environment-modules , that step is mandatory.

Owner

Brainiarc7 commented Dec 16, 2017

Hello @RyanRosario,

That step you mentioned is mandatory for all projects that require access to the CUDA shared libraries and runtimes, and should be a part of installing the CUDA SDK documentation.

On distributions where a repository based installation method is provided, that step may even be done for you automatically. Ubuntu, for one, does this. However, with manual installation methods, be it from the binary installer for a single user or for multiple users on a HPC cluster (for loading via environment-modules , that step is mandatory.

@MarkSonn

This comment has been minimized.

Show comment
Hide comment
@MarkSonn

MarkSonn Jan 2, 2018

Thanks for the great read @Brainiarc7

I was just wondering how you find out what the best Bazel (or other) optimisations are for any given hardware setup. Also, are these optimisations just for the CPU or also for other components such as the GPU too?

Here's my system:

  • GPU: 2x NVIDIA GTX 1080 Ti
  • CPU: Intel i7-6850K
  • RAM: 64GB
  • OS: Ubuntu 16.04

MarkSonn commented Jan 2, 2018

Thanks for the great read @Brainiarc7

I was just wondering how you find out what the best Bazel (or other) optimisations are for any given hardware setup. Also, are these optimisations just for the CPU or also for other components such as the GPU too?

Here's my system:

  • GPU: 2x NVIDIA GTX 1080 Ti
  • CPU: Intel i7-6850K
  • RAM: 64GB
  • OS: Ubuntu 16.04
@Brainiarc7

This comment has been minimized.

Show comment
Hide comment
@Brainiarc7

Brainiarc7 Jan 5, 2018

Hello @MarkSonn,

These optimizations are to enable native instruction support for the compiler running on the host when building the project.

For example, with the stock binaries Google builds, they lack AVX support because their default build target is for a generic processor without these instructions. However, most modern CPUs beyond Skylake have support for AVX, etc, and depending on the workload type, binaries compiled with these instructions enabled will run faster.

For the GPU, it's ideal to target the SM level of your card's CUDA architectures, and to build as few of these PTX targets as possible.

Run the configuration as explained here: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/docs_src/install/install_sources.md

Owner

Brainiarc7 commented Jan 5, 2018

Hello @MarkSonn,

These optimizations are to enable native instruction support for the compiler running on the host when building the project.

For example, with the stock binaries Google builds, they lack AVX support because their default build target is for a generic processor without these instructions. However, most modern CPUs beyond Skylake have support for AVX, etc, and depending on the workload type, binaries compiled with these instructions enabled will run faster.

For the GPU, it's ideal to target the SM level of your card's CUDA architectures, and to build as few of these PTX targets as possible.

Run the configuration as explained here: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/docs_src/install/install_sources.md

@MarkSonn

This comment has been minimized.

Show comment
Hide comment
@MarkSonn

MarkSonn Jan 7, 2018

These optimizations are to enable native instruction support for the compiler running on the host when building the project.

binaries compiled with these instructions enabled will run faster.

This confused me a bit, do these optimisations speed up training/inference, or do they only affect the build time of Tensorflow with Bazel?


What are PTX targets? Are they the Bazel flags such as "--copt=-mavx" or the config options such as "Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] n"?


it's ideal to target the SM level of your card's CUDA architectures

Also, how do you do this?


Sorry for asking so many questions, I really appreciate your help :)

MarkSonn commented Jan 7, 2018

These optimizations are to enable native instruction support for the compiler running on the host when building the project.

binaries compiled with these instructions enabled will run faster.

This confused me a bit, do these optimisations speed up training/inference, or do they only affect the build time of Tensorflow with Bazel?


What are PTX targets? Are they the Bazel flags such as "--copt=-mavx" or the config options such as "Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] n"?


it's ideal to target the SM level of your card's CUDA architectures

Also, how do you do this?


Sorry for asking so many questions, I really appreciate your help :)

@Brainiarc7

This comment has been minimized.

Show comment
Hide comment
@Brainiarc7

Brainiarc7 Jan 8, 2018

Hey @MarkSonn,

These optimizations will speed up training performance. However, that also increases the initial build time. To what extent depends on your exact setup (Processor, compiler versions, etc).

CUDA PTX targets are dictated by options passed to the NVCC compiler, which are offered as part of the configuration step when building the project. Refer to this document for more details: http://docs.nvidia.com/cuda/parallel-thread-execution/index.html

By default, NVCC will automatically pick up one PTX version targeted to your GPU's SM architecture, such as 6.1 for Pascal, etc. Only override this if you're building the target for a different host.

For further GCC optimizations (use with care), see https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#Optimize-Options

Owner

Brainiarc7 commented Jan 8, 2018

Hey @MarkSonn,

These optimizations will speed up training performance. However, that also increases the initial build time. To what extent depends on your exact setup (Processor, compiler versions, etc).

CUDA PTX targets are dictated by options passed to the NVCC compiler, which are offered as part of the configuration step when building the project. Refer to this document for more details: http://docs.nvidia.com/cuda/parallel-thread-execution/index.html

By default, NVCC will automatically pick up one PTX version targeted to your GPU's SM architecture, such as 6.1 for Pascal, etc. Only override this if you're building the target for a different host.

For further GCC optimizations (use with care), see https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#Optimize-Options

@Brainiarc7

This comment has been minimized.

Show comment
Hide comment
@Brainiarc7

Brainiarc7 Jan 8, 2018

And for available PTX targets depending on your CUDA compiler and the installed GPU, please refer to NVIDIA's documentation here: https://developer.nvidia.com/cuda-gpus

Owner

Brainiarc7 commented Jan 8, 2018

And for available PTX targets depending on your CUDA compiler and the installed GPU, please refer to NVIDIA's documentation here: https://developer.nvidia.com/cuda-gpus

@tim37021

This comment has been minimized.

Show comment
Hide comment
@tim37021

tim37021 Mar 19, 2018

Have everyone tried this with r1.2?

I got the following error, however incompatible_disallow_uncalled_set_constructor does not work for me. it still output the same error

/home/tim/.cache/bazel/_bazel_tim/439bea30bd5a24f814bf00e0ff130e68/external/io_bazel_rules_closure/closure/stylesheets/closure_css_library.bzl:27:13: The function 'set' has been removed in favor of 'depset', please use the latter. You can temporarily refer to the old 'set' constructor from unexecuted code by using --incompatible_disallow_uncalled_set_constructor=false

tim37021 commented Mar 19, 2018

Have everyone tried this with r1.2?

I got the following error, however incompatible_disallow_uncalled_set_constructor does not work for me. it still output the same error

/home/tim/.cache/bazel/_bazel_tim/439bea30bd5a24f814bf00e0ff130e68/external/io_bazel_rules_closure/closure/stylesheets/closure_css_library.bzl:27:13: The function 'set' has been removed in favor of 'depset', please use the latter. You can temporarily refer to the old 'set' constructor from unexecuted code by using --incompatible_disallow_uncalled_set_constructor=false

@eddywm

This comment has been minimized.

Show comment
Hide comment
@eddywm

eddywm Apr 15, 2018

I am getting this error

$HOME/.cache/bazel/_bazel_eddwm/a970f6278411c8a88863559be597c15c/external/io_bazel_rules_closure/closure/stylesheets/closure_css_library.bzl:27:13: The function 'set' has been removed in favor of 'depset', please use the latter. You can temporarily refer to the old 'set' constructor from unexecuted code by using --incompatible_disallow_uncalled_set_constructor=false
The same error as @tim37021

eddywm commented Apr 15, 2018

I am getting this error

$HOME/.cache/bazel/_bazel_eddwm/a970f6278411c8a88863559be597c15c/external/io_bazel_rules_closure/closure/stylesheets/closure_css_library.bzl:27:13: The function 'set' has been removed in favor of 'depset', please use the latter. You can temporarily refer to the old 'set' constructor from unexecuted code by using --incompatible_disallow_uncalled_set_constructor=false
The same error as @tim37021

@peshmerge

This comment has been minimized.

Show comment
Hide comment
@peshmerge

peshmerge May 14, 2018

Thanks for this gist! But one thing for installing bazel on Ubunut 18.04 you will need to specify the arch deb [arch=amd64]
The command should looks like this:
echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
https://docs.bazel.build/versions/master/install-ubuntu.html

peshmerge commented May 14, 2018

Thanks for this gist! But one thing for installing bazel on Ubunut 18.04 you will need to specify the arch deb [arch=amd64]
The command should looks like this:
echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
https://docs.bazel.build/versions/master/install-ubuntu.html

@Brainiarc7

This comment has been minimized.

Show comment
Hide comment
@Brainiarc7

Brainiarc7 May 16, 2018

You're correct. I'll update the documentation.

Also, if you'd prefer not to build from source but want similar performance, deploy the instances from NVIDIA's NGC cloud.

See how to get started on nvidia-docker here: https://gist.github.com/Brainiarc7/a8ab5f89494d053003454efc3be2d2ef

Owner

Brainiarc7 commented May 16, 2018

You're correct. I'll update the documentation.

Also, if you'd prefer not to build from source but want similar performance, deploy the instances from NVIDIA's NGC cloud.

See how to get started on nvidia-docker here: https://gist.github.com/Brainiarc7/a8ab5f89494d053003454efc3be2d2ef

@pascalwhoop

This comment has been minimized.

Show comment
Hide comment
@pascalwhoop

pascalwhoop Jun 15, 2018

For anyone coming here and wondering "how much longer" it will take:

I just built TF for Cuda 9.2 on my i7 quad core 7700k @4.4ghz with 16GB

$ bazel build --config=opt --config=cuda --action_env PATH="$PATH" //tensorflow/tools/pip_package:build_pip_package
INFO: Elapsed time: 2900.785s, Critical Path: 126.77s
INFO: 7823 processes, local.
INFO: Build completed successfully, 10065 total actions

took about 50 minutes. I'lll install this and then try again with some optimisations to see the difference in build time and performance

EDIT 1:

Cannot load tensorflow:

Python 3.6.5 (default, May 11 2018, 04:00:52) 
[GCC 8.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/__init__.py", line 22, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 81, in <module>
    from tensorflow.python import keras
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/__init__.py", line 24, in <module>
    from tensorflow.python.keras import activations
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/activations/__init__.py", line 22, in <module>
    from tensorflow.python.keras._impl.keras.activations import elu
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/__init__.py", line 21, in <module>
    from tensorflow.python.keras._impl.keras import activations
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/activations.py", line 23, in <module>
    from tensorflow.python.keras._impl.keras import backend as K
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/backend.py", line 38, in <module>
    from tensorflow.python.layers import base as tf_base_layers
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 25, in <module>
    from tensorflow.python.keras.engine import base_layer
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/__init__.py", line 21, in <module>
    from tensorflow.python.keras.engine.base_layer import InputSpec
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 33, in <module>
    from tensorflow.python.keras import backend
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/backend/__init__.py", line 22, in <module>
    from tensorflow.python.keras._impl.keras.backend import abs
ImportError: cannot import name 'abs'

I have protobuf 3.6.0, built TF 1.9 with CUDA compute 6.1. Will continue to investigate.

related issue

Working now! I had to manually delete the tensorflow folder inside my site-packages folder of the virtualenv. 1.8.0 -> 1.9.0 changed the location of the keras files massively and there was something still cached somewhere. If this doesn't help, also delete the __pycache__ file, that might also help.

pascalwhoop commented Jun 15, 2018

For anyone coming here and wondering "how much longer" it will take:

I just built TF for Cuda 9.2 on my i7 quad core 7700k @4.4ghz with 16GB

$ bazel build --config=opt --config=cuda --action_env PATH="$PATH" //tensorflow/tools/pip_package:build_pip_package
INFO: Elapsed time: 2900.785s, Critical Path: 126.77s
INFO: 7823 processes, local.
INFO: Build completed successfully, 10065 total actions

took about 50 minutes. I'lll install this and then try again with some optimisations to see the difference in build time and performance

EDIT 1:

Cannot load tensorflow:

Python 3.6.5 (default, May 11 2018, 04:00:52) 
[GCC 8.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/__init__.py", line 22, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 81, in <module>
    from tensorflow.python import keras
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/__init__.py", line 24, in <module>
    from tensorflow.python.keras import activations
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/activations/__init__.py", line 22, in <module>
    from tensorflow.python.keras._impl.keras.activations import elu
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/__init__.py", line 21, in <module>
    from tensorflow.python.keras._impl.keras import activations
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/activations.py", line 23, in <module>
    from tensorflow.python.keras._impl.keras import backend as K
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/backend.py", line 38, in <module>
    from tensorflow.python.layers import base as tf_base_layers
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 25, in <module>
    from tensorflow.python.keras.engine import base_layer
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/__init__.py", line 21, in <module>
    from tensorflow.python.keras.engine.base_layer import InputSpec
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 33, in <module>
    from tensorflow.python.keras import backend
  File "/home/pascalwhoop/Documents/Code/University/powerTAC/python-agent/venv/lib/python3.6/site-packages/tensorflow/python/keras/backend/__init__.py", line 22, in <module>
    from tensorflow.python.keras._impl.keras.backend import abs
ImportError: cannot import name 'abs'

I have protobuf 3.6.0, built TF 1.9 with CUDA compute 6.1. Will continue to investigate.

related issue

Working now! I had to manually delete the tensorflow folder inside my site-packages folder of the virtualenv. 1.8.0 -> 1.9.0 changed the location of the keras files massively and there was something still cached somewhere. If this doesn't help, also delete the __pycache__ file, that might also help.

@dwSun

This comment has been minimized.

Show comment
Hide comment
@dwSun

dwSun Jun 26, 2018

Is it possible to build multpile tensorflow versions with a config script without the interactive configure progress?

dwSun commented Jun 26, 2018

Is it possible to build multpile tensorflow versions with a config script without the interactive configure progress?

@Brainiarc7

This comment has been minimized.

Show comment
Hide comment
@Brainiarc7

Brainiarc7 Jun 28, 2018

Hey @dwSun,

I'll research on that later.

Owner

Brainiarc7 commented Jun 28, 2018

Hey @dwSun,

I'll research on that later.

@Brainiarc7

This comment has been minimized.

Show comment
Hide comment
@Brainiarc7

Brainiarc7 Jun 30, 2018

Hello @dwSun,

To automate the build via a script, some environment variables must be initialized (so as to respond to the interactive configuration utility), such as these shown here: https://gist.github.com/PatWie/0c915d5be59a518f934392219ca65c3d

All credit goes to @PatWie

Owner

Brainiarc7 commented Jun 30, 2018

Hello @dwSun,

To automate the build via a script, some environment variables must be initialized (so as to respond to the interactive configuration utility), such as these shown here: https://gist.github.com/PatWie/0c915d5be59a518f934392219ca65c3d

All credit goes to @PatWie

@third-meow

This comment has been minimized.

Show comment
Hide comment
@third-meow

third-meow Sep 4, 2018

Hello

thanks for this guide, this worked for me :)
Can I delete the source folder and package folder after pip installing?

third-meow commented Sep 4, 2018

Hello

thanks for this guide, this worked for me :)
Can I delete the source folder and package folder after pip installing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment