Skip to content

Instantly share code, notes, and snippets.

@erikbern
Last active November 24, 2021 13:56
Embed
What would you like to do?
Installing TensorFlow on EC2
# Note – this is not a bash script (some of the steps require reboot)
# I named it .sh just so Github does correct syntax highlighting.
#
# This is also available as an AMI in us-east-1 (virginia): ami-cf5028a5
#
# The CUDA part is mostly based on this excellent blog post:
# http://tleyden.github.io/blog/2014/10/25/cuda-6-dot-5-on-aws-gpu-instance-running-ubuntu-14-dot-04/
# Install various packages
sudo apt-get update
sudo apt-get upgrade -y # choose “install package maintainers version”
sudo apt-get install -y build-essential python-pip python-dev git python-numpy swig python-dev default-jdk zip zlib1g-dev
# Blacklist Noveau which has some kind of conflict with the nvidia driver
echo -e "blacklist nouveau\nblacklist lbm-nouveau\noptions nouveau modeset=0\nalias nouveau off\nalias lbm-nouveau off\n" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
sudo update-initramfs -u
sudo reboot # Reboot (annoying you have to do this in 2015!)
# Some other annoying thing we have to do
sudo apt-get install -y linux-image-extra-virtual
sudo reboot # Not sure why this is needed
# Install latest Linux headers
sudo apt-get install -y linux-source linux-headers-`uname -r`
# Install CUDA 7.0 (note – don't use any other version)
wget http://developer.download.nvidia.com/compute/cuda/7_0/Prod/local_installers/cuda_7.0.28_linux.run
chmod +x cuda_7.0.28_linux.run
./cuda_7.0.28_linux.run -extract=`pwd`/nvidia_installers
cd nvidia_installers
sudo ./NVIDIA-Linux-x86_64-346.46.run
sudo modprobe nvidia
sudo ./cuda-linux64-rel-7.0.28-19326674.run
cd
# Install CUDNN 6.5 (note – don't use any other version)
# YOU NEED TO SCP THIS ONE FROM SOMEWHERE ELSE – it's not available online.
# You need to register and get approved to get a download link. Very annoying.
tar -xzf cudnn-6.5-linux-x64-v2.tgz
sudo cp cudnn-6.5-linux-x64-v2/libcudnn* /usr/local/cuda/lib64
sudo cp cudnn-6.5-linux-x64-v2/cudnn.h /usr/local/cuda/include/
# At this point the root mount is getting a bit full
# I had a lot of issues where the disk would fill up and then Bazel would end up in this weird state complaining about random things
# Make sure you don't run out of disk space when building Tensorflow!
sudo mkdir /mnt/tmp
sudo chmod 777 /mnt/tmp
sudo rm -rf /tmp
sudo ln -s /mnt/tmp /tmp
# Note that /mnt is not saved when building an AMI, so don't put anything crucial on it
# Install Bazel
cd /mnt/tmp
git clone https://github.com/bazelbuild/bazel.git
cd bazel
git checkout tags/0.1.0
./compile.sh
sudo cp output/bazel /usr/bin
# Install TensorFlow
cd /mnt/tmp
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64"
export CUDA_HOME=/usr/local/cuda
git clone --recurse-submodules https://github.com/tensorflow/tensorflow
cd tensorflow
# Patch to support older K520 devices on AWS
# wget "https://gist.githubusercontent.com/infojunkie/cb6d1a4e8bf674c6e38e/raw/5e01e5b2b1f7afd3def83810f8373fbcf6e47e02/cuda_30.patch"
# git apply cuda_30.patch
# According to https://github.com/tensorflow/tensorflow/issues/25#issuecomment-156234658 this patch is no longer needed
# Instead, you need to run ./configure like below (not tested yet)
TF_UNOFFICIAL_SETTING=1 ./configure
bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer
# Build Python package
# Note: you have to specify --config=cuda here - this is not mentioned in the official docs
# https://github.com/tensorflow/tensorflow/issues/25#issuecomment-156173717
bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
sudo pip install /tmp/tensorflow_pkg/tensorflow-0.5.0-cp27-none-linux_x86_64.whl
# Test it!
cd tensorflow/models/image/cifar10/
python cifar10_multi_gpu_train.py
# On a g2.2xlarge: step 100, loss = 4.50 (325.2 examples/sec; 0.394 sec/batch)
# On a g2.8xlarge: step 100, loss = 4.49 (337.9 examples/sec; 0.379 sec/batch)
# doesn't seem like it is able to use the 4 GPU cards unfortunately :(
@hammer
Copy link

hammer commented Dec 16, 2015

Do you need python-dev twice on line 12?

@babakbehzad
Copy link

Yes, @hammer is right, python-dev is needed or the new python package will fail with a could not find <Python.h> error.

@marcotrombetti
Copy link

Hi,

Thanks for the great work.

I am trying to compare CPU / GPU and different hw and I am getting this:

Macbook Pro i5 2,6Ghz (cifar10_train.py)
2015-12-24 23:33:13.533470: step 50, loss = 4.58 (173.1 examples/sec; 0.739 sec/batch)

AWS g2.2xlarge GPU $2/hour - cifar10_multi_gpu_train.py
(Creating TensorFlow device (/gpu:0) -> (device: 0, name: GRID K520...)
2015-12-24 22:15:16.011190: step 50, loss = 4.59 (322.1 examples/sec; 0.397 sec/batch)

AWS g2.2xlarge NO GPU $2/hour - cifar10_multi_gpu_train.py
(Ignoring gpu device (device: 0, name: GRID K520... )
2015-12-24 22:00:05.110064: step 50, loss = 4.59 (254.6 examples/sec; 0.503 sec/batch)

Is it normal that my Mac is less than 2x slower than a g2.2xlarge that uses a GPU? I has expecting 10x...

@springcoil
Copy link

These were very useful Erik - I finally got around to using tensor flow today

@taion
Copy link

taion commented Dec 29, 2015

I'm seeing the same performance numbers as @marcotrombetti on g2.2xlarge instances, both on GPU and on CPU. This seems to be many times slower than Theano on the same hardware when running on GPU. Is this expected, or is this indicative of some misconfiguration on my side?

@closedLoop
Copy link

To correct Line 88 above, it CAN use all four cores

Performance Attributes

All measured at step = 50

Instance Type Num GPUs Examples / Sec Sec / Batch
g2.2xlarge 1 216.0 0.593
g2.2xlarge 1 225.2 0.568
g2.8xlarge 4 675.2 0.190

Notes

To run with 4 cores call example with --num_gpus = 4

Very important you set Compute Capability = 3.0 (thanks @wfbradley)

If you pulled the latest tensorflow version 0.6 you need to change Line 80
sudo pip install /tmp/tensorflow_pkg/tensorflow-0.6.0-cp27-none-linux_x86_64.whl

However for me the led to segfaults in the example due to an issue with the Eigen Kernel. This has temporarily been resolved. Please see:
tensorflow/tensorflow#726
tensorflow/tensorflow#713

@alexlenail
Copy link

Why does Erik install tensorflow the way he does? Why not use pip?

@shackenberg
Copy link

@zfrenchee You need to recompile tensorflow from source with the special configuration from line 72, otherwise it will not run on an ec2. See comments from line 67 on.

@shackenberg
Copy link

Thanks @closedLoop for all the information, I get the same numbers. Still strange that @erikbern and @marcotrombetti report much higher speed. (32_x_ examples / sec instead of 22_x_ examples / sec)

@raindeer
Copy link

To build for Python 3.4:

  • During configure specify /usr/bin/python3
  • Use bazel 0.1.1, git checkout tags/0.1.1
  • Use wheel 0.26
  • Don't use the current tensorflow master. It is not Python 3 compatible. Checkout the 0.6.0 tag.
  • Modify tensorflow/bazel-bin/tensorflow/tools/pip_package/build_pip_package. Add --python-tag py34 when building the wheel:
...
echo $(date) : "=== Building wheel"
python setup.py bdist_wheel --python-tag py34 >/dev/null
...
  • Make sure to use the correct wheel name when installing: sudo pip3 install /tmp/tensorflow_pkg/tensorflow-0.6.0-py34-none-any.whl

@Nodice
Copy link

Nodice commented Jan 14, 2016

Just for reference. Geforce 970, i7 Local Machine, examples/sec 903.3 examples/sec .142 sec/batch

@ggonzale
Copy link

Just got to run. Thank you for the code. My timing - measured at step 50

Instance Type s/batch
g2.2xlarge 0.26
g2.8xlarge 0.11 (with --num_gpus=4)
macbook-pro i7 quad end 2013 0.56

Quite unimpressive the usage of gpus, as @marcotrombetti says. I was also expecting an order of magnitude improvement.

@Razorwind
Copy link

I have done all the steps but the last pip install seems to be an issue
"Requirement '/tmp/tensorflow_pkg/tensorflow-0.5.0-cp27-none-linux_x86_64.whl' looks like a filename, but the file does not exist"

I used git "checkout tags/0.1.4" for basel instead.

*Edit ok silly me: Line 80 needs to be changed to 0.60 instead.

@Andyccs
Copy link

Andyccs commented Jan 31, 2016

@cancan101
Copy link

Using keras mnist_cnn script to compare the performance of Theano to Tensorflow on g2.2xlarges, I see 8s / epoch with Theano and 97s / epoch with Tensorflow!

@chrisconley
Copy link

In case anyone's interested, we documented how we installed TensorFlow along with Python 3.4 and Jupyter on EC2 based on this gist and many of the comments here. Thank you everyone!

@erikbern
Copy link
Author

erikbern commented Feb 4, 2016

Didn't notice all the comments here – Github doesn't send notifications on gists I guess. Anyway you should check out @chrisconley's link instead!

@AlexJoz
Copy link

AlexJoz commented Feb 20, 2016

Nice work! Thnx for guide ^^
Made another one with Python3, TensorFlow 0.7 and OpenCV 3.1:
https://gist.github.com/AlexJoz/1670baf0b32573ca7923
Public ami with my setup in N.Virginia: ami-9d0f3ff7

@axeltidemann
Copy link

If you for some reason found / to fill up unexplainably when compiling with bazel, this is due to bazel putting cache files in ~/.cache/bazel by default. Set export TEST_TMPDIR=/tmp/.cache to avoid this.

@ccywch
Copy link

ccywch commented Apr 10, 2016

There seems to be a new solution: https://aws.amazon.com/marketplace/pp/B01AOE205O

@aie0
Copy link

aie0 commented May 24, 2016

@AlexJoz's ami works great

@SpencerC
Copy link

Published new AMI in N. Virginia with 0.8.0 support: ami-1e19ee73

@hpssjellis
Copy link

Thanks. I will look into my changing my bash file that installs the cpu version of tensorflow (with video) to a gpu version on cloud9 http://c9.io

https://github.com/hpssjellis/forth-tensorflow

@shamak
Copy link

shamak commented Jul 9, 2016

When I run line 73, I get an error: Unrecognized option: --host_force_python=py2

Any idea why?

@zaheersm
Copy link

zaheersm commented Jul 11, 2016

@shamak I'm getting the exact same error.

@zaheersm
Copy link

zaheersm commented Jul 11, 2016

@shamak Install the latest version of Bazel from here

Later, you might also need to setup CUDNN v4 (7.0) instead of v2 (6.5). See this issue

@rnditdev
Copy link

It seems that Cudnn can be downloaded with

curl -fvSL http://developer.download.nvidia.com/compute/redist/cudnn/v2/cudnn-6.5-linux-x64-v2.tgz -o cudnn-6.5-linux-x64-v2.tgz

@standy66
Copy link

standy66 commented Aug 15, 2016

Just for reference:

GTX 1070, i7 6700k, local machine, tensorflow inside docker container, using nvidia-docker (but I doubt it adds any overhead)

1744.3 examples/sec; 0.073 sec/batch

@navoshta
Copy link

navoshta commented Sep 1, 2016

I've recently prepared a couple of convenience scripts for firing up your AWS instance with Jupyter Notebook on board that you may find useful:

@pvels
Copy link

pvels commented Sep 4, 2016

For stat:

Zotac GTX 1080 AMP Extreme, 2560 CUDA cores, 1771 MHz core clock, 10000 MHz mem clock. i7 930 3.8 GHz boost clock.

step 100000, loss = 0.72 (1780.0 examples/sec; 0.072 sec/batch); time: 2h 5m.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment