fast.ai course

README.md

List of files

File	Purpose
aws-alias.sh	Command aliases that make AWS server management easier.
instal-gpu-azure.sh	Installs required software on an Azure Ubuntu server. Instructions available on the wiki.
install anaconda.sh	Installs other software on an Ubuntu machine. FULL Instructions available in the file .
install-gpu.sh	Installs CUDA and nvidia drivers on an Ubuntu machine. FULL Instructions available in the file .
Install CUDA for DL.md	FULL Instructions to install CUDA and nvidia drivers on an Ubuntu machine.
setup_instance.sh	Sets up an AWS environment for use in the course including a server that has the required software installed. This script is used by setup_p2.sh or setup_t2.sh. You probably don't need to call it by itself.
setup_p2.sh and setup_t2.sh	Configure environment variables for use in setup_instance.sh. These files call setup_instance.sh, which does the actual work of setting up the AWS instance.

Setup Instructions

AWS

If you haven't already, view the video at http://course.fast.ai/lessons/aws.html for the steps you need to complete before running these scripts. More information is available on the wiki.

1. Decide if you will use a GPU server or a standard server. GPU servers process deep learning jobs faster than general purpose servers, but they cost more per hour. Compare server pricing at https://aws.amazon.com/ec2/pricing/on-demand/.
2. Download setup_p2.sh if you decided on the GPU server or setup_t2.sh for the general purpose server. Also download setup_instance.sh.
3. Run the command bash setup_p2.sh or bash setup_t2.sh, depending on the file you downloaded. Run the command locally from the folder where the files were downloaded. Running bash setup_p2.sh sets up a p2.xlarge GPU server. bash setup_p2.sh sets up a t2.xlarge general purpose server.
4. The script will set up the server you selected along with other pieces of AWS infrastructure. When it finishes, it will print out the command for connecting to the new server. The server is preloaded with the software required for the course.
5. Learn how to use the provided AWS aliases on the wiki.

Azure

Once you have an Azure GPU server set up, download and run the install-gpu-azure.sh script on that server. More instructions available on the wiki.

Ubuntu

Download and run the install-gpu.sh script to install required software on an Ubuntu machine. (modified to be able to be run on Ubuntu 18.04 LTS)

anaconda-install.sh

# Make sure first you have nvidia drivers and CUDA installed

# install Anaconda for current user
mkdir downloads
cd downloads
wget "https://repo.continuum.io/archive/Anaconda2-4.2.0-Linux-x86_64.sh" -O "Anaconda2-4.2.0-Linux-x86_64.sh"
bash "Anaconda2-4.2.0-Linux-x86_64.sh" -b

echo "export PATH=\"$HOME/anaconda2/bin:\$PATH\"" >> ~/.bashrc
export PATH="$HOME/anaconda2/bin:$PATH"
conda install -y bcolz
conda upgrade -y --all

# install and configure theano
pip install theano
echo "[global]
device = gpu
floatX = float32

[cuda]
root = /usr/local/cuda" > ~/.theanorc

# install and configure keras
pip install keras==1.2.2
mkdir ~/.keras
echo '{
    "image_dim_ordering": "th",
    "epsilon": 1e-07,
    "floatx": "float32",
    "backend": "theano"
}' > ~/.keras/keras.json

# install cudnn libraries
wget "https://developer.nvidia.com/compute/machine-learning/cudnn/secure/v7.4.1.5/prod/10.0_20181108/cudnn-10.0-linux-x64-v7.4.1.5.tgz" -O "cudnn.tgz"
tar -zxf cudnn.tgz
cd cuda
sudo cp lib64/* /usr/local/cuda/lib64/
sudo cp include/* /usr/local/cuda/include/


conda install matplotlib scikit-learn

# configure jupyter and prompt for password
jupyter notebook --generate-config
jupass=`python -c "from notebook.auth import passwd; print(passwd())"`
echo "c.NotebookApp.password = u'"$jupass"'" >> $HOME/.jupyter/jupyter_notebook_config.py
echo "c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False" >> $HOME/.jupyter/jupyter_notebook_config.py

# clone the fast.ai course repo and prompt to start notebook
cd ~/Desktop/
git clone https://github.com/fastai/courses.git
echo "\"jupyter notebook\" will start Jupyter on port 8888"
echo "If you get an error instead, try restarting your session so your $PATH is updated"

Install CUDA for DL Ubuntu 18.04.md

This is a README for setting up Ubuntu 18.04 on a laptop that has Nvidia Graphics Card. Use install-gpu.sh.

Note: Use this at your own risk. :)

First:

• To install an ubuntu, do the usual UEFI ISO buring on the USB.
• When booting from the live USB, follow the step given here(Dealing with the Open-Source version of graphic card drivers and replacing them with Nvidia drivers).
• After booting into the Ubuntu after the OS installation. Follow steps here to install the Nvidia Drivers found on ppa, repository of ubuntu. Steps linked here
• Now add a file at /etc/modprobe.d with the name disable-nouveau.conf and paste the following content in it.

   blacklist nouveau
   options nouveau modeset=0
  

  This would result in disabling of the stupid nouveau drivers that want to give problems with dual gpu (intel and nvidia here) setup.
• We have now a Ubuntu that uses nvidia graphics card for the xorg or the GUI of the system.
• (Optional) Next we set the OS to use only intel graphics for the display and want to set up nvidia graphics card for Deep Learning Purposes. We do this following the steps linked here.
• Have a look at the discussion here.

install-gpu.sh

# This script is designed to install cuda and nvidia drivers and works with ubuntu 18.04 LTS

# ensure system is updated and has basic build tools
sudo apt-get update
sudo apt-get --assume-yes upgrade
sudo apt-get --assume-yes install tmux build-essential gcc g++ make binutils
sudo apt-get --assume-yes install software-properties-common

# remove the existing CUDA and nvidia drivers
sudo rm -f /etc/apt/sources.list.d/cuda*
sudo apt remove nvidia-cuda-toolkit
sudo apt remove nvidia-*

# update the system
sudo apt update

# install key and add repo
sudo apt-key adv --fetch-keys  http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
sudo bash -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/cuda.list'

# update the system again
sudo apt update

# install the nvidia drivers and cuda resp.
sudo apt install nvidia-driver-410
sudo apt install cuda-10-0

############## MANUAL WORK NOW ########################
##################### https://askubuntu.com/questions/1077061/how-do-i-install-nvidia-and-cuda-drivers-into-ubuntu/1077063#1077063 ###################
# add this line to PATH so that it includes the binaries of CUDA go to /.profiles
if [ -d "/usr/local/cuda-10.0/bin/" ]; then
    export PATH=/usr/local/cuda-10.0/bin${PATH:+:${PATH}}
    export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
fi
# we need to run the above lines everytime we boot, paste thise lines at the end of ~/.profiles