CNN flower classification with data augmentation on multi-GPUs (cleared output)

CNN flower classification with data augmentation on multi-GPUs (cleared output).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FE7KNzPPVrVV"
   },
   "source": [
    "# Classifying Flowers 🌼🌻🌺🌹🌷 using tf.keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gN7G9GFmVrVY"
   },
   "source": [
    "This notebook is classifying images of five different flower types and shows how image augmentation can improve our validation accuracy. The demo is inspired by [this TensorFlow Example](https://github.com/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l05c04_exercise_flowers_with_data_augmentation_solution.ipynb). Find the github gist [here](https://gist.github.com/maxbonaparte/217903c52a4cd440cbc61608f171035f)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zF9uvbXNVrVY"
   },
   "source": [
    "# Importing Packages 📦"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VddxeYBEVrVZ"
   },
   "source": [
    "Standing on the shoulders of giants, we'll use the following packages:\n",
    "\n",
    "Besides TensorFlow 2.2, we'll use **os** to read files and directory structures, we'll use **numpy** convert python list to numpy array and to perform required matrix operations, **PIL** is an imaging library we use to open images, **glob** helps us finding pathnames matching a specified pattern and **matplotlib.pyplot** is used to plot the graph and display images in our training and validation data.\n",
    "\n",
    "Finally, the `ImageDataGenerator` from `tf.keras.preprocessing.image` is needed to perform image augmentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rtPGh2MAVrVa"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import os\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "import glob\n",
    "import matplotlib.pyplot as plt\n",
    "from random import randint\n",
    "\n",
    "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
    "\n",
    "print(\"Tensorflow Version: \", tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#In case this script has already run, remove the old folder structure:\n",
    "!rm -r /home/ubuntu/.keras/datasets/flower_photos/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UZZI6lNkVrVm"
   },
   "source": [
    "# Data Loading ⏳"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DPHx8-t-VrVo"
   },
   "source": [
    "First, we need to download & extract the flowers dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OYmOylPlVrVt"
   },
   "outputs": [],
   "source": [
    "_URL = \"https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz\"\n",
    "\n",
    "zip_file = tf.keras.utils.get_file(origin=_URL,\n",
    "                                   fname=\"flower_photos.tgz\",\n",
    "                                   extract=True)\n",
    "\n",
    "#define the base_dir that contains all the images:\n",
    "base_dir = os.path.join(os.path.dirname(zip_file), 'flower_photos')\n",
    "print(\"base directory: \", base_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inspect the Data and Shape 🕵️‍"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classes = ['roses', 'daisy', 'dandelion', 'sunflowers', 'tulips']\n",
    "total_images = 0;\n",
    "\n",
    "for cl in classes:\n",
    "    img_path = os.path.join(base_dir, cl)\n",
    "    images = glob.glob(img_path + '/*.jpg')\n",
    "    print(\"{}: {} Images.\".format(cl, len(images)))\n",
    "    \n",
    "    #total train and val counter:\n",
    "    total_images = total_images + len(images)\n",
    "\n",
    "print(\"-------------------------\")\n",
    "print('total images:', total_images)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#folders:\n",
    "!ls -d /home/ubuntu/.keras/datasets/flower_photos/*/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "G1ymuCPS0_eu"
   },
   "source": [
    "The downloaded dataset has following directory structure, representing the 5 classes of flowers we have:\n",
    "\n",
    "<pre style=\"font-size: 10.0pt; font-family: Arial; line-height: 2; letter-spacing: 1.0pt;\" >\n",
    "<b>flower_photos</b>\n",
    "|__ <b>diasy</b>\n",
    "|__ <b>dandelion</b>\n",
    "|__ <b>roses</b>\n",
    "|__ <b>sunflowers</b>\n",
    "|__ <b>tulips</b>\n",
    "</pre>\n",
    "\n",
    "There are no folders containing training and validation data. We will use `ImageDataGenerator` later to split training and validation set.\n",
    "\n",
    "If we have a look at some of the image formats, we realize they have all different sizes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Some Image Sizes:\")\n",
    "print(\"-------------------------\")\n",
    "for i in range(10):\n",
    "    print('Image Size:', Image.open(images[i]).size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#This function will plot 5 images in the form of a grid with 1 row and 5 columns where images are placed in each column.\n",
    "def plotXImages(images_arr, x):\n",
    "    fig, axes = plt.subplots(1, x, figsize=(20,20))\n",
    "    axes = axes.flatten()\n",
    "    i = 1\n",
    "    for img, ax in zip( images_arr, axes):\n",
    "        ax.imshow(img)\n",
    "        ax.set_title(i)\n",
    "        i += 1\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "    \n",
    "j = 0\n",
    "image_array = []\n",
    "    \n",
    "for cl in classes:\n",
    "    img_path = os.path.join(base_dir, cl)\n",
    "    images = glob.glob(img_path + '/*.jpg')\n",
    "    print(\"{}: {} Images.\".format(cl, len(images)))\n",
    "    images_example = Image.open(images[randint(0, 10)])\n",
    "    image_array.append(np.asarray(images_example))\n",
    "    j += 1\n",
    "\n",
    "plotXImages(image_array, 5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UOoVpxFwVrWy"
   },
   "source": [
    "# Data Augmentation 📶 through Image Transformation 🖼️"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Wn_QLciWVrWy"
   },
   "source": [
    "Overfitting generally occurs when we have small number of training examples. One way to fix this problem is to augment our dataset so that it has sufficient number of training examples. Data augmentation takes the approach of generating more training data from existing training samples, by augmenting the samples via a number of random transformations that yield believable-looking images. The goal is that at training time, your model will never see the exact same picture twice. This helps expose the model to more aspects of the data and generalize better.\n",
    "\n",
    "We can implement this using the `tf.keras.preprocessing.image.ImageDataGenerator` class as a data loader. We can simply pass different transformations we would want to our dataset as a form of arguments and it will take care of applying it to the dataset during our training process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rlVj6VqaVrW0"
   },
   "source": [
    "### Apply Random Horizontal Flip  🙃\n",
    "\n",
    "As a first image transformation we will use `ImageDataGenerator` to rescale the images by 255 and then apply a random horizontal flip and split the data into 80% training and 20% validation.\n",
    "\n",
    "Then use the `.flow_from_directory` method to apply the above transformation to the images in our training set. Make sure you indicate the batch size, the path to the directory of the training images, the target size for the images, and to shuffle the images.\n",
    "\n",
    "Also, as we're training a model with multiple GPUs, you can use the extra computing power effectively by increasing the batch size. In general, use the largest batch size that fits the GPU memory, and tune the learning rate accordingly. See our article here how to do that: https://blog.gws.genesiscloud.dev/2020/multi-gpu-training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Bi1_vHyBVrW2"
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE = 200\n",
    "IMG_SHAPE = (150, 150)\n",
    "VAL_SPLIT = 0.2\n",
    "\n",
    "data_gen = ImageDataGenerator(rescale=1./255, horizontal_flip=True, validation_split=VAL_SPLIT)\n",
    "\n",
    "train_data_gen = data_gen.flow_from_directory(batch_size=BATCH_SIZE, directory=base_dir,\n",
    "                                                class_mode='sparse',\n",
    "                                                shuffle=True,\n",
    "                                                target_size=IMG_SHAPE, subset=\"training\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zJpRSxJ-VrW7"
   },
   "source": [
    "Let's take 1 sample image from our training examples and repeat it 5 times so that the augmentation can be applied to the same image 5 times over randomly, to see the augmentation in action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jqb9OGoVKIOi"
   },
   "outputs": [],
   "source": [
    "# This function takes an array of numpy-images and plots them in the form of a grid with 1 row and X columns where images are placed in each column.\n",
    "def plotImages(images_arr):\n",
    "    fig, axes = plt.subplots(1, 6, figsize=(20,20))\n",
    "    axes = axes.flatten()\n",
    "    for img, ax in zip(images_arr, axes):\n",
    "        ax.imshow(img)\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "augmented_images = [train_data_gen[0][0][0] for i in range(6)]\n",
    "\n",
    "plotImages(augmented_images)\n",
    "plotXImages(augmented_images, 6)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OC8fIsalVrXd"
   },
   "source": [
    "### More Transformations\n",
    "\n",
    "We will use ImageDataGenerator to create now several more transformation with the images:\n",
    "\n",
    "- random 45 degree rotation\n",
    "- random zoom of up to 50%\n",
    "- random horizontal flip\n",
    "- width shift of 0.15\n",
    "- height shift of 0.15\n",
    "- spilt into 80% training and 20% validation\n",
    "\n",
    "Then use the `.flow_from_directory` method again to apply the above transformation to the images in our training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gnr2xujaVrXe"
   },
   "outputs": [],
   "source": [
    "data_gen = ImageDataGenerator(\n",
    "      rescale=1./255,\n",
    "      rotation_range=45,\n",
    "      width_shift_range=0.15,\n",
    "      height_shift_range=0.15,\n",
    "      zoom_range=0.5,\n",
    "      horizontal_flip=True,\n",
    "      fill_mode='nearest',\n",
    "      validation_split=VAL_SPLIT)\n",
    "\n",
    "\n",
    "train_data_gen = data_gen.flow_from_directory(batch_size=BATCH_SIZE,\n",
    "                                               directory=base_dir,\n",
    "                                               shuffle=True,\n",
    "                                               target_size=IMG_SHAPE,\n",
    "                                               seed=15,\n",
    "                                               class_mode='sparse', subset=\"training\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AW-pV5awVrXl"
   },
   "source": [
    "Let's visualize how a single image would look like 5 different times, when we pass these augmentations randomly to our dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "z2m68eMhVrXm"
   },
   "outputs": [],
   "source": [
    "augmented_images = [train_data_gen[0][0][0] for i in range(6)]\n",
    "plotImages(augmented_images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a99fDBt7VrXr"
   },
   "source": [
    "### Create a Data Generator for the Validation Set\n",
    "\n",
    "Generally, we only apply data augmentation to our training examples. So, in the cell below, we use `ImageDataGenerator ` and  `.flow_from_directory` to rescale all images in our validation set the images by 255. It is not necessary to shuffle the images in the validation set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "54x0aNbKVrXr"
   },
   "outputs": [],
   "source": [
    "data_gen_val = ImageDataGenerator(rescale=1./255, validation_split=VAL_SPLIT)\n",
    "\n",
    "val_data_gen = data_gen_val.flow_from_directory(batch_size=BATCH_SIZE, directory=base_dir, target_size=IMG_SHAPE, class_mode='sparse', subset=\"validation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wEgW4i18VrWZ"
   },
   "source": [
    "# Define & Compile the Model 🧠\n",
    "\n",
    "When training a model with multiple GPUs, you can use the extra computing power effectively by increasing the batch size. In general, use the largest batch size that fits the GPU memory, and tune the learning rate accordingly. See our article here how to do that: https://blog.genesiscloud.com/2020/multi-gpu-training\n",
    "\n",
    "We'll only define the model definition and compilation step as a function that will be called later within the strategy.scope().\n",
    "\n",
    "After the Convolutional &  MaxPool layers we add a Flatten layer followed by some fully connected (Dense) layers & Dropout layers. The CNN should output class probabilities based on 5 classes which is done by the **softmax** activation function. All other layers use a **relu** activation function.\n",
    "\n",
    "Eventually, we compile the model using the ADAM optimizer and the sparse cross entropy function as a loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Evjf8jZk2zi-"
   },
   "outputs": [],
   "source": [
    "# Model configuration\n",
    "NO_CLASSES = 5\n",
    "NO_EPOCHS = 60\n",
    "INPUT_SHAPE = (IMG_SHAPE[0], IMG_SHAPE[1], 3)\n",
    "print('INPUT_SHAPE:', INPUT_SHAPE)\n",
    "\n",
    "# Define the model\n",
    "def get_compiled_model():\n",
    "    model = tf.keras.models.Sequential([\n",
    "            tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=INPUT_SHAPE),\n",
    "            tf.keras.layers.MaxPooling2D(2, 2),\n",
    "\n",
    "            tf.keras.layers.Conv2D(32, (3,3), activation='relu'),\n",
    "            tf.keras.layers.MaxPooling2D(2,2),\n",
    "\n",
    "            tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "            tf.keras.layers.MaxPooling2D(2,2),\n",
    "\n",
    "            tf.keras.layers.Flatten(),\n",
    "            tf.keras.layers.Dropout(0.2),\n",
    "            tf.keras.layers.Dense(512, activation='relu'),\n",
    "            tf.keras.layers.Dropout(0.2),\n",
    "            tf.keras.layers.Dense(NO_CLASSES, activation='softmax')\n",
    "    ])\n",
    "    \n",
    "    # Compile the model\n",
    "    model.compile(optimizer='adam',\n",
    "                  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "    return model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Set up for multiple GPUs\n",
    "\n",
    "Here, we'll use **synchornous data parallelism** to take advantage of the multiple GPUs we have available (more about distributed stategies [here](https://blog.genesiscloud.com/2020/multi-gpu-training)). That means our model gets replicated on the multiple GPUs and after each of them has processed their batch of data, they merge the results. Here we assume a single host, multi-device (or multi-GPU) synchronous training, we'll use the [tf.distribute.MirroredStrategy](https://www.tensorflow.org/api_docs/python/tf/distribute/MirroredStrategy) API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a MirroredStrategy:\n",
    "strategy = tf.distribute.MirroredStrategy()\n",
    "\n",
    "print('Number of devices: {}'.format(strategy.num_replicas_in_sync))\n",
    "print(\"-------------------------\")\n",
    "\n",
    "#check GPUs: nvidia-smi cmd call:\n",
    "#!nvidia-smi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open a strategy scope.\n",
    "with strategy.scope():\n",
    "    # Everything that creates variables should be under the strategy scope.\n",
    "    # In general this is only model construction & `compile()`.\n",
    "    model = get_compiled_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oub9RtoFVrWk"
   },
   "source": [
    "# Train the Model 🏋️\n",
    "\n",
    "Since tensorflow 2.1.0 the **model.fit** function also supports generators like the **ImageDataGenerator** class we use to generate batches of training and validation data for our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tk5NT1PW3j_P"
   },
   "outputs": [],
   "source": [
    "total_val = int(total_images * VAL_SPLIT)\n",
    "total_train = int(total_images * (1 - VAL_SPLIT))\n",
    "\n",
    "print('total_train: ', total_train)\n",
    "print('total_val: ', total_val)\n",
    "\n",
    "#The ceil of the scalar x is the smallest integer i, such that i >= x.\n",
    "validation_steps=int(np.ceil(total_val / float(BATCH_SIZE)))\n",
    "steps_per_epoch=int(np.ceil(total_train / float(BATCH_SIZE)))\n",
    "\n",
    "print('validation_steps: ', validation_steps)\n",
    "print('steps_per_epoch: ', steps_per_epoch)\n",
    "\n",
    "\n",
    "history = model.fit(\n",
    "    train_data_gen,\n",
    "    steps_per_epoch=int(np.ceil(total_train / float(BATCH_SIZE))),\n",
    "    epochs=NO_EPOCHS,\n",
    "    validation_data=val_data_gen,\n",
    "    validation_steps=validation_steps\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LZPYT-EmVrWo"
   },
   "source": [
    "# Visualize the Training History 📈"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8CfngybnFHQR"
   },
   "outputs": [],
   "source": [
    "acc = history.history['accuracy']\n",
    "val_acc = history.history['val_accuracy']\n",
    "\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs_range = range(NO_EPOCHS)\n",
    "\n",
    "print('epochs_range: ', epochs_range)\n",
    "\n",
    "\n",
    "plt.figure(figsize=(16, 8))\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.plot(epochs_range, acc, label='Training Accuracy')\n",
    "plt.plot(epochs_range, val_acc, label='Validation Accuracy')\n",
    "plt.legend(loc='lower right')\n",
    "plt.title('Training and Validation Accuracy')\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "plt.plot(epochs_range, loss, label='Training Loss')\n",
    "plt.plot(epochs_range, val_loss, label='Validation Loss')\n",
    "plt.legend(loc='upper right')\n",
    "plt.title('Training and Validation Loss')\n",
    "plt.show()\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of l05c03_exercise_flowers_with_data_augmentation.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}