Last active
February 5, 2024 02:46
-
-
Save shlomihod/bfe325f238ddd23de5194c625c352e90 to your computer and use it in GitHub Desktop.
notebook.ipynb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "cells": [ | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "view-in-github", | |
| "colab_type": "text" | |
| }, | |
| "source": [ | |
| "<a href=\"https://colab.research.google.com/gist/shlomihod/bfe325f238ddd23de5194c625c352e90/notebook.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "ehvtYrh8nt7f" | |
| }, | |
| "source": [ | |
| "\n", | |
| "\n", | |
| "# Class 2 - Liability & Robustness: Autonomous Vehicle Case-Study\n", | |
| "\n", | |
| "https://learn.responsibly.ai" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "GTCNz-KW9hSy" | |
| }, | |
| "source": [ | |
| "## 1. Overview: Traffic sign detection" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "f0MpnWf4_pbw" | |
| }, | |
| "source": [ | |
| "### 1.1. Functional Description\n", | |
| "\n", | |
| "The autonomous car manufactured by \"Play\" detects traffic signs by using a multiclass Deep Neural Network, i.e. the model." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "clImKfFk96ad" | |
| }, | |
| "source": [ | |
| "#### Which task does the model perform?\n", | |
| "Given an image predict which traffic signs appear in it.\n", | |
| "\n", | |
| "\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "a4Ri4fft977l" | |
| }, | |
| "source": [ | |
| "#### What is the input and the output of the model?\n", | |
| "* input - image (jpg format)\n", | |
| "* output - indicator vector with length as the number of classes. For example, let's look on the following vector ${(1,1,0,0)}$ - out of the 4 possible classes, the first and the second classes appear in the input image.\n", | |
| "* In this exercise, we predict traffic signs in the picture. We divided the traffic signs into 4 categories following [Vienna convention for road signs](https://en.wikipedia.org/wiki/Vienna_Convention_on_Road_Signs_and_Signals) - Danger, Priority, Prohibitory and Mandatory.\n", | |
| "For example, under this division, stop sign is mapped to the category of priority signs and speed limit is mapped to prohibitory class.\n", | |
| "\n", | |
| "\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "5F-lcVhs99ac" | |
| }, | |
| "source": [ | |
| "### 1.2. Implementaions Details" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "nGCMKkjP-BqC" | |
| }, | |
| "source": [ | |
| "#### Dataset\n", | |
| "[Traffic Signs Dataset in YOLO format\n", | |
| "](https://www.kaggle.com/valentynsichkar/traffic-signs-dataset-in-yolo-format) by [Valentyn Sichkar\n", | |
| "](https://valentynsichkar.name/) (Originally: [GTSDB](http://benchmark.ini.rub.de/?section=gtsdb) (German Traffic Sign Detection Benchmark) by INI)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "7mIE6x1D-IWu" | |
| }, | |
| "source": [ | |
| "#### Model\n", | |
| "Fine tuned YOLOv5 on PyTourch [GitHub](https://github.com/ultralytics/yolov5)\n", | |
| "\n", | |
| "**DS Note:**\n", | |
| "YOLO is an object detection model, but in order to simplify the explanations for non-DS students we will consider the model as a binary multiclass classifier (i.e., each traffic sign is considered as exist if the model SoftMax layer got a higher score than a defined threshold)." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "-wJdvKWuNWM0" | |
| }, | |
| "source": [ | |
| "## 2. Accident Investigation Playground\n", | |
| "\n", | |
| "" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "kEjNN3rpKON7" | |
| }, | |
| "source": [ | |
| "### 2.0 Run this cell and restart the runtime" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "njMgJIweKON8" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "!pip install --upgrade Pillow\n", | |
| "\n", | |
| "import PIL\n", | |
| "PIL.__version__" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "3F-IwGyJN66U" | |
| }, | |
| "source": [ | |
| "### 2.1. Setup (just run me!)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "NJFX7msYpT3g" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "# https://databank.illinois.edu/datasets/IDB-9610843\n", | |
| "!wget https://stash.responsibly.ai/2-liability-robustness/activity/data.zip -O data.zip -q" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "iTdglGUmpT3h" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "!unzip -oq data.zip" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "Vb29pir7pT3h" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "%pip install -qqq git+https://github.com/ResponsiblyAI/railib.git" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "EfxJ3DR98EtX" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "%%bash\n", | |
| "\n", | |
| "git clone https://github.com/ultralytics/yolov5\n", | |
| "cd yolov5\n", | |
| "pip install -qqq -r requirements.txt" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "_li6h03XWMRA" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "!wget 'https://drive.google.com/uc?export=download&id=1u_9w0OlFFqcSXbKisFbZUVIwjjGMD_Ym' -O models.zip -q\n", | |
| "!unzip -oq models.zip" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "rr4c8dfuOGed" | |
| }, | |
| "source": [ | |
| "### 2.2. `traffic_signs_utils` Python Package\n", | |
| "The `traffic_signs_utils` Python package is used to investigate the performance of the traffic signs detection model.\n", | |
| "\n", | |
| "\n", | |
| "In the rest of the Netbook, useful functions are presented with a short documentation. The documentation is simplified to the most important parameters.\n", | |
| "\n", | |
| "### **IMPORTANT!**\n", | |
| "\n", | |
| "1. Start with running all cells in the notebook (Runtime > Run all)\n", | |
| "2. The notebook is written in a way that you get all the information you need\n", | |
| "3. Do not spend time understanding the details of the code. Focus on the text, comments, and outputs of cells\n", | |
| "4. The main goal is to cooperate and share the information with your teammates who have different backgrounds\n", | |
| "\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "S9Ln2LRB7Hb5" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "from railib import liability as tsu" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "8YYpS6o7NjyT" | |
| }, | |
| "source": [ | |
| "### 2.3. Image taken at the accident site" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "kH7fCYstrHnp" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "tsu.present_image('./data/site/site.jpg')" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "rzQTaARUOkvv" | |
| }, | |
| "source": [ | |
| "#### Applying the model\n", | |
| "The following image is presenting the traffic signs detected by the model of PLAY. \n", | |
| "\n", | |
| "In the left-top corner of the image you can see the traffic signs categories which were detected by PLAY's model* with there probabilities.\n", | |
| "\n", | |
| "\n", | |
| "*only the categories that crossed the model's threshold.\n", | |
| "\n", | |
| "**Note that STOP sign is a Priority sign.**" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "e_rIWK2Asrst" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "image_path = '/content/data/site/site.jpg'\n", | |
| "labels_path = tsu.run_detect(image_path)\n", | |
| "tsu.im_write(image_path, labels_path)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "T80IEsWrO-p4" | |
| }, | |
| "source": [ | |
| "### 2.4. Image taken by the car camera just a moment before the accident.\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "pNruIkaRPOeu" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "tsu.present_image('./data/accident/accident.jpg')" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "TMCQ6YMGfQcy" | |
| }, | |
| "source": [ | |
| "The same image with PLAY's model results." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "olAwKn_U6--8" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "image_path = './data/accident/accident.jpg'\n", | |
| "labels_path = tsu.run_detect(image_path)\n", | |
| "tsu.im_write(image_path, labels_path)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "bSeF46gkPul0" | |
| }, | |
| "source": [ | |
| "### 2.5. Exploring the Dataset" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "sBkPcCDgWpxj" | |
| }, | |
| "source": [ | |
| "#### Training Dataset" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "jPONoNW5gUb-" | |
| }, | |
| "source": [ | |
| "A random sample of the training dataset of PLAY's model with both the ground truth (GT) and the predictions on it." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "cellView": "form", | |
| "id": "Wu1jQBjYQTId" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "tsu.present_matrix_of_images(filename=tsu.TRAIN_TXT_PATH,\n", | |
| " labels_path=tsu.TRAIN_LABELS_PATH,\n", | |
| " with_prediction=True)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "OtzM5MHHWtmC" | |
| }, | |
| "source": [ | |
| "#### Test Dataset\n", | |
| "\n", | |
| "A random sample of the test dataset." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "ayd9SX-gWZOb" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "tsu.present_matrix_of_images(filename=tsu.TEST_TXT_PATH)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "dEcUDr49W70j" | |
| }, | |
| "source": [ | |
| "### 2.6. Evaluation\n", | |
| "\n", | |
| "Accuracy:\n", | |
| "${\\frac{TP + TN}{TP + TN + FP + FN}} $\n", | |
| "\n", | |
| "* TP - True Positive\n", | |
| "* TN - True Negative\n", | |
| "* FP - False Positive\n", | |
| "* FN - False Negative\n", | |
| "\n", | |
| "Or, in other words, the accuracy is the proportion of correctly predicted traffic signs.\n", | |
| "\n", | |
| "**Important Note: According to industry and regulatory standards, the accuracy of 90% and above is considered satisfactory.**\n", | |
| "\n", | |
| "\n", | |
| "\n", | |
| "\n", | |
| "\n", | |
| "\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "P44oHsW_R9L4" | |
| }, | |
| "source": [ | |
| "#### Training Dataset - evaluation" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "_KYd1nRlXK4k" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "tsu.evaluation(pred_path=tsu.TRAIN_JSON_LABELS_PATH,\n", | |
| " categories_reminder=True)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "lgU2qYrzSB2I" | |
| }, | |
| "source": [ | |
| "#### Test Dataset - evaluation" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "xxztdP2iXTD8" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "tsu.evaluation(pred_path=tsu.TEST_JSON_LABELS_PATH)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "KK_lhnF0fRsR" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "### SPLIT" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "TKgcX1DeY94q" | |
| }, | |
| "source": [ | |
| "## 3. Augmentation Experiment" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "ODC0s1yf_ssW" | |
| }, | |
| "source": [ | |
| "### 3.1. Demonstrating Augmentation\n", | |
| "\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "mNyxPbYzUZcl" | |
| }, | |
| "source": [ | |
| "For example, this is what happens when we augment the test dataset with \"drizzle\" rain:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "bwif5VLqdmfL" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "images_path = tsu.apply_augmentation(filename=tsu.TEST_TXT_PATH,\n", | |
| " aug_func=tsu.am.add_rain,\n", | |
| " rain_type='drizzle',\n", | |
| " drop_width=1)\n", | |
| "tsu.present_matrix_of_images(images_path)\n", | |
| "print(images_path)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "HC6S1bEkU3X4" | |
| }, | |
| "source": [ | |
| "Let's try to run the model on these images:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "gkdZ9DyDU2Np" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "# images_path - the one from last cell (where apply_augmentation is executed)\n", | |
| "labels_path = tsu.run_detect(images_path.replace('.txt',''))\n", | |
| "tsu.present_matrix_of_images(images_path,\n", | |
| " labels_path=labels_path,\n", | |
| " with_prediction=True)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "pDyLm_JSVCLE" | |
| }, | |
| "source": [ | |
| "We can also evaluate the performance of the model on these augmented images (with dizzle rain)." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "pCuYripNNsNr" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "# labels_path_json - the one from last cell\n", | |
| "labels_path_json = tsu.save_to_json(labels_path)\n", | |
| "tsu.evaluation(pred_path=labels_path_json)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "p5OOuPs5VWy1" | |
| }, | |
| "source": [ | |
| "Now with \"gravel\" augmentation.\n", | |
| "\n", | |
| "**Note**: Althoguh \"gravel\" means \"a loose aggregation of small water-worn or pounded stones\", we treat it here as \"mud/dirt\" on the camera." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "WKMwAyfQVe4C" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "images_path = tsu.apply_augmentation(filename=tsu.TEST_TXT_PATH,\n", | |
| " aug_func=tsu.am.add_gravel) # no need for rain_type and drop_with\n", | |
| "\n", | |
| "labels_path = tsu.run_detect(images_path.replace('.txt',''))\n", | |
| "tsu.present_matrix_of_images(images_path,\n", | |
| " labels_path=labels_path,\n", | |
| " with_prediction=True)\n", | |
| "\n", | |
| "labels_path_json = tsu.save_to_json(labels_path)\n", | |
| "tsu.evaluation(pred_path=labels_path_json)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "QKxp3PQu_x4Y" | |
| }, | |
| "source": [ | |
| "### 3.2. Accuracy Comparison by Augmentation" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "Ry4hzks-V3mA" | |
| }, | |
| "source": [ | |
| "An aggregated evaluation plot of pre-calculated results of relevant augmentation conditions." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "LKFM-XIanXxB" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "%matplotlib inline\n", | |
| "\n", | |
| "tsu.plot_all_aug_eval_bar_plot();" | |
| ] | |
| } | |
| ], | |
| "metadata": { | |
| "accelerator": "TPU", | |
| "colab": { | |
| "provenance": [], | |
| "include_colab_link": true | |
| }, | |
| "gpuClass": "standard", | |
| "kernel_info": { | |
| "name": "python3" | |
| }, | |
| "kernelspec": { | |
| "display_name": "Python 3 (ipykernel)", | |
| "language": "python", | |
| "name": "python3" | |
| }, | |
| "language_info": { | |
| "codemirror_mode": { | |
| "name": "ipython", | |
| "version": 3 | |
| }, | |
| "file_extension": ".py", | |
| "mimetype": "text/x-python", | |
| "name": "python", | |
| "nbconvert_exporter": "python", | |
| "pygments_lexer": "ipython3", | |
| "version": "3.11.3" | |
| }, | |
| "nteract": { | |
| "version": "0.12.3" | |
| } | |
| }, | |
| "nbformat": 4, | |
| "nbformat_minor": 0 | |
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment