Skip to content

Instantly share code, notes, and snippets.

@leggewie

leggewie/Python_for_DevOps.ipynb

Created January 7, 2021 23:53
Show Gist options
  • Save leggewie/32f8d912d37dad72e3ee46839d29d58e to your computer and use it in GitHub Desktop.
Save leggewie/32f8d912d37dad72e3ee46839d29d58e to your computer and use it in GitHub Desktop.
Download ZIP
Python for DevOps notebook
Display the source blob
Display the rendered blob
Raw
Python_for_DevOps.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "Python for DevOps.ipynb",
"provenance": [],
"collapsed_sections": [
"c_Id55m6Jsbu",
"hPWO_zyRopXN",
"MnsQIiDurIPp",
"ohA_HAOewNMT",
"A2sW1aQ-wuPH",
"0kmgxW7Sxa6j",
"kn-P-e_hjvK5",
"hQpf0vu_-Cnk",
"C95FIjW5-rJc",
"WrRrxofUSJqe",
"JgPZWw-cA_0B"
],
"toc_visible": true,
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"accelerator": "GPU"
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/github/noahgift/python-devops-course/blob/master/Python_for_DevOps.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4J2lIggJLUMS",
"colab_type": "text"
},
"source": [
"# **Python for DevOps**\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FlQO5nXbFPle",
"colab_type": "text"
},
"source": [
"![Python for DevOps](https://images-na.ssl-images-amazon.com/images/I/51pv-GUo2DL._SX379_BO1,204,203,200_.jpg)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c_Id55m6Jsbu",
"colab_type": "text"
},
"source": [
"## Pragmatic AI Labs\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e5p96AqpSDZa",
"colab_type": "text"
},
"source": [
"![alt text](https://paiml.com/images/logo_with_slogan_white_background.png)\n",
"\n",
"This notebook was produced by [Pragmatic AI Labs](https://paiml.com/). You can continue learning about these topics by:\n",
"\n",
"\n",
"* [Testing in Python ](http://leanpub.com/testinginpython/c/Yrv4sFCIUMd6)\n",
"* [Cloud Computing for Data Analysis](http://leanpub.com/cloud4data/c/SFInsbdIiLl2) \n",
"* Buying a copy of [Python for DevOps: Learn Ruthlessly Effective Automation](https://www.amazon.com/Python-DevOps-Ruthlessly-Effective-Automation/dp/149205769X)\n",
"* Buying a copy of [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](http://www.informit.com/store/pragmatic-ai-an-introduction-to-cloud-based-machine-9780134863917)\n",
"\n",
"* Reading an online copy of [Python for DevOps: Learn Ruthlessly Effective Automation](https://learning.oreilly.com/library/view/python-for-devops/9781492057680/)\n",
"* Reading an online copy of [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](https://www.safaribooksonline.com/library/view/pragmatic-ai-an/9780134863924/)\n",
"\n",
"* Viewing more content at [noahgift.com](https://noahgift.com/)\n",
"\n",
"### Additional Related Topics from Noah Gift\n",
"\n",
"His most recent books are:\n",
"\n",
"* [Pragmatic A.I.:   An introduction to Cloud-Based Machine Learning (Pearson, 2018)](https://www.amazon.com/Pragmatic-AI-Introduction-Cloud-Based-Analytics/dp/0134863860)\n",
"* [Python for DevOps (O'Reilly, 2020)](https://www.amazon.com/Python-DevOps-Ruthlessly-Effective-Automation/dp/149205769X). \n",
"\n",
"His most recent video courses are:\n",
"\n",
"* [Essential Machine Learning and A.I. with Python and Jupyter Notebook LiveLessons (Pearson, 2018)](https://learning.oreilly.com/videos/essential-machine-learning/9780135261118)\n",
"* [AWS Certified Machine Learning-Specialty (ML-S) (Pearson, 2019)](https://learning.oreilly.com/videos/aws-certified-machine/9780135556597)\n",
"* [Python for Data Science Complete Video Course Video Training (Pearson, 2019)](https://learning.oreilly.com/videos/python-for-data/9780135687253)\n",
"* [AWS Certified Big Data - Specialty Complete Video Course and Practice Test Video Training (Pearson, 2019)](https://learning.oreilly.com/videos/aws-certified-big/9780135772324)\n",
"* [Building A.I. Applications on Google Cloud Platform (Pearson, 2019)](https://learning.oreilly.com/videos/building-ai-applications/9780135973462)\n",
"* [Pragmatic AI and Machine Learning Core Principles (Pearson, 2019)](https://learning.oreilly.com/videos/pragmatic-ai-and/9780136554714)\n",
"* [Data Engineering with Python and AWS Lambda (Pearson, 2019)](https://learning.oreilly.com/videos/data-engineering-with/9780135964330)\n",
"\n",
"His most recent online courses are:\n",
"\n",
"* [Microservices with this Udacity DevOps Nanodegree (Udacity, 2019)](https://www.udacity.com/course/cloud-dev-ops-nanodegree--nd9991)\n",
"* [Command Line Automation in Python (DataCamp, 2019)](https://www.datacamp.com/instructors/ndgift)\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "V9ZjtXu7FbeR",
"colab_type": "text"
},
"source": [
"## Introduction to Python for DevOps"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uyS1mgtkF31O",
"colab_type": "text"
},
"source": [
"### Presentation: What is DevOps anyway?"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cBB-EIYmLqHI",
"colab_type": "text"
},
"source": [
"According to AWS these are [DevOps Practices](https://aws.amazon.com/devops/what-is-devops/):\n",
"\n",
"* Continuous Integration\n",
"* Continuous Delivery\n",
"* Microservices\n",
"* Monitoring and Logging (Use Data to make decisions)\n",
"* Communication and Collaboration\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nP_6tcHM7yFm",
"colab_type": "text"
},
"source": [
"### Using IPython, Jupyter, Colab and Python executable\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "n7JKyOrPkITH",
"colab_type": "text"
},
"source": [
"#### Using IPython\n",
"\n",
"Very similar to Jupyter, but run from terminal:\n",
"\n",
"* IPython predates Jupyter\n",
"* Both Jupyter and IPython accept *!ls -l* format to execute shell commands\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "YR3ZddpWM-uh",
"colab_type": "code",
"outputId": "47b0b3d1-6059-4a70-8867-01651279ec0e",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
}
},
"source": [
"!ls -l"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"total 4\n",
"drwxr-xr-x 1 root root 4096 Dec 18 16:52 sample_data\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "RETmEcecyCVf",
"colab_type": "code",
"outputId": "68fd6fb9-9d80-4a9a-ad9c-dfe37608c27d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"var = !ls -l\n",
"type(var)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"IPython.utils.text.SList"
]
},
"metadata": {
"tags": []
},
"execution_count": 2
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "dQMhyAm8sxSc",
"colab_type": "code",
"colab": {}
},
"source": [
"#var.fields?"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "qGSSYqVOyJEP",
"colab_type": "code",
"outputId": "9283356c-02f9-4a5e-a249-5bb372af946e",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"var.grep(\"data\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['drwxr-xr-x 1 root root 4096 Dec 18 16:52 sample_data']"
]
},
"metadata": {
"tags": []
},
"execution_count": 3
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Pci2aotqPIME",
"colab_type": "text"
},
"source": [
"#### Jupyter Notebook\n",
"\n",
"\n",
"Many flavors of Jupyter Notebook. A few popular ones:\n",
"\n",
"* ![Jupyter](https://user-images.githubusercontent.com/58792/40282633-395be25c-5c27-11e8-9e40-357ea4216562.png)\n",
"* ![JupyterHug](https://user-images.githubusercontent.com/58792/40282632-387efe32-5c27-11e8-9f02-6f95f2fee223.png)\n",
"* ![Colab](https://user-images.githubusercontent.com/58792/40282631-384f8cf6-5c27-11e8-9209-3f0d22de0d81.png)\n",
"* ![Kaggle](https://user-images.githubusercontent.com/58792/40282634-3985c3a6-5c27-11e8-8c53-06fbdedce847.png)\n",
"* ![Sagemaker](https://user-images.githubusercontent.com/58792/40282635-39a3bdd4-5c27-11e8-81d5-6533a3b84771.png)\n",
"\n",
"#### Hosted Commercial Flavors\n",
"\n",
"* [Google Colaboratory](https://colab.research.google.com/notebook): Free\n",
"* [Kaggle](https://www.kaggle.com/): Free\n",
"\n",
"\n",
"#### Pure Open Source\n",
"\n",
"* [Jupyter](http://jupyter.org/) standalone, original\n",
"* [JupyterHub](https://github.com/jupyterhub/jupyterhub) multi-user, docker friendly\n",
"\n",
"#### Hybrid Solutions\n",
"\n",
"* Running Jupyter on [AWS Spot Instances](https://aws.amazon.com/ec2/spot/)\n",
"* [Google Data Lab](https://cloud.google.com/datalab/)\n",
"* [Azure Data Science Virtual Machines](https://azure.microsoft.com/en-us/services/virtual-machines/data-science-virtual-machines/)\n",
"* [AWS Sagemaker](https://aws.amazon.com/sagemaker/)\n",
"* [Azure Machine Learning Studio](https://studio.azureml.net/)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lh4tsf3-p-lw",
"colab_type": "text"
},
"source": [
"#### Colab Notebook Key Features\n",
"\n",
"* [12.1 Perform Colaboratory basics ](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118/9780135261118-EMLA_01_12_01)\n",
"\n",
"* [12.2 Use Advanced Colab Features](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118/9780135261118-EMLA_01_12_02)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "V-xv1E3YqDK4",
"colab_type": "text"
},
"source": [
"* Can enable both TPU and GPU runtimes\n",
"* Can upload regular Jupyter Notebooks into colab\n",
"* Can have a Google Drive Full of Colab Notebooks\n",
"* Can sync colab notebooks to Github. Here is an example of a [gist of this notebook](https://gist.github.com/noahgift/c69200e05c057cf239fc7ea0be62e043)\n",
"* Can connect to a [local runtime](https://research.google.com/colaboratory/local-runtimes.html)\n",
"* Can create [forms in Colab](https://colab.research.google.com/notebooks/forms.ipynb)\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hPWO_zyRopXN",
"colab_type": "text"
},
"source": [
"##### Mount GDrive\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "XI73HZNLobp4",
"colab_type": "code",
"outputId": "96ba5445-43cb-4640-cd64-982dcebc0f29",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"source": [
"from google.colab import drive\n",
"drive.mount('/content/gdrive', force_remount=True)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n",
"\n",
"Enter your authorization code:\n",
"··········\n",
"Mounted at /content/gdrive\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "UNyzZwgmoxwm",
"colab_type": "code",
"outputId": "cc5cb731-0ce2-4b22-be9d-1f60aa7ad3f9",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"source": [
"import os;os.listdir(\"/content/gdrive/My Drive/awsml\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['kaggle.json', 'credentials', 'config', 'cloudai-7ab42a4d8a43.json']"
]
},
"metadata": {
"tags": []
},
"execution_count": 2
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3jX9349SpLbr",
"colab_type": "text"
},
"source": [
"#### Magic Commands"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Zf35CYV9PrjS",
"colab_type": "text"
},
"source": [
"##### %timeit"
]
},
{
"cell_type": "code",
"metadata": {
"id": "tMpMJH0rfAEg",
"colab_type": "code",
"outputId": "a2f4e7ee-4863-4917-92de-61cc3938a8e9",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 69
}
},
"source": [
"too_many_decimals = 1.912345897\n",
"\n",
"print(\"built in Python Round\")\n",
"%timeit round(too_many_decimals, 2)\n"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"built in Python Round\n",
"The slowest run took 65.50 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
"1000000 loops, best of 3: 419 ns per loop\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "p_KLutZKq5yN",
"colab_type": "text"
},
"source": [
"##### %alias"
]
},
{
"cell_type": "code",
"metadata": {
"id": "jyIZLH6UpMou",
"colab_type": "code",
"colab": {}
},
"source": [
"alias lscsv ls -l sample_data/*.csv "
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "BNJ6bKtGpVn3",
"colab_type": "code",
"outputId": "e7c82a8d-3ad6-42d6-a0a3-cbd2638982fe",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 87
}
},
"source": [
"lscsv"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"-rw-r--r-- 1 root root 301141 Dec 18 16:52 sample_data/california_housing_test.csv\n",
"-rw-r--r-- 1 root root 1706430 Dec 18 16:52 sample_data/california_housing_train.csv\n",
"-rw-r--r-- 1 root root 18289443 Dec 18 16:52 sample_data/mnist_test.csv\n",
"-rw-r--r-- 1 root root 36523880 Dec 18 16:52 sample_data/mnist_train_small.csv\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AE1cfp41rcwL",
"colab_type": "text"
},
"source": [
"[Reference These]\n",
"https://ipython.readthedocs.io/en/stable/interactive/magics.html"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MnsQIiDurIPp",
"colab_type": "text"
},
"source": [
"##### %who"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ljy17rdjrWD9",
"colab_type": "text"
},
"source": [
"Print variables"
]
},
{
"cell_type": "code",
"metadata": {
"id": "mUFS07KpPfE6",
"colab_type": "code",
"colab": {}
},
"source": [
"var1=1"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "ACyoq3IsrNZV",
"colab_type": "code",
"outputId": "68c7f887-6ae9-48a2-9608-81dc3a52197f",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"who"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"too_many_decimals\t var\t var1\t \n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "MURtyzAMrZ1Z",
"colab_type": "code",
"outputId": "f601d727-8340-4caa-a219-75fe40646174",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"too_many_decimals"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1.912345897"
]
},
"metadata": {
"tags": []
},
"execution_count": 9
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "l9z-cAQow-BO",
"colab_type": "text"
},
"source": [
"##### %writefile"
]
},
{
"cell_type": "code",
"metadata": {
"id": "luIP2qKqxJmB",
"colab_type": "code",
"outputId": "0ad988d1-6cef-409a-b912-291588f0c818",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"%%writefile magic_stuff.py\n",
"import pandas as pd\n",
"df = pd.read_csv(\n",
" \"https://raw.githubusercontent.com/noahgift/food/master/data/features.en.openfoodfacts.org.products.csv\")\n",
"df.drop([\"Unnamed: 0\", \"exceeded\", \"g_sum\", \"energy_100g\"], axis=1, inplace=True) #drop two rows we don't need\n",
"df = df.drop(df.index[[1,11877]]) #drop outlier\n",
"df.rename(index=str, columns={\"reconstructed_energy\": \"energy_100g\"}, inplace=True)\n",
"print(df.head())"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Writing magic_stuff.py\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "uok6ppvvxRo9",
"colab_type": "code",
"outputId": "a11e502f-ad0e-45bd-a9b8-f5ac944fe96c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 139
}
},
"source": [
"cat magic_stuff.py"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"import pandas as pd\n",
"df = pd.read_csv(\n",
" \"https://raw.githubusercontent.com/noahgift/food/master/data/features.en.openfoodfacts.org.products.csv\")\n",
"df.drop([\"Unnamed: 0\", \"exceeded\", \"g_sum\", \"energy_100g\"], axis=1, inplace=True) #drop two rows we don't need\n",
"df = df.drop(df.index[[1,11877]]) #drop outlier\n",
"df.rename(index=str, columns={\"reconstructed_energy\": \"energy_100g\"}, inplace=True)\n",
"print(df.head())"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "STjEo5XFPxM0",
"colab_type": "code",
"outputId": "ab3e00e3-20d6-448a-faf4-1e2653ed7dbc",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 156
}
},
"source": [
"!python magic_stuff.py"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
" fat_100g ... product\n",
"0 28.57 ... Banana Chips Sweetened (Whole)\n",
"2 57.14 ... Organic Salted Nut Mix\n",
"3 18.75 ... Organic Muesli\n",
"4 36.67 ... Zen Party Mix\n",
"5 18.18 ... Cinnamon Nut Granola\n",
"\n",
"[5 rows x 7 columns]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "0jUCwijQ0_-t",
"colab_type": "code",
"outputId": "8b18e753-b9ba-4a17-d760-a6a218e1e3f0",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 104
}
},
"source": [
"!pip install -q pylint"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"\u001b[K |████████████████████████████████| 307kB 8.3MB/s \n",
"\u001b[K |████████████████████████████████| 51kB 5.9MB/s \n",
"\u001b[K |████████████████████████████████| 215kB 15.7MB/s \n",
"\u001b[K |████████████████████████████████| 737kB 18.9MB/s \n",
"\u001b[K |████████████████████████████████| 61kB 8.5MB/s \n",
"\u001b[?25h"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "Xf8MwJfy1FqQ",
"colab_type": "code",
"outputId": "10e030c6-da7b-4ab9-a50e-c392c577dbaf",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 260
}
},
"source": [
"!pylint magic_stuff.py"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"************* Module magic_stuff\n",
"magic_stuff.py:3:0: C0301: Line too long (109/100) (line-too-long)\n",
"magic_stuff.py:4:0: C0301: Line too long (110/100) (line-too-long)\n",
"magic_stuff.py:5:24: C0326: Exactly one space required after comma\n",
"df = df.drop(df.index[[1,11877]]) #drop outlier\n",
" ^ (bad-whitespace)\n",
"magic_stuff.py:7:0: C0304: Final newline missing (missing-final-newline)\n",
"magic_stuff.py:1:0: C0114: Missing module docstring (missing-module-docstring)\n",
"magic_stuff.py:2:0: C0103: Constant name \"df\" doesn't conform to UPPER_CASE naming style (invalid-name)\n",
"magic_stuff.py:5:0: C0103: Constant name \"df\" doesn't conform to UPPER_CASE naming style (invalid-name)\n",
"\n",
"------------------------------------\n",
"Your code has been rated at -1.67/10\n",
"\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TtVDMB8EwJqG",
"colab_type": "text"
},
"source": [
"##### Bash"
]
},
{
"cell_type": "code",
"metadata": {
"id": "YC03T7XUuF1k",
"colab_type": "code",
"outputId": "7157036b-5a53-4d7c-ea90-f7ef7fa38c26",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 208
}
},
"source": [
"%%bash\n",
"uname -a\n",
"ls\n",
"ps"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Linux 6c859eb0a156 4.14.137+ #1 SMP Thu Aug 8 02:47:02 PDT 2019 x86_64 x86_64 x86_64 GNU/Linux\n",
"magic_stuff.py\n",
"sample_data\n",
" PID TTY TIME CMD\n",
" 1 ? 00:00:00 run.sh\n",
" 10 ? 00:00:01 node\n",
" 25 ? 00:00:03 jupyter-noteboo\n",
" 116 ? 00:00:00 tail\n",
" 124 ? 00:00:04 python3\n",
" 367 ? 00:00:00 bash\n",
" 370 ? 00:00:00 ps\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "HI8JuclMt0p2",
"colab_type": "code",
"outputId": "c4c3448d-22c8-46c6-951b-225729ee35bc",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"!uname -a"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Linux 6c859eb0a156 4.14.137+ #1 SMP Thu Aug 8 02:47:02 PDT 2019 x86_64 x86_64 x86_64 GNU/Linux\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ohA_HAOewNMT",
"colab_type": "text"
},
"source": [
"##### Python2"
]
},
{
"cell_type": "code",
"metadata": {
"id": "eioZW4txwO7-",
"colab_type": "code",
"outputId": "3bf7de3b-0de8-4453-9aec-4dd92f068264",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"%%python2\n",
"print \"old school\"\n"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"old school\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "A2sW1aQ-wuPH",
"colab_type": "text"
},
"source": [
"##### HTML"
]
},
{
"cell_type": "code",
"metadata": {
"id": "C8ueccG-wv8Z",
"colab_type": "code",
"outputId": "0d9f03e3-887e-43db-c282-079b5f459906",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 75
}
},
"source": [
"%%html\n",
"<h1>Only The Best Tags and People</h>"
],
"execution_count": 0,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/html": [
"<h1>Only The Best Tags and People</h>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {
"tags": []
}
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Lgxm56T_u38j",
"colab_type": "text"
},
"source": [
"##### Upload to Colab"
]
},
{
"cell_type": "code",
"metadata": {
"id": "iXE_tpgYvr7p",
"colab_type": "code",
"colab": {}
},
"source": [
"!pip install --upgrade -q gspread"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "C3OJaHBSvr69",
"colab_type": "code",
"outputId": "0100d30f-8c3b-4620-c837-cd0ee7e9e3f9",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 373
}
},
"source": [
"from google.colab import auth\n",
"auth.authenticate_user()\n",
"\n",
"import gspread\n",
"from oauth2client.client import GoogleCredentials\n",
"\n",
"gc = gspread.authorize(GoogleCredentials.get_application_default())\n",
"\n",
"worksheet = gc.open('DevOp-Wonder').sheet1\n",
"\n",
"# get_all_values gives a list of rows.\n",
"rows = worksheet.get_all_values()\n",
"print(rows)\n",
"\n",
"# Convert to a DataFrame and render.\n",
"import pandas as pd\n",
"df = pd.DataFrame.from_records(rows)\n",
"df"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"[['1'], ['2'], ['3'], ['4'], ['5'], ['6'], ['7'], ['8'], ['9'], ['10']]\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>6</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>8</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>9</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>10</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0\n",
"0 1\n",
"1 2\n",
"2 3\n",
"3 4\n",
"4 5\n",
"5 6\n",
"6 7\n",
"7 8\n",
"8 9\n",
"9 10"
]
},
"metadata": {
"tags": []
},
"execution_count": 23
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "6zYhaWrjXqxf",
"colab_type": "code",
"outputId": "3ae638a2-6b9c-4984-fe3a-1100d98ec7ce",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
}
},
"source": [
"df.median()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0 5.5\n",
"dtype: float64"
]
},
"metadata": {
"tags": []
},
"execution_count": 25
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "S64tLQHY16jh",
"colab_type": "text"
},
"source": [
"Upload your file here (request.csv)"
]
},
{
"cell_type": "code",
"metadata": {
"id": "ww-HcdVwuwEM",
"colab_type": "code",
"outputId": "dffc1597-274a-4793-e4b6-129abb615885",
"colab": {
"resources": {
"http://localhost:8080/nbextensions/google.colab/files.js": {
"data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7Ci8vIE1heCBhbW91bnQgb2YgdGltZSB0byBibG9jayB3YWl0aW5nIGZvciB0aGUgdXNlci4KY29uc3QgRklMRV9DSEFOR0VfVElNRU9VVF9NUyA9IDMwICogMTAwMDsKCmZ1bmN0aW9uIF91cGxvYWRGaWxlcyhpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IHN0ZXBzID0gdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKTsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIC8vIENhY2hlIHN0ZXBzIG9uIHRoZSBvdXRwdXRFbGVtZW50IHRvIG1ha2UgaXQgYXZhaWxhYmxlIGZvciB0aGUgbmV4dCBjYWxsCiAgLy8gdG8gdXBsb2FkRmlsZXNDb250aW51ZSBmcm9tIFB5dGhvbi4KICBvdXRwdXRFbGVtZW50LnN0ZXBzID0gc3RlcHM7CgogIHJldHVybiBfdXBsb2FkRmlsZXNDb250aW51ZShvdXRwdXRJZCk7Cn0KCi8vIFRoaXMgaXMgcm91Z2hseSBhbiBhc3luYyBnZW5lcmF0b3IgKG5vdCBzdXBwb3J0ZWQgaW4gdGhlIGJyb3dzZXIgeWV0KSwKLy8gd2hlcmUgdGhlcmUgYXJlIG11bHRpcGxlIGFzeW5jaHJvbm91cyBzdGVwcyBhbmQgdGhlIFB5dGhvbiBzaWRlIGlzIGdvaW5nCi8vIHRvIHBvbGwgZm9yIGNvbXBsZXRpb24gb2YgZWFjaCBzdGVwLgovLyBUaGlzIHVzZXMgYSBQcm9taXNlIHRvIGJsb2NrIHRoZSBweXRob24gc2lkZSBvbiBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcCwKLy8gdGhlbiBwYXNzZXMgdGhlIHJlc3VsdCBvZiB0aGUgcHJldmlvdXMgc3RlcCBhcyB0aGUgaW5wdXQgdG8gdGhlIG5leHQgc3RlcC4KZnVuY3Rpb24gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpIHsKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIGNvbnN0IHN0ZXBzID0gb3V0cHV0RWxlbWVudC5zdGVwczsKCiAgY29uc3QgbmV4dCA9IHN0ZXBzLm5leHQob3V0cHV0RWxlbWVudC5sYXN0UHJvbWlzZVZhbHVlKTsKICByZXR1cm4gUHJvbWlzZS5yZXNvbHZlKG5leHQudmFsdWUucHJvbWlzZSkudGhlbigodmFsdWUpID0+IHsKICAgIC8vIENhY2hlIHRoZSBsYXN0IHByb21pc2UgdmFsdWUgdG8gbWFrZSBpdCBhdmFpbGFibGUgdG8gdGhlIG5leHQKICAgIC8vIHN0ZXAgb2YgdGhlIGdlbmVyYXRvci4KICAgIG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSA9IHZhbHVlOwogICAgcmV0dXJuIG5leHQudmFsdWUucmVzcG9uc2U7CiAgfSk7Cn0KCi8qKgogKiBHZW5lcmF0b3IgZnVuY3Rpb24gd2hpY2ggaXMgY2FsbGVkIGJldHdlZW4gZWFjaCBhc3luYyBzdGVwIG9mIHRoZSB1cGxvYWQKICogcHJvY2Vzcy4KICogQHBhcmFtIHtzdHJpbmd9IGlucHV0SWQgRWxlbWVudCBJRCBvZiB0aGUgaW5wdXQgZmlsZSBwaWNrZXIgZWxlbWVudC4KICogQHBhcmFtIHtzdHJpbmd9IG91dHB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIG91dHB1dCBkaXNwbGF5LgogKiBAcmV0dXJuIHshSXRlcmFibGU8IU9iamVjdD59IEl0ZXJhYmxlIG9mIG5leHQgc3RlcHMuCiAqLwpmdW5jdGlvbiogdXBsb2FkRmlsZXNTdGVwKGlucHV0SWQsIG91dHB1dElkKSB7CiAgY29uc3QgaW5wdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoaW5wdXRJZCk7CiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gZmFsc2U7CgogIGNvbnN0IG91dHB1dEVsZW1lbnQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZChvdXRwdXRJZCk7CiAgb3V0cHV0RWxlbWVudC5pbm5lckhUTUwgPSAnJzsKCiAgY29uc3QgcGlja2VkUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBpbnB1dEVsZW1lbnQuYWRkRXZlbnRMaXN0ZW5lcignY2hhbmdlJywgKGUpID0+IHsKICAgICAgcmVzb2x2ZShlLnRhcmdldC5maWxlcyk7CiAgICB9KTsKICB9KTsKCiAgY29uc3QgY2FuY2VsID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnYnV0dG9uJyk7CiAgaW5wdXRFbGVtZW50LnBhcmVudEVsZW1lbnQuYXBwZW5kQ2hpbGQoY2FuY2VsKTsKICBjYW5jZWwudGV4dENvbnRlbnQgPSAnQ2FuY2VsIHVwbG9hZCc7CiAgY29uc3QgY2FuY2VsUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICBjYW5jZWwub25jbGljayA9ICgpID0+IHsKICAgICAgcmVzb2x2ZShudWxsKTsKICAgIH07CiAgfSk7CgogIC8vIENhbmNlbCB1cGxvYWQgaWYgdXNlciBoYXNuJ3QgcGlja2VkIGFueXRoaW5nIGluIHRpbWVvdXQuCiAgY29uc3QgdGltZW91dFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgc2V0VGltZW91dCgoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9LCBGSUxFX0NIQU5HRV9USU1FT1VUX01TKTsKICB9KTsKCiAgLy8gV2FpdCBmb3IgdGhlIHVzZXIgdG8gcGljayB0aGUgZmlsZXMuCiAgY29uc3QgZmlsZXMgPSB5aWVsZCB7CiAgICBwcm9taXNlOiBQcm9taXNlLnJhY2UoW3BpY2tlZFByb21pc2UsIHRpbWVvdXRQcm9taXNlLCBjYW5jZWxQcm9taXNlXSksCiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdzdGFydGluZycsCiAgICB9CiAgfTsKCiAgaWYgKCFmaWxlcykgewogICAgcmV0dXJuIHsKICAgICAgcmVzcG9uc2U6IHsKICAgICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICAgIH0KICAgIH07CiAgfQoKICBjYW5jZWwucmVtb3ZlKCk7CgogIC8vIERpc2FibGUgdGhlIGlucHV0IGVsZW1lbnQgc2luY2UgZnVydGhlciBwaWNrcyBhcmUgbm90IGFsbG93ZWQuCiAgaW5wdXRFbGVtZW50LmRpc2FibGVkID0gdHJ1ZTsKCiAgZm9yIChjb25zdCBmaWxlIG9mIGZpbGVzKSB7CiAgICBjb25zdCBsaSA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2xpJyk7CiAgICBsaS5hcHBlbmQoc3BhbihmaWxlLm5hbWUsIHtmb250V2VpZ2h0OiAnYm9sZCd9KSk7CiAgICBsaS5hcHBlbmQoc3BhbigKICAgICAgICBgKCR7ZmlsZS50eXBlIHx8ICduL2EnfSkgLSAke2ZpbGUuc2l6ZX0gYnl0ZXMsIGAgKwogICAgICAgIGBsYXN0IG1vZGlmaWVkOiAkewogICAgICAgICAgICBmaWxlLmxhc3RNb2RpZmllZERhdGUgPyBmaWxlLmxhc3RNb2RpZmllZERhdGUudG9Mb2NhbGVEYXRlU3RyaW5nKCkgOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbi9hJ30gLSBgKSk7CiAgICBjb25zdCBwZXJjZW50ID0gc3BhbignMCUgZG9uZScpOwogICAgbGkuYXBwZW5kQ2hpbGQocGVyY2VudCk7CgogICAgb3V0cHV0RWxlbWVudC5hcHBlbmRDaGlsZChsaSk7CgogICAgY29uc3QgZmlsZURhdGFQcm9taXNlID0gbmV3IFByb21pc2UoKHJlc29sdmUpID0+IHsKICAgICAgY29uc3QgcmVhZGVyID0gbmV3IEZpbGVSZWFkZXIoKTsKICAgICAgcmVhZGVyLm9ubG9hZCA9IChlKSA9PiB7CiAgICAgICAgcmVzb2x2ZShlLnRhcmdldC5yZXN1bHQpOwogICAgICB9OwogICAgICByZWFkZXIucmVhZEFzQXJyYXlCdWZmZXIoZmlsZSk7CiAgICB9KTsKICAgIC8vIFdhaXQgZm9yIHRoZSBkYXRhIHRvIGJlIHJlYWR5LgogICAgbGV0IGZpbGVEYXRhID0geWllbGQgewogICAgICBwcm9taXNlOiBmaWxlRGF0YVByb21pc2UsCiAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgYWN0aW9uOiAnY29udGludWUnLAogICAgICB9CiAgICB9OwoKICAgIC8vIFVzZSBhIGNodW5rZWQgc2VuZGluZyB0byBhdm9pZCBtZXNzYWdlIHNpemUgbGltaXRzLiBTZWUgYi82MjExNTY2MC4KICAgIGxldCBwb3NpdGlvbiA9IDA7CiAgICB3aGlsZSAocG9zaXRpb24gPCBmaWxlRGF0YS5ieXRlTGVuZ3RoKSB7CiAgICAgIGNvbnN0IGxlbmd0aCA9IE1hdGgubWluKGZpbGVEYXRhLmJ5dGVMZW5ndGggLSBwb3NpdGlvbiwgTUFYX1BBWUxPQURfU0laRSk7CiAgICAgIGNvbnN0IGNodW5rID0gbmV3IFVpbnQ4QXJyYXkoZmlsZURhdGEsIHBvc2l0aW9uLCBsZW5ndGgpOwogICAgICBwb3NpdGlvbiArPSBsZW5ndGg7CgogICAgICBjb25zdCBiYXNlNjQgPSBidG9hKFN0cmluZy5mcm9tQ2hhckNvZGUuYXBwbHkobnVsbCwgY2h1bmspKTsKICAgICAgeWllbGQgewogICAgICAgIHJlc3BvbnNlOiB7CiAgICAgICAgICBhY3Rpb246ICdhcHBlbmQnLAogICAgICAgICAgZmlsZTogZmlsZS5uYW1lLAogICAgICAgICAgZGF0YTogYmFzZTY0LAogICAgICAgIH0sCiAgICAgIH07CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPQogICAgICAgICAgYCR7TWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCl9JSBkb25lYDsKICAgIH0KICB9CgogIC8vIEFsbCBkb25lLgogIHlpZWxkIHsKICAgIHJlc3BvbnNlOiB7CiAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgIH0KICB9Owp9CgpzY29wZS5nb29nbGUgPSBzY29wZS5nb29nbGUgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYiA9IHNjb3BlLmdvb2dsZS5jb2xhYiB8fCB7fTsKc2NvcGUuZ29vZ2xlLmNvbGFiLl9maWxlcyA9IHsKICBfdXBsb2FkRmlsZXMsCiAgX3VwbG9hZEZpbGVzQ29udGludWUsCn07Cn0pKHNlbGYpOwo=",
"ok": true,
"headers": [
[
"content-type",
"application/javascript"
]
],
"status": 200,
"status_text": ""
}
},
"base_uri": "https://localhost:8080/",
"height": 346
}
},
"source": [
"from google.colab import files\n",
"uploaded = files.upload()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/html": [
"\n",
" <input type=\"file\" id=\"files-ad7a491e-0da2-4c6f-ac87-2e415fc45e76\" name=\"files[]\" multiple disabled />\n",
" <output id=\"result-ad7a491e-0da2-4c6f-ac87-2e415fc45e76\">\n",
" Upload widget is only available when the cell has been executed in the\n",
" current browser session. Please rerun this cell to enable.\n",
" </output>\n",
" <script src=\"/nbextensions/google.colab/files.js\"></script> "
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {
"tags": []
}
},
{
"output_type": "error",
"ename": "KeyboardInterrupt",
"evalue": "ignored",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-23-21dc3c638f66>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mgoogle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcolab\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mfiles\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0muploaded\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfiles\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m/usr/local/lib/python3.6/dist-packages/google/colab/files.py\u001b[0m in \u001b[0;36mupload\u001b[0;34m()\u001b[0m\n\u001b[1;32m 62\u001b[0m result = _output.eval_js(\n\u001b[1;32m 63\u001b[0m 'google.colab._files._uploadFiles(\"{input_id}\", \"{output_id}\")'.format(\n\u001b[0;32m---> 64\u001b[0;31m input_id=input_id, output_id=output_id))\n\u001b[0m\u001b[1;32m 65\u001b[0m \u001b[0mfiles\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_collections\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdefaultdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_six\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbinary_type\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 66\u001b[0m \u001b[0;31m# Mapping from original filename to filename as saved locally.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/local/lib/python3.6/dist-packages/google/colab/output/_js.py\u001b[0m in \u001b[0;36meval_js\u001b[0;34m(script, ignore_result)\u001b[0m\n\u001b[1;32m 37\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mignore_result\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 38\u001b[0m \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 39\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0m_message\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_reply_from_input\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrequest_id\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 40\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 41\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/local/lib/python3.6/dist-packages/google/colab/_message.py\u001b[0m in \u001b[0;36mread_reply_from_input\u001b[0;34m(message_id, timeout_sec)\u001b[0m\n\u001b[1;32m 99\u001b[0m \u001b[0mreply\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_read_next_input_message\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 100\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mreply\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0m_NOT_READY\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreply\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 101\u001b[0;31m \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msleep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0.025\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 102\u001b[0m \u001b[0;32mcontinue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 103\u001b[0m if (reply.get('type') == 'colab_reply' and\n",
"\u001b[0;31mKeyboardInterrupt\u001b[0m: "
]
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "O1cgxOH25NmQ",
"colab_type": "code",
"outputId": "8f314e55-1e71-404f-f38f-92c967dd160d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 103
}
},
"source": [
"!ls -l"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"total 712\n",
"drwx------ 4 root root 4096 Sep 9 16:16 gdrive\n",
"-rw-r--r-- 1 root root 407 Sep 9 16:20 magic_stuff.py\n",
"-rw-r--r-- 1 root root 712814 Sep 9 16:24 pytorch.pptx\n",
"drwxr-xr-x 1 root root 4096 Aug 27 16:17 sample_data\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "w4bssBq6QngE",
"colab_type": "code",
"outputId": "2d7678b6-95b9-4e9b-f8cf-f9039ea9d1f4",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 147
}
},
"source": [
"import pandas as pd\n",
"df = pd.read_csv(\"automl-tables_notebooks_census_income.csv\")\n",
"df.head(2)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>age</th>\n",
" <th>workclass</th>\n",
" <th>fnlwgt</th>\n",
" <th>education</th>\n",
" <th>education_num</th>\n",
" <th>marital_status</th>\n",
" <th>occupation</th>\n",
" <th>relationship</th>\n",
" <th>race</th>\n",
" <th>sex</th>\n",
" <th>capital_gain</th>\n",
" <th>capital_loss</th>\n",
" <th>hours_per_week</th>\n",
" <th>native_country</th>\n",
" <th>income</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>39</td>\n",
" <td>State-gov</td>\n",
" <td>77516</td>\n",
" <td>Bachelors</td>\n",
" <td>13</td>\n",
" <td>Never-married</td>\n",
" <td>Adm-clerical</td>\n",
" <td>Not-in-family</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>2174</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" <td>United-States</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>50</td>\n",
" <td>Self-emp-not-inc</td>\n",
" <td>83311</td>\n",
" <td>Bachelors</td>\n",
" <td>13</td>\n",
" <td>Married-civ-spouse</td>\n",
" <td>Exec-managerial</td>\n",
" <td>Husband</td>\n",
" <td>White</td>\n",
" <td>Male</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>13</td>\n",
" <td>United-States</td>\n",
" <td>&lt;=50K</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" age workclass fnlwgt ... hours_per_week native_country income\n",
"0 39 State-gov 77516 ... 40 United-States <=50K\n",
"1 50 Self-emp-not-inc 83311 ... 13 United-States <=50K\n",
"\n",
"[2 rows x 15 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 22
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "rUqRwiunUDxP",
"colab_type": "code",
"outputId": "e71f3190-8667-460b-cc03-23aabe3e999e",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 202
}
},
"source": [
"import pandas as pd\n",
"df = pd.read_csv(\"https://raw.githubusercontent.com/noahgift/sugar/master/data/education_sugar_cdc_2003.csv\")\n",
"df.head()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>State</th>\n",
" <th>Employed</th>\n",
" <th>Not employed</th>\n",
" <th>Retired</th>\n",
" <th>&lt;High school</th>\n",
" <th>High school</th>\n",
" <th>Some college</th>\n",
" <th>College graduate</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Alaska</td>\n",
" <td>26.2 (23.6–28.9)</td>\n",
" <td>32.1 (27.8–36.8)</td>\n",
" <td>16.0 (12.6–20.2)</td>\n",
" <td>47.1 (37.8–56.5)</td>\n",
" <td>34.9 (31.1–38.9)</td>\n",
" <td>24.2 (21.0–27.8)</td>\n",
" <td>12.9 (10.5–15.7)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Arizona</td>\n",
" <td>33.0 (28.5–37.8)</td>\n",
" <td>28.7 (23.5–34.5)</td>\n",
" <td>13.8 (10.8–17.5)</td>\n",
" <td>40.4 (30.9–50.7)</td>\n",
" <td>36.5 (30.7–42.7)</td>\n",
" <td>24.4 (19.9–29.4)</td>\n",
" <td>14.6 (11.6–18.3)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>California</td>\n",
" <td>22.9 (20.9–25.1)</td>\n",
" <td>30.2 (27.1–33.4)</td>\n",
" <td>15.0 (12.2–18.2)</td>\n",
" <td>38.5 (34.2–43.0)</td>\n",
" <td>29.9 (26.5–33.7)</td>\n",
" <td>21.4 (18.8–24.2)</td>\n",
" <td>11.5 (9.8–13.5)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Connecticut</td>\n",
" <td>18.9 (17.1–20.9)</td>\n",
" <td>24.3 (20.8–28.2)</td>\n",
" <td>15.0 (12.7–17.7)</td>\n",
" <td>27.8 (22.4–33.9)</td>\n",
" <td>26.9 (23.7–30.3)</td>\n",
" <td>19.9 (17.2–23.0)</td>\n",
" <td>10.2 (8.7–12.0)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>District of Columbia</td>\n",
" <td>18.5 (15.7–21.7)</td>\n",
" <td>34.6 (29.5–40.1)</td>\n",
" <td>18.5 (15.3–22.1)</td>\n",
" <td>45.6 (36.4–55.2)</td>\n",
" <td>39.0 (33.1–45.2)</td>\n",
" <td>28.9 (23.4–35.0)</td>\n",
" <td>8.4 (7.0–10.1)</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" State Employed ... Some college College graduate\n",
"0 Alaska 26.2 (23.6–28.9) ... 24.2 (21.0–27.8) 12.9 (10.5–15.7)\n",
"1 Arizona 33.0 (28.5–37.8) ... 24.4 (19.9–29.4) 14.6 (11.6–18.3)\n",
"2 California 22.9 (20.9–25.1) ... 21.4 (18.8–24.2) 11.5 (9.8–13.5)\n",
"3 Connecticut 18.9 (17.1–20.9) ... 19.9 (17.2–23.0) 10.2 (8.7–12.0)\n",
"4 District of Columbia 18.5 (15.7–21.7) ... 28.9 (23.4–35.0) 8.4 (7.0–10.1)\n",
"\n",
"[5 rows x 8 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 27
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "X97XO2e8_reX",
"colab_type": "code",
"outputId": "e9cb5064-e892-4b66-ab6f-17259a8cbea7",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 171
}
},
"source": [
"df.describe()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>State</th>\n",
" <th>Employed</th>\n",
" <th>Not employed</th>\n",
" <th>Retired</th>\n",
" <th>&lt;High school</th>\n",
" <th>High school</th>\n",
" <th>Some college</th>\n",
" <th>College graduate</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>24</td>\n",
" <td>24</td>\n",
" <td>24</td>\n",
" <td>24</td>\n",
" <td>24</td>\n",
" <td>24</td>\n",
" <td>24</td>\n",
" <td>24</td>\n",
" </tr>\n",
" <tr>\n",
" <th>unique</th>\n",
" <td>24</td>\n",
" <td>24</td>\n",
" <td>24</td>\n",
" <td>24</td>\n",
" <td>24</td>\n",
" <td>24</td>\n",
" <td>23</td>\n",
" <td>24</td>\n",
" </tr>\n",
" <tr>\n",
" <th>top</th>\n",
" <td>New York</td>\n",
" <td>18.9 (17.1–20.9)</td>\n",
" <td>47.7 (44.7–50.8)</td>\n",
" <td>20.8 (18.7–23.1)</td>\n",
" <td>35.5 (27.5–44.5)</td>\n",
" <td>29.5 (25.4–33.9)</td>\n",
" <td>24.2 (21.0–27.8)</td>\n",
" <td>14.6 (11.6–18.3)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>freq</th>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" State Employed ... Some college College graduate\n",
"count 24 24 ... 24 24\n",
"unique 24 24 ... 23 24\n",
"top New York 18.9 (17.1–20.9) ... 24.2 (21.0–27.8) 14.6 (11.6–18.3)\n",
"freq 1 1 ... 2 1\n",
"\n",
"[4 rows x 8 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 28
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0kmgxW7Sxa6j",
"colab_type": "text"
},
"source": [
"##### Forms in Colab"
]
},
{
"cell_type": "code",
"metadata": {
"id": "ZqHMr_brxdkA",
"colab_type": "code",
"colab": {}
},
"source": [
"Use_Python = False #@param [\"False\", \"True\"] {type:\"raw\"}"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "d1z9fz5_x_bf",
"colab_type": "code",
"outputId": "70b54aa1-9618-410c-c032-87702c2e02e3",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"source": [
"print(f\"You select it is {Use_Python} you use Python\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"You select it is True you use Python\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kn-P-e_hjvK5",
"colab_type": "text"
},
"source": [
"#### Python executable\n",
"\n",
"Can run scripts, REPL and even run python statements with -c flag and semicolon to string together multiple statements"
]
},
{
"cell_type": "code",
"metadata": {
"id": "p2aDegQxjegv",
"colab_type": "code",
"outputId": "5935e352-e9b0-459f-95fc-72d7e13aa4ed",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"!python -c \"import os;print(os.listdir())\""
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"['.config', 'magic_stuff.py', 'adc.json', 'sample_data']\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "WaXhDeUM_49R",
"colab_type": "code",
"outputId": "a7e93d6a-c415-4674-ae77-38313512825e",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 87
}
},
"source": [
"!ls -l"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"total 12\n",
"-rw-r--r-- 1 root root 2610 Jan 8 13:34 adc.json\n",
"-rw-r--r-- 1 root root 407 Jan 8 13:27 magic_stuff.py\n",
"drwxr-xr-x 1 root root 4096 Dec 18 16:52 sample_data\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "58zCAVUC_8JP",
"colab_type": "code",
"colab": {}
},
"source": [
"!pip install -q boto3"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "egHxTJUGNYCA",
"colab_type": "code",
"outputId": "2f00f806-4451-4874-f08c-e925457ad576",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 69
}
},
"source": [
"#this is how you capture input to a program\n",
"import sys;sys.argv"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py',\n",
" '-f',\n",
" '/root/.local/share/jupyter/runtime/kernel-10c490d0-6434-4157-b5c7-7cb291feb224.json']"
]
},
"metadata": {
"tags": []
},
"execution_count": 35
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6HJtGuiKUYFU",
"colab_type": "text"
},
"source": [
"### Introductory Concepts\n",
"* **Procedural Statements**\n",
"* Strings and String Formatting\n",
"* Numbers and Arithmetic Operations\n",
"* Data Structures\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BwmnQEziVQ9k",
"colab_type": "text"
},
"source": [
" #### Procedural Statements\n",
" Procedural statements are literally statements that can be issued one line at a time. Below are types of procedural statements. These statements can be run in:\n",
" * Jupyter Notebook\n",
" * IPython shell\n",
" * Python interpreter\n",
" * Python scripts"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yTePE3imVqfP",
"colab_type": "text"
},
"source": [
"**Printing**"
]
},
{
"cell_type": "code",
"metadata": {
"id": "n1r0n2V3Vs70",
"colab_type": "code",
"outputId": "3d894f98-a68b-485d-b8ae-bb2ab2bb485c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"print(\"Hello world\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Hello world\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8nUjNnApV6cC",
"colab_type": "text"
},
"source": [
"**Create Variable and Use Variable**"
]
},
{
"cell_type": "code",
"metadata": {
"id": "mhjO2LpSVx9V",
"colab_type": "code",
"outputId": "3f806751-3b00-4bcd-91a8-8f4e3ae1af9e",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"variable = \"armbar\"\n",
"variable"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'armbar'"
]
},
"metadata": {
"tags": []
},
"execution_count": 37
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "H004n8R-WDGF",
"colab_type": "text"
},
"source": [
"**Multiple procedural statements**"
]
},
{
"cell_type": "code",
"metadata": {
"id": "xrZWZEFmV_bN",
"colab_type": "code",
"outputId": "873196c3-1e1d-4635-a9e1-6ec5ed5f8b32",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
}
},
"source": [
"attack_one = \"kimura\"\n",
"attack_two = \"arm triangle\"\n",
"print(\"In Brazilian Jiu Jitsu a common attack is a:\", attack_one)\n",
"print(\"Another common attack is a:\", attack_two)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"In Brazilian Jiu Jitsu a common attack is a: kimura\n",
"Another common attack is a: arm triangle\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6IiZ6RnLWKsq",
"colab_type": "text"
},
"source": [
"**Adding Numbers**"
]
},
{
"cell_type": "code",
"metadata": {
"id": "h049FhlhWPRm",
"colab_type": "code",
"outputId": "a89116f8-0bec-4398-9f03-b3947dcd2aec",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"1+1"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"2"
]
},
"metadata": {
"tags": []
},
"execution_count": 39
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mlszsN87WmcO",
"colab_type": "text"
},
"source": [
"**Adding Phrases**"
]
},
{
"cell_type": "code",
"metadata": {
"id": "FaE5GnCaWgbI",
"colab_type": "code",
"outputId": "7a956ac3-30e5-4b18-a1f2-ccbcd75ff7c0",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"\"arm\" + \" bar\"+\" 4\"+\" morestuff \" + \"lemon\""
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'arm bar 4 morestuff lemon'"
]
},
"metadata": {
"tags": []
},
"execution_count": 40
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WSrfrIO1XYts",
"colab_type": "text"
},
"source": [
"**Complex statements**\n",
"\n",
"More complex statements can be created that use data structures like the belts variable, which is a list."
]
},
{
"cell_type": "code",
"metadata": {
"id": "0nf0W0e3XSQS",
"colab_type": "code",
"outputId": "9fdf8979-553e-4512-bc7f-ff5d74a2db83",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 104
}
},
"source": [
"belts = [\"white\", \"blue\", \"purple\", \"brown\", \"black\"]\n",
"for belt in belts:\n",
" if \"black\" in belt:\n",
" print(\"The belt I want to be is:\", belt)\n",
" else:\n",
" print(\"This is not the belt I want to end up at:\", belt)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"This is not the belt I want to end up at: white\n",
"This is not the belt I want to end up at: blue\n",
"This is not the belt I want to end up at: purple\n",
"This is not the belt I want to end up at: brown\n",
"The belt I want to be is: black\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UYzPEDtnXlbx",
"colab_type": "text"
},
"source": [
"#### Strings and String Formatting\n",
"\n",
"Strings are a sequence of characters and they are often programmatically formatted. Almost all Python programs have strings because they can be used to send messages to users who use the program. When creating strings there are few core concepts to understand:\n",
"\n",
"* Strings can be create with the single, double and triple/double quotes\n",
"* Strings are can be formatted\n",
"* One complication of strings is they can be encoded in several formats including unicode\n",
"* Many methods are available to operate on strings. In an editor or IPython shell you can see these methods by tab completion: \n",
"```\n",
"basic_string.\n",
" capitalize() format() islower() lower() rpartition() title() \n",
" casefold() format_map() isnumeric() lstrip() rsplit() translate() \n",
" center() index() isprintable() maketrans() rstrip() upper() \n",
" count() isalnum() isspace() partition() split() zfill() \n",
" encode() isalpha() istitle() replace() splitlines() \n",
" endswith() isdecimal() isupper() rfind() startswith() \n",
" expandtabs() isdigit() join() rindex() strip() \n",
" find() isidentifier() ljust() rjust() swapcase() \n",
"```"
]
},
{
"cell_type": "code",
"metadata": {
"id": "eHJX2ZFiXbFC",
"colab_type": "code",
"colab": {}
},
"source": [
"my_string = \"this is a string I am using this time\"\n",
"#my_string.split()\n",
"#my_string.upper()\n",
"#my_string.title()\n",
"#my_string.count(\"this\")"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "j1uXZqdXZdHf",
"colab_type": "code",
"outputId": "8d75b9f4-f194-485f-f49b-2099cc090628",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"my_string.title()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'This Is A String I Am Using This Time'"
]
},
"metadata": {
"tags": []
},
"execution_count": 46
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "DMBeWJv5claa",
"colab_type": "code",
"outputId": "921c2fb8-5536-46e5-9ad7-b72efd4e4fce",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"my_string.capitalize()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'This is a string i am using this time'"
]
},
"metadata": {
"tags": []
},
"execution_count": 47
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "-qK6mSH0xxCY",
"colab_type": "code",
"outputId": "ed7e1d92-3bd5-4804-8ab8-91924fd6afd1",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"my_string.isnumeric()\n",
"\n"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"False"
]
},
"metadata": {
"tags": []
},
"execution_count": 48
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "jKa_uTtpwwtm",
"colab_type": "code",
"outputId": "aa8097e7-da0f-4041-80ae-40dd73ca0e4c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 69
}
},
"source": [
"print(my_string)\n",
"var2 = my_string.swapcase()\n",
"print(var2)\n",
"print(var2.swapcase())"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"this is a string I am using this time\n",
"THIS IS A STRING i AM USING THIS TIME\n",
"this is a string I am using this time\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8afQjS25YDvQ",
"colab_type": "text"
},
"source": [
"**Basic String**"
]
},
{
"cell_type": "code",
"metadata": {
"id": "yfhfvyIpX-pc",
"colab_type": "code",
"colab": {}
},
"source": [
"basic_string = \"Brazilian Jiu Jitsu\""
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "GMYqc-cKYJlL",
"colab_type": "text"
},
"source": [
"**Splitting String**\n",
"\n",
"Turn a string in a list by splitting on spaces, or some other thing"
]
},
{
"cell_type": "code",
"metadata": {
"id": "eJaoPyTzYGSk",
"colab_type": "code",
"outputId": "c4fc61f8-61d7-4c44-80ac-79252280fbfd",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"#split on spaces (default)\n",
"basic_string.split()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['Brazilian', 'Jiu', 'Jitsu']"
]
},
"metadata": {
"tags": []
},
"execution_count": 54
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "jt4NbwL15MBl",
"colab_type": "code",
"outputId": "feab5868-bfec-4d7a-bd17-43ae6318bff4",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"result = basic_string.split()\n",
"len(result)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"3"
]
},
"metadata": {
"tags": []
},
"execution_count": 55
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "z94Ajl0NZXBP",
"colab_type": "code",
"outputId": "180e8459-d4bd-4c1a-8ada-56a60b359ce0",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"#split on hyphen\n",
"string_with_hyphen = \"Brazilian-Jiu-Jitsu\"\n",
"string_with_hyphen.split(\"-\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['Brazilian', 'Jiu', 'Jitsu']"
]
},
"metadata": {
"tags": []
},
"execution_count": 56
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "EsPKeSMA5dnP",
"colab_type": "code",
"outputId": "d772e707-db97-44ac-ba62-5d119bb7ffee",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"#split on comma\n",
"string_with_hyphen = \"Brazilian,Jiu,Jitsu\"\n",
"string_with_hyphen.split(\",\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['Brazilian', 'Jiu', 'Jitsu']"
]
},
"metadata": {
"tags": []
},
"execution_count": 58
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ti9hySuGZbbX",
"colab_type": "text"
},
"source": [
"**All Capital**\n",
"\n",
"Turn a string into all Capital Letter"
]
},
{
"cell_type": "code",
"metadata": {
"id": "y2NEPM8PZZT_",
"colab_type": "code",
"outputId": "34b86bba-b410-4d0b-bc87-98428be6b1f5",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"basic_string.capitalize()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'Brazilian jiu jitsu'"
]
},
"metadata": {
"tags": []
},
"execution_count": 59
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZlifO6c4Zk5W",
"colab_type": "text"
},
"source": [
"**Slicing Strings**\n",
"\n",
"Strings can be referenced by length and sliced"
]
},
{
"cell_type": "code",
"metadata": {
"id": "tBXlXz0TZh1Q",
"colab_type": "code",
"outputId": "ba2369c7-7c9f-49e5-9677-defa402fdf2d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"#Get the last character\n",
"basic_string[-1:]"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'u'"
]
},
"metadata": {
"tags": []
},
"execution_count": 60
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "dvNrJzBNyvZz",
"colab_type": "code",
"outputId": "3a785520-b723-4674-9abd-4fc12265598d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"len(basic_string[2:])"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"17"
]
},
"metadata": {
"tags": []
},
"execution_count": 61
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "EhgFdziaZtPv",
"colab_type": "code",
"outputId": "209c64d6-ae19-4b8a-eced-26e6c188ea23",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"#Get length of string\n",
"len(basic_string)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"19"
]
},
"metadata": {
"tags": []
},
"execution_count": 62
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "neXScIYHD75C",
"colab_type": "code",
"outputId": "55e768a8-7a0d-42f9-df9e-4a780eb07897",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"basic_string[-18:]"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'razilian Jiu Jitsu'"
]
},
"metadata": {
"tags": []
},
"execution_count": 63
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lxvtolozZy1f",
"colab_type": "text"
},
"source": [
"**Strings Can Be Added Together**"
]
},
{
"cell_type": "code",
"metadata": {
"id": "eIVgqMsRZvNQ",
"colab_type": "code",
"outputId": "9f76f4ab-2610-4955-fab4-d041206ea5d6",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"basic_string + \" is my favorite Martial Art\""
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'Brazilian Jiu Jitsu is my favorite Martial Art'"
]
},
"metadata": {
"tags": []
},
"execution_count": 64
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "tdga91RBzMb_",
"colab_type": "code",
"outputId": "9d3dfc12-cb4e-4b59-fec6-47f92afd6d08",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"items = [\"-\",1,2,3]\n",
"for item in items:\n",
" basic_string += str(item)\n",
"basic_string"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'Brazilian Jiu Jitsu-123'"
]
},
"metadata": {
"tags": []
},
"execution_count": 65
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "trK8HQJ1Mx6E",
"colab_type": "code",
"outputId": "780fa121-265e-4e58-dd76-a085e4928f8f",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"\"this is a string format: %s\" % \"wow\""
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'this is a string format: wow'"
]
},
"metadata": {
"tags": []
},
"execution_count": 66
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nNYxshNHZ6O8",
"colab_type": "text"
},
"source": [
"**F-Strings Can Be Formatted in More Complex Ways**\n",
"\n",
"One of the best ways to format a string in modern Python 3 is to use f-strings"
]
},
{
"cell_type": "code",
"metadata": {
"id": "u_NAcD5sZ1l4",
"colab_type": "code",
"outputId": "642376d6-16ca-44b7-9199-c747ed772f31",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"f'I love practicing my favorite Martial Art, {basic_string}'"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'I love practicing my favorite Martial Art, Brazilian Jiu Jitsu-123'"
]
},
"metadata": {
"tags": []
},
"execution_count": 67
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wlpMuwdmbui0",
"colab_type": "text"
},
"source": [
"**Strings Can Use Triple Quotes to Wrap**"
]
},
{
"cell_type": "code",
"metadata": {
"id": "YWmNAWzqbT4k",
"colab_type": "code",
"outputId": "82d3355c-d766-4d48-a65d-36965122b9be",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 55
}
},
"source": [
"f\"\"\"\n",
"This phrase is multiple sentenances long.\n",
"There phrase can be formatted like simpler sentances,\n",
"for example, I can still talk about my favorite Martial Art {basic_string}\n",
"\"\"\""
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'\\nThis phrase is multiple sentenances long.\\nThere phrase can be formatted like simpler sentances,\\nfor example, I can still talk about my favorite Martial Art Brazilian Jiu Jitsu-123\\n'"
]
},
"metadata": {
"tags": []
},
"execution_count": 69
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mBuMTMQWb239",
"colab_type": "text"
},
"source": [
"**Line Breaks Can Be Removed with Replace**\n",
"\n",
"The last long line contained line breaks, which are the **\\n** character, and they can be removed by using the replace method"
]
},
{
"cell_type": "code",
"metadata": {
"id": "K8AVf6Pibz3H",
"colab_type": "code",
"outputId": "3c43e6fd-35cb-4c48-ff65-5cf1e1e1ef59",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 55
}
},
"source": [
"f\"\"\"\n",
"This phrase is multiple sentenances long.\n",
"There phrase can be formatted like simpler sentances,\n",
"for example, I can still talk about my favorite Martial Art {basic_string}\n",
"\"\"\".replace(\"\\n\", \" \")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"' This phrase is multiple sentenances long. There phrase can be formatted like simpler sentances, for example, I can still talk about my favorite Martial Art Brazilian Jiu Jitsu-123 '"
]
},
"metadata": {
"tags": []
},
"execution_count": 70
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eRmipgSocZ9i",
"colab_type": "text"
},
"source": [
"#### Numbers and Arithmetic Operations\n",
"\n",
"Python is also a built-in calculator. Without installing any additional libraries it can do many simple and complex arithmetic operations."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xd6LxgEXcgP8",
"colab_type": "text"
},
"source": [
"**Adding and Subtracting Numbers**"
]
},
{
"cell_type": "code",
"metadata": {
"id": "mKwZGBoMb6_A",
"colab_type": "code",
"outputId": "791121d7-546b-4635-a65b-c4c15bab01e8",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"steps = (1+1)-1\n",
"print(f\"Two Steps Forward: One Step Back = {steps}\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Two Steps Forward: One Step Back = 1\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nMKS0ZYkdjvp",
"colab_type": "text"
},
"source": [
"**Multiplication with Decimals**\n",
"\n",
"Can use float type to solve decimal problems"
]
},
{
"cell_type": "code",
"metadata": {
"id": "Xgrw0LDucinh",
"colab_type": "code",
"outputId": "2072365a-bcaa-4ca9-bc34-d8f88c1b37a2",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"body_fat_percentage = 0.10\n",
"weight = 200\n",
"fat_total = body_fat_percentage * weight\n",
"print(f\"I weight 200lbs, and {fat_total}lbs of that is fat\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"I weight 200lbs, and 20.0lbs of that is fat\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gjMkBnuxiCQW",
"colab_type": "text"
},
"source": [
"Can also use Decimal Library to set precision and deal with repeating decimal\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "GN_PYDuDiBHg",
"colab_type": "code",
"outputId": "39f1669e-0b12-4473-9143-29cd3a4ac33c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"from decimal import (Decimal, getcontext)\n",
"\n",
"getcontext().prec = 5\n",
"Decimal(1)/Decimal(3)\n",
"\n"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"Decimal('0.33333')"
]
},
"metadata": {
"tags": []
},
"execution_count": 76
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f1_B6OUrdxlU",
"colab_type": "text"
},
"source": [
"**Using Exponents**\n",
"\n",
"Using the Python math library it is straightforward to call 2 to the 3rd power"
]
},
{
"cell_type": "code",
"metadata": {
"id": "Q2oC1HVhdmB9",
"colab_type": "code",
"outputId": "d522b640-d8e1-45d8-e2ec-8d5df5a5c4c7",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"import math\n",
"math.pow(2,4)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"16.0"
]
},
"metadata": {
"tags": []
},
"execution_count": 77
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nkPcDry7jWt-",
"colab_type": "text"
},
"source": [
"Can also use built in exponent operator to accomplish same thing"
]
},
{
"cell_type": "code",
"metadata": {
"id": "9aV-R-DljXEj",
"colab_type": "code",
"outputId": "75f8176c-2f8c-4154-8fb7-ab21270c5f18",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"2**4"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"16"
]
},
"metadata": {
"tags": []
},
"execution_count": 79
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "EOHXLdCU8R6i",
"colab_type": "code",
"outputId": "21408286-186c-4400-eb58-f6b8a958409a",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"2**4"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"16"
]
},
"metadata": {
"tags": []
},
"execution_count": 80
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aYA_mApaI-Y7",
"colab_type": "text"
},
"source": [
"this is regular multiplication"
]
},
{
"cell_type": "code",
"metadata": {
"id": "6A_Vdvl_I8lg",
"colab_type": "code",
"outputId": "2c50d568-3a66-4e30-bd82-a68d98a84045",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"2*3"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"6"
]
},
"metadata": {
"tags": []
},
"execution_count": 81
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "P6gOf9qtd6Nt",
"colab_type": "text"
},
"source": [
"**Converting Between different numerical types**\n",
"\n",
"There are many numerical forms to be aware of in Python.\n",
"A couple of the most common are:\n",
"\n",
"* Integers\n",
"* Floats"
]
},
{
"cell_type": "code",
"metadata": {
"id": "XCWg7yX-d210",
"colab_type": "code",
"outputId": "558f25c5-a077-4b0d-f22f-591ae821e570",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"number = 100\n",
"num_type = type(number).__name__\n",
"print(f\"{number} is type [{num_type}]\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"100 is type [int]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "iW2f1Ik7eAzd",
"colab_type": "code",
"outputId": "8ebeebca-b548-4ad2-b09b-db3007332426",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"number = float(100)\n",
"num_type = type(number).__name__\n",
"print(f\"{number} is type [{num_type}]\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"100.0 is type [float]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "zrlLg22C6KbR",
"colab_type": "code",
"outputId": "55fcd6b3-9516-45e7-abb3-fab587c26d78",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"num2 = 100.20\n",
"type(num2)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"float"
]
},
"metadata": {
"tags": []
},
"execution_count": 86
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "5dnSfoXZFQuP",
"colab_type": "code",
"colab": {}
},
"source": [
"class Foo:pass\n",
"f = Foo()"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "44v5W-HdFU4J",
"colab_type": "code",
"outputId": "ec912a86-418a-4469-c0e2-d986fd841d8c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"type(f)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"__main__.Foo"
]
},
"metadata": {
"tags": []
},
"execution_count": 88
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MrrFXb99gQ1Z",
"colab_type": "text"
},
"source": [
"**Numbers can also be rounded**\n",
"\n",
"Python Built in round "
]
},
{
"cell_type": "code",
"metadata": {
"id": "Wjtnfol2iZDL",
"colab_type": "code",
"outputId": "3e93a076-310b-437b-8457-0b408fa98800",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"too_many_decimals = 1.912345897\n",
"round(too_many_decimals, 6)\n",
"#get more info\n",
"#round?"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1.912346"
]
},
"metadata": {
"tags": []
},
"execution_count": 89
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SuJDUTLFWUJz",
"colab_type": "text"
},
"source": [
"Numpy round"
]
},
{
"cell_type": "code",
"metadata": {
"id": "PEabuAGGWXIF",
"colab_type": "code",
"outputId": "a64e2566-4af9-4797-a3c9-2e90815ecbd4",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"import numpy as np\n",
"np.round(too_many_decimals, 6)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1.912346"
]
},
"metadata": {
"tags": []
},
"execution_count": 90
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "X3aHIe6qW8ab",
"colab_type": "text"
},
"source": [
"Pandas round"
]
},
{
"cell_type": "code",
"metadata": {
"id": "J-x_sjiZW-gB",
"colab_type": "code",
"outputId": "8db357f2-253d-4dca-8cc6-07b75a5d0481",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 79
}
},
"source": [
"import pandas as pd\n",
"df = pd.DataFrame([too_many_decimals], columns=[\"A\"], index=[\"first\"])\n",
"df.round(2)\n"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>A</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>first</th>\n",
" <td>1.91</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" A\n",
"first 1.91"
]
},
"metadata": {
"tags": []
},
"execution_count": 92
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "elnncfa0XtOt",
"colab_type": "text"
},
"source": [
"Simple benchmark of all three (**Python**, **numpy** and **Pandas** round): using **%timeit**\n",
"\n",
"*Depending on what is getting rounded (i.e. a very large DataFrame, performance may very, so knowing how to benchmark performance is important with round) *\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "Yr1SXcF5Xw-L",
"colab_type": "code",
"outputId": "9731ad35-030a-4af8-9ba1-97962a003928",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 156
}
},
"source": [
"print(\"built in Python Round\")\n",
"%timeit round(too_many_decimals, 2)\n",
"\n",
"print(\"numpy round\")\n",
"%timeit np.round(too_many_decimals, 2)\n",
"\n",
"print(\"Pandas DataFrame round\")\n",
"%timeit df.round(2)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"built in Python Round\n",
"The slowest run took 28.19 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
"1000000 loops, best of 3: 431 ns per loop\n",
"numpy round\n",
"The slowest run took 9.56 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
"100000 loops, best of 3: 8.26 µs per loop\n",
"Pandas DataFrame round\n",
"1000 loops, best of 3: 891 µs per loop\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CuJ2OPSZiewO",
"colab_type": "text"
},
"source": [
"### Presentation: Lists and Dictionaries\n",
"Python has a couple of core Data Structures that are used very frequently\n",
"\n",
"* Lists\n",
"* Dictionaries\n",
"\n",
"Dictionaries and lists are the real workhorses of Python, but there are also other Data Structers like tuples, sets, Counters, etc, that are worth exploring too."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rYrg60aboNTc",
"colab_type": "text"
},
"source": [
"#### Python Dictionaries\n",
"\n",
"The workhorse of Python datastructures"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TJgbX5wb7yHD",
"colab_type": "text"
},
"source": [
"##### Creating Python Dictionaries\n",
"\n",
"Creating Python Dictionaries can be done with* brackets {}*"
]
},
{
"cell_type": "code",
"metadata": {
"id": "lPJxAjedZGGg",
"colab_type": "code",
"colab": {}
},
"source": [
"#bad_dictionary = {[2]:\"one\"}"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "VtlC50y3GLam",
"colab_type": "code",
"colab": {}
},
"source": [
"new_dictionary = {\"one\":1}"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "IEi6YTS2iZj7",
"colab_type": "code",
"outputId": "d5a9b5ea-d0b4-4a0f-a52e-12249f6c0efa",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 87
}
},
"source": [
"submissions = {\"armbar\": \"upper_body\", \n",
" \"arm_triangle\": \"upper_body\", \n",
" \"heel_hook\": \"lower_body\", \n",
" \"knee_bar\": \"lower_body\"}\n",
"#type(submissions)\n",
"#submissions.items?\n",
"submissions"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"{'arm_triangle': 'upper_body',\n",
" 'armbar': 'upper_body',\n",
" 'heel_hook': 'lower_body',\n",
" 'knee_bar': 'lower_body'}"
]
},
"metadata": {
"tags": []
},
"execution_count": 99
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "4guKbqgfink8",
"colab_type": "code",
"outputId": "5d88965e-2595-4f8a-9a1f-244c638ea34d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"new_dict =dict(upper_body=\"lower_body\")\n",
"new_dict"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"{'upper_body': 'lower_body'}"
]
},
"metadata": {
"tags": []
},
"execution_count": 100
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TjNRS-YOjYxw",
"colab_type": "text"
},
"source": [
"##### Using Python Dictionaries\n",
"A common dictionary usage pattern is to *iterate* on a dictionary by using the items method. In the example below the key and the value are printed:"
]
},
{
"cell_type": "code",
"metadata": {
"id": "oE1bzsiFGaOj",
"colab_type": "code",
"colab": {}
},
"source": [
"#submissions.items?\n"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "kW1UR_yMi5bl",
"colab_type": "code",
"outputId": "9f979027-7a76-4313-9690-56073000f6af",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 87
}
},
"source": [
"for submission, body_part in submissions.items():\n",
" print(f\"The {submission} is an attack on the {body_part}\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"The armbar is an attack on the upper_body\n",
"The arm_triangle is an attack on the upper_body\n",
"The heel_hook is an attack on the lower_body\n",
"The knee_bar is an attack on the lower_body\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5cue3xbzjfhF",
"colab_type": "text"
},
"source": [
"Dictionaries can also be used to *filter*. In the example below, only the submission attacks on the lower body are displayed:"
]
},
{
"cell_type": "code",
"metadata": {
"id": "c00Rxx84HAES",
"colab_type": "code",
"outputId": "1d5a0eb3-3c3f-4cbd-8419-a9add42f1969",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 87
}
},
"source": [
"for _, body_parts in submissions.items():\n",
" print(body_parts)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"upper_body\n",
"upper_body\n",
"lower_body\n",
"lower_body\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "buIoEo_Ajc1-",
"colab_type": "code",
"outputId": "de59f87e-6976-4f92-fb7b-16fa709b85ec",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 69
}
},
"source": [
"print(f\"These are lower_body submission attacks in Brazilian Jiu Jitsu:\")\n",
"for submission, body_part in submissions.items():\n",
" if body_part == \"lower_body\":\n",
" print(submission)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"These are lower_body submission attacks in Brazilian Jiu Jitsu:\n",
"heel_hook\n",
"knee_bar\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9jkR5VTHjsc2",
"colab_type": "text"
},
"source": [
"Dictionary keys and values can also be selected with built in *keys() * and *values()* methods"
]
},
{
"cell_type": "code",
"metadata": {
"id": "SviUgzEfjjD-",
"colab_type": "code",
"outputId": "d0edb393-56d1-47da-9e2c-ae60e7e3251a",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
}
},
"source": [
"print(f\"These are keys: {submissions.keys()}\")\n",
"print(f\"These are values: {submissions.values()}\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"These are keys: dict_keys(['armbar', 'arm_triangle', 'heel_hook', 'knee_bar'])\n",
"These are values: dict_values(['upper_body', 'upper_body', 'lower_body', 'lower_body'])\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sgPM6V_slmoy",
"colab_type": "text"
},
"source": [
"Key lookup is very performant, and one of the most common ways to use a dictionary."
]
},
{
"cell_type": "code",
"metadata": {
"id": "lOn5k2kRmk4K",
"colab_type": "code",
"outputId": "13599809-1f69-44dc-91fe-f7767db3f275",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"if \"armbar\" in submissions:\n",
" print(\"found key\")\n",
" "
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"found key\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "ORI3ZxbqntAG",
"colab_type": "code",
"outputId": "1b6f8d74-f698-472f-95f5-cae8946d6d45",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 69
}
},
"source": [
"print(\"timing key membership\")\n",
"%timeit if \"armbar\" in submissions: pass "
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"timing key membership\n",
"The slowest run took 43.71 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
"10000000 loops, best of 3: 32.1 ns per loop\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IyHkox1Rj66y",
"colab_type": "text"
},
"source": [
"#### Python Lists\n",
"\n",
"Lists are also very commonly used in Python. They allow for sequential collections. Lists can hold dictionaries, just as dictionaries can hold lists."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Xo7IMK567yHQ",
"colab_type": "text"
},
"source": [
"##### Creating Lists\n",
"\n",
"One way to create lists is with *[] syntax*"
]
},
{
"cell_type": "code",
"metadata": {
"id": "MI-S0FStj1o4",
"colab_type": "code",
"colab": {}
},
"source": [
"list_of_bjj_positions = [\"mount\", \"full-guard\", \"half-guard\", \n",
" \"turtle\", \"side-control\", \"rear-mount\", \n",
" \"knee-on-belly\", \"north-south\", \"open-guard\"]"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "WUgrRDDsoobr",
"colab_type": "text"
},
"source": [
"Another method os creating lists is with built in *list()* method\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "o-3IP2FOw9yi",
"colab_type": "code",
"outputId": "d3ce0d86-aa4f-4282-e36d-2d555d0a57ce",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"bjj_dominant_positions = list()\n",
"bjj_dominant_positions.append(\"side-control\")\n",
"bjj_dominant_positions.append(\"mount\")\n",
"bjj_dominant_positions\n"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['side-control', 'mount']"
]
},
"metadata": {
"tags": []
},
"execution_count": 91
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "i1gjso964T1T",
"colab_type": "text"
},
"source": [
"Yet another way, very performant way to create lists is to use list comprehension syntax"
]
},
{
"cell_type": "code",
"metadata": {
"id": "QeQt8XD_4T-D",
"colab_type": "code",
"outputId": "f96978bc-a7df-416f-879a-d2dd0f4fffe3",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"guards = \"full, half, open\"\n",
"guard_list = [f\"{guard}-guard\" for guard in guards.split(\",\")]\n",
"guard_list\n"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['full-guard', ' half-guard', ' open-guard']"
]
},
"metadata": {
"tags": []
},
"execution_count": 108
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CBDIYoFB7yHS",
"colab_type": "text"
},
"source": [
"##### Using Lists\n",
"\n",
"For loops are one of the simplist ways to use a list."
]
},
{
"cell_type": "code",
"metadata": {
"id": "EoFWVpKYj94o",
"colab_type": "code",
"outputId": "51f1213c-d174-49b3-ab20-6b84f6547efc",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"for position in list_of_bjj_positions:\n",
" if \"open\" in position: #explore on your own \"guard\"\n",
" print(position)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"open-guard\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YAMxQJobkCy8",
"colab_type": "text"
},
"source": [
"Lists can also be used to select elements by slicing."
]
},
{
"cell_type": "code",
"metadata": {
"id": "Zi6PKsRIkAl_",
"colab_type": "code",
"outputId": "b871af52-3a5d-4dac-e58c-257d8ec4be4d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 69
}
},
"source": [
"print(f'First position: {list_of_bjj_positions[:1]}')\n",
"print(f'Last position: {list_of_bjj_positions[-1:]}')\n",
"print(f'First three positions: {list_of_bjj_positions[0:3]}')"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"First position: ['mount']\n",
"Last position: ['open-guard']\n",
"First three positions: ['mount', 'full-guard', 'half-guard']\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XgxwgI1I7Ex3",
"colab_type": "text"
},
"source": [
"Lists can also be used to unpack powerful, succinct statements when used with built-in functions like zip.\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "nkaMbTx77ddw",
"colab_type": "code",
"outputId": "6befa394-dd61-488d-e158-a4448de18ae7",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 87
}
},
"source": [
"bjj_position_matrix = [\n",
" [\"dominant\", \"top-mount\", \"back-mount\", \"side-control\"],\n",
" [\"neutral\", \"open-guard\", \"full-guard\", \"standing\"],\n",
" [\"weak\", \"turtle\", \"bottom-back-mount\", \"bottom-mount\"]\n",
"]\n",
"list(zip(*bjj_position_matrix))"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"[('dominant', 'neutral', 'weak'),\n",
" ('top-mount', 'open-guard', 'turtle'),\n",
" ('back-mount', 'full-guard', 'bottom-back-mount'),\n",
" ('side-control', 'standing', 'bottom-mount')]"
]
},
"metadata": {
"tags": []
},
"execution_count": 113
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "9ZA34gKvCi39",
"colab_type": "code",
"colab": {}
},
"source": [
"zip?"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "YXalviiD7yHZ",
"colab_type": "text"
},
"source": [
"#### Python Sets\n",
"\n",
"Sets are unordered unique collections"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1y5ahhJ699Gp",
"colab_type": "text"
},
"source": [
"##### Creating Python Sets\n",
"\n",
"Sets can be created by using built-in *sets()* method\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "ZT4d5Dxw9V4X",
"colab_type": "code",
"outputId": "f90fe0ae-903a-4b35-f8f5-f1fce876706e",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
}
},
"source": [
"unique_attacks = set((\"armbar\",\"armbar\", \"armbar\", \"kimura\", \"kimura\", \"heel hook\"))\n",
"print(type(unique_attacks))\n",
"unique_attacks"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"<class 'set'>\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"{'armbar', 'heel hook', 'kimura'}"
]
},
"metadata": {
"tags": []
},
"execution_count": 114
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Kj_JgH317yHZ",
"colab_type": "text"
},
"source": [
"##### Using Sets\n",
"\n",
"One of the most powerful ways to use sets is to find the differences between to collections"
]
},
{
"cell_type": "code",
"metadata": {
"id": "zbAUyYgk_c0s",
"colab_type": "code",
"outputId": "16c32465-a804-4a77-f03c-7c2e5d9c8a8c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"attacks_set_one = set((\"armbar\", \"kimura\", \"heal-hook\"))\n",
"attacks_set_two = set((\"toe-hold\", \"knee-bar\", \"heal-hook\"))\n",
"unique_set_one_attacks = attacks_set_one - attacks_set_two\n",
"print(f\"Unique Set One Attacks {unique_set_one_attacks}\")\n"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Unique Set One Attacks {'armbar', 'kimura'}\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "s0y9Fv9eL1OR",
"colab_type": "text"
},
"source": [
"### Exercise: Write a dictionary with a mutable key."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-PugvjDSL2zD",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "code",
"metadata": {
"id": "AM6pTL2yvlSk",
"colab_type": "code",
"colab": {}
},
"source": [
"good = {'one': 'two'}"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "X-SkxPxPv4gZ",
"colab_type": "code",
"outputId": "e0449e9c-aee3-4311-dee6-57935203571c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 184
}
},
"source": [
"# A list is mutable\n",
"var = [\"two\"]\n",
"badone = {var:\"two\"}"
],
"execution_count": 0,
"outputs": [
{
"output_type": "error",
"ename": "TypeError",
"evalue": "ignored",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-118-a867e485e813>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0mvar\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m\"two\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mbadone\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0mvar\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\"two\"\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m: unhashable type: 'list'"
]
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "T0fbKGMfwIlj",
"colab_type": "code",
"colab": {}
},
"source": [
"# be careful about mutating objects\n",
"class Foo:pass\n",
"foo = Foo()\n",
"\n",
"bad_two = {foo: \"two\"}"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "hqdmW3A7wcsY",
"colab_type": "code",
"colab": {}
},
"source": [
"# try out why this is weird\n",
"\n",
"def fun():pass\n",
"bad_three = {fun:\"one\"}\n"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "1wY-GUQc3te-",
"colab_type": "text"
},
"source": [
"### Presentation: Functions, Lazy Expressions, Async and Concurrency\n",
"\n",
"* [Read related material covered in Chapter 1 (Functions Section) of Pragmatic AI](https://www.safaribooksonline.com/library/view/pragmatic-ai-an/9780134863924/ch01.xhtml#ch01lev1sub17)\n",
"\n",
"* [Watch video section 2: Writing and Applying Functions](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118/9780135261118-EMLA_01_02_00) \n",
"\n",
"* **Writing Functions**\n",
"* Function arguments: positional, keyword\n",
"* Functional Currying: Passing uncalled functions\n",
"* Functions that Yield\n",
"* Decorators: Functions that wrap other functions\n",
"* Making Classes Behave Like Functions\n",
"* Applying a Function to a Pandas DataFrame\n",
"* Writing Lambdas"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8JmjS2H33wZP",
"colab_type": "text"
},
"source": [
"#### Writing Functions\n",
"Learning to write a function is the most fundamental skill to learn in Python. With a basic mastery of functions, it is possible to have an almost full command of the language."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RHsiAy8N3-Cj",
"colab_type": "text"
},
"source": [
"**Simple function**\n",
"\n",
"The simplest functions just return a value."
]
},
{
"cell_type": "code",
"metadata": {
"id": "YsngH1Yh4BP3",
"colab_type": "code",
"colab": {}
},
"source": [
"def favorite_martial_art():\n",
" return \"bjj\""
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "hL4eSHZjjLSe",
"colab_type": "code",
"outputId": "a134b36d-92de-4eb3-d0e6-1a76055f70e8",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
}
},
"source": [
"print(favorite_martial_art())\n",
"# This is the same output\n",
"my_variable = \"bjj\"\n",
"my_variable"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"bjj\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'bjj'"
]
},
"metadata": {
"tags": []
},
"execution_count": 122
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "lq9VQm0V4BTE",
"colab_type": "code",
"colab": {}
},
"source": [
"def myfunc():pass"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "CCYhFgvvjGHt",
"colab_type": "code",
"outputId": "c98b1f37-ec70-4db0-b151-b133f48a2cf0",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"res = myfunc()\n",
"print(res)\n",
"#result = myfunc()\n",
"#print(result)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"None\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AMPU657F4JWF",
"colab_type": "text"
},
"source": [
"**Documenting Functions**\n",
"\n",
"It is a very good idea to document functions. \n",
"In Jupyter Notebook and IPython docstrings can be viewed by referring to the function with a ?. ie.\n",
"\n",
"```\n",
"In [2]: favorite_martial_art_with_docstring?\n",
"Signature: favorite_martial_art_with_docstring()\n",
"Docstring: This function returns the name of my favorite martial art\n",
"File: ~/src/functional_intro_to_python/<ipython-input-1-bef983c31735>\n",
"Type: function\n",
"```"
]
},
{
"cell_type": "code",
"metadata": {
"id": "ZKOHmIxg4Ply",
"colab_type": "code",
"colab": {}
},
"source": [
"def favorite_martial_art_with_docstring():\n",
" \"\"\"This function returns the name of my favorite martial art\n",
" This is more\n",
" This is even more\n",
" return \"string\"\n",
" \"\"\"\n",
" return \"bjj\""
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "8OW1SbDg4Rd1",
"colab_type": "text"
},
"source": [
"**Docstrings of functions can be printed out by referring to *```__doc__```*** "
]
},
{
"cell_type": "code",
"metadata": {
"id": "SuI6CO-E4ViO",
"colab_type": "code",
"outputId": "8aa47172-532a-41ea-9d54-47acc695368e",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"favorite_martial_art_with_docstring.__doc__\n",
"#favorite_martial_art_with_docstring?\n"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'This function returns the name of my favorite martial art\\n This is more\\n This is even more\\n return \"string\"\\n '"
]
},
"metadata": {
"tags": []
},
"execution_count": 127
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "DeanvrdvKTV6",
"colab_type": "code",
"colab": {}
},
"source": [
"#favorite_martial_art_with_docstring?"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "ew4-TwH84YJa",
"colab_type": "text"
},
"source": [
"#### Function arguments: positional, keyword\n",
"\n",
"A function is most useful when arguments are passed to the function. New values for times are processed inside the function. This function is also a 'positional' argument, vs a keyword argument. Positional arguments are processed in the order they are created in."
]
},
{
"cell_type": "code",
"metadata": {
"id": "rxL-hyuL4aws",
"colab_type": "code",
"colab": {}
},
"source": [
"def practice(times):\n",
" print(f\"I like to practice {times} times a day\")"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "WqDUN33f4cyf",
"colab_type": "code",
"outputId": "1aa30d27-abf8-479d-c7d0-51dc443a17d7",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"practice(2)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"I like to practice 2 times a day\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "C8h7Jo0y4ggH",
"colab_type": "code",
"outputId": "1ed860ec-26fa-4dee-ad69-ff0472bcd0c6",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"practice(3)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"I like to practice 3 times a day\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LILwF-Nt4iTm",
"colab_type": "text"
},
"source": [
"**Positional Arguments are processed in order**\n",
"\n",
"Note, *position* is the key to pay attention to.\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "Aw6DSXR64oVx",
"colab_type": "code",
"colab": {}
},
"source": [
"def practice(times, technique, duration):\n",
" print(f\"I like to practice {technique}, {times} times a day, for {duration} minutes\")"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "3Gq1XUkB4qtM",
"colab_type": "code",
"outputId": "a8531261-651d-41c9-ea45-f6219fea7819",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"practice(3, \"piano\", 45)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"I like to practice piano, 3 times a day, for 45 minutes\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "Sv9WBxWzuWAy",
"colab_type": "code",
"outputId": "729544f3-7ecd-48a0-d8f7-180d117de80a",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"#Order is important, now the entire is incorrect and prints out nonsense\n",
"practice(\"piano\", 7,60)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"I like to practice 7, piano times a day, for 60 minutes\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ntX-r2Bp4uTw",
"colab_type": "text"
},
"source": [
"**Keyword Arguments are processed by key, value and can have default values**\n",
"\n",
"One handy feature of keyword arguments is that you can set defaults and only change the defaults you want to change."
]
},
{
"cell_type": "code",
"metadata": {
"id": "cSIg9mKs4zg_",
"colab_type": "code",
"colab": {}
},
"source": [
"def practice(times=2, technique=\"python\", duration=60):\n",
" print(f\"I like to practice {technique}, {times} times a day, for {duration} minutes\")"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "Be5uDA_d41dg",
"colab_type": "code",
"outputId": "2d8e0cc5-5fe5-4afa-ccc0-68d6e0fdb831",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"practice()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"I like to practice python, 2 times a day, for 60 minutes\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "qOwB-4Dz43HQ",
"colab_type": "code",
"outputId": "2bc4a3fd-6003-4aa6-b4c7-3128697514aa",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"practice(duration=90, times=4)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"I like to practice python, 4 times a day, for 90 minutes\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2wBL_tQ846Qd",
"colab_type": "text"
},
"source": [
"*****args and ****kwargs\n",
"\n",
"allow dynamic argument passing to functions\n",
"Should be used with discretion because it can make code hard to understand"
]
},
{
"cell_type": "code",
"metadata": {
"id": "gJtpL1Ad5IUH",
"colab_type": "code",
"colab": {}
},
"source": [
"def attack_techniques(**kwargs):\n",
" \"\"\"This accepts any number of keyword arguments\"\"\"\n",
" \n",
" for name, attack in kwargs.items():\n",
" print(f\"This is an attack I would like to practice: {attack}\")"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "ODHgMleb5KVa",
"colab_type": "code",
"outputId": "492706cf-47c2-4c78-ba70-b7b06b452416",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 87
}
},
"source": [
"attack_techniques(arm_attack=\"kimura\", \n",
" leg_attack=\"straight_ankle_lock\", \n",
" neck_attack=\"arm_triangle\",\n",
" body_attack=\"charge\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"This is an attack I would like to practice: kimura\n",
"This is an attack I would like to practice: straight_ankle_lock\n",
"This is an attack I would like to practice: arm_triangle\n",
"This is an attack I would like to practice: charge\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "rdvgasCNvwKt",
"colab_type": "code",
"outputId": "dcb85f2c-f2b8-4dbf-a5e1-1619be8fd3b4",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 104
}
},
"source": [
"#I also can pass as many things as I wants\n",
"attack_techniques(arm_attack=\"kimura\", \n",
" leg_attack=\"straight_ankle_lock\", \n",
" neck_attach=\"arm_triangle\",\n",
" attack4=\"rear nake choke\", attack5=\"key lock\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"This is an attack I would like to practice: kimura\n",
"This is an attack I would like to practice: straight_ankle_lock\n",
"This is an attack I would like to practice: arm_triangle\n",
"This is an attack I would like to practice: rear nake choke\n",
"This is an attack I would like to practice: key lock\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6Bd0PwiD5NnI",
"colab_type": "text"
},
"source": [
"**passing dictionary of keywords to function**\n",
"\n",
"**kwargs syntax can also be used to pass in arguments all at once"
]
},
{
"cell_type": "code",
"metadata": {
"id": "Rxjc3U7-5QlB",
"colab_type": "code",
"colab": {}
},
"source": [
"attacks = {\"arm_attack\":\"kimura\", \n",
" \"leg_attack\":\"straight_ankle_lock\", \n",
" \"neck_attach\":\"arm_triangle\"}"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "0BmMRBOC5SG-",
"colab_type": "code",
"outputId": "c9436287-a3bb-4b1d-a579-42b381416701",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 69
}
},
"source": [
"attack_techniques(**attacks)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"This is an attack I would like to practice: kimura\n",
"This is an attack I would like to practice: straight_ankle_lock\n",
"This is an attack I would like to practice: arm_triangle\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3Srgz3C65Vdk",
"colab_type": "text"
},
"source": [
"**Passing Around Functions**\n",
"\n",
"Object-Oriented programming is a very popular way to program, but it isn't the only style available in Python. For concurrency and for Data Science, functional programming fits as a complementary style.\n",
"\n",
"In the example, below a function can be used inside of another function by being passed into the function itself as an argument."
]
},
{
"cell_type": "code",
"metadata": {
"id": "inzYwQSq5XPy",
"colab_type": "code",
"colab": {}
},
"source": [
"def attack_location(technique):\n",
" \"\"\"Return the location of an attack\"\"\"\n",
" \n",
" attacks = {\"kimura\": \"arm_attack\",\n",
" \"straight_ankle_lock\":\"leg_attack\", \n",
" \"arm_triangle\":\"neck_attach\"}\n",
" if technique in attacks:\n",
" return attacks[technique]\n",
" return \"Unknown\""
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "VfnYsD895ZFC",
"colab_type": "code",
"outputId": "6cbfc55a-fa1e-45cc-a864-b7e0b48babc4",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"attack_location(\"kimura\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'arm_attack'"
]
},
"metadata": {
"tags": []
},
"execution_count": 145
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "eK6NR4F-5aqT",
"colab_type": "code",
"outputId": "c2eae6ee-df4f-4411-8cf9-aeb8b3f348ef",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"attack_location(\"bear hug\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'Unknown'"
]
},
"metadata": {
"tags": []
},
"execution_count": 146
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "cOARK2b65f-k",
"colab_type": "code",
"colab": {}
},
"source": [
"def multiple_attacks(attack_location_function):\n",
" \"\"\"Takes a function that categorizes attacks and returns location\"\"\"\n",
" \n",
" new_attacks_list = [\"rear_naked_choke\", \"americana\", \"kimura\"]\n",
" for attack in new_attacks_list:\n",
" attack_location = attack_location_function(attack)\n",
" print(f\"The location of attack {attack} is {attack_location}\")"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "WzgM8VU85iju",
"colab_type": "code",
"outputId": "a0097509-a51f-4096-be85-a7957f9c5097",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 69
}
},
"source": [
"multiple_attacks(attack_location)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"The location of attack rear_naked_choke is Unknown\n",
"The location of attack americana is Unknown\n",
"The location of attack kimura is arm_attack\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bCdDA35S5mQ4",
"colab_type": "text"
},
"source": [
"#### Closures and Functional Currying\n",
"\n",
"Closures are functions that contain other nested functions with state from outer function.\n",
"\n",
"In Python, a common way to use them is to keep track of the state. In the example below, the outer function, attack_counter keeps track of counts of attacks. The inner fuction attack_filter uses the \"nonlocal\" keyword in Python3, to modify the variable in the outer function.\n",
"\n",
"This approach is called \"functional currying\". It allows for a specialized function to be created from general functions. As shown below, this style of function could be the basis of a simple video game or maybe for the statistics crew of a mma match."
]
},
{
"cell_type": "code",
"metadata": {
"id": "QeG9UZfL6THb",
"colab_type": "code",
"colab": {}
},
"source": [
"#nonlocal cannot modify this variable\n",
"#lower_body_counter=5\n",
"def attack_counter():\n",
" \"\"\"Counts number of attacks on part of body\"\"\"\n",
" lower_body_counter = 0\n",
" upper_body_counter = 0\n",
" #print(lower_body_counter)\n",
" def attack_filter(attack):\n",
" nonlocal lower_body_counter\n",
" nonlocal upper_body_counter\n",
" attacks = {\"kimura\": \"upper_body\",\n",
" \"straight_ankle_lock\":\"lower_body\", \n",
" \"arm_triangle\":\"upper_body\",\n",
" \"keylock\": \"upper_body\",\n",
" \"knee_bar\": \"lower_body\"}\n",
" if attack in attacks:\n",
" if attacks[attack] == \"upper_body\":\n",
" upper_body_counter +=1\n",
" if attacks[attack] == \"lower_body\":\n",
" lower_body_counter +=1\n",
" print(f\"Upper Body Attacks {upper_body_counter}, Lower Body Attacks {lower_body_counter}\")\n",
" return attack_filter"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "YT_Tk-iT6WiR",
"colab_type": "code",
"colab": {}
},
"source": [
"fight = attack_counter()"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "DdBJ_0_C1s89",
"colab_type": "code",
"outputId": "00bd53c6-2cc2-4ee9-bd7c-39d2d970e320",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"type(fight)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"function"
]
},
"metadata": {
"tags": []
},
"execution_count": 152
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "J1p7hvGb6Y3m",
"colab_type": "code",
"outputId": "70b6daf1-285d-4751-dd46-9c496b38dd72",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"fight(\"kimura\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Upper Body Attacks 1, Lower Body Attacks 0\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "gCoFnKDJ6aTS",
"colab_type": "code",
"outputId": "fe9e1ea9-1186-4756-bbc8-39bdad4f3647",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"fight(\"knee_bar\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Upper Body Attacks 1, Lower Body Attacks 1\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "_3MRMBkS6bmy",
"colab_type": "code",
"outputId": "dfd055da-7e91-427d-b341-6b87dd5c690f",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"fight(\"keylock\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Upper Body Attacks 2, Lower Body Attacks 1\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hZx708yd7lpU",
"colab_type": "text"
},
"source": [
"#### Partial Functions\n",
"\n",
"Useful to partial assign default values to functions"
]
},
{
"cell_type": "code",
"metadata": {
"id": "fOhajVSK7oeX",
"colab_type": "code",
"outputId": "eaa64020-7762-40b3-fdf5-6d02327931c1",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"from functools import partial\n",
"\n",
"def multiple_attacks(attack_one, attack_two):\n",
" \"\"\"Performs two attacks\"\"\"\n",
" \n",
" print(f\"First Attack {attack_one}\")\n",
" print(f\"Second Attack {attack_two}\")\n",
" \n",
"attack_this = partial(multiple_attacks, \"kimura\")\n",
"type(attack_this)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"functools.partial"
]
},
"metadata": {
"tags": []
},
"execution_count": 157
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ymZDwSka7rrh",
"colab_type": "text"
},
"source": [
"By using this partial function, only one argument is needed"
]
},
{
"cell_type": "code",
"metadata": {
"id": "jJv6HePw7tkH",
"colab_type": "code",
"outputId": "baed09ad-fed9-4474-d04d-57ba99b1eb16",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
}
},
"source": [
"attack_this(\"knee-bar\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"First Attack kimura\n",
"Second Attack knee-bar\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2CIWXwv27uPW",
"colab_type": "text"
},
"source": [
"Alternately, the original function can also be called with a different two attacks"
]
},
{
"cell_type": "code",
"metadata": {
"id": "QVudEj-h7wO_",
"colab_type": "code",
"outputId": "596b6285-c71b-4a6e-cea7-e91a9ca46f29",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
}
},
"source": [
"multiple_attacks(\"Darce Choke\", \"Bicep Slicer\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"First Attack Darce Choke\n",
"Second Attack Bicep Slicer\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5-tKvYXt6ftS",
"colab_type": "text"
},
"source": [
"#### Lazy Evaluated Functions (Generators)\n",
"\n",
"A very useful style of programming is \"lazy evaluation\". A generator is an example of that. Generators yield an items at a time.\n",
"\n",
"The example below return an \"infinite\" random sequence of attacks. The lazy portion comes into play in that while there is an infinite amount of values, they are only returned when the function is called."
]
},
{
"cell_type": "code",
"metadata": {
"id": "P-hOWZyV6jCt",
"colab_type": "code",
"colab": {}
},
"source": [
"def lazy_return_random_attacks():\n",
" \"\"\"Yield attacks each time\"\"\"\n",
" import random\n",
" attacks = {\"kimura\": \"upper_body\",\n",
" \"straight_ankle_lock\":\"lower_body\", \n",
" \"arm_triangle\":\"upper_body\",\n",
" \"keylock\": \"upper_body\",\n",
" \"knee_bar\": \"lower_body\"}\n",
" while True:\n",
" random_attack = random.choices(list(attacks.keys()))\n",
" yield random_attack"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "NrhrDOAL6u0b",
"colab_type": "code",
"colab": {}
},
"source": [
"attack = lazy_return_random_attacks()"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "NyxMi57a6wA4",
"colab_type": "code",
"outputId": "4f2270fa-56de-45cc-c80a-f73e89a0941c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"type(attack)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"generator"
]
},
"metadata": {
"tags": []
},
"execution_count": 162
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "0IxoCkJu6y-B",
"colab_type": "code",
"outputId": "50c2cda7-79d3-4c08-f58e-0cc6884fd402",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 121
}
},
"source": [
"for _ in range(6):\n",
" print(next(attack))"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"['arm_triangle']\n",
"['arm_triangle']\n",
"['arm_triangle']\n",
"['arm_triangle']\n",
"['knee_bar']\n",
"['arm_triangle']\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hlJXHUVq6391",
"colab_type": "text"
},
"source": [
"#### Decorators: Functions that wrap other functions"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9KQ2Cmcelcy-",
"colab_type": "text"
},
"source": [
"##### Randomized Sleep Decorator\n",
"\n",
"Another useful technique in Python is to use the decorator syntax to wrap one function with another function. In the example below, a decorator is written that adds random sleep to each function call. When combined with the previous \"infinite\" attack generator, it generates random sleeps between each function call."
]
},
{
"cell_type": "code",
"metadata": {
"id": "JvNzAVBT6695",
"colab_type": "code",
"colab": {}
},
"source": [
"def randomized_speed_attack_decorator(function):\n",
" \"\"\"Randomizes the speed of attacks\"\"\"\n",
" \n",
" import time\n",
" import random\n",
" \n",
" def wrapper_func(*args, **kwargs):\n",
" sleep_time = random.randint(0,3)\n",
" print(f\"Attacking after {sleep_time} seconds\")\n",
" time.sleep(sleep_time)\n",
" return function(*args, **kwargs)\n",
" return wrapper_func"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "5BGrwIl96-PX",
"colab_type": "code",
"colab": {}
},
"source": [
"@randomized_speed_attack_decorator\n",
"def lazy_return_random_attacks():\n",
" \"\"\"Yield attacks each time\"\"\"\n",
" import random\n",
" attacks = {\"kimura\": \"upper_body\",\n",
" \"straight_ankle_lock\":\"lower_body\", \n",
" \"arm_triangle\":\"upper_body\",\n",
" \"keylock\": \"upper_body\",\n",
" \"knee_bar\": \"lower_body\"}\n",
" while True:\n",
" random_attack = random.choices(list(attacks.keys()))\n",
" yield random_attack"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "HAEY8I7M7AL5",
"colab_type": "code",
"outputId": "89697340-abc2-45bf-e3c1-2ecf64c2609d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 191
}
},
"source": [
"for _ in range(5):\n",
" print(next(lazy_return_random_attacks()))"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Attacking after 3 seconds\n",
"['arm_triangle']\n",
"Attacking after 3 seconds\n",
"['straight_ankle_lock']\n",
"Attacking after 1 seconds\n",
"['knee_bar']\n",
"Attacking after 1 seconds\n",
"['keylock']\n",
"Attacking after 2 seconds\n",
"['straight_ankle_lock']\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qO-uoYUClvqr",
"colab_type": "text"
},
"source": [
"##### Timing Decorator"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "McePAV81l2Cm",
"colab_type": "text"
},
"source": [
"Using a decorator to time code is very common"
]
},
{
"cell_type": "code",
"metadata": {
"id": "pRtnp32Rl5jt",
"colab_type": "code",
"colab": {}
},
"source": [
"from functools import wraps\n",
"from time import time\n",
"\n",
"def timing(f):\n",
" @wraps(f)\n",
" def wrap(*args, **kw):\n",
" ts = time()\n",
" result = f(*args, **kw)\n",
" te = time()\n",
" print(f\"fun: {f.__name__}, args: [{args}, {kw}] took: {te-ts} sec\")\n",
" return result\n",
" return wrap"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "SFisqs8tmJer",
"colab_type": "text"
},
"source": [
"Using decorator to time execution of a function"
]
},
{
"cell_type": "code",
"metadata": {
"id": "h3B4RLEbmpie",
"colab_type": "code",
"outputId": "130292a7-f0ef-4e16-ed4e-1a29e9739623",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 156
}
},
"source": [
"@timing\n",
"def some_attacks(foo):\n",
" print(f\"this was passed in: {foo}\")\n",
" attack = lazy_return_random_attacks()\n",
" for _ in range(5):\n",
" print(next(attack))\n",
" \n",
"some_attacks(\"bar\")\n",
" "
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"this was passed in: bar\n",
"Attacking after 0 seconds\n",
"['arm_triangle']\n",
"['kimura']\n",
"['knee_bar']\n",
"['keylock']\n",
"['kimura']\n",
"fun: some_attacks, args: [('bar',), {}] took: 0.0030469894409179688 sec\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WT4sChux989o",
"colab_type": "text"
},
"source": [
"#### Making Classes Behave Like Functions\n",
"\n",
"Creating callable functions"
]
},
{
"cell_type": "code",
"metadata": {
"id": "y2ykyfI79_6E",
"colab_type": "code",
"colab": {}
},
"source": [
"class AttackFinder:\n",
" \"\"\"Finds the attack location\"\"\"\n",
" \n",
" \n",
" def __init__(self, attack):\n",
" self.attack = attack\n",
" \n",
" def __call__(self):\n",
" attacks = {\"kimura\": \"upper_body\",\n",
" \"straight_ankle_lock\":\"lower_body\", \n",
" \"arm_triangle\":\"upper_body\",\n",
" \"keylock\": \"upper_body\",\n",
" \"knee_bar\": \"lower_body\"}\n",
" if not self.attack in attacks:\n",
" return \"unknown location\"\n",
" return attacks[self.attack]\n",
" "
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "NKyd_XhOJCSk",
"colab_type": "code",
"outputId": "937ef1a6-0ec1-4a9e-9811-a0a1f1a7f3dd",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"my_attack = AttackFinder(\"kimura\")\n",
"my_attack()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'upper_body'"
]
},
"metadata": {
"tags": []
},
"execution_count": 178
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hQpf0vu_-Cnk",
"colab_type": "text"
},
"source": [
"#### Applying Functions to Pandas DataFrames\n",
"\n",
"The final lesson on functions is to take this knowledge and use it on a DataFrame in Pandas. One of the more fundamental concepts in Pandas is use apply on a column vs iterating through all of the values. An example is shown below where all of the numbers are rounded to a whole digit."
]
},
{
"cell_type": "code",
"metadata": {
"id": "WLWOOacP-KDz",
"colab_type": "code",
"outputId": "fc64b581-cd64-46a6-ffaa-04b5694b5266",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 141
}
},
"source": [
"import pandas as pd\n",
"iris = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv')\n",
"iris.head(3)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal_length</th>\n",
" <th>sepal_width</th>\n",
" <th>petal_length</th>\n",
" <th>petal_width</th>\n",
" <th>species</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>5.1</td>\n",
" <td>3.5</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4.9</td>\n",
" <td>3.0</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>4.7</td>\n",
" <td>3.2</td>\n",
" <td>1.3</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal_length sepal_width petal_length petal_width species\n",
"0 5.1 3.5 1.4 0.2 setosa\n",
"1 4.9 3.0 1.4 0.2 setosa\n",
"2 4.7 3.2 1.3 0.2 setosa"
]
},
"metadata": {
"tags": []
},
"execution_count": 179
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "EZTzZku3hZFM",
"colab_type": "code",
"outputId": "f6653ce9-ef6f-481a-bb5a-8977f415438a",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"iris.shape"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(150, 5)"
]
},
"metadata": {
"tags": []
},
"execution_count": 180
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "S8jXLfVq-Nrb",
"colab_type": "code",
"outputId": "49a332ad-fdad-40cf-de41-6965c3e5c7cb",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 202
}
},
"source": [
"iris['rounded_sepal_length'] = iris[['sepal_length']].apply(pd.Series.round)\n",
"iris.head()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal_length</th>\n",
" <th>sepal_width</th>\n",
" <th>petal_length</th>\n",
" <th>petal_width</th>\n",
" <th>species</th>\n",
" <th>rounded_sepal_length</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>5.1</td>\n",
" <td>3.5</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4.9</td>\n",
" <td>3.0</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>4.7</td>\n",
" <td>3.2</td>\n",
" <td>1.3</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>4.6</td>\n",
" <td>3.1</td>\n",
" <td>1.5</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>5.0</td>\n",
" <td>3.6</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal_length sepal_width ... species rounded_sepal_length\n",
"0 5.1 3.5 ... setosa 5.0\n",
"1 4.9 3.0 ... setosa 5.0\n",
"2 4.7 3.2 ... setosa 5.0\n",
"3 4.6 3.1 ... setosa 5.0\n",
"4 5.0 3.6 ... setosa 5.0\n",
"\n",
"[5 rows x 6 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 181
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "TBxKbJk8kjit",
"colab_type": "code",
"outputId": "68f4efd0-4efd-48dd-d5e0-37f433ae66a7",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"iris.shape"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(150, 6)"
]
},
"metadata": {
"tags": []
},
"execution_count": 182
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QrQiWvRC-QY9",
"colab_type": "text"
},
"source": [
"This was done with a built in function, but a custom function can also be written and applied to a column. In the example below, the values are multiplied by 100. The alternative way to accomplish this would be to create a loop, transform the data and then write it back. In Pandas, it is straightforward and simple to apply custom functions instead."
]
},
{
"cell_type": "code",
"metadata": {
"id": "6WlPQ823-SMS",
"colab_type": "code",
"outputId": "7371cd1f-83b0-4186-b3f3-ec7fa13c3586",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 202
}
},
"source": [
"def multiply_by_100(x):\n",
" \"\"\"Multiplies by 100\"\"\"\n",
" \n",
" res = x * 100\n",
" #print(f\"This was passed in {x}, and this result was generated {res}\")\n",
" return res\n",
" \n",
" \n",
"iris['100x_sepal_length'] = iris[['sepal_length']].apply(multiply_by_100)\n",
"iris.head()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal_length</th>\n",
" <th>sepal_width</th>\n",
" <th>petal_length</th>\n",
" <th>petal_width</th>\n",
" <th>species</th>\n",
" <th>rounded_sepal_length</th>\n",
" <th>100x_sepal_length</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>5.1</td>\n",
" <td>3.5</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" <td>510.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4.9</td>\n",
" <td>3.0</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" <td>490.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>4.7</td>\n",
" <td>3.2</td>\n",
" <td>1.3</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" <td>470.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>4.6</td>\n",
" <td>3.1</td>\n",
" <td>1.5</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" <td>460.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>5.0</td>\n",
" <td>3.6</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" <td>500.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal_length sepal_width ... rounded_sepal_length 100x_sepal_length\n",
"0 5.1 3.5 ... 5.0 510.0\n",
"1 4.9 3.0 ... 5.0 490.0\n",
"2 4.7 3.2 ... 5.0 470.0\n",
"3 4.6 3.1 ... 5.0 460.0\n",
"4 5.0 3.6 ... 5.0 500.0\n",
"\n",
"[5 rows x 7 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 183
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "pYg2Lz05ly7Q",
"colab_type": "code",
"outputId": "1e828dc9-f15d-4ec2-a7bd-47b382916150",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 202
}
},
"source": [
"iris[\"new_column\"] = iris[['sepal_length']]\n",
"iris.head()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal_length</th>\n",
" <th>sepal_width</th>\n",
" <th>petal_length</th>\n",
" <th>petal_width</th>\n",
" <th>species</th>\n",
" <th>rounded_sepal_length</th>\n",
" <th>100x_sepal_length</th>\n",
" <th>new_column</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>5.1</td>\n",
" <td>3.5</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" <td>510.0</td>\n",
" <td>5.1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4.9</td>\n",
" <td>3.0</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" <td>490.0</td>\n",
" <td>4.9</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>4.7</td>\n",
" <td>3.2</td>\n",
" <td>1.3</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" <td>470.0</td>\n",
" <td>4.7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>4.6</td>\n",
" <td>3.1</td>\n",
" <td>1.5</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" <td>460.0</td>\n",
" <td>4.6</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>5.0</td>\n",
" <td>3.6</td>\n",
" <td>1.4</td>\n",
" <td>0.2</td>\n",
" <td>setosa</td>\n",
" <td>5.0</td>\n",
" <td>500.0</td>\n",
" <td>5.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal_length sepal_width ... 100x_sepal_length new_column\n",
"0 5.1 3.5 ... 510.0 5.1\n",
"1 4.9 3.0 ... 490.0 4.9\n",
"2 4.7 3.2 ... 470.0 4.7\n",
"3 4.6 3.1 ... 460.0 4.6\n",
"4 5.0 3.6 ... 500.0 5.0\n",
"\n",
"[5 rows x 8 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 184
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "T66Ll8etnuxK",
"colab_type": "code",
"outputId": "ff39b95d-085a-4d43-fabb-db621ca1e459",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 171
}
},
"source": [
"iris.groupby(\"species\").max()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sepal_length</th>\n",
" <th>sepal_width</th>\n",
" <th>petal_length</th>\n",
" <th>petal_width</th>\n",
" <th>rounded_sepal_length</th>\n",
" <th>100x_sepal_length</th>\n",
" <th>new_column</th>\n",
" </tr>\n",
" <tr>\n",
" <th>species</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>setosa</th>\n",
" <td>5.8</td>\n",
" <td>4.4</td>\n",
" <td>1.9</td>\n",
" <td>0.6</td>\n",
" <td>6.0</td>\n",
" <td>580.0</td>\n",
" <td>5.8</td>\n",
" </tr>\n",
" <tr>\n",
" <th>versicolor</th>\n",
" <td>7.0</td>\n",
" <td>3.4</td>\n",
" <td>5.1</td>\n",
" <td>1.8</td>\n",
" <td>7.0</td>\n",
" <td>700.0</td>\n",
" <td>7.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>virginica</th>\n",
" <td>7.9</td>\n",
" <td>3.8</td>\n",
" <td>6.9</td>\n",
" <td>2.5</td>\n",
" <td>8.0</td>\n",
" <td>790.0</td>\n",
" <td>7.9</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sepal_length sepal_width ... 100x_sepal_length new_column\n",
"species ... \n",
"setosa 5.8 4.4 ... 580.0 5.8\n",
"versicolor 7.0 3.4 ... 700.0 7.0\n",
"virginica 7.9 3.8 ... 790.0 7.9\n",
"\n",
"[3 rows x 7 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 185
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "7nerC1KunT8u",
"colab_type": "code",
"colab": {}
},
"source": [
"#iris.apply(pd.Series.round, axis=1)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "PaPz8q3pjWZQ",
"colab_type": "code",
"colab": {}
},
"source": [
"#def sepal_category(x):\n",
" \n",
"# if x == 4:\n",
"# return \"big\"\n",
"\n",
"#iris['sepal_category'] = iris[['sepal_width']].apply(sepal_category)\n",
"#iris.head()\n",
" "
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "gq3i_yeDWWHt",
"colab_type": "code",
"outputId": "fea43304-cf2a-4a76-acd1-bdeb66e51476",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 87
}
},
"source": [
"#example of a smarter function\n",
"def smart_multiply_by_100(x):\n",
" if x > 5:\n",
" return 1\n",
" return x\n",
"\n",
"inputs = [1,2,6,10]\n",
"for input in inputs:\n",
" print(smart_multiply_by_100(input))\n",
" \n",
" "
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"1\n",
"2\n",
"1\n",
"1\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "6ox8_LbHW_vQ",
"colab_type": "code",
"colab": {}
},
"source": [
""
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "C95FIjW5-rJc",
"colab_type": "text"
},
"source": [
"#### Writing Lambdas\n",
"\n",
"Generally considered to be unnecessary. A Python lambda is an inline python and it can often lead to confusing code. \n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "1Zibl40rD6Qg",
"colab_type": "code",
"outputId": "8cd0c6ec-6438-41eb-fe37-5478070e13da",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 381
}
},
"source": [
"import this"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"The Zen of Python, by Tim Peters\n",
"\n",
"Beautiful is better than ugly.\n",
"Explicit is better than implicit.\n",
"Simple is better than complex.\n",
"Complex is better than complicated.\n",
"Flat is better than nested.\n",
"Sparse is better than dense.\n",
"Readability counts.\n",
"Special cases aren't special enough to break the rules.\n",
"Although practicality beats purity.\n",
"Errors should never pass silently.\n",
"Unless explicitly silenced.\n",
"In the face of ambiguity, refuse the temptation to guess.\n",
"There should be one-- and preferably only one --obvious way to do it.\n",
"Although that way may not be obvious at first unless you're Dutch.\n",
"Now is better than never.\n",
"Although never is often better than *right* now.\n",
"If the implementation is hard to explain, it's a bad idea.\n",
"If the implementation is easy to explain, it may be a good idea.\n",
"Namespaces are one honking great idea -- let's do more of those!\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "dEvuisbJ-tc2",
"colab_type": "code",
"outputId": "49021143-966c-4ba9-b5de-9e5548929bbf",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"func = lambda x: x**2\n",
"func(4)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"16"
]
},
"metadata": {
"tags": []
},
"execution_count": 187
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "Cq9-b_1N-tpI",
"colab_type": "code",
"outputId": "510ceb16-137a-4d9a-e2ec-69a2c2ff117b",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"def regular_func(x):\n",
" return x**2\n",
"\n",
"regular_func(4)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"16"
]
},
"metadata": {
"tags": []
},
"execution_count": 189
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "FlIAJo2GK7KL",
"colab_type": "code",
"colab": {}
},
"source": [
"def regular_func2(x):\n",
" \"\"\"This makes my variable go to the second power\"\"\"\n",
" return x**2"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "ZOMmTXtSLF9W",
"colab_type": "code",
"outputId": "4089a1d4-a04b-4e56-c949-2496b8740d60",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"regular_func2(2)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"4"
]
},
"metadata": {
"tags": []
},
"execution_count": 178
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "rnXLQ6rZLKj5",
"colab_type": "code",
"colab": {}
},
"source": [
""
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "QS-zJFR-DlPI",
"colab_type": "code",
"colab": {}
},
"source": [
"import random"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "YAcj5rn5DnEI",
"colab_type": "code",
"colab": {}
},
"source": [
"random.seed"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "EnmY9K44MD8V",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nqKDdhHRL7bs",
"colab_type": "text"
},
"source": [
"### Exercise: Write a function that returns infinite values"
]
},
{
"cell_type": "code",
"metadata": {
"id": "jxAK_PSN3iHt",
"colab_type": "code",
"colab": {}
},
"source": [
"def forever():\n",
" while True:\n",
" yield 1\n"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "NjSxSrZH3zP_",
"colab_type": "code",
"outputId": "54595cb2-3374-4dec-d0b8-9124e69e637d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"res = forever()\n",
"type(res)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"generator"
]
},
"metadata": {
"tags": []
},
"execution_count": 192
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "ci5oMfTY34yU",
"colab_type": "code",
"outputId": "11ea9cae-224d-412e-862f-63ec19c8037a",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"next(res)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1"
]
},
"metadata": {
"tags": []
},
"execution_count": 195
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "j7lDdPgS4RR8",
"colab_type": "code",
"colab": {}
},
"source": [
"#Finite\n",
"def not_forever():\n",
" vars = [1,2,3,5]\n",
" for var in vars:\n",
" yield var\n"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "Cv7qba4m4dIG",
"colab_type": "code",
"colab": {}
},
"source": [
"res2 = not_forever()"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "azWB311x4i2W",
"colab_type": "code",
"outputId": "a28510a4-4eca-4eef-df00-cb417a351c91",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 167
}
},
"source": [
"next(res2)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "error",
"ename": "StopIteration",
"evalue": "ignored",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mStopIteration\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-202-45c410d5a3dc>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;31mStopIteration\u001b[0m: "
]
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FfOtUt8NMFtR",
"colab_type": "text"
},
"source": [
"### Exercise: Capture the output of a shell command using the ! operator and parse with SList grep"
]
},
{
"cell_type": "code",
"metadata": {
"id": "72sCdojn4wiV",
"colab_type": "code",
"outputId": "4a8bbe77-22e6-4827-9b14-9c53c8023c30",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 87
}
},
"source": [
"!ls -l"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"total 12\n",
"-rw-r--r-- 1 root root 2610 Jan 8 13:34 adc.json\n",
"-rw-r--r-- 1 root root 407 Jan 8 13:27 magic_stuff.py\n",
"drwxr-xr-x 1 root root 4096 Dec 18 16:52 sample_data\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "HJcJA5Q444ii",
"colab_type": "code",
"outputId": "659d5ef3-4852-41c0-a0b0-351326622026",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"directory_listing = !ls -l\n",
"type(directory_listing)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"IPython.utils.text.SList"
]
},
"metadata": {
"tags": []
},
"execution_count": 204
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "mLJVR-pi5BMk",
"colab_type": "code",
"outputId": "ef8756bd-27a8-4297-c751-bbd0b20f8540",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"directory_listing.grep(\".py\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['-rw-r--r-- 1 root root 407 Jan 8 13:27 magic_stuff.py']"
]
},
"metadata": {
"tags": []
},
"execution_count": 207
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "I98BmWesoVG7",
"colab_type": "text"
},
"source": [
"Q/A: HOW DO I CONTINUE TO A NEW LINE"
]
},
{
"cell_type": "code",
"metadata": {
"id": "U0IbP1XhobdP",
"colab_type": "code",
"colab": {}
},
"source": [
"var = \"\"\"This is a a very long string\n",
" and it needs to be continued\"\"\""
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "9e8YB8_YomtP",
"colab_type": "code",
"colab": {}
},
"source": [
"long_list = [12,3,5,\n",
" 6,7,7,8]"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "2XF73XozovSF",
"colab_type": "code",
"colab": {}
},
"source": [
"#\\"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "c1MbWXS8GEYt",
"colab_type": "text"
},
"source": [
"## Automating Text and Filesystem"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_QcBIV2kHFXf",
"colab_type": "text"
},
"source": [
"### Presentation: Reading, Writing and Using files: TXT, CSV, and YAML"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Kfwkhkd6uFie",
"colab_type": "text"
},
"source": [
"#### Writing to a file\n",
"\n",
"* *[Watch Video Lesson 6.1: Use write file operations](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118/9780135261118-EMLA_01_06_01)*"
]
},
{
"cell_type": "code",
"metadata": {
"id": "BXIgWwb_u5To",
"colab_type": "code",
"outputId": "cd2f6b6b-df9c-4795-ebeb-0b0d0ad65f64",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 69
}
},
"source": [
"f = open('workfile.txt', 'w')\n",
"f.write(\"foo20\\n\")\n",
"f.write(\"foo3\\n\")\n",
"f.write(\"foo4\\n\")\n",
"f.close()\n",
"!cat workfile.txt\n",
"\n",
"#!ls -l"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"foo20\n",
"foo3\n",
"foo4\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "GURQsbCZApwI",
"colab_type": "code",
"outputId": "2c11da69-aae7-46f1-a865-a6ea8cd74178",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 69
}
},
"source": [
"my_writes = [\"foo2\\n\",\"foo3\\n\",\"foo4\\n\"]\n",
"f = open('workfile2.txt', 'w')\n",
"for line in my_writes:\n",
" f.write(line)\n",
"f.close()\n",
"!cat workfile2.txt"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"foo2\n",
"foo3\n",
"foo4\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ANVzyHdUu6vz",
"colab_type": "text"
},
"source": [
"#### Writing to a file with 'context'"
]
},
{
"cell_type": "code",
"metadata": {
"id": "h_Q944tSu62P",
"colab_type": "code",
"outputId": "7abb84f9-7df6-4a3a-f66a-57de56638be0",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"with open(\"workfile.txt\", \"w\") as workfile:\n",
" workfile.write(\"bam\")\n",
"!cat workfile.txt"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"bam"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8Flny7vIvBfE",
"colab_type": "text"
},
"source": [
"#### Reading a file in\n",
"\n",
"* *[Watch Video Lesson 6.2: Use read file operations](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118/9780135261118-EMLA_01_06_02)*\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "My_lNi5dvBm5",
"colab_type": "code",
"outputId": "0ef53c3a-03f2-4add-ebcf-aba5a2752b61",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
}
},
"source": [
"f = open(\"workfile.txt\", \"r\")\n",
"out = f.readlines() #r.read() works as well\n",
"# Maybe we want to create a generator pipeline\n",
"for line in out:\n",
" print(line)\n",
" # yield line\n",
"f.close()\n",
"print(out)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"bam\n",
"['bam']\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TEP1eI7SvHnu",
"colab_type": "text"
},
"source": [
"#### Reading a file with 'context'"
]
},
{
"cell_type": "code",
"metadata": {
"id": "-hk5VpDsvHxH",
"colab_type": "code",
"outputId": "4c09452f-0ac5-4d79-faba-50b84dc89646",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"with open(\"workfile.txt\", \"r\") as workfile:\n",
" print(workfile.readlines())\n",
" #print(workfile.read())\n"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"['bam']\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "A_qp5vlUSJIw",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rqj1rvHgwSwv",
"colab_type": "text"
},
"source": [
"#### Serialize a Python Dictionary to Pickle"
]
},
{
"cell_type": "code",
"metadata": {
"id": "xhNF4RNxwY3e",
"colab_type": "code",
"colab": {}
},
"source": [
"mydict = {\"one\":1, \"two\":2}"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "DMDCokXsweu8",
"colab_type": "code",
"colab": {}
},
"source": [
"import pickle"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "eVqqAXrEwhPS",
"colab_type": "code",
"colab": {}
},
"source": [
"pickle.dump(mydict, open('mydictionary.pickle', 'wb'))"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "QoRWp5MawitC",
"colab_type": "code",
"outputId": "326d4b8c-7081-4a4b-a001-bb27a2447c8f",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"!ls -lh mydictionary.pickle"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"-rw-r--r-- 1 root root 32 Jan 8 15:14 mydictionary.pickle\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "zwiN7WnpwkhZ",
"colab_type": "code",
"colab": {}
},
"source": [
"#!cat mydictionary.pickle"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "dGU7RVunwoAK",
"colab_type": "code",
"colab": {}
},
"source": [
"res = pickle.load(open('mydictionary.pickle', \"rb\"))"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "eC4gJSRkwp3C",
"colab_type": "code",
"outputId": "627cd778-11bf-42b2-f495-3f5dd6cd3868",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"print(res)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"{'one': 1, 'two': 2}\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hM20ft66wxm3",
"colab_type": "text"
},
"source": [
"#### Serialize a Python Dictionary to JSON\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "GLHcnEjyw0Sy",
"colab_type": "code",
"colab": {}
},
"source": [
"import json\n",
"with open('data.json', 'w') as outfile:\n",
" json.dump(res, outfile)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "yeFPhSkbw2Du",
"colab_type": "code",
"outputId": "ed8ad02c-2179-4d6b-dad5-b227e9b6d11e",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"!cat data.json"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"{\"one\": 1, \"two\": 2}"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "-IM3TA0Pw4tC",
"colab_type": "code",
"colab": {}
},
"source": [
"with open('data.json', 'rb') as outfile:\n",
" res2 = json.load(outfile)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "cWW27L2aw6lz",
"colab_type": "code",
"outputId": "53c41b9f-cbe1-4767-f99d-7b687616e70e",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
}
},
"source": [
"print(res2)\n",
"type(res2)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"{'one': 1, 'two': 2}\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"dict"
]
},
"metadata": {
"tags": []
},
"execution_count": 228
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rh8w9HQ7xbYp",
"colab_type": "text"
},
"source": [
"#### Save to Yaml"
]
},
{
"cell_type": "code",
"metadata": {
"id": "gbHq08eBxgH-",
"colab_type": "code",
"colab": {}
},
"source": [
"import yaml"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "jwXd4158xgT8",
"colab_type": "code",
"colab": {}
},
"source": [
"with open(\"data.yaml\", \"w\") as yamlfile: \n",
" yaml.safe_dump(res2, yamlfile, default_flow_style=False)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "OWvZMStHxgex",
"colab_type": "code",
"outputId": "e1afa4f8-56ad-4998-dd6d-056fc9d854bb",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
}
},
"source": [
"!cat data.yaml"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"one: 1\n",
"two: 2\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UfbzysQGxsR3",
"colab_type": "text"
},
"source": [
"#### Load Yaml"
]
},
{
"cell_type": "code",
"metadata": {
"id": "VgBxjVHExwtI",
"colab_type": "code",
"colab": {}
},
"source": [
"with open(\"data.yaml\", \"rb\") as yamlfile: \n",
" res3 = yaml.safe_load(yamlfile) "
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "dDrmL9p5xyFk",
"colab_type": "code",
"outputId": "ea59e502-6bfd-4d4d-a09b-e9a7e7da2d09",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
}
},
"source": [
"print(res3)\n",
"type(res3)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"{'one': 1, 'two': 2}\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"dict"
]
},
"metadata": {
"tags": []
},
"execution_count": 234
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-TNs6mfm97X0",
"colab_type": "text"
},
"source": [
"### TO DO Protobuff Example"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VawMwOlX9_r_",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WrRrxofUSJqe",
"colab_type": "text"
},
"source": [
"### Exercise: Read and Write a YAML file"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IJJPbByISJte",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "code",
"metadata": {
"id": "jJrxkgnx8Oq5",
"colab_type": "code",
"colab": {}
},
"source": [
"new_variable = {\"IP_Address\": \"Hostname\"}\n",
"with open(\"hosts.yaml\", \"w\") as yamlfile:\n",
" yaml.safe_dump(new_variable, yamlfile, default_flow_style=False)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "AMZa6pKp8rLJ",
"colab_type": "code",
"outputId": "2972cb77-e4c7-4331-dba0-fc0d5745276f",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"!cat hosts.yaml"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"IP_Address: Hostname\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ja91sLQ88zY-",
"colab_type": "text"
},
"source": [
"Q: So what is the difference between pickle and json libraries? they seem to do the same thing?![alt text](![alt text](https://))"
]
},
{
"cell_type": "code",
"metadata": {
"id": "JYKHIVSf81lj",
"colab_type": "code",
"colab": {}
},
"source": [
"var = {\"one\":\"two\"}\n",
"pickle.dump(var, open('unreadable.pickle', 'wb'))"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "DOpRigaC9IiD",
"colab_type": "code",
"outputId": "def12e9b-8d52-43b9-b18f-2a00af8b38bd",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"!ls -lh unreadable.pickle"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"-rw-r--r-- 1 root root 27 Nov 11 14:53 unreadable.pickle\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "M4s0eS3u9iNJ",
"colab_type": "code",
"colab": {}
},
"source": [
"with open('var.json', 'w') as outfile:\n",
" json.dump(var, outfile)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "wOTwJ0Vp9sl4",
"colab_type": "code",
"outputId": "bdf51732-4ef8-49f4-8ed4-0da3920c1540",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"!ls -lh var.json"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"-rw-r--r-- 1 root root 14 Nov 11 14:55 var.json\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "54BxP4dT9HyS",
"colab_type": "text"
},
"source": [
"Q: How would you really compare pickle size vs JSON (test it out)..?"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-hsimY7KSQCD",
"colab_type": "text"
},
"source": [
"### Presentation: Managing files and directories using os.path and pathlib"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "R1dKjjhpSQFK",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "code",
"metadata": {
"id": "I6jV7wQYgGVL",
"colab_type": "code",
"colab": {}
},
"source": [
"# Make some dirs\n",
"import pathlib\n",
"import os\n",
"\n",
"path = \"somedir\"\n",
"#Depending on the script you are writing maybe you want this False\n",
"pathlib.Path(path).mkdir(parents=True, exist_ok=True)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "tgOXVqFBgTef",
"colab_type": "code",
"outputId": "d419074f-14fe-41c1-eebd-69fe275e7926",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 69
}
},
"source": [
"ls -la somedir"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"total 12\n",
"drwxr-xr-x 2 root root 4096 Jan 8 15:20 \u001b[0m\u001b[01;34m.\u001b[0m/\n",
"drwxr-xr-x 1 root root 4096 Jan 8 15:20 \u001b[01;34m..\u001b[0m/\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "OAuM8rwjCs9x",
"colab_type": "code",
"colab": {}
},
"source": [
"import pathlib\n",
"pathvar = pathlib.Path(\"newfile.txt\")\n",
"pathvar2 = pathlib.Path(\"somedir\")"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "GX8yZ6SxDKDG",
"colab_type": "text"
},
"source": [
"Check if path exists?"
]
},
{
"cell_type": "code",
"metadata": {
"id": "QJhTeiWHC1Yu",
"colab_type": "code",
"outputId": "6724eab5-611e-45dc-b4d8-8325338f7fc9",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"pathvar.exists()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"False"
]
},
"metadata": {
"tags": []
},
"execution_count": 248
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "Vbsy-GMeDCFV",
"colab_type": "code",
"outputId": "498899ed-46df-4ac3-9670-bb63714d1113",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"pathvar2.exists()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"True"
]
},
"metadata": {
"tags": []
},
"execution_count": 244
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "rw-wpPqJDM0t",
"colab_type": "code",
"outputId": "bf14a7f3-64b0-4bdc-e3a7-8b557bfe8864",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"pathvar.write_text(\"foo\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"3"
]
},
"metadata": {
"tags": []
},
"execution_count": 249
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "EQOSDCm9DaOJ",
"colab_type": "code",
"outputId": "fd92d1b3-120f-4c1d-d78b-779c2d2f9bd5",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"cat newfile.txt"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"foo"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4NO023xTSVwE",
"colab_type": "text"
},
"source": [
"### Exercise: Use pathlib to rename files"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "81Q9__deSVzf",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "code",
"metadata": {
"id": "KYEP5xmuh0eI",
"colab_type": "code",
"outputId": "ee229ea7-5c79-4c5e-8802-8611f5c37f4d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 191
}
},
"source": [
"from pathlib import Path\n",
"p = Path('foo.txt')\n",
"p.open('w').write('bam')\n",
"\n",
"#This file is created\n",
"print(\"Before rename\")\n",
"!ls -l *.txt\n",
"\n",
"out = Path('bar.txt')\n",
"p.rename(out)\n",
"\n",
"#It is renamed\n",
"print(\"After rename\")\n",
"!ls -l *.txt\n",
"#!rm bar.txt"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Before rename\n",
"-rw-r--r-- 1 root root 3 Jan 8 15:25 foo.txt\n",
"-rw-r--r-- 1 root root 3 Jan 8 15:23 newfile.txt\n",
"-rw-r--r-- 1 root root 15 Jan 8 15:11 workfile2.txt\n",
"-rw-r--r-- 1 root root 3 Jan 8 15:12 workfile.txt\n",
"After rename\n",
"-rw-r--r-- 1 root root 3 Jan 8 15:25 bar.txt\n",
"-rw-r--r-- 1 root root 3 Jan 8 15:23 newfile.txt\n",
"-rw-r--r-- 1 root root 15 Jan 8 15:11 workfile2.txt\n",
"-rw-r--r-- 1 root root 3 Jan 8 15:12 workfile.txt\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "W6CzcDJwwnaO",
"colab_type": "code",
"outputId": "0ea3b031-7d30-4612-cd25-200bf9519a66",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"cat bar.txt"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"bam"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "xLfi4UR7w2sT",
"colab_type": "code",
"colab": {}
},
"source": [
"mypath = Path('bar.txt')"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "9VeytCZ1xCDG",
"colab_type": "text"
},
"source": [
"Try things out"
]
},
{
"cell_type": "code",
"metadata": {
"id": "LIv2Rc0mw70I",
"colab_type": "code",
"outputId": "8a10f395-235c-4de3-d234-3dd8175759ce",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"mypath.absolute()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"PosixPath('/content/bar.txt')"
]
},
"metadata": {
"tags": []
},
"execution_count": 256
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "zCJdK_qRxEEV",
"colab_type": "code",
"outputId": "f332eb12-bca5-4e42-b64a-9dc340bede36",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"mypath.cwd()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"PosixPath('/content')"
]
},
"metadata": {
"tags": []
},
"execution_count": 257
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IMgQrTLsBVUR",
"colab_type": "text"
},
"source": [
"\n",
"Fnmatch\n",
"```python\n",
"import fnmatch\n",
"\n",
"files = [\"data1.csv\", \"script.py\", \"image.png\", \"data2.csv\", \"all.py\"]\n",
"\n",
"\n",
"def csv_matches(files):\n",
" \"\"\"Return matches for csv files\"\"\"\n",
"\n",
" matches = fnmatch.filter(files, \"*.csv\")\n",
" return matches\n",
"\n",
"\n",
"# Print matches to pattern\n",
"print(f\"Found matches: {csv_matches(files)}\")\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cgnt87WDBnk8",
"colab_type": "text"
},
"source": [
"Archive?\n",
"\n",
"\n",
"\n",
"```python\n",
"from shutil import make_archive\n",
"import os\n",
"\n",
"username = \"user1\"\n",
"root_dir = \"/tmp\"\n",
"# archive path\n",
"path = f\"{root_dir}/{username}\"\n",
"\n",
"# create tar and gzipped archive\n",
"make_archive(username, \"gztar\", root_dir)\n",
"\n",
"# create zip archive\n",
"make_archive(username, \"zip\", root_dir)\n",
"\n",
"print(os.listdir(\"/tmp\"))\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "k2u1noABSY7Z",
"colab_type": "text"
},
"source": [
"### Presentation: Walking directory trees using os.walk"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9ZfRJfD2SY-b",
"colab_type": "text"
},
"source": [
"Example from standard library docs...but lets change.\n",
"\n",
"```python\n",
"import os\n",
"from os.path import join, getsize\n",
"for root, dirs, files in os.walk('somedir'):\n",
" print(root, \"consumes\", end=\" \")\n",
" print(sum(getsize(join(root, name)) for name in files), end=\" \")\n",
" print(\"bytes in\", len(files), \"non-directory files\")\n",
" if 'CVS' in dirs:\n",
" dirs.remove('CVS') # don't visit CVS directories\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7OSRDtFTGUQH",
"colab_type": "text"
},
"source": [
"These are the hooks to walk the file system..."
]
},
{
"cell_type": "code",
"metadata": {
"id": "xxDHcAJSFXCn",
"colab_type": "code",
"outputId": "f2a062f4-0306-4dd6-d419-383a04716e9a",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 159
}
},
"source": [
"import os\n",
"from os.path import join, getsize\n",
"for root, dirs, files in os.walk('sample_data'):\n",
" print(f\"files I found: {files}\")\n",
" #full path\n",
" for name in files:\n",
" full_path = join(root, name)\n",
" print(f\"full path to files: {full_path}\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"files I found: ['README.md', 'anscombe.json', 'mnist_test.csv', 'mnist_train_small.csv', 'california_housing_test.csv', 'california_housing_train.csv']\n",
"full path to files: sample_data/README.md\n",
"full path to files: sample_data/anscombe.json\n",
"full path to files: sample_data/mnist_test.csv\n",
"full path to files: sample_data/mnist_train_small.csv\n",
"full path to files: sample_data/california_housing_test.csv\n",
"full path to files: sample_data/california_housing_train.csv\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BnkhWhAVSZGG",
"colab_type": "text"
},
"source": [
"### Presentation: Getting stat information on files and directories"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "T44g0aDbBTQ9",
"colab_type": "text"
},
"source": [
"Use the same technique to explore the stat methods"
]
},
{
"cell_type": "code",
"metadata": {
"id": "XaDUwugRGt4X",
"colab_type": "code",
"colab": {}
},
"source": [
"import os\n",
"location = \"sample_data/anscombe.json\"\n",
"result = os.stat(location)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "Ke_p-0GjG7CH",
"colab_type": "code",
"outputId": "d340c94c-ca39-4008-8d46-0cfd09a46bcb",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"result.st_gid"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0"
]
},
"metadata": {
"tags": []
},
"execution_count": 260
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "e338KmC2ylFo",
"colab_type": "code",
"outputId": "2c5b62c7-28c5-4ea8-9745-d6f4f5130e4f",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"result.st_size"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1697"
]
},
"metadata": {
"tags": []
},
"execution_count": 262
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JgPZWw-cA_0B",
"colab_type": "text"
},
"source": [
"### Subprocess.run"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b9pNy1bTBDP0",
"colab_type": "text"
},
"source": [
"\n",
"\n",
"```python\n",
"import subprocess\n",
"import shlex\n",
"\n",
"print(\"Enter a list of directories to calculate storage total: \\n\")\n",
"user_input = \"pluto mars jupyter\"\n",
"sanitized_user_input = shlex.split(user_input)\n",
"print(f\"raw_user_input: {user_input} | sanitized_user_input: {sanitized_user_input}\")\n",
"cmd = [\"du\", \"-sh\", \"--total\"]\n",
"cmd.extend(sanitized_user_input)\n",
"print(f\"cmd: {cmd}\")\n",
"disk_total = subprocess.run(cmd, stdout=subprocess.PIPE)\n",
"print(disk_total)\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EsZu4zjxSfP_",
"colab_type": "text"
},
"source": [
"\n",
"# Example health check\n",
"```python\n",
"import subprocess\n",
"import os\n",
"\n",
"# setup\n",
"file_location = \"/tmp/file.txt\"\n",
"expected_uid = 501\n",
"# touch a file\n",
"proc = subprocess.Popen([\"touch\", file_location])\n",
"\n",
"# check user permissions\n",
"stat = os.stat(file_location)\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8Hd2HFK2SfbH",
"colab_type": "text"
},
"source": [
"### Presentation: Finding files: duplicates, globbing, and patterns"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Vp-hgWj2SffK",
"colab_type": "text"
},
"source": [
"\n",
"#subprocess find\n",
"```python\n",
"import subprocess\n",
"\n",
"print(\"Enter a path to search for directories: \\n\")\n",
"user_input = input()\n",
"print(f\"user_input: {user_input}\")\n",
"with subprocess.Popen(\n",
" [\"find\", user_input, \"-type\", \"d\"], stdout=subprocess.PIPE\n",
") as find:\n",
" result = find.stdout.readlines()\n",
" for line in result:\n",
" print(f\"Found Directory: {line}\")\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "PjXLYlsQfuPj",
"colab_type": "code",
"outputId": "837a02af-c9c3-4fc6-87ea-a041b0ee8693",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
}
},
"source": [
"!touch foo.csv && touch bar.txt\n",
"import glob\n",
"glob.glob(\"*.csv\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['foo.csv']"
]
},
"metadata": {
"tags": []
},
"execution_count": 264
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TEpmglHZSiGX",
"colab_type": "text"
},
"source": [
"### Exercise: Find files using glob"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Wal4LPEOSmBR",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "code",
"metadata": {
"id": "BPv951NsIRS4",
"colab_type": "code",
"outputId": "4b671e3d-adfd-47b0-8af3-cb1e30cac5dc",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 104
}
},
"source": [
"path = \"sample_data\"\n",
"results = glob.glob(f\"{path}/*.csv\")\n",
"\n",
"#how many many CSV did you find?\n",
"print(len(results))\n",
"results"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"4\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['sample_data/mnist_test.csv',\n",
" 'sample_data/mnist_train_small.csv',\n",
" 'sample_data/california_housing_test.csv',\n",
" 'sample_data/california_housing_train.csv']"
]
},
"metadata": {
"tags": []
},
"execution_count": 265
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "ekqeCDxsIqCh",
"colab_type": "code",
"outputId": "cce3761c-2a56-4dad-f6d1-6014cc13f409",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
}
},
"source": [
"path = \"sample_data\"\n",
"results = glob.glob(f\"{path}/*.json\")\n",
"\n",
"#how many many CSV did you find?\n",
"print(len(results))\n",
"results"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"1\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['sample_data/anscombe.json']"
]
},
"metadata": {
"tags": []
},
"execution_count": 266
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3rIS-eHrHGAX",
"colab_type": "text"
},
"source": [
"## Developing with the Command Line"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3-epgMWYHRIY",
"colab_type": "text"
},
"source": [
"### Presentation: Setting up a Python project with VSCode and pip"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8oevsZYBTFFX",
"colab_type": "text"
},
"source": [
"\n",
"Setup Virtual Environment workflow\n",
"```\n",
" 1 python3\n",
" 2 sudo yum update\n",
" 3 python3 -m venv ~/.python-devops-course\n",
" 4 which python3\n",
" 5 source ~/.python-devops-course/bin/activate\n",
" 6 which python3\n",
" 7 python3\n",
" 8 history\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "03sKQHy_TFIj",
"colab_type": "text"
},
"source": [
"### Presentation: Using sys.argv"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bHAEIX_-THFp",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M0dVUfeETK4a",
"colab_type": "text"
},
"source": [
"### Exercise: Write a command-line tool with sys.argv"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DdwBULNtTXcj",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VaNfwqzsTNcm",
"colab_type": "text"
},
"source": [
"### Presentation: Using click CLI Framework"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Y1bWeR0wTPzO",
"colab_type": "text"
},
"source": [
"### Write a command-line tool with click"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yHi_dqaZTSd8",
"colab_type": "text"
},
"source": [
"Reference: https://click.palletsprojects.com/en/5.x/quickstart/"
]
},
{
"cell_type": "code",
"metadata": {
"id": "cEMmQESUe8WR",
"colab_type": "code",
"outputId": "65247bfa-988e-41a0-a514-56d6309b781e",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"%%writefile hello-click.py\n",
"import click\n",
"\n",
"@click.command()\n",
"def hello():\n",
" click.echo('Hello World!')\n",
"\n",
"if __name__ == '__main__':\n",
" hello()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Writing hello-click.py\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "s9k7usChfMPW",
"colab_type": "code",
"outputId": "65d025ef-b858-42bb-cba3-bf2b7bd1643c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"!python hello-click.py"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Hello World!\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "slemlZcgTT2W",
"colab_type": "text"
},
"source": [
"### Presentation: click, subprocess.run, and mixing shell and Python together"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "21UVe2NETVCP",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iD-9dU8DI6j9",
"colab_type": "text"
},
"source": [
"## Continuous Integration and Delivery"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nNeuZROzI-pI",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "K-rwZGi1UJPb",
"colab_type": "text"
},
"source": [
"### Presentation: Creating Makefiles"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wqUr1XEAUL4-",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8yduXBcZUL8-",
"colab_type": "text"
},
"source": [
"### Exercise: Create a Makefile"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EYw4GHuLUORG",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZUiGf3pSUOUN",
"colab_type": "text"
},
"source": [
"### Presentation: Creating Python virtualenv"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AzmzrRQfUTAK",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dKcauKdOUTDQ",
"colab_type": "text"
},
"source": [
"### Exercise: Create and use virtualenv"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kUEq2tVsUWlx",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dJ2jbaVHUWo0",
"colab_type": "text"
},
"source": [
"### Presentation: Linting and testing Python code with Pylint and PyTest"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xrrKjQ6iUZOo",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qimXAGPDUZaE",
"colab_type": "text"
},
"source": [
"### Exercise: Lint code with Pylint"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ns4fQErGVQYN",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kS6oxmbhVQ5f",
"colab_type": "text"
},
"source": [
"### Presentation: Configuring CircleCI Continuous Integration with Github"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_Hehf5WXXfZY",
"colab_type": "text"
},
"source": [
"### Exercise: Setup github repo to build with CircleCI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f6LzSpGRXgQo",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DAq-rESbXgmS",
"colab_type": "text"
},
"source": [
"### Presentation: What is Continuous Delivery?"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cICr0LtxXgvx",
"colab_type": "text"
},
"source": [
""
]
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment