Skip to content

Instantly share code, notes, and snippets.

@psychemedia
Last active April 28, 2020 19:23
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save psychemedia/df5fc6d0b4a9cb4d3e61202fc970ac58 to your computer and use it in GitHub Desktop.
Save psychemedia/df5fc6d0b4a9cb4d3e61202fc970ac58 to your computer and use it in GitHub Desktop.
UK Covid-19 NHS Daily deaths
pandas
sqlite-utils
parse
dateparser
Display the source blob
Display the rendered blob
Raw
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Covid-19 Daily Deaths - UK\n",
"\n",
"Via: https://www.england.nhs.uk/statistics/statistical-work-areas/covid-19-daily-deaths/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Daily reports are published as an Excel spreadhseet linked from the following page:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# Reporting page\n",
"url = 'https://www.england.nhs.uk/statistics/statistical-work-areas/covid-19-daily-deaths/'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load the page:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import requests\n",
"\n",
"page = requests.get(url)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Get the HTML page data into a form we can scrape it:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"from bs4 import BeautifulSoup, SoupStrainer\n",
"\n",
"soup = BeautifulSoup(page.text)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Get the relevant links to the daily spreadseets:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'COVID 19 daily announced deaths 28 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-28-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 27 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-27-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 26 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-26-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 25 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-25-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 24 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-24-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 23 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-23-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 22 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-22-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 21 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-21-April-2020-1.xlsx',\n",
" 'COVID 19 daily announced deaths 20 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-20-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 19 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-19-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 18 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-18-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 17 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-17-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 16 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-16-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 15 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-15-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 14 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-14-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 13 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-13-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 12 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-12-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 11 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-11-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 10 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-10-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 9 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-9-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 8 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-8-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 7 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-7-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 6 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-6-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 5 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-5-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 4 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-4-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 3 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-3-April-2020.xlsx',\n",
" 'COVID 19 daily announced deaths 2 April 2020': 'https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/04/COVID-19-daily-announced-deaths-2-April-2020-1.xlsx'}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"links = {}\n",
"for link in soup.find(\"article\", {\"class\": \"rich-text\"}).find_all('a'):\n",
" if link.text.startswith('COVID 19 daily announced deaths'):\n",
" if link.text not in links:\n",
" links[link.text] = link.get('href')\n",
" elif link.text.startswith('COVID 19 total announced deaths'):\n",
" totals_link = link.get('href')\n",
"links"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Start to sketch out how we can parse the data out of one of the spreadsheets. The following has been arrivied though a little bit of iteration an previewing of the data:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"odict_keys(['COVID19 daily deaths by region', 'COVID19 daily deaths by age', 'COVID19 daily deaths by trust'])"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sheets = pd.read_excel(links['COVID 19 daily announced deaths 9 April 2020'],\n",
" sheet_name=None)\n",
"\n",
"# What sheets are available in the spreadsheet\n",
"sheet_names = sheets.keys()\n",
"sheet_names"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The spreadsheet contains the following sheets:\n",
"\n",
"- `COVID19 daily deaths by region`\n",
"- `COVID19 daily deaths by age`\n",
"- `COVID19 daily deaths by trust`\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Cleaning the Sheets\n",
"\n",
"Clean the sheets to get the actual data:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Unnamed: 0</th>\n",
" <th>Unnamed: 1</th>\n",
" <th>Unnamed: 2</th>\n",
" <th>Unnamed: 3</th>\n",
" <th>Unnamed: 4</th>\n",
" <th>Unnamed: 5</th>\n",
" <th>Unnamed: 6</th>\n",
" <th>Unnamed: 7</th>\n",
" <th>Unnamed: 8</th>\n",
" <th>Unnamed: 9</th>\n",
" <th>...</th>\n",
" <th>Unnamed: 32</th>\n",
" <th>Unnamed: 33</th>\n",
" <th>Unnamed: 34</th>\n",
" <th>Unnamed: 35</th>\n",
" <th>Unnamed: 36</th>\n",
" <th>Unnamed: 37</th>\n",
" <th>Unnamed: 38</th>\n",
" <th>Unnamed: 39</th>\n",
" <th>Unnamed: 40</th>\n",
" <th>Unnamed: 41</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>NaN</td>\n",
" <td>Title:</td>\n",
" <td>COVID-19 daily deaths announcement</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>NaN</td>\n",
" <td>Summary:</td>\n",
" <td>Deaths of patients who have died in hospitals ...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>Period:</td>\n",
" <td>5pm 07 April 2020 - 5pm 08 April 2020</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>NaN</td>\n",
" <td>Source:</td>\n",
" <td>COVID-19 Patient Notification System</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>NaN</td>\n",
" <td>Basis:</td>\n",
" <td>Provider</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>NaN</td>\n",
" <td>Published:</td>\n",
" <td>2020-04-09 00:00:00</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>NaN</td>\n",
" <td>Revised:</td>\n",
" <td>-</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>NaN</td>\n",
" <td>Status:</td>\n",
" <td>Published</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>NaN</td>\n",
" <td>Contact:</td>\n",
" <td>england.nhsdata@nhs.net</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>NaN</td>\n",
" <td>Regional summary</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>NaN</td>\n",
" <td>NHS England Region</td>\n",
" <td>NaN</td>\n",
" <td>2020-03-16 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>2020-03-23 00:00:00</td>\n",
" <td>2020-03-24 00:00:00</td>\n",
" <td>2020-03-26 00:00:00</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>NaN</td>\n",
" <td>England</td>\n",
" <td>NaN</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>...</td>\n",
" <td>104.0</td>\n",
" <td>105.0</td>\n",
" <td>106.0</td>\n",
" <td>107.0</td>\n",
" <td>108.0</td>\n",
" <td>109.0</td>\n",
" <td>110.0</td>\n",
" <td>111.0</td>\n",
" <td>112.0</td>\n",
" <td>113.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>15 rows × 42 columns</p>\n",
"</div>"
],
"text/plain": [
" Unnamed: 0 Unnamed: 1 \\\n",
"0 NaN Title: \n",
"1 NaN Summary: \n",
"2 NaN Period: \n",
"3 NaN Source: \n",
"4 NaN Basis: \n",
"5 NaN Published: \n",
"6 NaN Revised: \n",
"7 NaN Status: \n",
"8 NaN Contact: \n",
"9 NaN NaN \n",
"10 NaN Regional summary \n",
"11 NaN NaN \n",
"12 NaN NHS England Region \n",
"13 NaN England \n",
"14 NaN NaN \n",
"\n",
" Unnamed: 2 Unnamed: 3 \\\n",
"0 COVID-19 daily deaths announcement NaN \n",
"1 Deaths of patients who have died in hospitals ... NaN \n",
"2 5pm 07 April 2020 - 5pm 08 April 2020 NaN \n",
"3 COVID-19 Patient Notification System NaN \n",
"4 Provider NaN \n",
"5 2020-04-09 00:00:00 NaN \n",
"6 - NaN \n",
"7 Published NaN \n",
"8 england.nhsdata@nhs.net NaN \n",
"9 NaN NaN \n",
"10 NaN NaN \n",
"11 NaN NaN \n",
"12 NaN 2020-03-16 00:00:00 \n",
"13 NaN 1 \n",
"14 NaN ) \n",
"\n",
" Unnamed: 4 Unnamed: 5 Unnamed: 6 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 NaN NaN NaN \n",
"4 NaN NaN NaN \n",
"5 NaN NaN NaN \n",
"6 NaN NaN NaN \n",
"7 NaN NaN NaN \n",
"8 NaN NaN NaN \n",
"9 NaN NaN NaN \n",
"10 NaN NaN NaN \n",
"11 NaN NaN NaN \n",
"12 2020-03-19 00:00:00 2020-03-20 00:00:00 2020-03-22 00:00:00 \n",
"13 2 1 2 \n",
"14 ) ) ) \n",
"\n",
" Unnamed: 7 Unnamed: 8 Unnamed: 9 ... \\\n",
"0 NaN NaN NaN ... \n",
"1 NaN NaN NaN ... \n",
"2 NaN NaN NaN ... \n",
"3 NaN NaN NaN ... \n",
"4 NaN NaN NaN ... \n",
"5 NaN NaN NaN ... \n",
"6 NaN NaN NaN ... \n",
"7 NaN NaN NaN ... \n",
"8 NaN NaN NaN ... \n",
"9 NaN NaN NaN ... \n",
"10 NaN NaN NaN ... \n",
"11 NaN NaN NaN ... \n",
"12 2020-03-23 00:00:00 2020-03-24 00:00:00 2020-03-26 00:00:00 ... \n",
"13 1 2 3 ... \n",
"14 ) ) ) ... \n",
"\n",
" Unnamed: 32 Unnamed: 33 Unnamed: 34 Unnamed: 35 Unnamed: 36 Unnamed: 37 \\\n",
"0 NaN NaN NaN NaN NaN NaN \n",
"1 NaN NaN NaN NaN NaN NaN \n",
"2 NaN NaN NaN NaN NaN NaN \n",
"3 NaN NaN NaN NaN NaN NaN \n",
"4 NaN NaN NaN NaN NaN NaN \n",
"5 NaN NaN NaN NaN NaN NaN \n",
"6 NaN NaN NaN NaN NaN NaN \n",
"7 NaN NaN NaN NaN NaN NaN \n",
"8 NaN NaN NaN NaN NaN NaN \n",
"9 NaN NaN NaN NaN NaN NaN \n",
"10 NaN NaN NaN NaN NaN NaN \n",
"11 NaN NaN NaN NaN NaN NaN \n",
"12 NaN NaN NaN NaN NaN NaN \n",
"13 NaN NaN NaN NaN NaN NaN \n",
"14 104.0 105.0 106.0 107.0 108.0 109.0 \n",
"\n",
" Unnamed: 38 Unnamed: 39 Unnamed: 40 Unnamed: 41 \n",
"0 NaN NaN NaN NaN \n",
"1 NaN NaN NaN NaN \n",
"2 NaN NaN NaN NaN \n",
"3 NaN NaN NaN NaN \n",
"4 NaN NaN NaN NaN \n",
"5 NaN NaN NaN NaN \n",
"6 NaN NaN NaN NaN \n",
"7 NaN NaN NaN NaN \n",
"8 NaN NaN NaN NaN \n",
"9 NaN NaN NaN NaN \n",
"10 NaN NaN NaN NaN \n",
"11 NaN NaN NaN NaN \n",
"12 NaN NaN NaN NaN \n",
"13 NaN NaN NaN NaN \n",
"14 110.0 111.0 112.0 113.0 \n",
"\n",
"[15 rows x 42 columns]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sheet = 'COVID19 daily deaths by region'\n",
"sheets[sheet].head(15)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We don't necessarily know how much metadata there is at the start of the sheet so we need to emply heuristics. If *NHS England Region* is used consistently as a column heading, we can use that as a crib:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/site-packages/pandas/core/ops/__init__.py:1115: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison\n",
" result = method(y)\n"
]
},
{
"data": {
"text/plain": [
"(array([12]), array([1]))"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy as np\n",
"\n",
"rows, cols = np.where(sheets[sheet] == 'NHS England Region')\n",
"rows, cols"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>12</th>\n",
" <th>NHS England Region</th>\n",
" <th>2020-03-16 00:00:00</th>\n",
" <th>2020-03-19 00:00:00</th>\n",
" <th>2020-03-20 00:00:00</th>\n",
" <th>2020-03-22 00:00:00</th>\n",
" <th>2020-03-23 00:00:00</th>\n",
" <th>2020-03-24 00:00:00</th>\n",
" <th>2020-03-26 00:00:00</th>\n",
" <th>2020-03-27 00:00:00</th>\n",
" <th>2020-03-28 00:00:00</th>\n",
" <th>...</th>\n",
" <th>2020-04-01 00:00:00</th>\n",
" <th>2020-04-02 00:00:00</th>\n",
" <th>2020-04-03 00:00:00</th>\n",
" <th>2020-04-04 00:00:00</th>\n",
" <th>2020-04-05 00:00:00</th>\n",
" <th>2020-04-06 00:00:00</th>\n",
" <th>2020-04-07 00:00:00</th>\n",
" <th>2020-04-08 00:00:00</th>\n",
" <th>Awaiting verification</th>\n",
" <th>Total</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>East Of England</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>1</td>\n",
" <td>7</td>\n",
" <td>2</td>\n",
" <td>8</td>\n",
" <td>15</td>\n",
" <td>15</td>\n",
" <td>42</td>\n",
" <td>20</td>\n",
" <td>0</td>\n",
" <td>113</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>London</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" <td>6</td>\n",
" <td>5</td>\n",
" <td>...</td>\n",
" <td>7</td>\n",
" <td>10</td>\n",
" <td>15</td>\n",
" <td>16</td>\n",
" <td>12</td>\n",
" <td>25</td>\n",
" <td>61</td>\n",
" <td>29</td>\n",
" <td>0</td>\n",
" <td>213</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>Midlands</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>...</td>\n",
" <td>8</td>\n",
" <td>3</td>\n",
" <td>5</td>\n",
" <td>7</td>\n",
" <td>13</td>\n",
" <td>22</td>\n",
" <td>41</td>\n",
" <td>17</td>\n",
" <td>0</td>\n",
" <td>124</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>North East And Yorkshire</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" <td>6</td>\n",
" <td>48</td>\n",
" <td>32</td>\n",
" <td>0</td>\n",
" <td>92</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>North West</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>2</td>\n",
" <td>4</td>\n",
" <td>6</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>6</td>\n",
" <td>35</td>\n",
" <td>18</td>\n",
" <td>0</td>\n",
" <td>77</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 23 columns</p>\n",
"</div>"
],
"text/plain": [
"12 NHS England Region 2020-03-16 00:00:00 2020-03-19 00:00:00 \\\n",
"15 East Of England 0 0 \n",
"16 London 1 2 \n",
"17 Midlands 0 0 \n",
"18 North East And Yorkshire 0 0 \n",
"19 North West 0 0 \n",
"\n",
"12 2020-03-20 00:00:00 2020-03-22 00:00:00 2020-03-23 00:00:00 \\\n",
"15 0 0 0 \n",
"16 0 2 1 \n",
"17 0 0 0 \n",
"18 0 0 0 \n",
"19 0 0 0 \n",
"\n",
"12 2020-03-24 00:00:00 2020-03-26 00:00:00 2020-03-27 00:00:00 \\\n",
"15 0 0 0 \n",
"16 2 2 6 \n",
"17 0 0 0 \n",
"18 0 0 0 \n",
"19 0 0 0 \n",
"\n",
"12 2020-03-28 00:00:00 ... 2020-04-01 00:00:00 2020-04-02 00:00:00 \\\n",
"15 0 ... 1 7 \n",
"16 5 ... 7 10 \n",
"17 1 ... 8 3 \n",
"18 0 ... 0 2 \n",
"19 0 ... 2 4 \n",
"\n",
"12 2020-04-03 00:00:00 2020-04-04 00:00:00 2020-04-05 00:00:00 \\\n",
"15 2 8 15 \n",
"16 15 16 12 \n",
"17 5 7 13 \n",
"18 0 0 4 \n",
"19 6 4 2 \n",
"\n",
"12 2020-04-06 00:00:00 2020-04-07 00:00:00 2020-04-08 00:00:00 \\\n",
"15 15 42 20 \n",
"16 25 61 29 \n",
"17 22 41 17 \n",
"18 6 48 32 \n",
"19 6 35 18 \n",
"\n",
"12 Awaiting verification Total \n",
"15 0 113 \n",
"16 0 213 \n",
"17 0 124 \n",
"18 0 92 \n",
"19 0 77 \n",
"\n",
"[5 rows x 23 columns]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"colnames = sheets[sheet].iloc[12]\n",
"sheets[sheet] = sheets[sheet].iloc[15:]\n",
"sheets[sheet].columns = colnames\n",
"sheets[sheet].dropna(axis=1, how='all', inplace=True)\n",
"sheets[sheet].dropna(axis=0, how='all', inplace=True)\n",
"#sheets[sheet].dropna(axis=0, subset=[sheets[sheet].columns[0]], inplace=True)\n",
"sheets[sheet].head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The ages data is structured differently, but we can perhaps use *Age Group* as a crib?"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Unnamed: 0</th>\n",
" <th>Unnamed: 1</th>\n",
" <th>Unnamed: 2</th>\n",
" <th>Unnamed: 3</th>\n",
" <th>Unnamed: 4</th>\n",
" <th>Unnamed: 5</th>\n",
" <th>Unnamed: 6</th>\n",
" <th>Unnamed: 7</th>\n",
" <th>Unnamed: 8</th>\n",
" <th>Unnamed: 9</th>\n",
" <th>...</th>\n",
" <th>Unnamed: 17</th>\n",
" <th>Unnamed: 18</th>\n",
" <th>Unnamed: 19</th>\n",
" <th>Unnamed: 20</th>\n",
" <th>Unnamed: 21</th>\n",
" <th>Unnamed: 22</th>\n",
" <th>Unnamed: 23</th>\n",
" <th>Unnamed: 24</th>\n",
" <th>Unnamed: 25</th>\n",
" <th>Unnamed: 26</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>NaN</td>\n",
" <td>Title:</td>\n",
" <td>COVID-19 daily deaths announcement</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>AE_Closures</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>NaN</td>\n",
" <td>Summary:</td>\n",
" <td>Deaths of patients who have died in hospitals ...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>NaN</td>\n",
" <td>Period:</td>\n",
" <td>5pm 07 April 2020 - 5pm 08 April 2020</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>NaN</td>\n",
" <td>Source:</td>\n",
" <td>COVID-19 Patient Notification System</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>NaN</td>\n",
" <td>Basis:</td>\n",
" <td>Provider</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>NaN</td>\n",
" <td>Published:</td>\n",
" <td>2020-04-09 00:00:00</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>NaN</td>\n",
" <td>Revised:</td>\n",
" <td>-</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>NaN</td>\n",
" <td>Status:</td>\n",
" <td>Published</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>NaN</td>\n",
" <td>Contact:</td>\n",
" <td>england.nhsdata@nhs.net</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>NaN</td>\n",
" <td>Breakdown by age group</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>NaN</td>\n",
" <td>Age group</td>\n",
" <td>NaN</td>\n",
" <td>2020-03-16 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>2020-03-23 00:00:00</td>\n",
" <td>2020-03-24 00:00:00</td>\n",
" <td>2020-03-26 00:00:00</td>\n",
" <td>...</td>\n",
" <td>2020-04-03 00:00:00</td>\n",
" <td>2020-04-04 00:00:00</td>\n",
" <td>2020-04-05 00:00:00</td>\n",
" <td>2020-04-06 00:00:00</td>\n",
" <td>2020-04-07 00:00:00</td>\n",
" <td>2020-04-08 00:00:00</td>\n",
" <td>NaN</td>\n",
" <td>Awaiting verification</td>\n",
" <td>NaN</td>\n",
" <td>Total</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>NaN</td>\n",
" <td>Total</td>\n",
" <td>NaN</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" <td>...</td>\n",
" <td>31</td>\n",
" <td>47</td>\n",
" <td>57</td>\n",
" <td>100</td>\n",
" <td>284</td>\n",
" <td>140</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>765</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>...</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>)</td>\n",
" <td>95.0</td>\n",
" <td>NaN</td>\n",
" <td>95.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>NaN</td>\n",
" <td>0-19</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>NaN</td>\n",
" <td>20-39</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>6</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>NaN</td>\n",
" <td>40-59</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>5</td>\n",
" <td>3</td>\n",
" <td>5</td>\n",
" <td>16</td>\n",
" <td>18</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>54</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>NaN</td>\n",
" <td>60-79</td>\n",
" <td>NaN</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>17</td>\n",
" <td>19</td>\n",
" <td>21</td>\n",
" <td>38</td>\n",
" <td>112</td>\n",
" <td>42</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>290</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>NaN</td>\n",
" <td>80+</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>3</td>\n",
" <td>...</td>\n",
" <td>14</td>\n",
" <td>23</td>\n",
" <td>33</td>\n",
" <td>57</td>\n",
" <td>152</td>\n",
" <td>80</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>415</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>NaN</td>\n",
" <td>TBC</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>22 rows × 27 columns</p>\n",
"</div>"
],
"text/plain": [
" Unnamed: 0 Unnamed: 1 \\\n",
"0 NaN Title: \n",
"1 NaN Summary: \n",
"2 NaN NaN \n",
"3 NaN Period: \n",
"4 NaN Source: \n",
"5 NaN Basis: \n",
"6 NaN Published: \n",
"7 NaN Revised: \n",
"8 NaN Status: \n",
"9 NaN Contact: \n",
"10 NaN NaN \n",
"11 NaN Breakdown by age group \n",
"12 NaN NaN \n",
"13 NaN Age group \n",
"14 NaN Total \n",
"15 NaN NaN \n",
"16 NaN 0-19 \n",
"17 NaN 20-39 \n",
"18 NaN 40-59 \n",
"19 NaN 60-79 \n",
"20 NaN 80+ \n",
"21 NaN TBC \n",
"\n",
" Unnamed: 2 Unnamed: 3 \\\n",
"0 COVID-19 daily deaths announcement NaN \n",
"1 Deaths of patients who have died in hospitals ... NaN \n",
"2 NaN NaN \n",
"3 5pm 07 April 2020 - 5pm 08 April 2020 NaN \n",
"4 COVID-19 Patient Notification System NaN \n",
"5 Provider NaN \n",
"6 2020-04-09 00:00:00 NaN \n",
"7 - NaN \n",
"8 Published NaN \n",
"9 england.nhsdata@nhs.net NaN \n",
"10 NaN NaN \n",
"11 NaN NaN \n",
"12 NaN NaN \n",
"13 NaN 2020-03-16 00:00:00 \n",
"14 NaN 1 \n",
"15 NaN ) \n",
"16 NaN 0 \n",
"17 NaN 0 \n",
"18 NaN 0 \n",
"19 NaN 1 \n",
"20 NaN 0 \n",
"21 NaN 0 \n",
"\n",
" Unnamed: 4 Unnamed: 5 Unnamed: 6 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 NaN NaN NaN \n",
"4 NaN NaN NaN \n",
"5 NaN NaN NaN \n",
"6 NaN NaN NaN \n",
"7 NaN NaN NaN \n",
"8 NaN NaN NaN \n",
"9 NaN NaN NaN \n",
"10 NaN NaN NaN \n",
"11 NaN NaN NaN \n",
"12 NaN NaN NaN \n",
"13 2020-03-19 00:00:00 2020-03-20 00:00:00 2020-03-22 00:00:00 \n",
"14 2 1 2 \n",
"15 ) ) ) \n",
"16 0 0 0 \n",
"17 0 0 0 \n",
"18 0 0 0 \n",
"19 2 1 0 \n",
"20 0 0 2 \n",
"21 0 0 0 \n",
"\n",
" Unnamed: 7 Unnamed: 8 Unnamed: 9 ... \\\n",
"0 NaN NaN NaN ... \n",
"1 NaN NaN NaN ... \n",
"2 NaN NaN NaN ... \n",
"3 NaN NaN NaN ... \n",
"4 NaN NaN NaN ... \n",
"5 NaN NaN NaN ... \n",
"6 NaN NaN NaN ... \n",
"7 NaN NaN NaN ... \n",
"8 NaN NaN NaN ... \n",
"9 NaN NaN NaN ... \n",
"10 NaN NaN NaN ... \n",
"11 NaN NaN NaN ... \n",
"12 NaN NaN NaN ... \n",
"13 2020-03-23 00:00:00 2020-03-24 00:00:00 2020-03-26 00:00:00 ... \n",
"14 1 2 3 ... \n",
"15 ) ) ) ... \n",
"16 0 0 0 ... \n",
"17 0 0 0 ... \n",
"18 0 0 0 ... \n",
"19 0 1 0 ... \n",
"20 1 1 3 ... \n",
"21 0 0 0 ... \n",
"\n",
" Unnamed: 17 Unnamed: 18 Unnamed: 19 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 NaN NaN NaN \n",
"4 NaN NaN NaN \n",
"5 NaN NaN NaN \n",
"6 NaN NaN NaN \n",
"7 NaN NaN NaN \n",
"8 NaN NaN NaN \n",
"9 NaN NaN NaN \n",
"10 NaN NaN NaN \n",
"11 NaN NaN NaN \n",
"12 NaN NaN NaN \n",
"13 2020-04-03 00:00:00 2020-04-04 00:00:00 2020-04-05 00:00:00 \n",
"14 31 47 57 \n",
"15 ) ) ) \n",
"16 0 0 0 \n",
"17 0 0 0 \n",
"18 0 5 3 \n",
"19 17 19 21 \n",
"20 14 23 33 \n",
"21 0 0 0 \n",
"\n",
" Unnamed: 20 Unnamed: 21 Unnamed: 22 Unnamed: 23 \\\n",
"0 NaN AE_Closures NaN NaN \n",
"1 NaN NaN NaN NaN \n",
"2 NaN NaN NaN NaN \n",
"3 NaN NaN NaN NaN \n",
"4 NaN NaN NaN NaN \n",
"5 NaN NaN NaN NaN \n",
"6 NaN NaN NaN NaN \n",
"7 NaN NaN NaN NaN \n",
"8 NaN NaN NaN NaN \n",
"9 NaN NaN NaN NaN \n",
"10 NaN NaN NaN NaN \n",
"11 NaN NaN NaN NaN \n",
"12 NaN NaN NaN NaN \n",
"13 2020-04-06 00:00:00 2020-04-07 00:00:00 2020-04-08 00:00:00 NaN \n",
"14 100 284 140 NaN \n",
"15 ) ) ) 95.0 \n",
"16 0 0 0 NaN \n",
"17 0 4 0 NaN \n",
"18 5 16 18 NaN \n",
"19 38 112 42 NaN \n",
"20 57 152 80 NaN \n",
"21 0 0 0 NaN \n",
"\n",
" Unnamed: 24 Unnamed: 25 Unnamed: 26 \n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 NaN NaN NaN \n",
"4 NaN NaN NaN \n",
"5 NaN NaN NaN \n",
"6 NaN NaN NaN \n",
"7 NaN NaN NaN \n",
"8 NaN NaN NaN \n",
"9 NaN NaN NaN \n",
"10 NaN NaN NaN \n",
"11 NaN NaN NaN \n",
"12 NaN NaN NaN \n",
"13 Awaiting verification NaN Total \n",
"14 0 NaN 765 \n",
"15 NaN 95.0 NaN \n",
"16 0 NaN 0 \n",
"17 0 NaN 6 \n",
"18 0 NaN 54 \n",
"19 0 NaN 290 \n",
"20 0 NaN 415 \n",
"21 0 NaN 0 \n",
"\n",
"[22 rows x 27 columns]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sheet = 'COVID19 daily deaths by age'\n",
"sheets[sheet].head(25)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can extract the published date to provide an additional metadata column:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Timestamp('2020-04-09 00:00:00')"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rows, cols = np.where(sheets[sheet] == 'Published:')\n",
"published_date = sheets[sheet].iat[rows[0], cols[0]+1]\n",
"published_date"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Try the crib:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(array([13]), array([1]))"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rows, cols = np.where(sheets[sheet] == 'Age group')\n",
"rows, cols"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Does the same cleaning pattern work?"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>13</th>\n",
" <th>Age group</th>\n",
" <th>2020-03-16 00:00:00</th>\n",
" <th>2020-03-19 00:00:00</th>\n",
" <th>2020-03-20 00:00:00</th>\n",
" <th>2020-03-22 00:00:00</th>\n",
" <th>2020-03-23 00:00:00</th>\n",
" <th>2020-03-24 00:00:00</th>\n",
" <th>2020-03-26 00:00:00</th>\n",
" <th>2020-03-27 00:00:00</th>\n",
" <th>2020-03-28 00:00:00</th>\n",
" <th>...</th>\n",
" <th>2020-04-01 00:00:00</th>\n",
" <th>2020-04-02 00:00:00</th>\n",
" <th>2020-04-03 00:00:00</th>\n",
" <th>2020-04-04 00:00:00</th>\n",
" <th>2020-04-05 00:00:00</th>\n",
" <th>2020-04-06 00:00:00</th>\n",
" <th>2020-04-07 00:00:00</th>\n",
" <th>2020-04-08 00:00:00</th>\n",
" <th>Awaiting verification</th>\n",
" <th>Total</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>0-19</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>20-39</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>2</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>6</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>40-59</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>0</td>\n",
" <td>5</td>\n",
" <td>3</td>\n",
" <td>5</td>\n",
" <td>16</td>\n",
" <td>18</td>\n",
" <td>0</td>\n",
" <td>54</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>60-79</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>3</td>\n",
" <td>...</td>\n",
" <td>7</td>\n",
" <td>15</td>\n",
" <td>17</td>\n",
" <td>19</td>\n",
" <td>21</td>\n",
" <td>38</td>\n",
" <td>112</td>\n",
" <td>42</td>\n",
" <td>0</td>\n",
" <td>290</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>80+</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>3</td>\n",
" <td>...</td>\n",
" <td>10</td>\n",
" <td>13</td>\n",
" <td>14</td>\n",
" <td>23</td>\n",
" <td>33</td>\n",
" <td>57</td>\n",
" <td>152</td>\n",
" <td>80</td>\n",
" <td>0</td>\n",
" <td>415</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 23 columns</p>\n",
"</div>"
],
"text/plain": [
"13 Age group 2020-03-16 00:00:00 2020-03-19 00:00:00 2020-03-20 00:00:00 \\\n",
"16 0-19 0 0 0 \n",
"17 20-39 0 0 0 \n",
"18 40-59 0 0 0 \n",
"19 60-79 1 2 1 \n",
"20 80+ 0 0 0 \n",
"\n",
"13 2020-03-22 00:00:00 2020-03-23 00:00:00 2020-03-24 00:00:00 \\\n",
"16 0 0 0 \n",
"17 0 0 0 \n",
"18 0 0 0 \n",
"19 0 0 1 \n",
"20 2 1 1 \n",
"\n",
"13 2020-03-26 00:00:00 2020-03-27 00:00:00 2020-03-28 00:00:00 ... \\\n",
"16 0 0 0 ... \n",
"17 0 0 0 ... \n",
"18 0 1 1 ... \n",
"19 0 1 3 ... \n",
"20 3 4 3 ... \n",
"\n",
"13 2020-04-01 00:00:00 2020-04-02 00:00:00 2020-04-03 00:00:00 \\\n",
"16 0 0 0 \n",
"17 2 0 0 \n",
"18 0 2 0 \n",
"19 7 15 17 \n",
"20 10 13 14 \n",
"\n",
"13 2020-04-04 00:00:00 2020-04-05 00:00:00 2020-04-06 00:00:00 \\\n",
"16 0 0 0 \n",
"17 0 0 0 \n",
"18 5 3 5 \n",
"19 19 21 38 \n",
"20 23 33 57 \n",
"\n",
"13 2020-04-07 00:00:00 2020-04-08 00:00:00 Awaiting verification Total \n",
"16 0 0 0 0 \n",
"17 4 0 0 6 \n",
"18 16 18 0 54 \n",
"19 112 42 0 290 \n",
"20 152 80 0 415 \n",
"\n",
"[5 rows x 23 columns]"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"colnames = sheets[sheet].iloc[13]\n",
"sheets[sheet] = sheets[sheet].iloc[16:]\n",
"sheets[sheet].columns = colnames\n",
"sheets[sheet].dropna(axis=0, how='all', inplace=True)\n",
"sheets[sheet].dropna(axis=1, how='all', inplace=True)\n",
"#sheets[sheet].dropna(axis=0, subset=[sheets[sheet].columns[0]], inplace=True)\n",
"sheets[sheet].head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Again, *NHS England Region* may be a handy crib in the following sheet:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Unnamed: 0</th>\n",
" <th>Unnamed: 1</th>\n",
" <th>Unnamed: 2</th>\n",
" <th>Unnamed: 3</th>\n",
" <th>Unnamed: 4</th>\n",
" <th>Unnamed: 5</th>\n",
" <th>Unnamed: 6</th>\n",
" <th>Unnamed: 7</th>\n",
" <th>Unnamed: 8</th>\n",
" <th>Unnamed: 9</th>\n",
" <th>...</th>\n",
" <th>Unnamed: 19</th>\n",
" <th>Unnamed: 20</th>\n",
" <th>Unnamed: 21</th>\n",
" <th>Unnamed: 22</th>\n",
" <th>Unnamed: 23</th>\n",
" <th>Unnamed: 24</th>\n",
" <th>Unnamed: 25</th>\n",
" <th>Unnamed: 26</th>\n",
" <th>Unnamed: 27</th>\n",
" <th>Unnamed: 28</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>NaN</td>\n",
" <td>Title:</td>\n",
" <td>COVID-19 daily deaths announcement</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>AE_Closures</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>NaN</td>\n",
" <td>Summary:</td>\n",
" <td>Deaths of patients who have died in hospitals ...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>Period:</td>\n",
" <td>5pm 07 April 2020 - 5pm 08 April 2020</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>NaN</td>\n",
" <td>Source:</td>\n",
" <td>COVID-19 Patient Notification System</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>NaN</td>\n",
" <td>Basis:</td>\n",
" <td>Provider</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>NaN</td>\n",
" <td>Published:</td>\n",
" <td>2020-04-09 00:00:00</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>NaN</td>\n",
" <td>Revised:</td>\n",
" <td>-</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>NaN</td>\n",
" <td>Status:</td>\n",
" <td>Published</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>NaN</td>\n",
" <td>Contact:</td>\n",
" <td>england.nhsdata@nhs.net</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>NaN</td>\n",
" <td>Trust level data</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>NaN</td>\n",
" <td>NHS England Region</td>\n",
" <td>NaN</td>\n",
" <td>Code</td>\n",
" <td>Name</td>\n",
" <td>2020-03-16 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>2020-03-23 00:00:00</td>\n",
" <td>...</td>\n",
" <td>2020-04-03 00:00:00</td>\n",
" <td>2020-04-04 00:00:00</td>\n",
" <td>2020-04-05 00:00:00</td>\n",
" <td>2020-04-06 00:00:00</td>\n",
" <td>2020-04-07 00:00:00</td>\n",
" <td>2020-04-08 00:00:00</td>\n",
" <td>NaN</td>\n",
" <td>Awaiting verification</td>\n",
" <td>NaN</td>\n",
" <td>Total</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>NaN</td>\n",
" <td>-</td>\n",
" <td>NaN</td>\n",
" <td>-</td>\n",
" <td>ENGLAND</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>...</td>\n",
" <td>31</td>\n",
" <td>47</td>\n",
" <td>57</td>\n",
" <td>100</td>\n",
" <td>284</td>\n",
" <td>140</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>765</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>15 rows × 29 columns</p>\n",
"</div>"
],
"text/plain": [
" Unnamed: 0 Unnamed: 1 \\\n",
"0 NaN Title: \n",
"1 NaN Summary: \n",
"2 NaN Period: \n",
"3 NaN Source: \n",
"4 NaN Basis: \n",
"5 NaN Published: \n",
"6 NaN Revised: \n",
"7 NaN Status: \n",
"8 NaN Contact: \n",
"9 NaN NaN \n",
"10 NaN Trust level data \n",
"11 NaN NaN \n",
"12 NaN NHS England Region \n",
"13 NaN - \n",
"14 NaN NaN \n",
"\n",
" Unnamed: 2 Unnamed: 3 Unnamed: 4 \\\n",
"0 COVID-19 daily deaths announcement NaN NaN \n",
"1 Deaths of patients who have died in hospitals ... NaN NaN \n",
"2 5pm 07 April 2020 - 5pm 08 April 2020 NaN NaN \n",
"3 COVID-19 Patient Notification System NaN NaN \n",
"4 Provider NaN NaN \n",
"5 2020-04-09 00:00:00 NaN NaN \n",
"6 - NaN NaN \n",
"7 Published NaN NaN \n",
"8 england.nhsdata@nhs.net NaN NaN \n",
"9 NaN NaN NaN \n",
"10 NaN NaN NaN \n",
"11 NaN NaN NaN \n",
"12 NaN Code Name \n",
"13 NaN - ENGLAND \n",
"14 NaN NaN NaN \n",
"\n",
" Unnamed: 5 Unnamed: 6 Unnamed: 7 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 NaN NaN NaN \n",
"4 NaN NaN NaN \n",
"5 NaN NaN NaN \n",
"6 NaN NaN NaN \n",
"7 NaN NaN NaN \n",
"8 NaN NaN NaN \n",
"9 NaN NaN NaN \n",
"10 NaN NaN NaN \n",
"11 NaN NaN NaN \n",
"12 2020-03-16 00:00:00 2020-03-19 00:00:00 2020-03-20 00:00:00 \n",
"13 1 2 1 \n",
"14 NaN NaN NaN \n",
"\n",
" Unnamed: 8 Unnamed: 9 ... Unnamed: 19 \\\n",
"0 NaN NaN ... NaN \n",
"1 NaN NaN ... NaN \n",
"2 NaN NaN ... NaN \n",
"3 NaN NaN ... NaN \n",
"4 NaN NaN ... NaN \n",
"5 NaN NaN ... NaN \n",
"6 NaN NaN ... NaN \n",
"7 NaN NaN ... NaN \n",
"8 NaN NaN ... NaN \n",
"9 NaN NaN ... NaN \n",
"10 NaN NaN ... NaN \n",
"11 NaN NaN ... NaN \n",
"12 2020-03-22 00:00:00 2020-03-23 00:00:00 ... 2020-04-03 00:00:00 \n",
"13 2 1 ... 31 \n",
"14 NaN NaN ... NaN \n",
"\n",
" Unnamed: 20 Unnamed: 21 Unnamed: 22 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 NaN NaN NaN \n",
"4 NaN NaN NaN \n",
"5 NaN NaN NaN \n",
"6 NaN NaN NaN \n",
"7 NaN NaN NaN \n",
"8 NaN NaN NaN \n",
"9 NaN NaN NaN \n",
"10 NaN NaN NaN \n",
"11 NaN NaN NaN \n",
"12 2020-04-04 00:00:00 2020-04-05 00:00:00 2020-04-06 00:00:00 \n",
"13 47 57 100 \n",
"14 NaN NaN NaN \n",
"\n",
" Unnamed: 23 Unnamed: 24 Unnamed: 25 \\\n",
"0 AE_Closures NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 NaN NaN NaN \n",
"4 NaN NaN NaN \n",
"5 NaN NaN NaN \n",
"6 NaN NaN NaN \n",
"7 NaN NaN NaN \n",
"8 NaN NaN NaN \n",
"9 NaN NaN NaN \n",
"10 NaN NaN NaN \n",
"11 NaN NaN NaN \n",
"12 2020-04-07 00:00:00 2020-04-08 00:00:00 NaN \n",
"13 284 140 NaN \n",
"14 NaN NaN NaN \n",
"\n",
" Unnamed: 26 Unnamed: 27 Unnamed: 28 \n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 NaN NaN NaN \n",
"4 NaN NaN NaN \n",
"5 NaN NaN NaN \n",
"6 NaN NaN NaN \n",
"7 NaN NaN NaN \n",
"8 NaN NaN NaN \n",
"9 NaN NaN NaN \n",
"10 NaN NaN NaN \n",
"11 NaN NaN NaN \n",
"12 Awaiting verification NaN Total \n",
"13 0 NaN 765 \n",
"14 NaN NaN NaN \n",
"\n",
"[15 rows x 29 columns]"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sheet = 'COVID19 daily deaths by trust'\n",
"sheets[sheet].head(15)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The same cleaning pattern we used before seems to work fine:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>12</th>\n",
" <th>NHS England Region</th>\n",
" <th>Code</th>\n",
" <th>Name</th>\n",
" <th>2020-03-16 00:00:00</th>\n",
" <th>2020-03-19 00:00:00</th>\n",
" <th>2020-03-20 00:00:00</th>\n",
" <th>2020-03-22 00:00:00</th>\n",
" <th>2020-03-23 00:00:00</th>\n",
" <th>2020-03-24 00:00:00</th>\n",
" <th>2020-03-26 00:00:00</th>\n",
" <th>...</th>\n",
" <th>2020-04-01 00:00:00</th>\n",
" <th>2020-04-02 00:00:00</th>\n",
" <th>2020-04-03 00:00:00</th>\n",
" <th>2020-04-04 00:00:00</th>\n",
" <th>2020-04-05 00:00:00</th>\n",
" <th>2020-04-06 00:00:00</th>\n",
" <th>2020-04-07 00:00:00</th>\n",
" <th>2020-04-08 00:00:00</th>\n",
" <th>Awaiting verification</th>\n",
" <th>Total</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>East Of England</td>\n",
" <td>NQ1</td>\n",
" <td>ANGLIAN COMMUNITY ENTERPRISE CIC - CLACTON HOS...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>East Of England</td>\n",
" <td>NQ1</td>\n",
" <td>ANGLIAN COMMUNITY ENTERPRISE CIC - FRYATT HOSP...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>East Of England</td>\n",
" <td>RDD</td>\n",
" <td>BASILDON AND THURROCK UNIVERSITY HOSPITALS NHS...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" <td>7</td>\n",
" <td>5</td>\n",
" <td>3</td>\n",
" <td>0</td>\n",
" <td>24</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>East Of England</td>\n",
" <td>RC1</td>\n",
" <td>BEDFORD HOSPITAL NHS TRUST</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>East Of England</td>\n",
" <td>RGT</td>\n",
" <td>CAMBRIDGE UNIVERSITY HOSPITALS NHS FOUNDATION ...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 25 columns</p>\n",
"</div>"
],
"text/plain": [
"12 NHS England Region Code Name \\\n",
"15 East Of England NQ1 ANGLIAN COMMUNITY ENTERPRISE CIC - CLACTON HOS... \n",
"16 East Of England NQ1 ANGLIAN COMMUNITY ENTERPRISE CIC - FRYATT HOSP... \n",
"17 East Of England RDD BASILDON AND THURROCK UNIVERSITY HOSPITALS NHS... \n",
"18 East Of England RC1 BEDFORD HOSPITAL NHS TRUST \n",
"19 East Of England RGT CAMBRIDGE UNIVERSITY HOSPITALS NHS FOUNDATION ... \n",
"\n",
"12 2020-03-16 00:00:00 2020-03-19 00:00:00 2020-03-20 00:00:00 \\\n",
"15 0 0 0 \n",
"16 0 0 0 \n",
"17 0 0 0 \n",
"18 0 0 0 \n",
"19 0 0 0 \n",
"\n",
"12 2020-03-22 00:00:00 2020-03-23 00:00:00 2020-03-24 00:00:00 \\\n",
"15 0 0 0 \n",
"16 0 0 0 \n",
"17 0 0 0 \n",
"18 0 0 0 \n",
"19 0 0 0 \n",
"\n",
"12 2020-03-26 00:00:00 ... 2020-04-01 00:00:00 2020-04-02 00:00:00 \\\n",
"15 0 ... 0 0 \n",
"16 0 ... 0 0 \n",
"17 0 ... 0 2 \n",
"18 0 ... 0 0 \n",
"19 0 ... 0 0 \n",
"\n",
"12 2020-04-03 00:00:00 2020-04-04 00:00:00 2020-04-05 00:00:00 \\\n",
"15 0 0 0 \n",
"16 0 0 0 \n",
"17 0 2 3 \n",
"18 0 0 0 \n",
"19 0 0 0 \n",
"\n",
"12 2020-04-06 00:00:00 2020-04-07 00:00:00 2020-04-08 00:00:00 \\\n",
"15 0 1 0 \n",
"16 0 1 0 \n",
"17 7 5 3 \n",
"18 1 1 0 \n",
"19 0 1 0 \n",
"\n",
"12 Awaiting verification Total \n",
"15 0 1 \n",
"16 0 1 \n",
"17 0 24 \n",
"18 0 2 \n",
"19 0 1 \n",
"\n",
"[5 rows x 25 columns]"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"colnames = sheets[sheet].iloc[12]\n",
"sheets[sheet] = sheets[sheet].iloc[15:]\n",
"sheets[sheet].columns = colnames\n",
"sheets[sheet].dropna(axis=1, how='all', inplace=True)\n",
"sheets[sheet].dropna(axis=0, how='all', inplace=True)\n",
"#sheets[sheet].dropna(axis=0, subset=[sheets[sheet].columns[0]], inplace=True)\n",
"sheets[sheet].head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following tries to clean things automatically - we drop the national aggregate values:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"def cleaner(sheets):\n",
" for sheet in sheets:\n",
" if 'chart' in sheet:\n",
" continue\n",
" rows, cols = np.where(sheets[sheet] == 'Published:')\n",
" published_date = sheets[sheet].iat[rows[0], cols[0]+1]\n",
"\n",
" if 'age' in sheet:\n",
" rows, cols = np.where(sheets[sheet] == 'Age group')\n",
" #print((rows, cols))\n",
" _ix= rows[0]\n",
" elif 'ethnicity' in sheet:\n",
" rows, cols = np.where(sheets[sheet] == 'Ethnic group')\n",
" #print((rows, cols))\n",
" _ix= rows[0]\n",
" else:\n",
" rows, cols = np.where(sheets[sheet] == 'NHS England Region')\n",
" #print((sheet, rows, cols))\n",
" _ix= rows[0] #ix[sheet][0]\n",
"\n",
" colnames = sheets[sheet].iloc[_ix]\n",
" sheets[sheet] = sheets[sheet].iloc[_ix+3:]\n",
" sheets[sheet].columns = colnames\n",
" sheets[sheet].dropna(axis=1, how='all', inplace=True)\n",
" sheets[sheet].dropna(axis=0, how='all', inplace=True)\n",
" sheets[sheet] = sheets[sheet].loc[:, sheets[sheet].columns.notnull()]\n",
" \n",
" sheets[sheet]['Published'] = published_date\n",
" #sheets[sheet].dropna(axis=0, subset=[sheets[sheet].columns[0]], inplace=True)\n",
" return sheets"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Grab all the daily reports:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"COVID 19 daily announced deaths 28 April 2020\n",
"COVID 19 daily announced deaths 27 April 2020\n",
"COVID 19 daily announced deaths 26 April 2020\n",
"COVID 19 daily announced deaths 25 April 2020\n",
"COVID 19 daily announced deaths 24 April 2020\n",
"COVID 19 daily announced deaths 23 April 2020\n",
"COVID 19 daily announced deaths 22 April 2020\n",
"COVID 19 daily announced deaths 21 April 2020\n",
"COVID 19 daily announced deaths 20 April 2020\n",
"COVID 19 daily announced deaths 19 April 2020\n",
"COVID 19 daily announced deaths 18 April 2020\n",
"COVID 19 daily announced deaths 17 April 2020\n",
"COVID 19 daily announced deaths 16 April 2020\n",
"COVID 19 daily announced deaths 15 April 2020\n",
"COVID 19 daily announced deaths 14 April 2020\n",
"COVID 19 daily announced deaths 13 April 2020\n",
"COVID 19 daily announced deaths 12 April 2020\n",
"COVID 19 daily announced deaths 11 April 2020\n",
"COVID 19 daily announced deaths 10 April 2020\n",
"COVID 19 daily announced deaths 9 April 2020\n",
"COVID 19 daily announced deaths 8 April 2020\n",
"COVID 19 daily announced deaths 7 April 2020\n",
"COVID 19 daily announced deaths 6 April 2020\n",
"COVID 19 daily announced deaths 5 April 2020\n",
"COVID 19 daily announced deaths 4 April 2020\n",
"COVID 19 daily announced deaths 3 April 2020\n",
"COVID 19 daily announced deaths 2 April 2020\n"
]
}
],
"source": [
"data = {}\n",
"\n",
"for link in links:\n",
" print(link)\n",
" sheets = pd.read_excel(links[link], sheet_name=None)\n",
" sheets = cleaner(sheets)\n",
" data[link] = sheets\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"dict_keys(['COVID 19 daily announced deaths 28 April 2020', 'COVID 19 daily announced deaths 27 April 2020', 'COVID 19 daily announced deaths 26 April 2020', 'COVID 19 daily announced deaths 25 April 2020', 'COVID 19 daily announced deaths 24 April 2020', 'COVID 19 daily announced deaths 23 April 2020', 'COVID 19 daily announced deaths 22 April 2020', 'COVID 19 daily announced deaths 21 April 2020', 'COVID 19 daily announced deaths 20 April 2020', 'COVID 19 daily announced deaths 19 April 2020', 'COVID 19 daily announced deaths 18 April 2020', 'COVID 19 daily announced deaths 17 April 2020', 'COVID 19 daily announced deaths 16 April 2020', 'COVID 19 daily announced deaths 15 April 2020', 'COVID 19 daily announced deaths 14 April 2020', 'COVID 19 daily announced deaths 13 April 2020', 'COVID 19 daily announced deaths 12 April 2020', 'COVID 19 daily announced deaths 11 April 2020', 'COVID 19 daily announced deaths 10 April 2020', 'COVID 19 daily announced deaths 9 April 2020', 'COVID 19 daily announced deaths 8 April 2020', 'COVID 19 daily announced deaths 7 April 2020', 'COVID 19 daily announced deaths 6 April 2020', 'COVID 19 daily announced deaths 5 April 2020', 'COVID 19 daily announced deaths 4 April 2020', 'COVID 19 daily announced deaths 3 April 2020', 'COVID 19 daily announced deaths 2 April 2020'])"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data.keys()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Just as an aside, we can informally extract the publication date of a spreadheet from the associated link text on the original web page (trusting that the link does refer to the correctly linked document):"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"datetime.datetime(2020, 4, 15, 0, 0)"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from parse import parse\n",
"import dateparser\n",
"\n",
"def getLinkDate(link):\n",
" \"\"\"Get date from link text.\"\"\"\n",
" _date = parse('COVID 19 daily announced deaths {date}', link)['date']\n",
" return dateparser.parse(_date)\n",
"\n",
"#Test the date extractor\n",
"getLinkDate('COVID 19 daily announced deaths 15 April 2020')"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"odict_keys(['COVID19 daily deaths by region', 'COVID19 daily deaths by age', 'COVID19 daily deaths by trust'])"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data['COVID 19 daily announced deaths 9 April 2020'].keys()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Preview what sort of data we've got:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>12</th>\n",
" <th>NHS England Region</th>\n",
" <th>Code</th>\n",
" <th>Name</th>\n",
" <th>2020-03-16 00:00:00</th>\n",
" <th>2020-03-19 00:00:00</th>\n",
" <th>2020-03-20 00:00:00</th>\n",
" <th>2020-03-22 00:00:00</th>\n",
" <th>2020-03-23 00:00:00</th>\n",
" <th>2020-03-24 00:00:00</th>\n",
" <th>2020-03-26 00:00:00</th>\n",
" <th>...</th>\n",
" <th>2020-04-02 00:00:00</th>\n",
" <th>2020-04-03 00:00:00</th>\n",
" <th>2020-04-04 00:00:00</th>\n",
" <th>2020-04-05 00:00:00</th>\n",
" <th>2020-04-06 00:00:00</th>\n",
" <th>2020-04-07 00:00:00</th>\n",
" <th>2020-04-08 00:00:00</th>\n",
" <th>Awaiting verification</th>\n",
" <th>Total</th>\n",
" <th>Published</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>East Of England</td>\n",
" <td>NQ1</td>\n",
" <td>ANGLIAN COMMUNITY ENTERPRISE CIC - CLACTON HOS...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2020-04-09</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>East Of England</td>\n",
" <td>NQ1</td>\n",
" <td>ANGLIAN COMMUNITY ENTERPRISE CIC - FRYATT HOSP...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2020-04-09</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>East Of England</td>\n",
" <td>RDD</td>\n",
" <td>BASILDON AND THURROCK UNIVERSITY HOSPITALS NHS...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>2</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" <td>7</td>\n",
" <td>5</td>\n",
" <td>3</td>\n",
" <td>0</td>\n",
" <td>24</td>\n",
" <td>2020-04-09</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>East Of England</td>\n",
" <td>RC1</td>\n",
" <td>BEDFORD HOSPITAL NHS TRUST</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>2020-04-09</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>East Of England</td>\n",
" <td>RGT</td>\n",
" <td>CAMBRIDGE UNIVERSITY HOSPITALS NHS FOUNDATION ...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2020-04-09</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>181</th>\n",
" <td>South West</td>\n",
" <td>RA9</td>\n",
" <td>TORBAY AND SOUTH DEVON NHS FOUNDATION TRUST</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2020-04-09</td>\n",
" </tr>\n",
" <tr>\n",
" <th>182</th>\n",
" <td>South West</td>\n",
" <td>RA7</td>\n",
" <td>UNIVERSITY HOSPITALS BRISTOL NHS FOUNDATION TRUST</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" <td>2020-04-09</td>\n",
" </tr>\n",
" <tr>\n",
" <th>183</th>\n",
" <td>South West</td>\n",
" <td>RK9</td>\n",
" <td>UNIVERSITY HOSPITALS PLYMOUTH NHS TRUST</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2020-04-09</td>\n",
" </tr>\n",
" <tr>\n",
" <th>184</th>\n",
" <td>South West</td>\n",
" <td>RA3</td>\n",
" <td>WESTON AREA HEALTH NHS TRUST</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2020-04-09</td>\n",
" </tr>\n",
" <tr>\n",
" <th>185</th>\n",
" <td>South West</td>\n",
" <td>RA4</td>\n",
" <td>YEOVIL DISTRICT HOSPITAL NHS FOUNDATION TRUST</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>2020-04-09</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>171 rows × 26 columns</p>\n",
"</div>"
],
"text/plain": [
"12 NHS England Region Code \\\n",
"15 East Of England NQ1 \n",
"16 East Of England NQ1 \n",
"17 East Of England RDD \n",
"18 East Of England RC1 \n",
"19 East Of England RGT \n",
".. ... ... \n",
"181 South West RA9 \n",
"182 South West RA7 \n",
"183 South West RK9 \n",
"184 South West RA3 \n",
"185 South West RA4 \n",
"\n",
"12 Name 2020-03-16 00:00:00 \\\n",
"15 ANGLIAN COMMUNITY ENTERPRISE CIC - CLACTON HOS... 0 \n",
"16 ANGLIAN COMMUNITY ENTERPRISE CIC - FRYATT HOSP... 0 \n",
"17 BASILDON AND THURROCK UNIVERSITY HOSPITALS NHS... 0 \n",
"18 BEDFORD HOSPITAL NHS TRUST 0 \n",
"19 CAMBRIDGE UNIVERSITY HOSPITALS NHS FOUNDATION ... 0 \n",
".. ... ... \n",
"181 TORBAY AND SOUTH DEVON NHS FOUNDATION TRUST 0 \n",
"182 UNIVERSITY HOSPITALS BRISTOL NHS FOUNDATION TRUST 0 \n",
"183 UNIVERSITY HOSPITALS PLYMOUTH NHS TRUST 0 \n",
"184 WESTON AREA HEALTH NHS TRUST 0 \n",
"185 YEOVIL DISTRICT HOSPITAL NHS FOUNDATION TRUST 0 \n",
"\n",
"12 2020-03-19 00:00:00 2020-03-20 00:00:00 2020-03-22 00:00:00 \\\n",
"15 0 0 0 \n",
"16 0 0 0 \n",
"17 0 0 0 \n",
"18 0 0 0 \n",
"19 0 0 0 \n",
".. ... ... ... \n",
"181 0 0 0 \n",
"182 0 0 0 \n",
"183 0 0 0 \n",
"184 0 0 0 \n",
"185 0 0 0 \n",
"\n",
"12 2020-03-23 00:00:00 2020-03-24 00:00:00 2020-03-26 00:00:00 ... \\\n",
"15 0 0 0 ... \n",
"16 0 0 0 ... \n",
"17 0 0 0 ... \n",
"18 0 0 0 ... \n",
"19 0 0 0 ... \n",
".. ... ... ... ... \n",
"181 0 0 0 ... \n",
"182 0 0 0 ... \n",
"183 0 0 0 ... \n",
"184 0 0 0 ... \n",
"185 0 0 0 ... \n",
"\n",
"12 2020-04-02 00:00:00 2020-04-03 00:00:00 2020-04-04 00:00:00 \\\n",
"15 0 0 0 \n",
"16 0 0 0 \n",
"17 2 0 2 \n",
"18 0 0 0 \n",
"19 0 0 0 \n",
".. ... ... ... \n",
"181 0 0 0 \n",
"182 0 0 0 \n",
"183 0 0 0 \n",
"184 0 0 0 \n",
"185 0 0 0 \n",
"\n",
"12 2020-04-05 00:00:00 2020-04-06 00:00:00 2020-04-07 00:00:00 \\\n",
"15 0 0 1 \n",
"16 0 0 1 \n",
"17 3 7 5 \n",
"18 0 1 1 \n",
"19 0 0 1 \n",
".. ... ... ... \n",
"181 0 0 0 \n",
"182 0 1 2 \n",
"183 0 0 1 \n",
"184 0 0 0 \n",
"185 0 1 1 \n",
"\n",
"12 2020-04-08 00:00:00 Awaiting verification Total Published \n",
"15 0 0 1 2020-04-09 \n",
"16 0 0 1 2020-04-09 \n",
"17 3 0 24 2020-04-09 \n",
"18 0 0 2 2020-04-09 \n",
"19 0 0 1 2020-04-09 \n",
".. ... ... ... ... \n",
"181 0 0 0 2020-04-09 \n",
"182 1 0 4 2020-04-09 \n",
"183 0 0 1 2020-04-09 \n",
"184 0 0 0 2020-04-09 \n",
"185 0 0 2 2020-04-09 \n",
"\n",
"[171 rows x 26 columns]"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = data['COVID 19 daily announced deaths 9 April 2020']['COVID19 daily deaths by trust']\n",
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Preview a specific area, albeit with quite an informal search term:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>12</th>\n",
" <th>NHS England Region</th>\n",
" <th>Code</th>\n",
" <th>Name</th>\n",
" <th>2020-03-03 00:00:00</th>\n",
" <th>2020-03-16 00:00:00</th>\n",
" <th>2020-03-18 00:00:00</th>\n",
" <th>2020-03-20 00:00:00</th>\n",
" <th>2020-03-21 00:00:00</th>\n",
" <th>2020-03-22 00:00:00</th>\n",
" <th>2020-03-23 00:00:00</th>\n",
" <th>...</th>\n",
" <th>2020-03-28 00:00:00</th>\n",
" <th>2020-03-29 00:00:00</th>\n",
" <th>2020-03-30 00:00:00</th>\n",
" <th>2020-03-31 00:00:00</th>\n",
" <th>2020-04-01 00:00:00</th>\n",
" <th>2020-04-02 00:00:00</th>\n",
" <th>2020-04-03 00:00:00</th>\n",
" <th>Awaiting verification</th>\n",
" <th>Total</th>\n",
" <th>Published</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>132</th>\n",
" <td>South East</td>\n",
" <td>R1F</td>\n",
" <td>ISLE OF WIGHT NHS TRUST</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2020-04-04</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>1 rows × 24 columns</p>\n",
"</div>"
],
"text/plain": [
"12 NHS England Region Code Name 2020-03-03 00:00:00 \\\n",
"132 South East R1F ISLE OF WIGHT NHS TRUST 0 \n",
"\n",
"12 2020-03-16 00:00:00 2020-03-18 00:00:00 2020-03-20 00:00:00 \\\n",
"132 0 0 0 \n",
"\n",
"12 2020-03-21 00:00:00 2020-03-22 00:00:00 2020-03-23 00:00:00 ... \\\n",
"132 0 0 0 ... \n",
"\n",
"12 2020-03-28 00:00:00 2020-03-29 00:00:00 2020-03-30 00:00:00 \\\n",
"132 0 0 0 \n",
"\n",
"12 2020-03-31 00:00:00 2020-04-01 00:00:00 2020-04-02 00:00:00 \\\n",
"132 0 0 0 \n",
"\n",
"12 2020-04-03 00:00:00 Awaiting verification Total Published \n",
"132 1 0 1 2020-04-04 \n",
"\n",
"[1 rows x 24 columns]"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = data['COVID 19 daily announced deaths 4 April 2020']['COVID19 daily deaths by trust']\n",
"df[df['Name'].str.contains('WIGHT')]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Grab the totals:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"odict_keys(['COVID19 total deaths chart', 'COVID19 total deaths by trust', 'COVID19 total deaths by region', 'Deaths by region- negative test', 'COVID19 total deaths by age', 'COVID19 total by ethnicity'])"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"totals_xl = pd.read_excel(totals_link, sheet_name=None)\n",
"totals_xl.keys()"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"odict_keys(['COVID19 total deaths chart', 'COVID19 total deaths by trust', 'COVID19 total deaths by region', 'Deaths by region- negative test', 'COVID19 total deaths by age', 'COVID19 total by ethnicity'])"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"totals_xl = cleaner(totals_xl)\n",
"totals_xl.keys()"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>14</th>\n",
" <th>NHS England Region</th>\n",
" <th>Code</th>\n",
" <th>Name</th>\n",
" <th>Up to 01-Mar-20</th>\n",
" <th>2020-03-01 00:00:00</th>\n",
" <th>2020-03-02 00:00:00</th>\n",
" <th>2020-03-03 00:00:00</th>\n",
" <th>2020-03-04 00:00:00</th>\n",
" <th>2020-03-05 00:00:00</th>\n",
" <th>2020-03-06 00:00:00</th>\n",
" <th>...</th>\n",
" <th>2020-04-21 00:00:00</th>\n",
" <th>2020-04-22 00:00:00</th>\n",
" <th>2020-04-23 00:00:00</th>\n",
" <th>2020-04-24 00:00:00</th>\n",
" <th>2020-04-25 00:00:00</th>\n",
" <th>2020-04-26 00:00:00</th>\n",
" <th>2020-04-27 00:00:00</th>\n",
" <th>Awaiting verification</th>\n",
" <th>Total</th>\n",
" <th>Published</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>179</th>\n",
" <td>South East</td>\n",
" <td>R1F</td>\n",
" <td>ISLE OF WIGHT NHS TRUST</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>22</td>\n",
" <td>2020-04-28</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>1 rows × 65 columns</p>\n",
"</div>"
],
"text/plain": [
"14 NHS England Region Code Name Up to 01-Mar-20 \\\n",
"179 South East R1F ISLE OF WIGHT NHS TRUST 0 \n",
"\n",
"14 2020-03-01 00:00:00 2020-03-02 00:00:00 2020-03-03 00:00:00 \\\n",
"179 0 0 0 \n",
"\n",
"14 2020-03-04 00:00:00 2020-03-05 00:00:00 2020-03-06 00:00:00 ... \\\n",
"179 0 0 0 ... \n",
"\n",
"14 2020-04-21 00:00:00 2020-04-22 00:00:00 2020-04-23 00:00:00 \\\n",
"179 0 1 1 \n",
"\n",
"14 2020-04-24 00:00:00 2020-04-25 00:00:00 2020-04-26 00:00:00 \\\n",
"179 2 0 0 \n",
"\n",
"14 2020-04-27 00:00:00 Awaiting verification Total Published \n",
"179 0 0 22 2020-04-28 \n",
"\n",
"[1 rows x 65 columns]"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dfs = totals_xl['COVID19 total deaths by trust']\n",
"dfs[dfs['Name'].str.contains('WIGHT')]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Adding NHS Daily Data to a Database\n",
"\n",
"The data is perhaps most easily managed in a long form. We could normalise the data properly across several tables, or for mow we can just grab perhaps slightly denormalised tables for the dates and separate tables for totals and result awaiting verification:"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>NHS England Region</th>\n",
" <th>Code</th>\n",
" <th>Name</th>\n",
" <th>Published</th>\n",
" <th>Date</th>\n",
" <th>value</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>East Of England</td>\n",
" <td>RDD</td>\n",
" <td>BASILDON AND THURROCK UNIVERSITY HOSPITALS NHS...</td>\n",
" <td>2020-04-04</td>\n",
" <td>2020-03-03</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>East Of England</td>\n",
" <td>RC1</td>\n",
" <td>BEDFORD HOSPITAL NHS TRUST</td>\n",
" <td>2020-04-04</td>\n",
" <td>2020-03-03</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>East Of England</td>\n",
" <td>RGT</td>\n",
" <td>CAMBRIDGE UNIVERSITY HOSPITALS NHS FOUNDATION ...</td>\n",
" <td>2020-04-04</td>\n",
" <td>2020-03-03</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>East Of England</td>\n",
" <td>RWH</td>\n",
" <td>EAST AND NORTH HERTFORDSHIRE NHS TRUST</td>\n",
" <td>2020-04-04</td>\n",
" <td>2020-03-03</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>East Of England</td>\n",
" <td>RDE</td>\n",
" <td>EAST SUFFOLK AND NORTH ESSEX NHS FOUNDATION TRUST</td>\n",
" <td>2020-04-04</td>\n",
" <td>2020-03-03</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" NHS England Region Code Name \\\n",
"0 East Of England RDD BASILDON AND THURROCK UNIVERSITY HOSPITALS NHS... \n",
"1 East Of England RC1 BEDFORD HOSPITAL NHS TRUST \n",
"2 East Of England RGT CAMBRIDGE UNIVERSITY HOSPITALS NHS FOUNDATION ... \n",
"3 East Of England RWH EAST AND NORTH HERTFORDSHIRE NHS TRUST \n",
"4 East Of England RDE EAST SUFFOLK AND NORTH ESSEX NHS FOUNDATION TRUST \n",
"\n",
" Published Date value \n",
"0 2020-04-04 2020-03-03 0 \n",
"1 2020-04-04 2020-03-03 0 \n",
"2 2020-04-04 2020-03-03 0 \n",
"3 2020-04-04 2020-03-03 0 \n",
"4 2020-04-04 2020-03-03 0 "
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_dailies = df.drop(columns=['Awaiting verification', 'Total'])\n",
"tmp = df_dailies.melt(id_vars=['NHS England Region','Code','Name', 'Published'],\n",
" var_name='Date',\n",
" value_name='value')\n",
"tmp.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Find the days lag between published and strike date:"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>NHS England Region</th>\n",
" <th>Code</th>\n",
" <th>Name</th>\n",
" <th>Published</th>\n",
" <th>Date</th>\n",
" <th>value</th>\n",
" <th>lag</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>East Of England</td>\n",
" <td>RDD</td>\n",
" <td>BASILDON AND THURROCK UNIVERSITY HOSPITALS NHS...</td>\n",
" <td>2020-04-04</td>\n",
" <td>2020-03-03</td>\n",
" <td>0</td>\n",
" <td>32 days</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>East Of England</td>\n",
" <td>RC1</td>\n",
" <td>BEDFORD HOSPITAL NHS TRUST</td>\n",
" <td>2020-04-04</td>\n",
" <td>2020-03-03</td>\n",
" <td>0</td>\n",
" <td>32 days</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>East Of England</td>\n",
" <td>RGT</td>\n",
" <td>CAMBRIDGE UNIVERSITY HOSPITALS NHS FOUNDATION ...</td>\n",
" <td>2020-04-04</td>\n",
" <td>2020-03-03</td>\n",
" <td>0</td>\n",
" <td>32 days</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>East Of England</td>\n",
" <td>RWH</td>\n",
" <td>EAST AND NORTH HERTFORDSHIRE NHS TRUST</td>\n",
" <td>2020-04-04</td>\n",
" <td>2020-03-03</td>\n",
" <td>0</td>\n",
" <td>32 days</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>East Of England</td>\n",
" <td>RDE</td>\n",
" <td>EAST SUFFOLK AND NORTH ESSEX NHS FOUNDATION TRUST</td>\n",
" <td>2020-04-04</td>\n",
" <td>2020-03-03</td>\n",
" <td>0</td>\n",
" <td>32 days</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" NHS England Region Code Name \\\n",
"0 East Of England RDD BASILDON AND THURROCK UNIVERSITY HOSPITALS NHS... \n",
"1 East Of England RC1 BEDFORD HOSPITAL NHS TRUST \n",
"2 East Of England RGT CAMBRIDGE UNIVERSITY HOSPITALS NHS FOUNDATION ... \n",
"3 East Of England RWH EAST AND NORTH HERTFORDSHIRE NHS TRUST \n",
"4 East Of England RDE EAST SUFFOLK AND NORTH ESSEX NHS FOUNDATION TRUST \n",
"\n",
" Published Date value lag \n",
"0 2020-04-04 2020-03-03 0 32 days \n",
"1 2020-04-04 2020-03-03 0 32 days \n",
"2 2020-04-04 2020-03-03 0 32 days \n",
"3 2020-04-04 2020-03-03 0 32 days \n",
"4 2020-04-04 2020-03-03 0 32 days "
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tmp['Date'] = pd.to_datetime(tmp['Date'])\n",
"tmp['lag'] = tmp['Published'] - tmp['Date']\n",
"tmp.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create a simple SQLite database:"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"import sqlite_utils\n",
"!rm nhs_dailies.db\n",
"DB = sqlite_utils.Database(\"nhs_dailies.db\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Add the daily data to the db:"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/site-packages/pandas/core/generic.py:2712: UserWarning: The spaces in these column names will not be changed. In pandas versions < 0.14, spaces were converted to underscores.\n",
" method=method,\n"
]
}
],
"source": [
"idx = {'trust': ['NHS England Region','Code','Name', 'Published'],\n",
" 'age': ['Age group', 'Published'],\n",
" 'region': ['NHS England Region', 'Published'] }\n",
"\n",
"# TO DO - ethnicity\n",
"for daily in data.keys():\n",
" #print(daily)\n",
" #linkDate = getLinkDate(daily)\n",
" # TO DO - get data from excluded sheets\n",
" for sheet in data[daily].keys():\n",
" if 'chart' in sheet or 'ethnicity' in sheet or 'negative test ' in sheet:\n",
" continue\n",
" #print(sheet)\n",
" table = parse('COVID19 daily deaths by {table}', sheet)['table']\n",
" df_dailies = data[daily][sheet].drop(columns=['Awaiting verification', 'Total'])\n",
" #df_dailies['Link_date'] = linkDate\n",
" idx_cols = idx[table]#+['Link_date']\n",
" df_long = df_dailies.melt(id_vars=idx_cols,\n",
" var_name='Date',\n",
" value_name='value')\n",
" df_long['Date'] = pd.to_datetime(df_long['Date'])\n",
" df_long['lag'] = (df_long['Published'] - df_long['Date']).dt.days\n",
"\n",
" _table = f'nhs_dailies_{table}'\n",
" df_long.to_sql(_table, DB.conn, index=False, if_exists='append')\n",
" \n",
" cols = idx[table] + ['Awaiting verification', 'Total']\n",
" data[daily][sheet][cols].to_sql(f'{_table}_summary',\n",
" DB.conn, index=False, if_exists='append')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Dummy query on `age` sheet:"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Age group</th>\n",
" <th>Published</th>\n",
" <th>Date</th>\n",
" <th>value</th>\n",
" <th>lag</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0-19</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20-39</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>40-59</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>60-79</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>1</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>80+</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>TBC</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>0-19</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>0</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>20-39</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>0</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>40-59</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>0</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>60-79</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>0</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>80+</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>1</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>TBC</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>0</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>0-19</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>0</td>\n",
" <td>37</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>20-39</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>0</td>\n",
" <td>37</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>40-59</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>0</td>\n",
" <td>37</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>60-79</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>1</td>\n",
" <td>37</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>80+</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>1</td>\n",
" <td>37</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>TBC</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>0</td>\n",
" <td>37</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>0-19</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-23 00:00:00</td>\n",
" <td>0</td>\n",
" <td>36</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>20-39</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-23 00:00:00</td>\n",
" <td>0</td>\n",
" <td>36</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>40-59</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-23 00:00:00</td>\n",
" <td>0</td>\n",
" <td>36</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>60-79</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-23 00:00:00</td>\n",
" <td>0</td>\n",
" <td>36</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>80+</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-23 00:00:00</td>\n",
" <td>1</td>\n",
" <td>36</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>TBC</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-23 00:00:00</td>\n",
" <td>0</td>\n",
" <td>36</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>0-19</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-24 00:00:00</td>\n",
" <td>0</td>\n",
" <td>35</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Age group Published Date value lag\n",
"0 0-19 2020-04-28 00:00:00 2020-03-19 00:00:00 0 40\n",
"1 20-39 2020-04-28 00:00:00 2020-03-19 00:00:00 0 40\n",
"2 40-59 2020-04-28 00:00:00 2020-03-19 00:00:00 0 40\n",
"3 60-79 2020-04-28 00:00:00 2020-03-19 00:00:00 1 40\n",
"4 80+ 2020-04-28 00:00:00 2020-03-19 00:00:00 0 40\n",
"5 TBC 2020-04-28 00:00:00 2020-03-19 00:00:00 0 40\n",
"6 0-19 2020-04-28 00:00:00 2020-03-20 00:00:00 0 39\n",
"7 20-39 2020-04-28 00:00:00 2020-03-20 00:00:00 0 39\n",
"8 40-59 2020-04-28 00:00:00 2020-03-20 00:00:00 0 39\n",
"9 60-79 2020-04-28 00:00:00 2020-03-20 00:00:00 0 39\n",
"10 80+ 2020-04-28 00:00:00 2020-03-20 00:00:00 1 39\n",
"11 TBC 2020-04-28 00:00:00 2020-03-20 00:00:00 0 39\n",
"12 0-19 2020-04-28 00:00:00 2020-03-22 00:00:00 0 37\n",
"13 20-39 2020-04-28 00:00:00 2020-03-22 00:00:00 0 37\n",
"14 40-59 2020-04-28 00:00:00 2020-03-22 00:00:00 0 37\n",
"15 60-79 2020-04-28 00:00:00 2020-03-22 00:00:00 1 37\n",
"16 80+ 2020-04-28 00:00:00 2020-03-22 00:00:00 1 37\n",
"17 TBC 2020-04-28 00:00:00 2020-03-22 00:00:00 0 37\n",
"18 0-19 2020-04-28 00:00:00 2020-03-23 00:00:00 0 36\n",
"19 20-39 2020-04-28 00:00:00 2020-03-23 00:00:00 0 36\n",
"20 40-59 2020-04-28 00:00:00 2020-03-23 00:00:00 0 36\n",
"21 60-79 2020-04-28 00:00:00 2020-03-23 00:00:00 0 36\n",
"22 80+ 2020-04-28 00:00:00 2020-03-23 00:00:00 1 36\n",
"23 TBC 2020-04-28 00:00:00 2020-03-23 00:00:00 0 36\n",
"24 0-19 2020-04-28 00:00:00 2020-03-24 00:00:00 0 35"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.read_sql(\"SELECT * FROM nhs_dailies_age LIMIT 25\", DB.conn)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Dummy query on `age_summary` sheet:"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Age group</th>\n",
" <th>Published</th>\n",
" <th>Awaiting verification</th>\n",
" <th>Total</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0-19</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>20-39</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>40-59</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>41</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>60-79</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>216</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>80+</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>285</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Age group Published Awaiting verification Total\n",
"0 0-19 2020-04-28 00:00:00 0 0\n",
"1 20-39 2020-04-28 00:00:00 0 4\n",
"2 40-59 2020-04-28 00:00:00 0 41\n",
"3 60-79 2020-04-28 00:00:00 0 216\n",
"4 80+ 2020-04-28 00:00:00 0 285"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.read_sql(\"SELECT * FROM nhs_dailies_age_summary LIMIT 5\", DB.conn)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Dummy query on `trust` sheet:"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>NHS England Region</th>\n",
" <th>Code</th>\n",
" <th>Name</th>\n",
" <th>Published</th>\n",
" <th>Date</th>\n",
" <th>value</th>\n",
" <th>lag</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>East Of England</td>\n",
" <td>NQ1</td>\n",
" <td>ANGLIAN COMMUNITY ENTERPRISE CIC - CLACTON HOS...</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>East Of England</td>\n",
" <td>NQ1</td>\n",
" <td>ANGLIAN COMMUNITY ENTERPRISE CIC - FRYATT HOSP...</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>East Of England</td>\n",
" <td>RDD</td>\n",
" <td>BASILDON AND THURROCK UNIVERSITY HOSPITALS NHS...</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>East Of England</td>\n",
" <td>RC1</td>\n",
" <td>BEDFORD HOSPITAL NHS TRUST</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>East Of England</td>\n",
" <td>RGT</td>\n",
" <td>CAMBRIDGE UNIVERSITY HOSPITALS NHS FOUNDATION ...</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" NHS England Region Code Name \\\n",
"0 East Of England NQ1 ANGLIAN COMMUNITY ENTERPRISE CIC - CLACTON HOS... \n",
"1 East Of England NQ1 ANGLIAN COMMUNITY ENTERPRISE CIC - FRYATT HOSP... \n",
"2 East Of England RDD BASILDON AND THURROCK UNIVERSITY HOSPITALS NHS... \n",
"3 East Of England RC1 BEDFORD HOSPITAL NHS TRUST \n",
"4 East Of England RGT CAMBRIDGE UNIVERSITY HOSPITALS NHS FOUNDATION ... \n",
"\n",
" Published Date value lag \n",
"0 2020-04-28 00:00:00 2020-03-19 00:00:00 0 40 \n",
"1 2020-04-28 00:00:00 2020-03-19 00:00:00 0 40 \n",
"2 2020-04-28 00:00:00 2020-03-19 00:00:00 0 40 \n",
"3 2020-04-28 00:00:00 2020-03-19 00:00:00 0 40 \n",
"4 2020-04-28 00:00:00 2020-03-19 00:00:00 0 40 "
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.read_sql(\"SELECT * FROM nhs_dailies_trust LIMIT 5\", DB.conn)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Dummy query on `trust_summary` sheet:"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>NHS England Region</th>\n",
" <th>Code</th>\n",
" <th>Name</th>\n",
" <th>Published</th>\n",
" <th>Awaiting verification</th>\n",
" <th>Total</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>East Of England</td>\n",
" <td>NQ1</td>\n",
" <td>ANGLIAN COMMUNITY ENTERPRISE CIC - CLACTON HOS...</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>East Of England</td>\n",
" <td>NQ1</td>\n",
" <td>ANGLIAN COMMUNITY ENTERPRISE CIC - FRYATT HOSP...</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>East Of England</td>\n",
" <td>RDD</td>\n",
" <td>BASILDON AND THURROCK UNIVERSITY HOSPITALS NHS...</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>East Of England</td>\n",
" <td>RC1</td>\n",
" <td>BEDFORD HOSPITAL NHS TRUST</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>6</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>East Of England</td>\n",
" <td>RGT</td>\n",
" <td>CAMBRIDGE UNIVERSITY HOSPITALS NHS FOUNDATION ...</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" NHS England Region Code Name \\\n",
"0 East Of England NQ1 ANGLIAN COMMUNITY ENTERPRISE CIC - CLACTON HOS... \n",
"1 East Of England NQ1 ANGLIAN COMMUNITY ENTERPRISE CIC - FRYATT HOSP... \n",
"2 East Of England RDD BASILDON AND THURROCK UNIVERSITY HOSPITALS NHS... \n",
"3 East Of England RC1 BEDFORD HOSPITAL NHS TRUST \n",
"4 East Of England RGT CAMBRIDGE UNIVERSITY HOSPITALS NHS FOUNDATION ... \n",
"\n",
" Published Awaiting verification Total \n",
"0 2020-04-28 00:00:00 0 0 \n",
"1 2020-04-28 00:00:00 0 0 \n",
"2 2020-04-28 00:00:00 0 1 \n",
"3 2020-04-28 00:00:00 0 6 \n",
"4 2020-04-28 00:00:00 0 2 "
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.read_sql(\"SELECT * FROM nhs_dailies_trust_summary LIMIT 5\", DB.conn)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Dummy query on `region` sheet:"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>NHS England Region</th>\n",
" <th>Published</th>\n",
" <th>Date</th>\n",
" <th>value</th>\n",
" <th>lag</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>East Of England</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>London</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>1</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Midlands</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>North East And Yorkshire</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>North West</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>South East</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>South West</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-19 00:00:00</td>\n",
" <td>0</td>\n",
" <td>40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>East Of England</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>0</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>London</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>0</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Midlands</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>1</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>North East And Yorkshire</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>0</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>North West</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>0</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>South East</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>0</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>South West</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-20 00:00:00</td>\n",
" <td>0</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>East Of England</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>0</td>\n",
" <td>37</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>London</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>1</td>\n",
" <td>37</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>Midlands</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>0</td>\n",
" <td>37</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>North East And Yorkshire</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>1</td>\n",
" <td>37</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>North West</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>0</td>\n",
" <td>37</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>South East</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>0</td>\n",
" <td>37</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>South West</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-22 00:00:00</td>\n",
" <td>0</td>\n",
" <td>37</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>East Of England</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-23 00:00:00</td>\n",
" <td>0</td>\n",
" <td>36</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>London</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-23 00:00:00</td>\n",
" <td>1</td>\n",
" <td>36</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>Midlands</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-23 00:00:00</td>\n",
" <td>0</td>\n",
" <td>36</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>North East And Yorkshire</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-03-23 00:00:00</td>\n",
" <td>0</td>\n",
" <td>36</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" NHS England Region Published Date value \\\n",
"0 East Of England 2020-04-28 00:00:00 2020-03-19 00:00:00 0 \n",
"1 London 2020-04-28 00:00:00 2020-03-19 00:00:00 1 \n",
"2 Midlands 2020-04-28 00:00:00 2020-03-19 00:00:00 0 \n",
"3 North East And Yorkshire 2020-04-28 00:00:00 2020-03-19 00:00:00 0 \n",
"4 North West 2020-04-28 00:00:00 2020-03-19 00:00:00 0 \n",
"5 South East 2020-04-28 00:00:00 2020-03-19 00:00:00 0 \n",
"6 South West 2020-04-28 00:00:00 2020-03-19 00:00:00 0 \n",
"7 East Of England 2020-04-28 00:00:00 2020-03-20 00:00:00 0 \n",
"8 London 2020-04-28 00:00:00 2020-03-20 00:00:00 0 \n",
"9 Midlands 2020-04-28 00:00:00 2020-03-20 00:00:00 1 \n",
"10 North East And Yorkshire 2020-04-28 00:00:00 2020-03-20 00:00:00 0 \n",
"11 North West 2020-04-28 00:00:00 2020-03-20 00:00:00 0 \n",
"12 South East 2020-04-28 00:00:00 2020-03-20 00:00:00 0 \n",
"13 South West 2020-04-28 00:00:00 2020-03-20 00:00:00 0 \n",
"14 East Of England 2020-04-28 00:00:00 2020-03-22 00:00:00 0 \n",
"15 London 2020-04-28 00:00:00 2020-03-22 00:00:00 1 \n",
"16 Midlands 2020-04-28 00:00:00 2020-03-22 00:00:00 0 \n",
"17 North East And Yorkshire 2020-04-28 00:00:00 2020-03-22 00:00:00 1 \n",
"18 North West 2020-04-28 00:00:00 2020-03-22 00:00:00 0 \n",
"19 South East 2020-04-28 00:00:00 2020-03-22 00:00:00 0 \n",
"20 South West 2020-04-28 00:00:00 2020-03-22 00:00:00 0 \n",
"21 East Of England 2020-04-28 00:00:00 2020-03-23 00:00:00 0 \n",
"22 London 2020-04-28 00:00:00 2020-03-23 00:00:00 1 \n",
"23 Midlands 2020-04-28 00:00:00 2020-03-23 00:00:00 0 \n",
"24 North East And Yorkshire 2020-04-28 00:00:00 2020-03-23 00:00:00 0 \n",
"\n",
" lag \n",
"0 40 \n",
"1 40 \n",
"2 40 \n",
"3 40 \n",
"4 40 \n",
"5 40 \n",
"6 40 \n",
"7 39 \n",
"8 39 \n",
"9 39 \n",
"10 39 \n",
"11 39 \n",
"12 39 \n",
"13 39 \n",
"14 37 \n",
"15 37 \n",
"16 37 \n",
"17 37 \n",
"18 37 \n",
"19 37 \n",
"20 37 \n",
"21 36 \n",
"22 36 \n",
"23 36 \n",
"24 36 "
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.read_sql(\"SELECT * FROM nhs_dailies_region LIMIT 25\", DB.conn)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Dummy query on `region_summary` sheet:"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>NHS England Region</th>\n",
" <th>Published</th>\n",
" <th>Awaiting verification</th>\n",
" <th>Total</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>East Of England</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>72</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>London</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>88</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Midlands</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>99</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>North East And Yorkshire</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>87</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>North West</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>102</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>South East</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>72</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>South West</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>26</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>East Of England</td>\n",
" <td>2020-04-27 00:00:00</td>\n",
" <td>0</td>\n",
" <td>34</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>London</td>\n",
" <td>2020-04-27 00:00:00</td>\n",
" <td>0</td>\n",
" <td>87</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Midlands</td>\n",
" <td>2020-04-27 00:00:00</td>\n",
" <td>0</td>\n",
" <td>55</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>North East And Yorkshire</td>\n",
" <td>2020-04-27 00:00:00</td>\n",
" <td>0</td>\n",
" <td>55</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>North West</td>\n",
" <td>2020-04-27 00:00:00</td>\n",
" <td>0</td>\n",
" <td>41</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>South East</td>\n",
" <td>2020-04-27 00:00:00</td>\n",
" <td>0</td>\n",
" <td>38</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>South West</td>\n",
" <td>2020-04-27 00:00:00</td>\n",
" <td>0</td>\n",
" <td>19</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>East Of England</td>\n",
" <td>2020-04-26 00:00:00</td>\n",
" <td>0</td>\n",
" <td>49</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>London</td>\n",
" <td>2020-04-26 00:00:00</td>\n",
" <td>0</td>\n",
" <td>64</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>Midlands</td>\n",
" <td>2020-04-26 00:00:00</td>\n",
" <td>0</td>\n",
" <td>57</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>North East And Yorkshire</td>\n",
" <td>2020-04-26 00:00:00</td>\n",
" <td>0</td>\n",
" <td>66</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>North West</td>\n",
" <td>2020-04-26 00:00:00</td>\n",
" <td>0</td>\n",
" <td>46</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>South East</td>\n",
" <td>2020-04-26 00:00:00</td>\n",
" <td>0</td>\n",
" <td>43</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>South West</td>\n",
" <td>2020-04-26 00:00:00</td>\n",
" <td>0</td>\n",
" <td>11</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>East Of England</td>\n",
" <td>2020-04-25 00:00:00</td>\n",
" <td>0</td>\n",
" <td>88</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>London</td>\n",
" <td>2020-04-25 00:00:00</td>\n",
" <td>0</td>\n",
" <td>116</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>Midlands</td>\n",
" <td>2020-04-25 00:00:00</td>\n",
" <td>0</td>\n",
" <td>145</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>North East And Yorkshire</td>\n",
" <td>2020-04-25 00:00:00</td>\n",
" <td>0</td>\n",
" <td>117</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" NHS England Region Published Awaiting verification \\\n",
"0 East Of England 2020-04-28 00:00:00 0 \n",
"1 London 2020-04-28 00:00:00 0 \n",
"2 Midlands 2020-04-28 00:00:00 0 \n",
"3 North East And Yorkshire 2020-04-28 00:00:00 0 \n",
"4 North West 2020-04-28 00:00:00 0 \n",
"5 South East 2020-04-28 00:00:00 0 \n",
"6 South West 2020-04-28 00:00:00 0 \n",
"7 East Of England 2020-04-27 00:00:00 0 \n",
"8 London 2020-04-27 00:00:00 0 \n",
"9 Midlands 2020-04-27 00:00:00 0 \n",
"10 North East And Yorkshire 2020-04-27 00:00:00 0 \n",
"11 North West 2020-04-27 00:00:00 0 \n",
"12 South East 2020-04-27 00:00:00 0 \n",
"13 South West 2020-04-27 00:00:00 0 \n",
"14 East Of England 2020-04-26 00:00:00 0 \n",
"15 London 2020-04-26 00:00:00 0 \n",
"16 Midlands 2020-04-26 00:00:00 0 \n",
"17 North East And Yorkshire 2020-04-26 00:00:00 0 \n",
"18 North West 2020-04-26 00:00:00 0 \n",
"19 South East 2020-04-26 00:00:00 0 \n",
"20 South West 2020-04-26 00:00:00 0 \n",
"21 East Of England 2020-04-25 00:00:00 0 \n",
"22 London 2020-04-25 00:00:00 0 \n",
"23 Midlands 2020-04-25 00:00:00 0 \n",
"24 North East And Yorkshire 2020-04-25 00:00:00 0 \n",
"\n",
" Total \n",
"0 72 \n",
"1 88 \n",
"2 99 \n",
"3 87 \n",
"4 102 \n",
"5 72 \n",
"6 26 \n",
"7 34 \n",
"8 87 \n",
"9 55 \n",
"10 55 \n",
"11 41 \n",
"12 38 \n",
"13 19 \n",
"14 49 \n",
"15 64 \n",
"16 57 \n",
"17 66 \n",
"18 46 \n",
"19 43 \n",
"20 11 \n",
"21 88 \n",
"22 116 \n",
"23 145 \n",
"24 117 "
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.read_sql(\"SELECT * FROM nhs_dailies_region_summary LIMIT 25\", DB.conn)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Adding NHS Totals Data to Database"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"odict_keys(['COVID19 total deaths chart', 'COVID19 total deaths by trust', 'COVID19 total deaths by region', 'Deaths by region- negative test', 'COVID19 total deaths by age', 'COVID19 total by ethnicity'])"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"totals_xl.keys()"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [],
"source": [
"for sheet in totals_xl.keys():\n",
" if 'chart' in sheet or 'ethnicity' in sheet or 'negative test' in sheet:\n",
" continue\n",
" table = parse('COVID19 total deaths by {table}', sheet)['table']\n",
" df_totals = totals_xl[sheet].drop(columns=['Awaiting verification', 'Total', 'Up to 01-Mar-20'])\n",
" idx_cols = idx[table]\n",
" df_long = df_totals.melt(id_vars=idx_cols,\n",
" var_name='Date',\n",
" value_name='value')\n",
" df_long['Date'] = pd.to_datetime(df_long['Date'])\n",
" df_long['lag'] = (df_long['Published'] - df_long['Date']).dt.days\n",
"\n",
" _table = f'nhs_totals_{table}'\n",
" df_long.to_sql(_table, DB.conn, index=False, if_exists='append')\n",
" \n",
" cols = idx_cols + ['Up to 01-Mar-20', 'Awaiting verification', 'Total']\n",
" totals_xl[sheet][cols].to_sql(f'{_table}_summary',\n",
" DB.conn, index=False, if_exists='append')"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>NHS England Region</th>\n",
" <th>Published</th>\n",
" <th>Up to 01-Mar-20</th>\n",
" <th>Awaiting verification</th>\n",
" <th>Total</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>East Of England</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2210</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>London</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>4781</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Midlands</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>3794</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>North East And Yorkshire</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2550</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>North West</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2851</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>South East</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2226</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>South West</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>883</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" NHS England Region Published Up to 01-Mar-20 \\\n",
"0 East Of England 2020-04-28 00:00:00 0 \n",
"1 London 2020-04-28 00:00:00 0 \n",
"2 Midlands 2020-04-28 00:00:00 0 \n",
"3 North East And Yorkshire 2020-04-28 00:00:00 0 \n",
"4 North West 2020-04-28 00:00:00 0 \n",
"5 South East 2020-04-28 00:00:00 0 \n",
"6 South West 2020-04-28 00:00:00 0 \n",
"\n",
" Awaiting verification Total \n",
"0 0 2210 \n",
"1 0 4781 \n",
"2 0 3794 \n",
"3 0 2550 \n",
"4 1 2851 \n",
"5 0 2226 \n",
"6 0 883 "
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.read_sql(\"SELECT * FROM nhs_totals_region_summary LIMIT 25\", DB.conn)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Basic Charts\n",
"\n",
"Let's try some basic charts. For example, "
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>NHS England Region</th>\n",
" <th>Published</th>\n",
" <th>Date</th>\n",
" <th>value</th>\n",
" <th>lag</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>London</td>\n",
" <td>2020-04-28 00:00:00</td>\n",
" <td>2020-04-09 00:00:00</td>\n",
" <td>187</td>\n",
" <td>19</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" NHS England Region Published Date value lag\n",
"0 London 2020-04-28 00:00:00 2020-04-09 00:00:00 187 19"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"zz = pd.read_sql(\"SELECT * FROM nhs_totals_region WHERE `NHS England Region`='London' and Date=DATETIME('2020-04-09')\", DB.conn)\n",
"zz"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"How long does it take for a particular hospital to report deaths (i.e. what's the lag distribution between the publication date and the strike date?)?\n",
"\n",
"The following chart sums the number of deaths reported relative to the delay in reporting them:"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.axes._subplots.AxesSubplot at 0x11b446b50>"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAEJCAYAAABmA8c1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAZX0lEQVR4nO3dfZxcVZ3n8c8vJBiZIGLSkEgIyToKQ2DkoUVdcHBFFAXE2QGRYVyU0ezsDo/iuhnZEWeXdRhmdXVfOLuvDCAqLIhRBGVHYDNho2xgScJDQsKDEwI0BNKAPBtI4Owf93RSqVR311OnOyef9+tVr646dercc2+d+tapW7duR0oJSdL2bdxod0CS1DnDXJIKYJhLUgEMc0kqgGEuSQUwzCWpAOO35cKmTJmSZs6cuS0XKUnbvaVLlz6dUuoZqs42DfOZM2eyZMmSbblISdruRcQjw9VxN4skFcAwl6QCGOaSVIBtus9ckoazYcMG+vr6WL9+/Wh3ZZubOHEi06dPZ8KECS0/1jCXNKb09fWx6667MnPmTCJitLuzzaSUeOaZZ+jr62PWrFktP37Y3SwRcXlErIuIFQ3uOy8iUkRMaXnJktTA+vXrmTx58g4V5AARweTJk9v+RNLMPvMrgGMaLHhv4CPAo20tWZIGsaMF+YBO1nvYME8pLQKebXDXfwW+DHhCdEk7tEmTJo12F9rbZx4RJwCPp5TuGe6dJCLmAHMAZsyY0c7ixoyZc28EYM1FxzZdt9n6khqrfS11Q6mvx5YPTYyIXYCvAF9tpn5KaV5KqTel1NvTM+SvUSVpTJg7dy7f+c53Nt3+2te+xoUXXshRRx3FIYccwoEHHsj111+/1eNuvfVWjjvuuE23zzjjDK644goAli5dypFHHsmhhx7KRz/6UdauXdvVPrdznPk7gFnAPRGxBpgOLIuIqd3smCSNlpNPPplrr7120+1rr72W0047jeuuu45ly5axcOFCzjvvPJr9t5sbNmzgzDPPZP78+SxdupTTTz+d888/v6t9bnk3S0ppObDHwO0c6L0ppae72C9JGjUHH3ww69at44knnqC/v5/dd9+dqVOncu6557Jo0SLGjRvH448/zlNPPcXUqcPPYx944AFWrFjB0UcfDcDrr7/OtGnTutrnYcM8Iq4GPghMiYg+4IKU0mVd7YUkjTEnnXQS8+fP58knn+Tkk0/mqquuor+/n6VLlzJhwgRmzpy51WGE48eP54033th0e+D+lBKzZ89m8eLFI9bfZo5mOSWlNC2lNCGlNL0+yFNKM52VSyrNySefzDXXXMP8+fM56aSTeP7559ljjz2YMGECCxcu5JFHtj6R4T777MPKlSt59dVXee6551iwYAEA++67L/39/ZvCfMOGDdx3331d7a+/AJWkBmbPns2LL77IXnvtxbRp0zj11FM5/vjjOfDAA+nt7WW//fbb6jF77703n/rUpzjggAOYNWsWBx98MAA777wz8+fP56yzzuL5559n48aNnHPOOcyePbtr/TXMJY1po3ko4fLlyzddnzJlyqC7SV566aVN1y+++GIuvvjireocdNBBLFq0qPudzDxroiQVwDCXpAIY5pJUAMNc0pjT7I9xStPJehvmksaUiRMn8swzz+xwgT5wPvOJEye29XiPZpE0pkyfPp2+vj76+/tHuyvb3MB/GmqHYS5pTJkwYUJb/2lnR2eYqykjeUpfTxcsdc595pJUAMNckgpgmEtSAQxzSSqAYS5JBTDMJakAhrkkFcAwl6QCGOaSVADDXJIKMGyYR8TlEbEuIlbUlP1tRNwfEfdGxHUR8daR7aYkaSjNzMyvAI6pK7sFOCCl9PvAg8BfdLlfkqQWDBvmKaVFwLN1ZTenlDbmm7cD7Z2zUZLUFd3YZ3468A9daEeS1KaOwjwizgc2AlcNUWdORCyJiCU74snmJWlbaDvMI+KzwHHAqWmI/++UUpqXUupNKfX29PS0uzhJ0hDa+ucUEXEM8GXgyJTSK93tkiSpVc0cmng1sBjYNyL6IuJPgUuAXYFbIuLuiPgfI9xPSdIQhp2Zp5ROaVB82Qj0RZLUJn8BKkkFMMwlqQCGuSQVwDCXpAIY5pJUAMNckgpgmEtSAQxzSSqAYS5JBTDMJakAhrkkFcAwl6QCGOaSVADDXJIKYJhLUgEMc0kqgGEuSQUwzCWpAIa5JBXAMJekAhjmklSAYcM8Ii6PiHURsaKm7G0RcUtEPJT/7j6y3ZQkDaWZmfkVwDF1ZXOBBSmldwIL8m1J0igZNsxTSouAZ+uKTwC+l69/D/hkl/slSWrB+DYft2dKaW2+/iSw52AVI2IOMAdgxowZbS5uZMyce+Om62suOnbY8pFc5lhvu1Xbui9jad2l0dDxF6AppQSkIe6fl1LqTSn19vT0dLo4SVID7Yb5UxExDSD/Xde9LkmSWtVumN8AnJavnwZc353uSJLa0cyhiVcDi4F9I6IvIv4UuAg4OiIeAj6cb0uSRsmwX4CmlE4Z5K6jutwXSVKb/AWoJBXAMJekAhjmklQAw1ySCmCYS1IBDHNJKoBhLkkFMMwlqQCGuSQVoN1T4I5Jo3FK25HUrX5vr+s/mNLWR+oGZ+aSVADDXJIKYJhLUgEMc0kqgGEuSQUwzCWpAIa5JBXAMJekAhjmklQAw1ySCtBRmEfEuRFxX0SsiIirI2JitzomSWpe22EeEXsBZwG9KaUDgJ2AT3erY5Kk5nW6m2U88OaIGA/sAjzReZckSa1qO8xTSo8D/wV4FFgLPJ9SurlbHZMkNa/tU+BGxO7ACcAs4DngRxHxJymlK+vqzQHmAMyYMaODrmrAWDoF7Ej2ZXttWxoNnexm+TDwcEqpP6W0AfgJ8M/rK6WU5qWUelNKvT09PR0sTpI0mE7C/FHgfRGxS0QEcBSwqjvdkiS1opN95ncA84FlwPLc1rwu9UuS1IKO/m1cSukC4IIu9UWS1CZ/ASpJBTDMJakAhrkkFcAwl6QCGOaSVADDXJIKYJhLUgEMc0kqgGEuSQUwzCWpAB39nF/l8bSz0vbJmbkkFcAwl6QCGOaSVADDXJIKYJhLUgEMc0kqgGEuSQUwzCWpAIa5JBXAMJekAnQU5hHx1oiYHxH3R8SqiHh/tzomSWpep+dm+Tbwi5TSiRGxM7BLF/okSWpR22EeEbsBfwB8FiCl9BrwWne6JUlqRScz81lAP/DdiHg3sBQ4O6X0cm2liJgDzAGYMWPGoI21eka90s7AV9r6SNq2OtlnPh44BPjvKaWDgZeBufWVUkrzUkq9KaXenp6eDhYnSRpMJ2HeB/SllO7It+dThbskaRtrO8xTSk8Cj0XEvrnoKGBlV3olSWpJp0eznAlclY9kWQ18rvMuSZJa1VGYp5TuBnq71BdJUpv8BagkFcAwl6QCGOaSVADDXJIKYJhLUgEMc0kqgGEuSQUwzCWpAIa5JBWg05/zj7ixfmrYsd6/oQz0fXvr90gajedzex5DGjucmUtSAQxzSSqAYS5JBTDMJakAhrkkFcAwl6QCGOaSVADDXJIKYJhLUgEMc0kqQMdhHhE7RcRdEfHzbnRIktS6bszMzwZWdaEdSVKbOgrziJgOHAtc2p3uSJLa0enM/FvAl4E3utAXSVKb2g7ziDgOWJdSWjpMvTkRsSQilvT397e7OEnSEDqZmR8OfCIi1gDXAB+KiCvrK6WU5qWUelNKvT09PR0sTpI0mLbDPKX0Fyml6SmlmcCngX9MKf1J13omSWqax5lLUgG68m/jUkq3Ard2oy1JUuucmUtSAQxzSSqAYS5JBTDMJakAhrkkFcAwl6QCGOaSVADDXJIKYJhLUgEMc0kqQFd+zi9peDPn3rjp+pqLjh21NlQmZ+aSVADDXJIKYJhLUgEMc0kqgGEuSQUwzCWpAIa5JBXAMJekAhjmklQAw1ySCtB2mEfE3hGxMCJWRsR9EXF2NzsmSWpeJ+dm2Qicl1JaFhG7Aksj4paU0sou9U2S1KS2Z+YppbUppWX5+ovAKmCvbnVMktS8rpw1MSJmAgcDdzS4bw4wB2DGjBme9U3bpYFxWztmBxvLnh1Ro6HjL0AjYhLwY+CclNIL9fenlOallHpTSr09PT2dLk6S1EBHYR4RE6iC/KqU0k+60yVJUqs6OZolgMuAVSmlb3avS5KkVnUyMz8c+AzwoYi4O18+3qV+SZJa0PYXoCmlXwHRxb5IktrkL0AlqQCGuSQVwDCXpAIY5pJUAMNckgpgmEtSAQxzSSqAYS5JBTDMJakAXTkFrjRW7Sinkt1R1rMVO9o2cWYuSQUwzCWpAIa5JBXAMJekAhjmklQAw1ySCmCYS1IBDHNJKoBhLkkFMMwlqQCGuSQVoKMwj4hjIuKBiPh1RMztVqckSa1pO8wjYifgO8DHgP2BUyJi/251TJLUvE5m5ocBv04prU4pvQZcA5zQnW5JkloRKaX2HhhxInBMSunz+fZngPemlM6oqzcHmJNv7gs8kK9PAZ5u0HSj8lbqjqW2R2OZrs/YaHs0lun6jI22R2KZ+6SUehrU2Syl1NYFOBG4tOb2Z4BLWnj8kmbLW6k7ltp2fXbctl2fHbftkV7mYJdOdrM8Duxdc3t6LpMkbWOdhPmdwDsjYlZE7Ax8GrihO92SJLWi7X8bl1LaGBFnADcBOwGXp5Tua6GJeS2Ut1J3LLU9Gst0fcZG26OxTNdnbLQ90stsqO0vQCVJY4e/AJWkAhjmklQAw1ySCjCqYR4R+0XEURExqa78rIh4T76+f0R8MSI+3uDx329QdkREfCsiPplvvzki/ioifhYR/7vRKQciYueI+FcR8eF8+48j4pKI+POIeFdEfCkivh0R34yIP4uIt3RrG6gSEXu0WH/ySPVFalYr43bEx2wrB6V38wKcRfVr0GXAGuCEXH4B8DKwBPhr4B+BvwSeAVZRHf54A/Az4CXgN8AN+bFfAO4G1gG3AXOpvhH+FnAEsB74LfBL4N8CPflxVwE/zG3+ALiO6kdQdwBPAP8B+L9U56L5z8BK4IPbcFvt0WL9yV1Y5m7ARcD9wLM12/8i4K0N6r8F+Ke8/f64pnwqsCJvu8nA1/L2+ynwe8Dbcvkaqh+iva1m+ZcB9wL3Ab+Xy3uB1cCvgVeBS4F31CyvF1gIXEn1O4hbgOeBpXks3Jdv9wO3A6cD/xr4RV7WvcA/AH8GTGiwnjvl8fOfgMNz2S7Al3MbE4HP5jF6MTCp7vEPAr9fc3tCHl83UB0Ztncu/11gEfBaXtfD6tr5Z8DlwIXAJODv83b+EfDvgBuBe6heXz8EvtHJOtas5y9y+4OuZxvr+BzVGJvbYHsNtp53UmXDwDpeAxzLyI3Z5Ww5ZgfG7VPA7vl2W2N2mHF7J3BwU6/ZbRVIDTbk8vzkPArMpArvs3P5XXngvAC8Jde/iyq4Pwgcmf+uBR4Cjsx17gR68hP4O7mtZTXLvIsq7D+SN3p/HpyPAbtSHar5FLBTTR/vrRnIt+brs3O9YQdNHjB/TfWDqnYHzBpaC7r1VC+QU5ocMI2C7jngWmBqXZ+/DSwGDqm7LKB6c/0k1Yv2x8Cbarbv3Nzffw+8kbfZy8DD+bKBaqCvzsu6lOoFvA/VG+pPc/lC4D35+mPAk1Rj6P8B5+bn+GPAKfn+E3Pd26heTNOBL1JNEN6Zt9dS4H35vun5+uVUb+pvq7v8AHgFOCc/7pt5O30jb7cFwCXAB6iC+DWqcfxivryeLy/kfn0DuIJqTD8DfD+X3wj8IdW4+T95+1yby3amCsF/k7frCuC8/Jz+Km/PI6gmMf+RKvT68rLaWsfcp2upxv3f1aznK/l5e63ddczX+6l+uv5sE+v5Q+D63LeBdTw6t309IzNm96Yat7Vj9mEg5b+raX/Mvj1fbzRujwIWj5kwZ/NsoPayniq8Xs11JuWNuA64eyB8a9oYRzWwbwEOymWrqd6Zd6cKvSW5/EfA56he2N8FenP5fcCdNW1OAD5BFVz9uZ0X2RyaK4D78/Xda9q/KT8hzQyaBbkPz3YwYFoKuvyYy/JjmhkwjYJuNfA94Ot1z+XruX8L6y4vAr+tqXd+bvde8hsq8Gj+e15e//tr6j/Mlm+8d9dcX1UzJm6vKV8GLM/XP0AVMhtyf+YMLC/ffw9bjqc7898Ha/tRt56v1T0Pq3PZa7nOeKo3wefyc3pXHhcDh/z+N6oJyJ5161nbj7vJs2OqT6r31vXvrpqx+Bngf1GN1WeAj9Ru15rXWm37t+d1fBOwqoN1/MnANgRiYD3zOn4fWNnuOg6sZ+77W4Zbz5rHD2yb22ueyy3WsVtjtmbcvgAcWFP2ahfG7JO5L3Pql1mfg0Pm7EiEd4ON+RRwEFX4DFxuAz4OPFFTb3x+Al/Pt8fV3Ldb3hDTqcL6Eqp3tzV5AA4MxGm57pVUYXYH1Qt8dd5g727Qv3Pz4x+h2v2zgOrj3BN5Q/891Sz8c7n+r4FFTQ6aF/Pf33YwYFoKupp2lzczYGgQdMDNVG8wD9WU75nbua3Buq8CHqsr+yzVm/Yj+faFNfdNp/oU8E2qT0WrqWaPX8zbYDWbQ/HM3PcPUX2C+TbVLG8t8IO6ZS4GvpK3+SPAJ2vCcOCN+RPATTVh9zhbjrVxue2tXkR5HNSv59r8nD5E9eO52vsepJoZn5XbXZ0v/xL4I2rCh2oX3rNUuxa+QjUzXkE1Mfl5Tb3Jed0WU5299Gk2T1hq1/MQqpnt7cBJbBm4ra7jBVSz2Ify7ctr7js0Pz/16/iHTa7jPlSv45/XLbN2Pd8zsJ5UM/IPUb2GDiG/FqnG7FNs+ebZtTFb85gfsXncbqTzMbtTbvem/DzVjtsjafIcLdsqzC8Djqgrm041m/2fdeVvomY/XU35FLYMuGOpmzXW1d8FOBB4dx5sewLvGqL+24G35+tvpdqtcRjVLpUTgf1q6t5MtY902EGTn6RxtYOmjQHTatA9CPxV7aAZZsBsFXRUn0T+huoN8TdUL8BVVLuBDmuw/S4Gvtqg/CqqUyXXl/8uMD8v7/a87S6ouwx8pzE19+mHVDO45VQztzup2+ebn++bqPYJ75e3x3NUb8Ar87r8Ctg31z+EapKwLm+3B/P1pcDHGvT7SuDv6souBf4c2FBX/o68rHFUQfdLqgnCd+sue9as50qqCcjT+Xl9Gfg6sFtd20dRzXJXUe1S+THVm8lvqCZED1FNAt5LtRvzp1S7RB7M97W0jrn8l/XrWLeetet4RQvruJJqYrZbg7YbrWcf1cRpbV7H99WMqTuo3oxGbMzWvE5uz/3vaMzm+gex9bj9DdXehK3ysGGGNVPJy1YbfiDoBvaZ1w6a9zQYMB8mB2ebA6bVoHuCajfD+Lq2mw26d+X676f6BFT/pdTn84us2fIv1Jfn5c+l2r32ZuCANtveqpzqO4dGX6Z9IT8X9eVnU71xTwYOB75E9anxMDbv69yf6s10sPJzGpQdW1f3A8BXcxvvbaLt2VRv3o3arm9jdk2/39+ofr49OV+uHGRsf7+dcjZPLqYBz7TYxg9arP9zqjfIKUPVzdv7PPJumpryI/I26bT8A1Rf7H6kybqt9KVh/SFzqdmKXprcoHlXTDPlzdRly6Bruu1OlplvD3a00VlUnyp+Wld+ZrPlNW0/12QbTS8z172/QduDtXEBWx49tYDqO4M1VLPF+qOqGpUvoNo11jdM3cHaHqq8G23fz9ZHg71ENUl4kq2PEGulfGOD8kZ1Wy0fbJk3DLHM2iPbPk81ubmAavY8N5cPHPHWjfK78vMycOTc5wep205fBurfNlA+bPaMdviVdqHuy4uhylup263yZusy9NFG9+Q67ZY/kdu+a4y03ejoqRVU+2SHLc9tDBw91VYbI9k21Rvy1VSH4R3JlkeC3UzjI8SaLd+Qy85uso0HR3iZR+btcCebP73ew+YvHbtaTjV2Bp6fkVjm7wyUD3fxF6BtiIh7B7n8FpheXzZYeYOybpV3vEyqw/YWU+3rXEP1gvkY1cfoBNBB+a5Uh5PFGGh7z1z3FeCfUkov5Kd5A/BGk+UbU0ovU+3HbbeNkWy7l+pIpj2A51NKt1L93mJfql1u53dQ/lguO67JNvYbwWW+Atybf5wTKaV+NksjVD6O6hDKNBLLzM/9RprRTOJ72WoW2+jonH2ovnha16Ds4/kxw9XtVnk3ljnY0UZryUcbdVC+kOpj9OtjoO3Bjp5awuZD34Ysp/rSbSrVDLitNrZB27tRzR43HQlWc9/0Tsu70Uany6TBkW253iNUX+R3vTwv8+FcPhLLnETN0WtD5tJoB+P2eKHB0Tk15bc0qsvWR+1sVbdb5d1YJoMfbTQdOL5BG02X17R9+Bhoe7Cjp95OzdFTQ5XnNqY0KG+6jW3Q9qY2GORIsG6Uj2TbrS6z5v5dgFnbsnykl9no4vnMJakA7jOXpAIY5pJUAMNcO5SIeGm0+yCNBMNckgpgmGuHFBGTImJBRCyLiOURcULNfX8ZEQ9ExK8i4uqI+NJo9lVqxvjR7oA0StZTnUv7hYiYAtweETdQ/cDmj6jOYzOB6vjupaPXTak5hrl2VAF8PSL+gOoc8ntR/Rr0cOD6lNJ6YH1E/GwU+yg1zTDXjupUqnNrHJpS2hARa6j+FZq0XXKfuXZUuwHrcpD/C6pTGEB1GoPjI2Ji/kfjx41aD6UWODPXjuoq4GcRsZzqfCb3A6SU7sz7zu+lOrfNcqr/iCSNaf6cX6oTEZNSSi9FxC5U/3ZtTkpp2Wj3SxqKM3Npa/MiYn+qfejfM8i1PXBmLkkF8AtQSSqAYS5JBTDMJakAhrkkFcAwl6QCGOaSVID/D2w9FX00J9/3AAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"pd.read_sql(\"SELECT value, lag FROM nhs_totals_trust WHERE Name='WEST HERTFORDSHIRE HOSPITALS NHS TRUST'\", DB.conn).groupby(['lag']).sum().plot(kind='bar')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Public Health England\n",
"\n",
"Data published by Public Health England:\n",
" \n",
"- [Cases](https://coronavirus.data.gov.uk/downloads/csv/coronavirus-cases_latest.csv)\n",
"- [Deaths](https://coronavirus.data.gov.uk/downloads/csv/coronavirus-deaths_latest.csv)"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [],
"source": [
"#via https://stackoverflow.com/questions/61415090/python-pandas-handling-of-308-request\n",
"import requests\n",
"import io\n",
"\n",
"def get_308_csv(url):\n",
" datastr = requests.get(url, allow_redirects=True).text\n",
" data_file = io.StringIO(datastr)\n",
" _df = pd.read_csv(data_file)\n",
" _df['Specimen date'] = pd.to_datetime(_df['Specimen date'])\n",
" return _df"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/site-packages/pandas/core/generic.py:2712: UserWarning: The spaces in these column names will not be changed. In pandas versions < 0.14, spaces were converted to underscores.\n",
" method=method,\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Area name</th>\n",
" <th>Area code</th>\n",
" <th>Area type</th>\n",
" <th>Specimen date</th>\n",
" <th>Daily lab-confirmed cases</th>\n",
" <th>Previously reported daily cases</th>\n",
" <th>Change in daily cases</th>\n",
" <th>Cumulative lab-confirmed cases</th>\n",
" <th>Previously reported cumulative cases</th>\n",
" <th>Change in cumulative cases</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>England</td>\n",
" <td>E92000001</td>\n",
" <td>Nation</td>\n",
" <td>2020-04-27</td>\n",
" <td>103.0</td>\n",
" <td>0.0</td>\n",
" <td>103.0</td>\n",
" <td>114456</td>\n",
" <td>112467.0</td>\n",
" <td>1989.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>South West</td>\n",
" <td>E12000009</td>\n",
" <td>Region</td>\n",
" <td>2020-04-27</td>\n",
" <td>4.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>5986</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>South East</td>\n",
" <td>E12000008</td>\n",
" <td>Region</td>\n",
" <td>2020-04-27</td>\n",
" <td>7.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>16323</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>London</td>\n",
" <td>E12000007</td>\n",
" <td>Region</td>\n",
" <td>2020-04-27</td>\n",
" <td>6.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>23979</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>East of England</td>\n",
" <td>E12000006</td>\n",
" <td>Region</td>\n",
" <td>2020-04-27</td>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>9907</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Area name Area code Area type Specimen date \\\n",
"0 England E92000001 Nation 2020-04-27 \n",
"1 South West E12000009 Region 2020-04-27 \n",
"2 South East E12000008 Region 2020-04-27 \n",
"3 London E12000007 Region 2020-04-27 \n",
"4 East of England E12000006 Region 2020-04-27 \n",
"\n",
" Daily lab-confirmed cases Previously reported daily cases \\\n",
"0 103.0 0.0 \n",
"1 4.0 NaN \n",
"2 7.0 NaN \n",
"3 6.0 NaN \n",
"4 1.0 NaN \n",
"\n",
" Change in daily cases Cumulative lab-confirmed cases \\\n",
"0 103.0 114456 \n",
"1 NaN 5986 \n",
"2 NaN 16323 \n",
"3 NaN 23979 \n",
"4 NaN 9907 \n",
"\n",
" Previously reported cumulative cases Change in cumulative cases \n",
"0 112467.0 1989.0 \n",
"1 NaN NaN \n",
"2 NaN NaN \n",
"3 NaN NaN \n",
"4 NaN NaN "
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"phe_cases_url = 'https://coronavirus.data.gov.uk/downloads/csv/coronavirus-cases_latest.csv'\n",
"phe_cases_df = get_308_csv(phe_cases_url)\n",
"\n",
"_table = f'phe_cases'\n",
"phe_cases_df.to_sql(_table, DB.conn, index=False, if_exists='append')\n",
" \n",
"phe_cases_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Area name</th>\n",
" <th>Area code</th>\n",
" <th>Area type</th>\n",
" <th>Specimen date</th>\n",
" <th>Daily lab-confirmed cases</th>\n",
" <th>Previously reported daily cases</th>\n",
" <th>Change in daily cases</th>\n",
" <th>Cumulative lab-confirmed cases</th>\n",
" <th>Previously reported cumulative cases</th>\n",
" <th>Change in cumulative cases</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>England</td>\n",
" <td>E92000001</td>\n",
" <td>Nation</td>\n",
" <td>2020-04-27 00:00:00</td>\n",
" <td>103.0</td>\n",
" <td>0.0</td>\n",
" <td>103.0</td>\n",
" <td>114456</td>\n",
" <td>112467.0</td>\n",
" <td>1989.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>South West</td>\n",
" <td>E12000009</td>\n",
" <td>Region</td>\n",
" <td>2020-04-27 00:00:00</td>\n",
" <td>4.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>5986</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>South East</td>\n",
" <td>E12000008</td>\n",
" <td>Region</td>\n",
" <td>2020-04-27 00:00:00</td>\n",
" <td>7.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>16323</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Area name Area code Area type Specimen date \\\n",
"0 England E92000001 Nation 2020-04-27 00:00:00 \n",
"1 South West E12000009 Region 2020-04-27 00:00:00 \n",
"2 South East E12000008 Region 2020-04-27 00:00:00 \n",
"\n",
" Daily lab-confirmed cases Previously reported daily cases \\\n",
"0 103.0 0.0 \n",
"1 4.0 NaN \n",
"2 7.0 NaN \n",
"\n",
" Change in daily cases Cumulative lab-confirmed cases \\\n",
"0 103.0 114456 \n",
"1 NaN 5986 \n",
"2 NaN 16323 \n",
"\n",
" Previously reported cumulative cases Change in cumulative cases \n",
"0 112467.0 1989.0 \n",
"1 NaN NaN \n",
"2 NaN NaN "
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.read_sql(\"SELECT * FROM phe_cases LIMIT 3\", DB.conn)"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Area name</th>\n",
" <th>Area code</th>\n",
" <th>Area type</th>\n",
" <th>Specimen date</th>\n",
" <th>Daily lab-confirmed cases</th>\n",
" <th>Previously reported daily cases</th>\n",
" <th>Change in daily cases</th>\n",
" <th>Cumulative lab-confirmed cases</th>\n",
" <th>Previously reported cumulative cases</th>\n",
" <th>Change in cumulative cases</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>England</td>\n",
" <td>E92000001</td>\n",
" <td>Nation</td>\n",
" <td>2020-04-27</td>\n",
" <td>103.0</td>\n",
" <td>0.0</td>\n",
" <td>103.0</td>\n",
" <td>114456</td>\n",
" <td>112467.0</td>\n",
" <td>1989.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>South West</td>\n",
" <td>E12000009</td>\n",
" <td>Region</td>\n",
" <td>2020-04-27</td>\n",
" <td>4.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>5986</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>South East</td>\n",
" <td>E12000008</td>\n",
" <td>Region</td>\n",
" <td>2020-04-27</td>\n",
" <td>7.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>16323</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>London</td>\n",
" <td>E12000007</td>\n",
" <td>Region</td>\n",
" <td>2020-04-27</td>\n",
" <td>6.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>23979</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>East of England</td>\n",
" <td>E12000006</td>\n",
" <td>Region</td>\n",
" <td>2020-04-27</td>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>9907</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Area name Area code Area type Specimen date \\\n",
"0 England E92000001 Nation 2020-04-27 \n",
"1 South West E12000009 Region 2020-04-27 \n",
"2 South East E12000008 Region 2020-04-27 \n",
"3 London E12000007 Region 2020-04-27 \n",
"4 East of England E12000006 Region 2020-04-27 \n",
"\n",
" Daily lab-confirmed cases Previously reported daily cases \\\n",
"0 103.0 0.0 \n",
"1 4.0 NaN \n",
"2 7.0 NaN \n",
"3 6.0 NaN \n",
"4 1.0 NaN \n",
"\n",
" Change in daily cases Cumulative lab-confirmed cases \\\n",
"0 103.0 114456 \n",
"1 NaN 5986 \n",
"2 NaN 16323 \n",
"3 NaN 23979 \n",
"4 NaN 9907 \n",
"\n",
" Previously reported cumulative cases Change in cumulative cases \n",
"0 112467.0 1989.0 \n",
"1 NaN NaN \n",
"2 NaN NaN \n",
"3 NaN NaN \n",
"4 NaN NaN "
]
},
"execution_count": 44,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"phe_deaths_url = 'https://coronavirus.data.gov.uk/downloads/csv/coronavirus-deaths_latest.csv'\n",
"phe_deaths_df = get_308_csv(phe_cases_url)\n",
"\n",
"_table = f'phe_deaths'\n",
"phe_cases_df.to_sql(_table, DB.conn, index=False, if_exists='append')\n",
"\n",
"phe_deaths_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Area name</th>\n",
" <th>Area code</th>\n",
" <th>Area type</th>\n",
" <th>Specimen date</th>\n",
" <th>Daily lab-confirmed cases</th>\n",
" <th>Previously reported daily cases</th>\n",
" <th>Change in daily cases</th>\n",
" <th>Cumulative lab-confirmed cases</th>\n",
" <th>Previously reported cumulative cases</th>\n",
" <th>Change in cumulative cases</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>England</td>\n",
" <td>E92000001</td>\n",
" <td>Nation</td>\n",
" <td>2020-04-27 00:00:00</td>\n",
" <td>103.0</td>\n",
" <td>0.0</td>\n",
" <td>103.0</td>\n",
" <td>114456</td>\n",
" <td>112467.0</td>\n",
" <td>1989.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>South West</td>\n",
" <td>E12000009</td>\n",
" <td>Region</td>\n",
" <td>2020-04-27 00:00:00</td>\n",
" <td>4.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>5986</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>South East</td>\n",
" <td>E12000008</td>\n",
" <td>Region</td>\n",
" <td>2020-04-27 00:00:00</td>\n",
" <td>7.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>16323</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Area name Area code Area type Specimen date \\\n",
"0 England E92000001 Nation 2020-04-27 00:00:00 \n",
"1 South West E12000009 Region 2020-04-27 00:00:00 \n",
"2 South East E12000008 Region 2020-04-27 00:00:00 \n",
"\n",
" Daily lab-confirmed cases Previously reported daily cases \\\n",
"0 103.0 0.0 \n",
"1 4.0 NaN \n",
"2 7.0 NaN \n",
"\n",
" Change in daily cases Cumulative lab-confirmed cases \\\n",
"0 103.0 114456 \n",
"1 NaN 5986 \n",
"2 NaN 16323 \n",
"\n",
" Previously reported cumulative cases Change in cumulative cases \n",
"0 112467.0 1989.0 \n",
"1 NaN NaN \n",
"2 NaN NaN "
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.read_sql(\"SELECT * FROM phe_deaths LIMIT 3\", DB.conn)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### NHS - A&E\n",
"\n",
"Monthly data:\n",
"https://www.england.nhs.uk/statistics/statistical-work-areas/ae-waiting-times-and-activity/ae-attendances-and-emergency-admissions-2019-20/\n",
"\n",
"Hospital Episode Statistics:\n",
"https://digital.nhs.uk/data-and-information/publications/statistical/hospital-episode-statistics-for-admitted-patient-care-outpatient-and-accident-and-emergency-data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## ONS\n",
"\n",
"Death registrations, 2020: https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/causesofdeath/datasets/deathregistrationsandoccurrencesbylocalauthorityandhealthboard\n",
"\n",
"Weekly Death registrations (provisional):\n",
"https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/datasets/weeklyprovisionalfiguresondeathsregisteredinenglandandwales\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Weekly deaths, ONS:"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [],
"source": [
"ons_weekly_url = 'https://www.ons.gov.uk/file?uri=%2fpeoplepopulationandcommunity%2fbirthsdeathsandmarriages%2fdeaths%2fdatasets%2fweeklyprovisionalfiguresondeathsregisteredinenglandandwales%2f2020/publishedweek1620201.xlsx'\n"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"odict_keys(['Analysis', 'Contents', 'Information', 'Terms and conditions', 'Weekly figures 2020', 'Covid-19 - Weekly registrations', 'Covid-19 - Weekly occurrences', 'UK - Covid-19 - Weekly reg', 'Covid-19 - E&W comparisons', 'Covid-19 - England comparisons', 'Covid-19 - Wales comparison', 'Covid-19 - Place of occurrence ', 'Related publications'])"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"r = requests.get(ons_weekly_url)\n",
"\n",
"fn = ons_weekly_url.split('/')[-1]\n",
" \n",
"with open(fn, 'wb') as f:\n",
" f.write(r.content)\n",
"\n",
"ons_sheets = pd.read_excel(fn, sheet_name=None)\n",
"\n",
"# What sheets are available in the spreadsheet\n",
"ons_sheet_names = ons_sheets.keys()\n",
"ons_sheet_names"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Contents</th>\n",
" <th>Unnamed: 1</th>\n",
" <th>Unnamed: 2</th>\n",
" <th>Unnamed: 3</th>\n",
" <th>Unnamed: 4</th>\n",
" <th>Unnamed: 5</th>\n",
" <th>Unnamed: 6</th>\n",
" <th>Unnamed: 7</th>\n",
" <th>Unnamed: 8</th>\n",
" <th>Unnamed: 9</th>\n",
" <th>...</th>\n",
" <th>Unnamed: 45</th>\n",
" <th>Unnamed: 46</th>\n",
" <th>Unnamed: 47</th>\n",
" <th>Unnamed: 48</th>\n",
" <th>Unnamed: 49</th>\n",
" <th>Unnamed: 50</th>\n",
" <th>Unnamed: 51</th>\n",
" <th>Unnamed: 52</th>\n",
" <th>Unnamed: 53</th>\n",
" <th>Unnamed: 54</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Weekly provisional figures on deaths registere...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Note that up-to-date counts of the total numbe...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Week number</td>\n",
" <td>NaN</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" <td>...</td>\n",
" <td>44</td>\n",
" <td>45</td>\n",
" <td>46</td>\n",
" <td>47</td>\n",
" <td>48</td>\n",
" <td>49</td>\n",
" <td>50</td>\n",
" <td>51</td>\n",
" <td>52</td>\n",
" <td>53</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Week ended</td>\n",
" <td>NaN</td>\n",
" <td>2020-01-03 00:00:00</td>\n",
" <td>2020-01-10 00:00:00</td>\n",
" <td>2020-01-17 00:00:00</td>\n",
" <td>2020-01-24 00:00:00</td>\n",
" <td>2020-01-31 00:00:00</td>\n",
" <td>2020-02-07 00:00:00</td>\n",
" <td>2020-02-14 00:00:00</td>\n",
" <td>2020-02-21 00:00:00</td>\n",
" <td>...</td>\n",
" <td>2020-10-30 00:00:00</td>\n",
" <td>2020-11-06 00:00:00</td>\n",
" <td>2020-11-13 00:00:00</td>\n",
" <td>2020-11-20 00:00:00</td>\n",
" <td>2020-11-27 00:00:00</td>\n",
" <td>2020-12-04 00:00:00</td>\n",
" <td>2020-12-11 00:00:00</td>\n",
" <td>2020-12-18 00:00:00</td>\n",
" <td>2020-12-25 00:00:00</td>\n",
" <td>2021-01-01 00:00:00</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 55 columns</p>\n",
"</div>"
],
"text/plain": [
" Contents Unnamed: 1 \\\n",
"0 Weekly provisional figures on deaths registere... NaN \n",
"1 NaN NaN \n",
"2 Note that up-to-date counts of the total numbe... NaN \n",
"3 Week number NaN \n",
"4 Week ended NaN \n",
"\n",
" Unnamed: 2 Unnamed: 3 Unnamed: 4 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 1 2 3 \n",
"4 2020-01-03 00:00:00 2020-01-10 00:00:00 2020-01-17 00:00:00 \n",
"\n",
" Unnamed: 5 Unnamed: 6 Unnamed: 7 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 4 5 6 \n",
"4 2020-01-24 00:00:00 2020-01-31 00:00:00 2020-02-07 00:00:00 \n",
"\n",
" Unnamed: 8 Unnamed: 9 ... Unnamed: 45 \\\n",
"0 NaN NaN ... NaN \n",
"1 NaN NaN ... NaN \n",
"2 NaN NaN ... NaN \n",
"3 7 8 ... 44 \n",
"4 2020-02-14 00:00:00 2020-02-21 00:00:00 ... 2020-10-30 00:00:00 \n",
"\n",
" Unnamed: 46 Unnamed: 47 Unnamed: 48 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 45 46 47 \n",
"4 2020-11-06 00:00:00 2020-11-13 00:00:00 2020-11-20 00:00:00 \n",
"\n",
" Unnamed: 49 Unnamed: 50 Unnamed: 51 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 48 49 50 \n",
"4 2020-11-27 00:00:00 2020-12-04 00:00:00 2020-12-11 00:00:00 \n",
"\n",
" Unnamed: 52 Unnamed: 53 Unnamed: 54 \n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 51 52 53 \n",
"4 2020-12-18 00:00:00 2020-12-25 00:00:00 2021-01-01 00:00:00 \n",
"\n",
"[5 rows x 55 columns]"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ons_weekly_reg = ons_sheets['Covid-19 - Weekly registrations']\n",
"ons_weekly_reg.head()"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Contents</th>\n",
" <th>Unnamed: 1</th>\n",
" <th>Unnamed: 2</th>\n",
" <th>Unnamed: 3</th>\n",
" <th>Unnamed: 4</th>\n",
" <th>Unnamed: 5</th>\n",
" <th>Unnamed: 6</th>\n",
" <th>Unnamed: 7</th>\n",
" <th>Unnamed: 8</th>\n",
" <th>Unnamed: 9</th>\n",
" <th>...</th>\n",
" <th>Unnamed: 46</th>\n",
" <th>Unnamed: 47</th>\n",
" <th>Unnamed: 48</th>\n",
" <th>Unnamed: 49</th>\n",
" <th>Unnamed: 50</th>\n",
" <th>Unnamed: 51</th>\n",
" <th>Unnamed: 52</th>\n",
" <th>Unnamed: 53</th>\n",
" <th>Unnamed: 54</th>\n",
" <th>Unnamed: 55</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Weekly provisional figures on death occurrence...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Note that up-to-date counts of the total numbe...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Week number</td>\n",
" <td>NaN</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" <td>...</td>\n",
" <td>44</td>\n",
" <td>45</td>\n",
" <td>46</td>\n",
" <td>47</td>\n",
" <td>48</td>\n",
" <td>49</td>\n",
" <td>50</td>\n",
" <td>51</td>\n",
" <td>52</td>\n",
" <td>53</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Week ended</td>\n",
" <td>NaN</td>\n",
" <td>2020-01-03 00:00:00</td>\n",
" <td>2020-01-10 00:00:00</td>\n",
" <td>2020-01-17 00:00:00</td>\n",
" <td>2020-01-24 00:00:00</td>\n",
" <td>2020-01-31 00:00:00</td>\n",
" <td>2020-02-07 00:00:00</td>\n",
" <td>2020-02-14 00:00:00</td>\n",
" <td>2020-02-21 00:00:00</td>\n",
" <td>...</td>\n",
" <td>2020-10-30 00:00:00</td>\n",
" <td>2020-11-06 00:00:00</td>\n",
" <td>2020-11-13 00:00:00</td>\n",
" <td>2020-11-20 00:00:00</td>\n",
" <td>2020-11-27 00:00:00</td>\n",
" <td>2020-12-04 00:00:00</td>\n",
" <td>2020-12-11 00:00:00</td>\n",
" <td>2020-12-18 00:00:00</td>\n",
" <td>2020-12-25 00:00:00</td>\n",
" <td>2021-01-01 00:00:00</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 56 columns</p>\n",
"</div>"
],
"text/plain": [
" Contents Unnamed: 1 \\\n",
"0 Weekly provisional figures on death occurrence... NaN \n",
"1 NaN NaN \n",
"2 Note that up-to-date counts of the total numbe... NaN \n",
"3 Week number NaN \n",
"4 Week ended NaN \n",
"\n",
" Unnamed: 2 Unnamed: 3 Unnamed: 4 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 1 2 3 \n",
"4 2020-01-03 00:00:00 2020-01-10 00:00:00 2020-01-17 00:00:00 \n",
"\n",
" Unnamed: 5 Unnamed: 6 Unnamed: 7 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 4 5 6 \n",
"4 2020-01-24 00:00:00 2020-01-31 00:00:00 2020-02-07 00:00:00 \n",
"\n",
" Unnamed: 8 Unnamed: 9 ... Unnamed: 46 \\\n",
"0 NaN NaN ... NaN \n",
"1 NaN NaN ... NaN \n",
"2 NaN NaN ... NaN \n",
"3 7 8 ... 44 \n",
"4 2020-02-14 00:00:00 2020-02-21 00:00:00 ... 2020-10-30 00:00:00 \n",
"\n",
" Unnamed: 47 Unnamed: 48 Unnamed: 49 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 45 46 47 \n",
"4 2020-11-06 00:00:00 2020-11-13 00:00:00 2020-11-20 00:00:00 \n",
"\n",
" Unnamed: 50 Unnamed: 51 Unnamed: 52 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 48 49 50 \n",
"4 2020-11-27 00:00:00 2020-12-04 00:00:00 2020-12-11 00:00:00 \n",
"\n",
" Unnamed: 53 Unnamed: 54 Unnamed: 55 \n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 51 52 53 \n",
"4 2020-12-18 00:00:00 2020-12-25 00:00:00 2021-01-01 00:00:00 \n",
"\n",
"[5 rows x 56 columns]"
]
},
"execution_count": 49,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ons_weekly_occ = ons_sheets['Covid-19 - Weekly occurrences']\n",
"ons_weekly_occ.head()"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [],
"source": [
" def ons_weeklies(ons_weekly, typ):\n",
" ons_weekly_long = {}\n",
" rows, cols = np.where(ons_weekly == 'Week ended')\n",
" colnames = ons_weekly.iloc[rows[0]].tolist()\n",
" colnames[1] = 'Age'\n",
"\n",
" rows, cols = np.where(ons_weekly == 'Deaths by age group')\n",
" _rows, _ = np.where(ons_weekly == '90+')\n",
" _ix = rows[0]\n",
"\n",
" tables = []\n",
"\n",
"\n",
" #Get the first three tables - for Persons, Males and Females\n",
" for r, c in zip(rows, cols):\n",
" tables.append(ons_weekly.iloc[r-1, c].split()[0])\n",
"\n",
" for r, _r, t in zip(rows, _rows, tables):\n",
" ons_weekly_long[t] = ons_weekly.iloc[r+1: _r+1]\n",
" ons_weekly_long[t].columns = colnames\n",
" ons_weekly_long[t].dropna(axis=1, how='all', inplace=True)\n",
" if 'Year to date' in ons_weekly_long[t].columns:\n",
" ons_weekly_long[t].drop(columns=['Year to date'], inplace=True)\n",
" ons_weekly_long[t] = ons_weekly_long[t].melt(id_vars=['Age'], var_name='Date', value_name='value')\n",
" ons_weekly_long[t]['measure'] = typ\n",
" ons_weekly_long[t]['Date'] = pd.to_datetime(ons_weekly_long[t]['Date'])\n",
"\n",
" ons_weekly_long['Any'] = pd.DataFrame()\n",
" for t in tables:\n",
" ons_weekly_long[t]['Group'] = t\n",
" ons_weekly_long['Any'] = pd.concat([ons_weekly_long['Any'], ons_weekly_long[t]])\n",
" \n",
" ons_weekly_long['Any'].reset_index(inplace=True, drop=True)\n",
" \n",
" return ons_weekly_long"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/site-packages/ipykernel_launcher.py:21: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame\n",
"\n",
"See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Age</th>\n",
" <th>Date</th>\n",
" <th>value</th>\n",
" <th>measure</th>\n",
" <th>Group</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>&lt;1</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1-4</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>5-9</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>10-14</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>15-19</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>315</th>\n",
" <td>70-74</td>\n",
" <td>2020-04-17</td>\n",
" <td>298</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" <tr>\n",
" <th>316</th>\n",
" <td>75-79</td>\n",
" <td>2020-04-17</td>\n",
" <td>465</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" <tr>\n",
" <th>317</th>\n",
" <td>80-84</td>\n",
" <td>2020-04-17</td>\n",
" <td>652</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" <tr>\n",
" <th>318</th>\n",
" <td>85-89</td>\n",
" <td>2020-04-17</td>\n",
" <td>781</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" <tr>\n",
" <th>319</th>\n",
" <td>90+</td>\n",
" <td>2020-04-17</td>\n",
" <td>954</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>320 rows × 5 columns</p>\n",
"</div>"
],
"text/plain": [
" Age Date value measure Group\n",
"0 <1 2020-01-03 0 Weekly registrations Females\n",
"1 1-4 2020-01-03 0 Weekly registrations Females\n",
"2 5-9 2020-01-03 0 Weekly registrations Females\n",
"3 10-14 2020-01-03 0 Weekly registrations Females\n",
"4 15-19 2020-01-03 0 Weekly registrations Females\n",
".. ... ... ... ... ...\n",
"315 70-74 2020-04-17 298 Weekly registrations Females\n",
"316 75-79 2020-04-17 465 Weekly registrations Females\n",
"317 80-84 2020-04-17 652 Weekly registrations Females\n",
"318 85-89 2020-04-17 781 Weekly registrations Females\n",
"319 90+ 2020-04-17 954 Weekly registrations Females\n",
"\n",
"[320 rows x 5 columns]"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ons_weekly_reg_long = ons_weeklies(ons_weekly_reg, 'Weekly registrations')\n",
"ons_weekly_reg_long['Females']"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Age</th>\n",
" <th>Date</th>\n",
" <th>value</th>\n",
" <th>measure</th>\n",
" <th>Group</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>&lt;1</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Persons</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1-4</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Persons</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>5-9</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Persons</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>10-14</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Persons</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>15-19</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Persons</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>955</th>\n",
" <td>70-74</td>\n",
" <td>2020-04-17</td>\n",
" <td>298</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" <tr>\n",
" <th>956</th>\n",
" <td>75-79</td>\n",
" <td>2020-04-17</td>\n",
" <td>465</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" <tr>\n",
" <th>957</th>\n",
" <td>80-84</td>\n",
" <td>2020-04-17</td>\n",
" <td>652</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" <tr>\n",
" <th>958</th>\n",
" <td>85-89</td>\n",
" <td>2020-04-17</td>\n",
" <td>781</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" <tr>\n",
" <th>959</th>\n",
" <td>90+</td>\n",
" <td>2020-04-17</td>\n",
" <td>954</td>\n",
" <td>Weekly registrations</td>\n",
" <td>Females</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>960 rows × 5 columns</p>\n",
"</div>"
],
"text/plain": [
" Age Date value measure Group\n",
"0 <1 2020-01-03 0 Weekly registrations Persons\n",
"1 1-4 2020-01-03 0 Weekly registrations Persons\n",
"2 5-9 2020-01-03 0 Weekly registrations Persons\n",
"3 10-14 2020-01-03 0 Weekly registrations Persons\n",
"4 15-19 2020-01-03 0 Weekly registrations Persons\n",
".. ... ... ... ... ...\n",
"955 70-74 2020-04-17 298 Weekly registrations Females\n",
"956 75-79 2020-04-17 465 Weekly registrations Females\n",
"957 80-84 2020-04-17 652 Weekly registrations Females\n",
"958 85-89 2020-04-17 781 Weekly registrations Females\n",
"959 90+ 2020-04-17 954 Weekly registrations Females\n",
"\n",
"[960 rows x 5 columns]"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ons_weekly_reg_long['Any']"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/site-packages/ipykernel_launcher.py:21: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame\n",
"\n",
"See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
"/usr/local/lib/python3.7/site-packages/pandas/core/frame.py:4117: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame\n",
"\n",
"See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
" errors=errors,\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Age</th>\n",
" <th>Date</th>\n",
" <th>value</th>\n",
" <th>measure</th>\n",
" <th>Group</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>&lt;1</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly occurrences</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1-4</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly occurrences</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>5-9</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly occurrences</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>10-14</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly occurrences</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>15-19</td>\n",
" <td>2020-01-03</td>\n",
" <td>0</td>\n",
" <td>Weekly occurrences</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>315</th>\n",
" <td>70-74</td>\n",
" <td>2020-04-17</td>\n",
" <td>459</td>\n",
" <td>Weekly occurrences</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>316</th>\n",
" <td>75-79</td>\n",
" <td>2020-04-17</td>\n",
" <td>597</td>\n",
" <td>Weekly occurrences</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>317</th>\n",
" <td>80-84</td>\n",
" <td>2020-04-17</td>\n",
" <td>803</td>\n",
" <td>Weekly occurrences</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>318</th>\n",
" <td>85-89</td>\n",
" <td>2020-04-17</td>\n",
" <td>793</td>\n",
" <td>Weekly occurrences</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>319</th>\n",
" <td>90+</td>\n",
" <td>2020-04-17</td>\n",
" <td>627</td>\n",
" <td>Weekly occurrences</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>320 rows × 5 columns</p>\n",
"</div>"
],
"text/plain": [
" Age Date value measure Group\n",
"0 <1 2020-01-03 0 Weekly occurrences Males\n",
"1 1-4 2020-01-03 0 Weekly occurrences Males\n",
"2 5-9 2020-01-03 0 Weekly occurrences Males\n",
"3 10-14 2020-01-03 0 Weekly occurrences Males\n",
"4 15-19 2020-01-03 0 Weekly occurrences Males\n",
".. ... ... ... ... ...\n",
"315 70-74 2020-04-17 459 Weekly occurrences Males\n",
"316 75-79 2020-04-17 597 Weekly occurrences Males\n",
"317 80-84 2020-04-17 803 Weekly occurrences Males\n",
"318 85-89 2020-04-17 793 Weekly occurrences Males\n",
"319 90+ 2020-04-17 627 Weekly occurrences Males\n",
"\n",
"[320 rows x 5 columns]"
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ons_weekly_occ_long = ons_weeklies(ons_weekly_occ, 'Weekly occurrences')\n",
"ons_weekly_occ_long['Males']"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Contents</th>\n",
" <th>Unnamed: 1</th>\n",
" <th>Unnamed: 2</th>\n",
" <th>Unnamed: 3</th>\n",
" <th>Unnamed: 4</th>\n",
" <th>Unnamed: 5</th>\n",
" <th>Unnamed: 6</th>\n",
" <th>Unnamed: 7</th>\n",
" <th>Unnamed: 8</th>\n",
" <th>Unnamed: 9</th>\n",
" <th>...</th>\n",
" <th>Unnamed: 45</th>\n",
" <th>Unnamed: 46</th>\n",
" <th>Unnamed: 47</th>\n",
" <th>Unnamed: 48</th>\n",
" <th>Unnamed: 49</th>\n",
" <th>Unnamed: 50</th>\n",
" <th>Unnamed: 51</th>\n",
" <th>Unnamed: 52</th>\n",
" <th>Unnamed: 53</th>\n",
" <th>Unnamed: 54</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Weekly provisional figures on deaths registere...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Note that up-to-date counts of the total numbe...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Week number</td>\n",
" <td>NaN</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" <td>6</td>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" <td>...</td>\n",
" <td>44</td>\n",
" <td>45</td>\n",
" <td>46</td>\n",
" <td>47</td>\n",
" <td>48</td>\n",
" <td>49</td>\n",
" <td>50</td>\n",
" <td>51</td>\n",
" <td>52</td>\n",
" <td>53</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Week ended</td>\n",
" <td>NaN</td>\n",
" <td>2020-01-03 00:00:00</td>\n",
" <td>2020-01-10 00:00:00</td>\n",
" <td>2020-01-17 00:00:00</td>\n",
" <td>2020-01-24 00:00:00</td>\n",
" <td>2020-01-31 00:00:00</td>\n",
" <td>2020-02-07 00:00:00</td>\n",
" <td>2020-02-14 00:00:00</td>\n",
" <td>2020-02-21 00:00:00</td>\n",
" <td>...</td>\n",
" <td>2020-10-30 00:00:00</td>\n",
" <td>2020-11-06 00:00:00</td>\n",
" <td>2020-11-13 00:00:00</td>\n",
" <td>2020-11-20 00:00:00</td>\n",
" <td>2020-11-27 00:00:00</td>\n",
" <td>2020-12-04 00:00:00</td>\n",
" <td>2020-12-11 00:00:00</td>\n",
" <td>2020-12-18 00:00:00</td>\n",
" <td>2020-12-25 00:00:00</td>\n",
" <td>2021-01-01 00:00:00</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 55 columns</p>\n",
"</div>"
],
"text/plain": [
" Contents Unnamed: 1 \\\n",
"0 Weekly provisional figures on deaths registere... NaN \n",
"1 NaN NaN \n",
"2 Note that up-to-date counts of the total numbe... NaN \n",
"3 Week number NaN \n",
"4 Week ended NaN \n",
"\n",
" Unnamed: 2 Unnamed: 3 Unnamed: 4 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 1 2 3 \n",
"4 2020-01-03 00:00:00 2020-01-10 00:00:00 2020-01-17 00:00:00 \n",
"\n",
" Unnamed: 5 Unnamed: 6 Unnamed: 7 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 4 5 6 \n",
"4 2020-01-24 00:00:00 2020-01-31 00:00:00 2020-02-07 00:00:00 \n",
"\n",
" Unnamed: 8 Unnamed: 9 ... Unnamed: 45 \\\n",
"0 NaN NaN ... NaN \n",
"1 NaN NaN ... NaN \n",
"2 NaN NaN ... NaN \n",
"3 7 8 ... 44 \n",
"4 2020-02-14 00:00:00 2020-02-21 00:00:00 ... 2020-10-30 00:00:00 \n",
"\n",
" Unnamed: 46 Unnamed: 47 Unnamed: 48 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 45 46 47 \n",
"4 2020-11-06 00:00:00 2020-11-13 00:00:00 2020-11-20 00:00:00 \n",
"\n",
" Unnamed: 49 Unnamed: 50 Unnamed: 51 \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 48 49 50 \n",
"4 2020-11-27 00:00:00 2020-12-04 00:00:00 2020-12-11 00:00:00 \n",
"\n",
" Unnamed: 52 Unnamed: 53 Unnamed: 54 \n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 51 52 53 \n",
"4 2020-12-18 00:00:00 2020-12-25 00:00:00 2021-01-01 00:00:00 \n",
"\n",
"[5 rows x 55 columns]"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ons_weekly_all = ons_sheets['Weekly figures 2020']\n",
"ons_weekly_all.head()"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/site-packages/ipykernel_launcher.py:21: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame\n",
"\n",
"See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Age</th>\n",
" <th>Date</th>\n",
" <th>value</th>\n",
" <th>measure</th>\n",
" <th>Group</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>&lt;1</td>\n",
" <td>2020-01-03</td>\n",
" <td>30</td>\n",
" <td>Weekly all mortality</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1-4</td>\n",
" <td>2020-01-03</td>\n",
" <td>5</td>\n",
" <td>Weekly all mortality</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>5-9</td>\n",
" <td>2020-01-03</td>\n",
" <td>2</td>\n",
" <td>Weekly all mortality</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>10-14</td>\n",
" <td>2020-01-03</td>\n",
" <td>1</td>\n",
" <td>Weekly all mortality</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>15-19</td>\n",
" <td>2020-01-03</td>\n",
" <td>5</td>\n",
" <td>Weekly all mortality</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>315</th>\n",
" <td>70-74</td>\n",
" <td>2020-04-17</td>\n",
" <td>1280</td>\n",
" <td>Weekly all mortality</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>316</th>\n",
" <td>75-79</td>\n",
" <td>2020-04-17</td>\n",
" <td>1635</td>\n",
" <td>Weekly all mortality</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>317</th>\n",
" <td>80-84</td>\n",
" <td>2020-04-17</td>\n",
" <td>2100</td>\n",
" <td>Weekly all mortality</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>318</th>\n",
" <td>85-89</td>\n",
" <td>2020-04-17</td>\n",
" <td>2111</td>\n",
" <td>Weekly all mortality</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" <tr>\n",
" <th>319</th>\n",
" <td>90+</td>\n",
" <td>2020-04-17</td>\n",
" <td>1879</td>\n",
" <td>Weekly all mortality</td>\n",
" <td>Males</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>320 rows × 5 columns</p>\n",
"</div>"
],
"text/plain": [
" Age Date value measure Group\n",
"0 <1 2020-01-03 30 Weekly all mortality Males\n",
"1 1-4 2020-01-03 5 Weekly all mortality Males\n",
"2 5-9 2020-01-03 2 Weekly all mortality Males\n",
"3 10-14 2020-01-03 1 Weekly all mortality Males\n",
"4 15-19 2020-01-03 5 Weekly all mortality Males\n",
".. ... ... ... ... ...\n",
"315 70-74 2020-04-17 1280 Weekly all mortality Males\n",
"316 75-79 2020-04-17 1635 Weekly all mortality Males\n",
"317 80-84 2020-04-17 2100 Weekly all mortality Males\n",
"318 85-89 2020-04-17 2111 Weekly all mortality Males\n",
"319 90+ 2020-04-17 1879 Weekly all mortality Males\n",
"\n",
"[320 rows x 5 columns]"
]
},
"execution_count": 55,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ons_weekly_all_long = ons_weeklies(ons_weekly_all, 'Weekly all mortality')\n",
"ons_weekly_all_long['Males']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Add to database..."
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [],
"source": [
"_table = 'ons_deaths'\n",
"\n",
"ons_weekly_occ_long['Any'].to_sql(_table, DB.conn, index=False, if_exists='append')\n",
"ons_weekly_reg_long['Any'].to_sql(_table, DB.conn, index=False, if_exists='append')\n",
"ons_weekly_all_long['Any'].to_sql(_table, DB.conn, index=False, if_exists='append')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ONS Death Registrations, 2020\n",
"\n",
"https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/causesofdeath/datasets/deathregistrationsandoccurrencesbylocalauthorityandhealthboard"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [],
"source": [
"ons_death_reg_url = 'https://www.ons.gov.uk/file?uri=%2fpeoplepopulationandcommunity%2fhealthandsocialcare%2fcausesofdeath%2fdatasets%2fdeathregistrationsandoccurrencesbylocalauthorityandhealthboard%2f2020/lahbtablesweek16.xlsx'"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"odict_keys(['Contents', 'Information', 'Terms and conditions', 'Registrations - All data', 'Registrations - Pivot table', 'Occurrences - All data', 'Occurrences - Pivot table'])"
]
},
"execution_count": 58,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"r = requests.get(ons_death_reg_url)\n",
"\n",
"fn = ons_death_reg_url.split('/')[-1]\n",
" \n",
"with open(fn, 'wb') as f:\n",
" f.write(r.content)\n",
"\n",
"ons_reg_sheets = pd.read_excel(fn, sheet_name=None)\n",
"\n",
"# What sheets are available in the spreadsheet\n",
"ons_reg_sheet_names = ons_reg_sheets.keys()\n",
"ons_reg_sheet_names"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'Deaths (numbers) by local authority and cause of death, registered up to the 17th April 2020, England and Wales'"
]
},
"execution_count": 59,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ons_death_reg = ons_reg_sheets['Registrations - All data']\n",
"ons_death_reg_metadata = ons_death_reg.iloc[0, 0]\n",
"ons_death_reg_metadata"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Area code</th>\n",
" <th>Geography type</th>\n",
" <th>Area name</th>\n",
" <th>Cause of death</th>\n",
" <th>Week number</th>\n",
" <th>Place of death</th>\n",
" <th>Number of deaths</th>\n",
" <th>Registered up to</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>E06000001</td>\n",
" <td>Local Authority</td>\n",
" <td>Hartlepool</td>\n",
" <td>All causes</td>\n",
" <td>1</td>\n",
" <td>Care home</td>\n",
" <td>8</td>\n",
" <td>2020-04-17</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>E06000001</td>\n",
" <td>Local Authority</td>\n",
" <td>Hartlepool</td>\n",
" <td>COVID 19</td>\n",
" <td>1</td>\n",
" <td>Care home</td>\n",
" <td>0</td>\n",
" <td>2020-04-17</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>E06000001</td>\n",
" <td>Local Authority</td>\n",
" <td>Hartlepool</td>\n",
" <td>All causes</td>\n",
" <td>1</td>\n",
" <td>Elsewhere</td>\n",
" <td>0</td>\n",
" <td>2020-04-17</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>E06000001</td>\n",
" <td>Local Authority</td>\n",
" <td>Hartlepool</td>\n",
" <td>COVID 19</td>\n",
" <td>1</td>\n",
" <td>Elsewhere</td>\n",
" <td>0</td>\n",
" <td>2020-04-17</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>E06000001</td>\n",
" <td>Local Authority</td>\n",
" <td>Hartlepool</td>\n",
" <td>All causes</td>\n",
" <td>1</td>\n",
" <td>Home</td>\n",
" <td>2</td>\n",
" <td>2020-04-17</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>64766</th>\n",
" <td>W11000030</td>\n",
" <td>Health Board</td>\n",
" <td>Cwm Taf Morgannwg University Health Board</td>\n",
" <td>COVID 19</td>\n",
" <td>16</td>\n",
" <td>Home</td>\n",
" <td>4</td>\n",
" <td>2020-04-17</td>\n",
" </tr>\n",
" <tr>\n",
" <th>64767</th>\n",
" <td>W11000030</td>\n",
" <td>Health Board</td>\n",
" <td>Cwm Taf Morgannwg University Health Board</td>\n",
" <td>COVID 19</td>\n",
" <td>16</td>\n",
" <td>Hospital</td>\n",
" <td>53</td>\n",
" <td>2020-04-17</td>\n",
" </tr>\n",
" <tr>\n",
" <th>64768</th>\n",
" <td>W11000031</td>\n",
" <td>Health Board</td>\n",
" <td>Swansea Bay University Health Board</td>\n",
" <td>COVID 19</td>\n",
" <td>16</td>\n",
" <td>Care home</td>\n",
" <td>6</td>\n",
" <td>2020-04-17</td>\n",
" </tr>\n",
" <tr>\n",
" <th>64769</th>\n",
" <td>W11000031</td>\n",
" <td>Health Board</td>\n",
" <td>Swansea Bay University Health Board</td>\n",
" <td>COVID 19</td>\n",
" <td>16</td>\n",
" <td>Hospital</td>\n",
" <td>47</td>\n",
" <td>2020-04-17</td>\n",
" </tr>\n",
" <tr>\n",
" <th>64770</th>\n",
" <td>W11000031</td>\n",
" <td>Health Board</td>\n",
" <td>Swansea Bay University Health Board</td>\n",
" <td>COVID 19</td>\n",
" <td>16</td>\n",
" <td>Other communal establishment</td>\n",
" <td>6</td>\n",
" <td>2020-04-17</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>64771 rows × 8 columns</p>\n",
"</div>"
],
"text/plain": [
" Area code Geography type Area name \\\n",
"0 E06000001 Local Authority Hartlepool \n",
"1 E06000001 Local Authority Hartlepool \n",
"2 E06000001 Local Authority Hartlepool \n",
"3 E06000001 Local Authority Hartlepool \n",
"4 E06000001 Local Authority Hartlepool \n",
"... ... ... ... \n",
"64766 W11000030 Health Board Cwm Taf Morgannwg University Health Board \n",
"64767 W11000030 Health Board Cwm Taf Morgannwg University Health Board \n",
"64768 W11000031 Health Board Swansea Bay University Health Board \n",
"64769 W11000031 Health Board Swansea Bay University Health Board \n",
"64770 W11000031 Health Board Swansea Bay University Health Board \n",
"\n",
" Cause of death Week number Place of death \\\n",
"0 All causes 1 Care home \n",
"1 COVID 19 1 Care home \n",
"2 All causes 1 Elsewhere \n",
"3 COVID 19 1 Elsewhere \n",
"4 All causes 1 Home \n",
"... ... ... ... \n",
"64766 COVID 19 16 Home \n",
"64767 COVID 19 16 Hospital \n",
"64768 COVID 19 16 Care home \n",
"64769 COVID 19 16 Hospital \n",
"64770 COVID 19 16 Other communal establishment \n",
"\n",
" Number of deaths Registered up to \n",
"0 8 2020-04-17 \n",
"1 0 2020-04-17 \n",
"2 0 2020-04-17 \n",
"3 0 2020-04-17 \n",
"4 2 2020-04-17 \n",
"... ... ... \n",
"64766 4 2020-04-17 \n",
"64767 53 2020-04-17 \n",
"64768 6 2020-04-17 \n",
"64769 47 2020-04-17 \n",
"64770 6 2020-04-17 \n",
"\n",
"[64771 rows x 8 columns]"
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from parse import parse\n",
"import dateparser\n",
"\n",
"upto = parse('Deaths (numbers) by local authority and cause of death, registered up to the {date}, England and Wales',\n",
" ons_death_reg_metadata)['date']\n",
"upto = dateparser.parse(upto)\n",
"\n",
"rows, cols = np.where(ons_death_reg == 'Area code')\n",
"colnames = ons_death_reg.iloc[rows[0]].tolist()\n",
" \n",
"ons_death_reg = ons_death_reg.iloc[rows[0]+1:].reset_index(drop=True)\n",
"ons_death_reg.columns = colnames\n",
"\n",
"\n",
"ons_death_reg['Registered up to'] = upto\n",
"ons_death_reg"
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Area code</th>\n",
" <th>Geography type</th>\n",
" <th>Area name</th>\n",
" <th>Cause of death</th>\n",
" <th>Week number</th>\n",
" <th>Place of death</th>\n",
" <th>Number of deaths</th>\n",
" <th>Occurred up to</th>\n",
" <th>Registered up to</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>E06000001</td>\n",
" <td>Local Authority</td>\n",
" <td>Hartlepool</td>\n",
" <td>All causes</td>\n",
" <td>1</td>\n",
" <td>Care home</td>\n",
" <td>3</td>\n",
" <td>2020-04-17</td>\n",
" <td>2020-04-25</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>E06000001</td>\n",
" <td>Local Authority</td>\n",
" <td>Hartlepool</td>\n",
" <td>All causes</td>\n",
" <td>1</td>\n",
" <td>Elsewhere</td>\n",
" <td>0</td>\n",
" <td>2020-04-17</td>\n",
" <td>2020-04-25</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>E06000001</td>\n",
" <td>Local Authority</td>\n",
" <td>Hartlepool</td>\n",
" <td>All causes</td>\n",
" <td>1</td>\n",
" <td>Home</td>\n",
" <td>7</td>\n",
" <td>2020-04-17</td>\n",
" <td>2020-04-25</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>E06000001</td>\n",
" <td>Local Authority</td>\n",
" <td>Hartlepool</td>\n",
" <td>All causes</td>\n",
" <td>1</td>\n",
" <td>Hospice</td>\n",
" <td>1</td>\n",
" <td>2020-04-17</td>\n",
" <td>2020-04-25</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>E06000001</td>\n",
" <td>Local Authority</td>\n",
" <td>Hartlepool</td>\n",
" <td>All causes</td>\n",
" <td>1</td>\n",
" <td>Hospital</td>\n",
" <td>13</td>\n",
" <td>2020-04-17</td>\n",
" <td>2020-04-25</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>66427</th>\n",
" <td>W11000031</td>\n",
" <td>Health Board</td>\n",
" <td>Swansea Bay University Health Board</td>\n",
" <td>COVID 19</td>\n",
" <td>16</td>\n",
" <td>Elsewhere</td>\n",
" <td>0</td>\n",
" <td>2020-04-17</td>\n",
" <td>2020-04-25</td>\n",
" </tr>\n",
" <tr>\n",
" <th>66428</th>\n",
" <td>W11000031</td>\n",
" <td>Health Board</td>\n",
" <td>Swansea Bay University Health Board</td>\n",
" <td>COVID 19</td>\n",
" <td>16</td>\n",
" <td>Home</td>\n",
" <td>1</td>\n",
" <td>2020-04-17</td>\n",
" <td>2020-04-25</td>\n",
" </tr>\n",
" <tr>\n",
" <th>66429</th>\n",
" <td>W11000031</td>\n",
" <td>Health Board</td>\n",
" <td>Swansea Bay University Health Board</td>\n",
" <td>COVID 19</td>\n",
" <td>16</td>\n",
" <td>Hospice</td>\n",
" <td>0</td>\n",
" <td>2020-04-17</td>\n",
" <td>2020-04-25</td>\n",
" </tr>\n",
" <tr>\n",
" <th>66430</th>\n",
" <td>W11000031</td>\n",
" <td>Health Board</td>\n",
" <td>Swansea Bay University Health Board</td>\n",
" <td>COVID 19</td>\n",
" <td>16</td>\n",
" <td>Hospital</td>\n",
" <td>40</td>\n",
" <td>2020-04-17</td>\n",
" <td>2020-04-25</td>\n",
" </tr>\n",
" <tr>\n",
" <th>66431</th>\n",
" <td>W11000031</td>\n",
" <td>Health Board</td>\n",
" <td>Swansea Bay University Health Board</td>\n",
" <td>COVID 19</td>\n",
" <td>16</td>\n",
" <td>Other communal establishment</td>\n",
" <td>4</td>\n",
" <td>2020-04-17</td>\n",
" <td>2020-04-25</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>66432 rows × 9 columns</p>\n",
"</div>"
],
"text/plain": [
" Area code Geography type Area name \\\n",
"0 E06000001 Local Authority Hartlepool \n",
"1 E06000001 Local Authority Hartlepool \n",
"2 E06000001 Local Authority Hartlepool \n",
"3 E06000001 Local Authority Hartlepool \n",
"4 E06000001 Local Authority Hartlepool \n",
"... ... ... ... \n",
"66427 W11000031 Health Board Swansea Bay University Health Board \n",
"66428 W11000031 Health Board Swansea Bay University Health Board \n",
"66429 W11000031 Health Board Swansea Bay University Health Board \n",
"66430 W11000031 Health Board Swansea Bay University Health Board \n",
"66431 W11000031 Health Board Swansea Bay University Health Board \n",
"\n",
" Cause of death Week number Place of death \\\n",
"0 All causes 1 Care home \n",
"1 All causes 1 Elsewhere \n",
"2 All causes 1 Home \n",
"3 All causes 1 Hospice \n",
"4 All causes 1 Hospital \n",
"... ... ... ... \n",
"66427 COVID 19 16 Elsewhere \n",
"66428 COVID 19 16 Home \n",
"66429 COVID 19 16 Hospice \n",
"66430 COVID 19 16 Hospital \n",
"66431 COVID 19 16 Other communal establishment \n",
"\n",
" Number of deaths Occurred up to Registered up to \n",
"0 3 2020-04-17 2020-04-25 \n",
"1 0 2020-04-17 2020-04-25 \n",
"2 7 2020-04-17 2020-04-25 \n",
"3 1 2020-04-17 2020-04-25 \n",
"4 13 2020-04-17 2020-04-25 \n",
"... ... ... ... \n",
"66427 0 2020-04-17 2020-04-25 \n",
"66428 1 2020-04-17 2020-04-25 \n",
"66429 0 2020-04-17 2020-04-25 \n",
"66430 40 2020-04-17 2020-04-25 \n",
"66431 4 2020-04-17 2020-04-25 \n",
"\n",
"[66432 rows x 9 columns]"
]
},
"execution_count": 61,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ons_death_occ = ons_reg_sheets['Occurrences - All data']\n",
"ons_death_occ_metadata = ons_death_occ.iloc[0, 0]\n",
"ons_death_occ_metadata\n",
"\n",
"uptos = parse('Deaths (numbers) by local authority and cause of death, for deaths that occurred up to {date_occ} but were registered up to the {date_reg}, England and Wales',\n",
" ons_death_occ_metadata)\n",
"\n",
"upto_occ = uptos['date_occ']\n",
"if '2020' not in upto_occ: upto_occ = f'{upto_occ} 2020'\n",
" \n",
"upto_reg = uptos['date_reg']\n",
"if '2020' not in upto_occ: upto_occ = f'{upto_reg} 2020'\n",
"\n",
"upto_occ = dateparser.parse(upto_occ)\n",
"upto_reg = dateparser.parse(upto_reg)\n",
"\n",
"rows, cols = np.where(ons_death_occ == 'Area code')\n",
"colnames = ons_death_occ.iloc[rows[0]].tolist()\n",
" \n",
"ons_death_occ = ons_death_occ.iloc[rows[0]+1:].reset_index(drop=True)\n",
"ons_death_occ.columns = colnames\n",
"\n",
"\n",
"ons_death_occ['Occurred up to'] = upto_occ\n",
"ons_death_occ['Registered up to'] = upto_reg\n",
"ons_death_occ"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [],
"source": [
"_table = 'ons_deaths_reg'\n",
"ons_death_reg.to_sql(_table, DB.conn, index=False, if_exists='append')\n",
"\n",
"_table = 'ons_deaths_reg_occ'\n",
"ons_death_occ.to_sql(_table, DB.conn, index=False, if_exists='append')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Deployment via datasette\n",
"\n",
"`datasette publish fly nhs_dailies.db --app=\"nhs-orgs\"`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Simple Chat"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {},
"outputs": [],
"source": [
"# It takes tiny amounts of code to post s/thing from a notebook to an API and display a result\n",
"\n",
"\n",
"# Create some magic to call and API\n",
"from IPython.core.magic import register_cell_magic, register_line_magic\n",
"import requests\n",
"import pandas as pd\n",
"\n",
"from urllib.parse import urlencode\n",
" \n",
"_datasette_url = 'https://nhs-orgs.fly.dev/nhs_dailies/phe_cases.csv?{}'\n",
"\n",
"@register_line_magic\n",
"def phe_cases(line):\n",
" \"Query datasette.\"\n",
" payload = {'_sort': 'rowid',\n",
" 'Area name__contains': line,\n",
" '_size': 'max'}\n",
" _url = _datasette_url.format(urlencode(payload))\n",
" return pd.read_csv( _url)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Pass a string to the API via some magic and display the result\n",
"\n",
"%phe_cases isle of wight"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Looking Inside Downloaded Zip Files"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#https://techoverflow.net/2018/01/16/downloading-reading-a-zip-file-in-memory-using-python/\n",
"import zipfile\n",
"\n",
"def download_extract_xml(url):\n",
" \"\"\"\n",
" Download a ZIP file and extract its contents in memory\n",
" yields (filename, file-like object) pairs\n",
" \"\"\"\n",
" response = requests.get(url)\n",
" with zipfile.ZipFile(io.BytesIO(response.content)) as thezip:\n",
" for zipinfo in thezip.infolist():\n",
" with thezip.open(zipinfo) as thefile:\n",
" yield zipinfo.filename, thefile\n",
" \n",
"r = download_extract_xml(ons_weekly_url)\n",
"for f in r:\n",
" print(f)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment