Skip to content

Instantly share code, notes, and snippets.

@ihnorton

ihnorton/TileDB-Test.ipynb

Last active June 24, 2020 03:49
Show Gist options
  • Save ihnorton/cd3490cf5b9d63790b1597a41020e7bb to your computer and use it in GitHub Desktop.
Save ihnorton/cd3490cf5b9d63790b1597a41020e7bb to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
TileDB-Test.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# TileDB Test\n",
"\n",
"<br>\n",
"\n",
"### Imports"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import tiledb\n",
"import requests\n",
"import io"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"\n",
"### Reading in Data\n",
"\n",
"We'll start by reading in data on the UL electricity generation between 2015 and 2019"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ccgt</th>\n",
" <th>oil</th>\n",
" <th>coal</th>\n",
" <th>nuclear</th>\n",
" <th>wind</th>\n",
" <th>ps</th>\n",
" <th>npshyd</th>\n",
" <th>ocgt</th>\n",
" <th>other</th>\n",
" <th>intfr</th>\n",
" <th>intirl</th>\n",
" <th>intned</th>\n",
" <th>intew</th>\n",
" <th>biomass</th>\n",
" <th>intnem</th>\n",
" </tr>\n",
" <tr>\n",
" <th>datetime</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2015-02-22 00:00:00</th>\n",
" <td>5293</td>\n",
" <td>0</td>\n",
" <td>12980</td>\n",
" <td>7777</td>\n",
" <td>1456</td>\n",
" <td>0</td>\n",
" <td>681</td>\n",
" <td>0</td>\n",
" <td>784</td>\n",
" <td>1496</td>\n",
" <td>0</td>\n",
" <td>1002</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-02-22 00:30:00</th>\n",
" <td>5547</td>\n",
" <td>0</td>\n",
" <td>12947</td>\n",
" <td>7772</td>\n",
" <td>1401</td>\n",
" <td>0</td>\n",
" <td>682</td>\n",
" <td>0</td>\n",
" <td>784</td>\n",
" <td>1496</td>\n",
" <td>0</td>\n",
" <td>1028</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-02-22 01:00:00</th>\n",
" <td>5168</td>\n",
" <td>0</td>\n",
" <td>13314</td>\n",
" <td>7779</td>\n",
" <td>1604</td>\n",
" <td>0</td>\n",
" <td>594</td>\n",
" <td>0</td>\n",
" <td>782</td>\n",
" <td>1496</td>\n",
" <td>0</td>\n",
" <td>982</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-02-22 01:30:00</th>\n",
" <td>4498</td>\n",
" <td>0</td>\n",
" <td>13421</td>\n",
" <td>7768</td>\n",
" <td>1548</td>\n",
" <td>0</td>\n",
" <td>592</td>\n",
" <td>0</td>\n",
" <td>781</td>\n",
" <td>1496</td>\n",
" <td>0</td>\n",
" <td>1010</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-02-22 02:00:00</th>\n",
" <td>4660</td>\n",
" <td>0</td>\n",
" <td>13304</td>\n",
" <td>7778</td>\n",
" <td>1351</td>\n",
" <td>0</td>\n",
" <td>584</td>\n",
" <td>0</td>\n",
" <td>780</td>\n",
" <td>1496</td>\n",
" <td>0</td>\n",
" <td>1016</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ccgt oil coal nuclear wind ps npshyd ocgt other \\\n",
"datetime \n",
"2015-02-22 00:00:00 5293 0 12980 7777 1456 0 681 0 784 \n",
"2015-02-22 00:30:00 5547 0 12947 7772 1401 0 682 0 784 \n",
"2015-02-22 01:00:00 5168 0 13314 7779 1604 0 594 0 782 \n",
"2015-02-22 01:30:00 4498 0 13421 7768 1548 0 592 0 781 \n",
"2015-02-22 02:00:00 4660 0 13304 7778 1351 0 584 0 780 \n",
"\n",
" intfr intirl intned intew biomass intnem \n",
"datetime \n",
"2015-02-22 00:00:00 1496 0 1002 0 0 0 \n",
"2015-02-22 00:30:00 1496 0 1028 0 0 0 \n",
"2015-02-22 01:00:00 1496 0 982 0 0 0 \n",
"2015-02-22 01:30:00 1496 0 1010 0 0 0 \n",
"2015-02-22 02:00:00 1496 0 1016 0 0 0 "
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#df_FUELHH_raw = pd.read_csv('https://raw.githubusercontent.com/AyrtonB/Sage/master/data/elec_data.csv')\n",
"df_FUELHH_raw = pd.read_csv('elec_data.csv')\n",
"\n",
"df_FUELHH = df_FUELHH_raw.drop(columns=['Unnamed: 0']).copy().astype(np.int32)\n",
"df_FUELHH.index = pd.to_datetime(df_FUELHH_raw['Unnamed: 0'])\n",
"df_FUELHH.index.name = 'datetime'\n",
"\n",
"df_FUELHH.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"\n",
"We'll quickly check the datatypes"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"ccgt int32\n",
"oil int32\n",
"coal int32\n",
"nuclear int32\n",
"wind int32\n",
"ps int32\n",
"npshyd int32\n",
"ocgt int32\n",
"other int32\n",
"intfr int32\n",
"intirl int32\n",
"intned int32\n",
"intew int32\n",
"biomass int32\n",
"intnem int32\n",
"dtype: object"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_FUELHH.dtypes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"\n",
"### Creating TileDB Array\n",
"\n",
"Now we want to save the dataframe to a TileDB array.\n",
"\n",
"To do this we'll start by defining the domain of the array. Ideally this will a datetime domain with cells every half hour, not knowing the best approach to achieve non-standard time units minutes are used instead."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Domain(Dim(name='datetime', domain=(numpy.datetime64('2015-02-22T00:00:00.000'), numpy.datetime64('2019-10-01T00:00:00.000')), tile=525600 milliseconds, dtype='datetime64[ms]'))"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"TileDB_dir = 'data/TileDB'\n",
"array_name = 'FUELHH'\n",
"\n",
"#array_dir = f'{TileDB_dir}/{array_name}'\n",
"array_dir = array_name\n",
"\n",
"if tiledb.VFS().is_dir(array_dir):\n",
" tiledb.remove(array_dir)\n",
"\n",
"dom = tiledb.Domain(tiledb.Dim(name='datetime', \n",
" domain=(np.datetime64('2015-02-22T00:00'), np.datetime64('2019-10-01T00:00')), \n",
" tile=525600, # Minutes in a year \n",
" dtype='datetime64[ms]'))\n",
"\n",
"dom"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"\n",
"We'll now define the schema, having a separate attribute for each fuel type"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"ArraySchema(\n",
" domain=Domain(*[\n",
" Dim(name='datetime', domain=(numpy.datetime64('2015-02-22T00:00:00.000'), numpy.datetime64('2019-10-01T00:00:00.000')), tile=525600 milliseconds, dtype='datetime64[ms]'),\n",
" ]),\n",
" attrs=[\n",
" Attr(name='ccgt', dtype='int32'),\n",
" Attr(name='oil', dtype='int32'),\n",
" Attr(name='coal', dtype='int32'),\n",
" Attr(name='nuclear', dtype='int32'),\n",
" Attr(name='wind', dtype='int32'),\n",
" Attr(name='ps', dtype='int32'),\n",
" Attr(name='npshyd', dtype='int32'),\n",
" Attr(name='ocgt', dtype='int32'),\n",
" Attr(name='other', dtype='int32'),\n",
" Attr(name='intfr', dtype='int32'),\n",
" Attr(name='intirl', dtype='int32'),\n",
" Attr(name='intned', dtype='int32'),\n",
" Attr(name='intew', dtype='int32'),\n",
" Attr(name='biomass', dtype='int32'),\n",
" Attr(name='intnem', dtype='int32'),\n",
" ],\n",
" cell_order='row-major',\n",
" tile_order='row-major', sparse=True)\n",
"# note: filters omitted"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"FUELHH_col_order = ['ccgt', 'oil', 'coal', 'nuclear', 'wind', 'ps', 'npshyd', 'ocgt', 'other', 'intfr', 'intirl', 'intned', 'intew', 'biomass', 'intnem']\n",
"\n",
"attrs = [tiledb.Attr(name=fuel, dtype=np.int32) for fuel in FUELHH_col_order]\n",
"schema = tiledb.ArraySchema(domain=dom, sparse=True, attrs=attrs)\n",
"\n",
"schema"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"\n",
"We're now ready to initialise our array"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"#tiledb.Array.create(array_dir.replace('/', '\\\\'), schema)\n",
"tiledb.Array.create(array_dir, schema)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"\n",
"We'll put the values into a dictionary of value arrays, with each key mapping to an attribute in the TileDB array"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'ccgt': array([5293, 5547, 5168, ..., 4502, 3893, 4460], dtype=int32),\n",
" 'oil': array([0, 0, 0, ..., 0, 0, 0], dtype=int32),\n",
" 'coal': array([12980, 12947, 13314, ..., 0, 0, 0], dtype=int32),\n",
" 'nuclear': array([7777, 7772, 7779, ..., 6339, 6336, 6319], dtype=int32),\n",
" 'wind': array([1456, 1401, 1604, ..., 7933, 8228, 7995], dtype=int32),\n",
" 'ps': array([0, 0, 0, ..., 0, 0, 0], dtype=int32),\n",
" 'npshyd': array([681, 682, 594, ..., 411, 408, 400], dtype=int32),\n",
" 'ocgt': array([0, 0, 0, ..., 1, 0, 0], dtype=int32),\n",
" 'other': array([784, 784, 782, ..., 101, 101, 99], dtype=int32),\n",
" 'intfr': array([1496, 1496, 1496, ..., 910, 912, 812], dtype=int32),\n",
" 'intirl': array([ 0, 0, 0, ..., 160, 160, 160], dtype=int32),\n",
" 'intned': array([1002, 1028, 982, ..., 0, 0, 0], dtype=int32),\n",
" 'intew': array([ 0, 0, 0, ..., 120, 112, 118], dtype=int32),\n",
" 'biomass': array([ 0, 0, 0, ..., 1635, 1605, 1502], dtype=int32),\n",
" 'intnem': array([ 0, 0, 0, ..., 324, 328, 562], dtype=int32)}"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"attr_data_dict = df_FUELHH.T.apply(np.array, axis=1).to_dict()\n",
"\n",
"attr_data_dict"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<br>\n",
"\n",
"Finally we can save the array\n",
"\n",
"<b>This step is failing, I believe due to the original issue around using non-standard datetime steps in the domain</b>"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"with tiledb.open(array_dir.replace('/', '\\\\'), 'w') as TileDB_array:\n",
" TileDB_array[df_FUELHH.index.values] = attr_data_dict"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ccgt (80737,)\n",
"oil (80737,)\n",
"coal (80737,)\n",
"nuclear (80737,)\n",
"wind (80737,)\n",
"ps (80737,)\n",
"npshyd (80737,)\n",
"ocgt (80737,)\n",
"other (80737,)\n",
"intfr (80737,)\n",
"intirl (80737,)\n",
"intned (80737,)\n",
"intew (80737,)\n",
"biomass (80737,)\n",
"intnem (80737,)\n"
]
}
],
"source": [
"for k,v in attr_data_dict.items():\n",
" print(k, v.shape)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"OrderedDict([('ccgt',\n",
" array([5293, 5547, 5168, ..., 4502, 3893, 4460], dtype=int32)),\n",
" ('oil', array([0, 0, 0, ..., 0, 0, 0], dtype=int32)),\n",
" ('coal',\n",
" array([12980, 12947, 13314, ..., 0, 0, 0], dtype=int32)),\n",
" ('nuclear',\n",
" array([7777, 7772, 7779, ..., 6339, 6336, 6319], dtype=int32)),\n",
" ('wind',\n",
" array([1456, 1401, 1604, ..., 7933, 8228, 7995], dtype=int32)),\n",
" ('ps', array([0, 0, 0, ..., 0, 0, 0], dtype=int32)),\n",
" ('npshyd',\n",
" array([681, 682, 594, ..., 411, 408, 400], dtype=int32)),\n",
" ('ocgt', array([0, 0, 0, ..., 1, 0, 0], dtype=int32)),\n",
" ('other',\n",
" array([784, 784, 782, ..., 101, 101, 99], dtype=int32)),\n",
" ('intfr',\n",
" array([1496, 1496, 1496, ..., 910, 912, 812], dtype=int32)),\n",
" ('intirl',\n",
" array([ 0, 0, 0, ..., 160, 160, 160], dtype=int32)),\n",
" ('intned',\n",
" array([1002, 1028, 982, ..., 0, 0, 0], dtype=int32)),\n",
" ('intew',\n",
" array([ 0, 0, 0, ..., 120, 112, 118], dtype=int32)),\n",
" ('biomass',\n",
" array([ 0, 0, 0, ..., 1635, 1605, 1502], dtype=int32)),\n",
" ('intnem',\n",
" array([ 0, 0, 0, ..., 324, 328, 562], dtype=int32)),\n",
" ('datetime',\n",
" array(['2015-02-22T00:00:00.000', '2015-02-22T00:30:00.000',\n",
" '2015-02-22T01:00:00.000', ..., '2019-09-30T23:00:00.000',\n",
" '2019-09-30T23:30:00.000', '2019-10-01T00:00:00.000'],\n",
" dtype='datetime64[ms]'))])"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with tiledb.open(array_dir.replace('/', '\\\\'), 'r') as TileDB_array:\n",
" read_in_vals = TileDB_array[:]\n",
" \n",
"read_in_vals"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ccgt</th>\n",
" <th>oil</th>\n",
" <th>coal</th>\n",
" <th>nuclear</th>\n",
" <th>wind</th>\n",
" <th>ps</th>\n",
" <th>npshyd</th>\n",
" <th>ocgt</th>\n",
" <th>other</th>\n",
" <th>intfr</th>\n",
" <th>intirl</th>\n",
" <th>intned</th>\n",
" <th>intew</th>\n",
" <th>biomass</th>\n",
" <th>intnem</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2015-02-22 00:00:00</th>\n",
" <td>5293</td>\n",
" <td>0</td>\n",
" <td>12980</td>\n",
" <td>7777</td>\n",
" <td>1456</td>\n",
" <td>0</td>\n",
" <td>681</td>\n",
" <td>0</td>\n",
" <td>784</td>\n",
" <td>1496</td>\n",
" <td>0</td>\n",
" <td>1002</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-02-22 00:30:00</th>\n",
" <td>5547</td>\n",
" <td>0</td>\n",
" <td>12947</td>\n",
" <td>7772</td>\n",
" <td>1401</td>\n",
" <td>0</td>\n",
" <td>682</td>\n",
" <td>0</td>\n",
" <td>784</td>\n",
" <td>1496</td>\n",
" <td>0</td>\n",
" <td>1028</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-02-22 01:00:00</th>\n",
" <td>5168</td>\n",
" <td>0</td>\n",
" <td>13314</td>\n",
" <td>7779</td>\n",
" <td>1604</td>\n",
" <td>0</td>\n",
" <td>594</td>\n",
" <td>0</td>\n",
" <td>782</td>\n",
" <td>1496</td>\n",
" <td>0</td>\n",
" <td>982</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-02-22 01:30:00</th>\n",
" <td>4498</td>\n",
" <td>0</td>\n",
" <td>13421</td>\n",
" <td>7768</td>\n",
" <td>1548</td>\n",
" <td>0</td>\n",
" <td>592</td>\n",
" <td>0</td>\n",
" <td>781</td>\n",
" <td>1496</td>\n",
" <td>0</td>\n",
" <td>1010</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015-02-22 02:00:00</th>\n",
" <td>4660</td>\n",
" <td>0</td>\n",
" <td>13304</td>\n",
" <td>7778</td>\n",
" <td>1351</td>\n",
" <td>0</td>\n",
" <td>584</td>\n",
" <td>0</td>\n",
" <td>780</td>\n",
" <td>1496</td>\n",
" <td>0</td>\n",
" <td>1016</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2019-09-30 22:00:00</th>\n",
" <td>6740</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>6418</td>\n",
" <td>7627</td>\n",
" <td>0</td>\n",
" <td>453</td>\n",
" <td>0</td>\n",
" <td>103</td>\n",
" <td>1210</td>\n",
" <td>80</td>\n",
" <td>110</td>\n",
" <td>58</td>\n",
" <td>1636</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2019-09-30 22:30:00</th>\n",
" <td>4614</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>6408</td>\n",
" <td>7725</td>\n",
" <td>0</td>\n",
" <td>432</td>\n",
" <td>0</td>\n",
" <td>103</td>\n",
" <td>1594</td>\n",
" <td>160</td>\n",
" <td>114</td>\n",
" <td>204</td>\n",
" <td>1643</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2019-09-30 23:00:00</th>\n",
" <td>4502</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>6339</td>\n",
" <td>7933</td>\n",
" <td>0</td>\n",
" <td>411</td>\n",
" <td>1</td>\n",
" <td>101</td>\n",
" <td>910</td>\n",
" <td>160</td>\n",
" <td>0</td>\n",
" <td>120</td>\n",
" <td>1635</td>\n",
" <td>324</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2019-09-30 23:30:00</th>\n",
" <td>3893</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>6336</td>\n",
" <td>8228</td>\n",
" <td>0</td>\n",
" <td>408</td>\n",
" <td>0</td>\n",
" <td>101</td>\n",
" <td>912</td>\n",
" <td>160</td>\n",
" <td>0</td>\n",
" <td>112</td>\n",
" <td>1605</td>\n",
" <td>328</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2019-10-01 00:00:00</th>\n",
" <td>4460</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>6319</td>\n",
" <td>7995</td>\n",
" <td>0</td>\n",
" <td>400</td>\n",
" <td>0</td>\n",
" <td>99</td>\n",
" <td>812</td>\n",
" <td>160</td>\n",
" <td>0</td>\n",
" <td>118</td>\n",
" <td>1502</td>\n",
" <td>562</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>80737 rows × 15 columns</p>\n",
"</div>"
],
"text/plain": [
" ccgt oil coal nuclear wind ps npshyd ocgt other \\\n",
"2015-02-22 00:00:00 5293 0 12980 7777 1456 0 681 0 784 \n",
"2015-02-22 00:30:00 5547 0 12947 7772 1401 0 682 0 784 \n",
"2015-02-22 01:00:00 5168 0 13314 7779 1604 0 594 0 782 \n",
"2015-02-22 01:30:00 4498 0 13421 7768 1548 0 592 0 781 \n",
"2015-02-22 02:00:00 4660 0 13304 7778 1351 0 584 0 780 \n",
"... ... ... ... ... ... .. ... ... ... \n",
"2019-09-30 22:00:00 6740 0 0 6418 7627 0 453 0 103 \n",
"2019-09-30 22:30:00 4614 0 0 6408 7725 0 432 0 103 \n",
"2019-09-30 23:00:00 4502 0 0 6339 7933 0 411 1 101 \n",
"2019-09-30 23:30:00 3893 0 0 6336 8228 0 408 0 101 \n",
"2019-10-01 00:00:00 4460 0 0 6319 7995 0 400 0 99 \n",
"\n",
" intfr intirl intned intew biomass intnem \n",
"2015-02-22 00:00:00 1496 0 1002 0 0 0 \n",
"2015-02-22 00:30:00 1496 0 1028 0 0 0 \n",
"2015-02-22 01:00:00 1496 0 982 0 0 0 \n",
"2015-02-22 01:30:00 1496 0 1010 0 0 0 \n",
"2015-02-22 02:00:00 1496 0 1016 0 0 0 \n",
"... ... ... ... ... ... ... \n",
"2019-09-30 22:00:00 1210 80 110 58 1636 0 \n",
"2019-09-30 22:30:00 1594 160 114 204 1643 0 \n",
"2019-09-30 23:00:00 910 160 0 120 1635 324 \n",
"2019-09-30 23:30:00 912 160 0 112 1605 328 \n",
"2019-10-01 00:00:00 812 160 0 118 1502 562 \n",
"\n",
"[80737 rows x 15 columns]"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# the OrderedDict is convertible to a dataframe\n",
"idx = read_in_vals.pop('datetime')\n",
"pd.DataFrame(read_in_vals, index=idx)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python (tdb-6)",
"language": "python",
"name": "tdb-6"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.2"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
@qiuwei
Copy link

qiuwei commented Jun 24, 2020

These kinds of examples are very helpful for users. Thanks a lot.
Could you provide more examples to demonstrate other features of tiledb, such as versioning, time-traveling, etc.?

The documents on the official website are somewhat vague and fluffing.
Thanks in advance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment