Skip to content

Instantly share code, notes, and snippets.

@odewahn

odewahn/pandas-jvdp.ipynb

Created June 2, 2015 10:18
Show Gist options
  • Save odewahn/24eab111f696171cf49f to your computer and use it in GitHub Desktop.
Save odewahn/24eab111f696171cf49f to your computer and use it in GitHub Desktop.
Download ZIP
Pandas Recipe by Jake BanderPlas
Display the source blob
Display the rendered blob
Raw
pandas-jvdp.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Introducing Pandas Objects"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"At the very basic level, Pandas objects can be thought of as enhanced versions of NumPy structured arrays in which the rows and columns are identified with labels rather than simple integer indices.\n",
"As we will see through the rest of this chapter, Pandas provides a host of useful tools, methods, and functionality on top of these data structures, but nearly everything that follows will require an understanding of what these structures are.\n",
"This first section will cover the two fundamental Pandas data structures: the ``Series``, ``DataFrame``, and ``Index``."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pandas ``Series``\n",
"\n",
"A Pandas series is a mapping of a set of indices to a set of data values, and can be thought of as a generalization of a one-dimensional NumPy array.\n",
"Let's show this by constructing a simple Pandas ``Series``.\n",
"We'll start by importing ``pandas`` under the standard alias ``pd``:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 0.25\n",
"1 0.50\n",
"2 0.75\n",
"3 1.00\n",
"dtype: float64"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = pd.Series([0.25, 0.5, 0.75, 1.0])\n",
"data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we see in the output above, the series has both a sequence of values and a sequence of indices, which we can access with the ``values`` and ``index`` attributes.\n",
"The ``values`` are simply a familiar NumPy array:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array([ 0.25, 0.5 , 0.75, 1. ])"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data.values"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"while the ``index`` is an array-like object of type ``pd.Index``:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Int64Index([0, 1, 2, 3], dtype='int64')"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data.index"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Like with a NumPy array, the index can be passed with the typical square-bracket notation to access a particular value:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"0.5"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data[1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ``Series`` as Generalized NumPy Array"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"From what we've seen so far, it may look like the ``Series`` object is essentially interchangeable with a one-dimensional NumPy array.\n",
"The essential difference is the presence is the index: while the Numpy Array has an implicitly defined integer index used to access the values, the Pandas Series has an *explicitly defined* index associated with the values.\n",
"\n",
"This explicit index definition gives the ``Series`` object additional capabilities. For example, the index need not be an integer, but can consist of values of any desired type.\n",
"For example, if we wish, we can use strings as an index:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"a 0.25\n",
"b 0.50\n",
"c 0.75\n",
"d 1.00\n",
"dtype: float64"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = pd.Series([0.25, 0.5, 0.75, 1.0],\n",
" index=['a', 'b', 'c', 'd'])\n",
"data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and the item access works as expected:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"0.5"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data['b']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can even use non-contiguous or non-sequential indices"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"2 0.25\n",
"5 0.50\n",
"3 0.75\n",
"7 1.00\n",
"dtype: float64"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = pd.Series([0.25, 0.5, 0.75, 1.0],\n",
" index=[2, 5, 3, 7])\n",
"data"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"0.25"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data[2]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ``Series`` as Specialized Dictionary\n",
"\n",
"In this way, you can think of a Pandas ``Series`` a bit like a specialization of a Python dictionary.\n",
"A dictionary is a structure which maps unique keys to a set of values, and a series is a structure which which maps uniqued *typed* keys to a set of *typed* values.\n",
"This typing is important: just as the included type information of a NumPy array makes it more efficient than Python lists, the type information of a Pandas Series makes it much more efficient than Python dictionaries.\n",
"\n",
"This analogy can be made even more clear by constructing a ``Series`` object directly from a Python dictionary:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"California 38332521\n",
"Florida 19552860\n",
"Illinois 12882135\n",
"New York 19651127\n",
"Texas 26448193\n",
"dtype: int64"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"population_dict = {'California': 38332521,\n",
" 'Texas': 26448193,\n",
" 'New York': 19651127,\n",
" 'Florida': 19552860,\n",
" 'Illinois': 12882135}\n",
"population = pd.Series(population_dict)\n",
"population"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By default, a series will be created where the index is drawn from the sorted keys.\n",
"From here, typical item access and slicing operations can be performed"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"38332521"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"population['California']"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"California 38332521\n",
"Florida 19552860\n",
"Illinois 12882135\n",
"dtype: int64"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"population['California':'Illinois']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Constructing ``Series`` Objects\n",
"\n",
"We've already seen a few ways of constructing a Pandas ``Series`` from scratch; all of them are some version of the following,\n",
"\n",
"```\n",
">>> pd.Series(data, index=index)\n",
"```\n",
"\n",
"where ``index`` is an optional argument.\n",
"\n",
"We saw some of these above, but to be explicit:\n",
"\n",
"``data`` can be a list or NumPy array, in which case ``index`` defaults to an integer sequence:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 2\n",
"1 4\n",
"2 6\n",
"dtype: int64"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.Series([2, 4, 6])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"``data`` can be a scalar, which is broadcast across the specified index:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"100 5\n",
"200 5\n",
"300 5\n",
"dtype: int64"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.Series(5, index=[100, 200, 300])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"``data`` can be a dictionary, in which ``index`` defaults to the sorted dictionary keys:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1 b\n",
"2 a\n",
"dtype: object"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.Series({2:'a', 1:'b'})"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In each case, the index can be explicitly set if a different result is preferred."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pandas ``DataFrame``\n",
"\n",
"The next fundamental structure in Pandas is the ``DataFrame``.\n",
"Like the ``Series`` above, the ``DataFrame`` can be thought of either as a generalization of a NumPy array, or as a specialization of a Python dictionary.\n",
"We'll discuss these views below."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ``DataFrame`` as a Generalized NumPy array\n",
"If a ``Series`` is an analog of a one-dimensional array with flexible indices, a ``DataFrame`` is an analog of a two-dimensional array with both flexible row indices and flexible column names.\n",
"Just as you might think of a two-dimensional array as an ordered sequence of aligned one-dimensional columns, you can think of a ``DataFrame`` as a sequence of aligned ``Series`` objects.\n",
"Here, by \"aligned\" we mean that they share the same index.\n",
"\n",
"To demonstrate this, let's first construct a new ``Series`` listing the area of each of the five states mentioned above:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"California 423967\n",
"Florida 170312\n",
"Illinois 149995\n",
"New York 141297\n",
"Texas 695662\n",
"dtype: int64"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"area_dict = {'California': 423967, 'Texas': 695662, 'New York': 141297, 'Florida': 170312, 'Illinois': 149995}\n",
"area = pd.Series(area_dict)\n",
"area"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have this along with the ``population`` Series from above, we can use a dictionary to construct a single two-dimensional object containing this information:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>area</th>\n",
" <th>population</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>California</th>\n",
" <td> 423967</td>\n",
" <td> 38332521</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Florida</th>\n",
" <td> 170312</td>\n",
" <td> 19552860</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Illinois</th>\n",
" <td> 149995</td>\n",
" <td> 12882135</td>\n",
" </tr>\n",
" <tr>\n",
" <th>New York</th>\n",
" <td> 141297</td>\n",
" <td> 19651127</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Texas</th>\n",
" <td> 695662</td>\n",
" <td> 26448193</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" area population\n",
"California 423967 38332521\n",
"Florida 170312 19552860\n",
"Illinois 149995 12882135\n",
"New York 141297 19651127\n",
"Texas 695662 26448193"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"states = pd.DataFrame({'population': population,\n",
" 'area': area})\n",
"states"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Like the ``Series`` object, the ``DataFrame`` has an ``index`` attribute which gives access to the index names:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Index(['California', 'Florida', 'Illinois', 'New York', 'Texas'], dtype='object')"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"states.index"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Additionally, the ``DataFrame`` has a ``columns`` attribute which is an ``Index`` object holding the column names:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Index(['area', 'population'], dtype='object')"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"states.columns"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Thus the ``DataFrame`` can be thought of as a generalization of a two-dimensional NumPy array, where both the rows and columns have a generalized index for accessing the data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ``DataFrame`` as Specialized Dictionary\n",
"\n",
"Similarly, we can think of a dataframe as a specialization of a dictionary.\n",
"Where a dictionary maps a key to a value, a data frame maps a column name to a Series of column data.\n",
"For example, asking for the ``'area'`` attribute returns the ``Series`` object containing the areas we saw above:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"California 423967\n",
"Florida 170312\n",
"Illinois 149995\n",
"New York 141297\n",
"Texas 695662\n",
"Name: area, dtype: int64"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"states['area']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice the potential gotcha here: in a two-dimesnional NumPy array, ``data[0]`` will return the first *row*. For a dataframe, ``data['col0']`` will return the first *column*.\n",
"Because of this, it is probably better to think about dataframes as generalized dictionaries rather than generalized arrays, though both ways of looking at the situation can be useful."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Constructing ``DataFrame`` Objects\n",
"\n",
"A Pandas ``DataFrame`` can be constructed in a variety of ways.\n",
"Here we'll give several examples:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### From a single ``Series`` object"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>population</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>California</th>\n",
" <td> 38332521</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Florida</th>\n",
" <td> 19552860</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Illinois</th>\n",
" <td> 12882135</td>\n",
" </tr>\n",
" <tr>\n",
" <th>New York</th>\n",
" <td> 19651127</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Texas</th>\n",
" <td> 26448193</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" population\n",
"California 38332521\n",
"Florida 19552860\n",
"Illinois 12882135\n",
"New York 19651127\n",
"Texas 26448193"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.DataFrame(population, columns=['population'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### From a list of ``dict``s\n",
"\n",
"Any list of dictionaries can be made into a dataframe.\n",
"We'll use a simple list comprehension to create some data:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>a</th>\n",
" <th>b</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td> 0</td>\n",
" <td> 0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td> 1</td>\n",
" <td> 2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td> 2</td>\n",
" <td> 4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" a b\n",
"0 0 0\n",
"1 1 2\n",
"2 2 4"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = [{'a': i, 'b': 2 * i}\n",
" for i in range(3)]\n",
"pd.DataFrame(data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Even if some keys in the dictionary are missing, Pandas will fill them in with ``NaN`` (i.e. \"not a number\") values:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>a</th>\n",
" <th>b</th>\n",
" <th>c</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td> 1</td>\n",
" <td> 2</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>NaN</td>\n",
" <td> 3</td>\n",
" <td> 4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" a b c\n",
"0 1 2 NaN\n",
"1 NaN 3 4"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.DataFrame([{'a': 1, 'b': 2}, {'b': 3, 'c': 4}])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### From a dictionary of ``Series`` objects\n",
"\n",
"We saw this above, but a dictionary of ``Series`` objects works as well:"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>area</th>\n",
" <th>population</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>California</th>\n",
" <td> 423967</td>\n",
" <td> 38332521</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Florida</th>\n",
" <td> 170312</td>\n",
" <td> 19552860</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Illinois</th>\n",
" <td> 149995</td>\n",
" <td> 12882135</td>\n",
" </tr>\n",
" <tr>\n",
" <th>New York</th>\n",
" <td> 141297</td>\n",
" <td> 19651127</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Texas</th>\n",
" <td> 695662</td>\n",
" <td> 26448193</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" area population\n",
"California 423967 38332521\n",
"Florida 170312 19552860\n",
"Illinois 149995 12882135\n",
"New York 141297 19651127\n",
"Texas 695662 26448193"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.DataFrame({'population': population,\n",
" 'area': area})"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### From a two-dimensional NumPy array\n",
"\n",
"Given a two-dimensional array of data, we can create a dataframe with any specified column and index names.\n",
"If left out, an integer index will be used for each."
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>foo</th>\n",
" <th>bar</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>a</th>\n",
" <td> 0.153318</td>\n",
" <td> 0.460967</td>\n",
" </tr>\n",
" <tr>\n",
" <th>b</th>\n",
" <td> 0.816986</td>\n",
" <td> 0.984606</td>\n",
" </tr>\n",
" <tr>\n",
" <th>c</th>\n",
" <td> 0.737356</td>\n",
" <td> 0.013190</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" foo bar\n",
"a 0.153318 0.460967\n",
"b 0.816986 0.984606\n",
"c 0.737356 0.013190"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy as np\n",
"pd.DataFrame(np.random.rand(3, 2),\n",
" columns=['foo', 'bar'],\n",
" index=['a', 'b', 'c'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### From a numpy structured array\n",
"\n",
"We covered structured arrays in section X.X.\n",
"A Pandas dataframe operates much like a structured array, and can be created directly from one:"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>A</th>\n",
" <th>B</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td> 0</td>\n",
" <td> 0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td> 0</td>\n",
" <td> 0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td> 0</td>\n",
" <td> 0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" A B\n",
"0 0 0\n",
"1 0 0\n",
"2 0 0"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A = np.zeros(3, dtype=[('A', 'i8'), ('B', 'f8')])\n",
"pd.DataFrame(A)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pandas ``Index``\n",
"\n",
"Above we saw that both the ``Series`` and ``DataFrame`` contain an explicit *index* which lets you reference and modify data.\n",
"This ``Index`` object is an interesting structure in itself, and it can be thought of either as an *immutable array* or as an *ordered set*.\n",
"We'll see some of the consequences of those operations here."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Int64Index([2, 3, 5, 7, 11], dtype='int64')"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ind = pd.Index([2, 3, 5, 7, 11])\n",
"ind"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ``Index`` as Immutable Array\n",
"\n",
"The index in many ways operates like an array.\n",
"For example, we can use standard Python indexing notation to to retrieve values or slices:"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"3"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ind[1]"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Int64Index([2, 5, 11], dtype='int64')"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ind[::2]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"``Index`` objects also have many of the attributes familiar from NumPy arrays:"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"5 (5,) 1 int64\n"
]
}
],
"source": [
"print(ind.size, ind.shape, ind.ndim, ind.dtype)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One difference between ``Index`` objects and NumPy arrays is that indices are immutable: that is, they cannot be modified via the normal means:"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"collapsed": false
},
"outputs": [
{
"ename": "TypeError",
"evalue": "'<class 'pandas.core.index.Int64Index'>' does not support mutable operations.",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-31-40e631c82e8a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mind\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m/Users/jakevdp/anaconda/envs/py3k/lib/python3.3/site-packages/pandas/core/base.py\u001b[0m in \u001b[0;36m_disabled\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 180\u001b[0m \u001b[0;34m\"\"\"This method will not function because object is immutable.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 181\u001b[0m raise TypeError(\"'%s' does not support mutable operations.\" %\n\u001b[0;32m--> 182\u001b[0;31m self.__class__)\n\u001b[0m\u001b[1;32m 183\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 184\u001b[0m \u001b[0m__setitem__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m__setslice__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m__delitem__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m__delslice__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_disabled\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mTypeError\u001b[0m: '<class 'pandas.core.index.Int64Index'>' does not support mutable operations."
]
}
],
"source": [
"ind[1] = 0"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This immutability makes it safter to share indices between multiple dataframes and arrays, without nasty side-effects from inadvertent index modification.\n",
"Another difference between indices and arrays is that arithmetic operators such as ``A + B`` and ``A - B`` implement *set operations* rather than standard element-wise arithmetic operations."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ``Index`` as Ordered Set\n",
"\n",
"Pandas objects are designed to facilitate operations such as joins across datasets, which depend on many aspects of set arithmetic.\n",
"Recall that Python has a built-in ``set`` object, which we explored in section X.X.\n",
"The ``Index`` object follows many of the conventions of this built-in set object, so that unions, intersections, differences, and other combinations can be computed in a familiar way:"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"indA = pd.Index([1, 3, 5, 7, 9])\n",
"indB = pd.Index([2, 3, 5, 7, 11])"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Int64Index([3, 5, 7], dtype='int64')"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"indA & indB # intersection"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Int64Index([1, 2, 3, 5, 7, 9, 11], dtype='int64')"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"indA | indB # union"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Int64Index([1, 9], dtype='int64')"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"indA - indB # difference"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Int64Index([1, 2, 9, 11], dtype='int64')"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"indA ^ indB # symmetric difference"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These operations may also be accessed via object methods, e.g. ``indA.intersection(indB)``.\n",
"For more information on the variety of set operations implemented, see section X.X.\n",
"Nearly every syntax listed there, with the exception of operations which modify the set, can also be performed on ``Index`` objects."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Looking Forward\n",
"\n",
"Above we saw the basics of the ``Series``, ``DataFrame``, and ``Index`` objects, which form the foundation of data-oriented computing with Pandas.\n",
"We saw how they are similar to and different from other Python data structures, and how they can be created from scratch from these more familiar objects.\n",
"Through this chapter, we'll go more into more detail about creation of these structures (including very useful interfaces for creating them from various file types) and manipulating data within these structures.\n",
"Just as understanding the effective use of NumPy arrays is fundamental to effective numerical computing in Python, understanding the effective use of Pandas structures is fundamental to the data munging required for data science in Python."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.3.5"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment