A comparison of aggregation methods in R using the Gapminder data from Data Carpentry

aggregation.Rmd

---
title: "Aggregation in R"
author: "K. Arthur Endsley"
date: "August 4, 2016"
output: html_document
---

Let's explore multiple ways to aggregate data in R.
We'll do this to answer two questions:

* What is the change in life expectancy, `lifeExp`, in each country over all years?
* What is the mean per capita GDP, `gdpPercap`, in each country?

```{r loading}
gapminder <- read.csv('/Users/arthur/Downloads/gapminder-sql-data/gapminder-FiveYearData.csv')
head(gapminder)
```

# Aggregation Trials

## Aggregation with For Loops

Nested `for` loops are the most basic way to aggregate in R.
It involves looping over each group.

```{r with_for}
num_countries <- length(unique(gapminder$country))

# Pre-allocate the output data structure
life_exp_change <- vector(mode = 'numeric', length = num_countries)
mean_gdp_per_capita <- vector(mode = 'numeric', length = num_countries)
output <- data.frame(
  country = unique(gapminder$country),
  life_exp_change = life_exp_change,
  mean_gdp_per_capita = mean_gdp_per_capita)

for (country in unique(gapminder$country)) {
  subset <- gapminder[gapminder$country == country,]
  output[output == country, 'life_exp_change'] <- with(subset, 
    max(lifeExp) - min(lifeExp))
  output[output == country, 'mean_gdp_per_capita'] <- with(subset,
    mean(gdpPercap))
}

head(output)
```

## Aggregation with aggregate()

The built-in `aggregate()` function can also be used.

```{r with_aggregate}
life_exp_change <- aggregate(gapminder[,c('lifeExp')],
  by = list(gapminder$country), FUN = function (values) {
    max(values) - min(values)
})

mean_gdp_per_capita <- aggregate(gapminder[,c('gdpPercap')], 
  by = list(gapminder$country), FUN = mean)

output <- data.frame(
  country = unique(gapminder$country),
  life_exp_change = life_exp_change$x,
  mean_gdp_per_capita = mean_gdp_per_capita$x)

head(output)
```

## Aggregation with tapply()

`tapply()` is a slightly more advanced way to get the same result by **applying** a function over a sequence (in this case, a data frame).

```{r with_tapply}
life_exp_change <- tapply(gapminder$lifeExp, 
  gapminder$country, function (values) {
    return(max(values) - min(values))
})

mean_gdp_per_capita <- tapply(gapminder$gdpPercap, 
  gapminder$country, mean)

output <- data.frame(
  life_exp_change = life_exp_change,
  mean_gdp_per_capita = mean_gdp_per_capita)

head(output)
```

Note that with `tapply()` we row names for our output `data.frame` for free.

## Aggregation in Parallel with foreach

For very large datasets (not this one), this construction will be faster than `aggregate()` but possibly not faster than using the `plyr` package or even `tapply()` (based on [these benchmarks](http://jaredlander.com/content/2015/04/MakingRGoFasterAndBigger.html#29)).

```{r with_foreach, message=FALSE}
library(doParallel)

# Setup the compute cluster
cl <- makeCluster(detectCores())
registerDoParallel(cl)

# Describet the output
output <- foreach(country = unique(gapminder$country), 
  .combine = rbind) %dopar% data.frame(
    country = country, 
    life_exp_change = max(gapminder[gapminder$country == country, 'lifeExp']) - min(gapminder[gapminder$country == country, 'lifeExp']),
    mean_gdp_per_capita = mean(gapminder[gapminder$country == country, 'gdpPercap']))

stopCluster(cl) # Stop the cluster
head(output)
```

## Aggregation with Melting and Casting

We can use the `melt()` and `cast()` functions in the `reshape2` package to aggregate as well.
If we want to use the `subset` argument, we also need the `plyr` package here in order to get the "as quoted" function `.()`; see the documentation on this function (`?.`).

```{r with_melt_cast, message=FALSE, warning=FALSE}
library(plyr)
library(reshape2)
melted <- melt(gapminder[,c('country', 'lifeExp', 'gdpPercap')], 
  id.vars='country')

# Calculate mean per capita GDP
mean_gdp_per_capita <- dcast(melted, country ~ variable, 
  subset = .(variable == 'gdpPercap'), fun.aggregate = mean)

life_exp_change <- dcast(melted, country ~ variable, 
  subset = .(variable == 'lifeExp'), fun.aggregate = function (values) {
    return(max(values, na.rm = TRUE) - min(values, na.rm = TRUE))
})

output <- data.frame(
  country = unique(gapminder$country),
  life_exp_change = life_exp_change['lifeExp'],
  mean_gdp_per_capita = mean_gdp_per_capita['gdpPercap'])

head(output)
```

The warnings here look scary but we get the right result.
To understand these warnings, [read this article](http://stackoverflow.com/questions/24282550/no-non-missing-arguments-warning-when-using-min-or-max-in-reshape2).

## Aggregation with plyr

We can use the `summarize()` and `ddply()` functions to aggregate the data as well.

```{r with_plyr}
library(plyr)

output <- ddply(gapminder, c('country'), summarize, 
  life_exp_diff = max(lifeExp) - min(lifeExp),
  mean_gdp_per_capita = mean(gdpPercap))

head(output)
```

## Aggregation with Data Tables

```{r with_data_tables, message=FALSE}
library(data.table)
gapminder_dt <- data.table(gapminder)
output <- gapminder_dt[, list(
  life_exp_change = max(lifeExp) - min(lifeExp),
  mean_gdp_per_capita = mean(gdpPercap)), by = country]

head(output)
```

## Aggregation with dplyr

```{r with_dplyr, warning=FALSE, message=FALSE}
library(dplyr)
gapminder %>% group_by(country) %>% summarize(
  life_exp_change = max(lifeExp) - min(lifeExp),
  mean_gdp_per_capita = mean(gdpPercap)) %>% head(10)
```

# Summary

* Writing nested `for` loops is a clear loser in many ways; it's slow, and it takes a long time to write.
* Other than that, **you should use whatever method is most appealing to you;** whatever method makes the most sense and is easiest for you to come back to with a different problem.
* `dplyr` and `ddply()` in the `plyr` package are tied for the shortest solution (in terms of writing code).

gapminder-FiveYearData.csv