briatte

porngram <- function(x = c("hardcore", "softcore"), ..., adjust = "xxx") {
  library(ggplot2)
  library(XML)
  library(reshape)
  library(rPython)
  
  x = c(x, ...)
  if (length(x) > 10) {
    x <- x[1:10]
    warning("Porngram API limit: only using first 10 phrases.")

briatte

Collapsing a 4-column data frame of real data from 500,000 rows to 91,000 by pasting and counting row values. Execution on a 1.8GHz Intel Core i5 shows that dplyr is 1.5 times quicker than base R.

See this Gist for a simpler test over twice more rows and roughly as many groups. In both tests, dplyr is as concise as plyr, as fast as data.table, and clearly more readable than base R.

briatte

Here's a simple timing test of aggregation functions in R, using 1.3 million rows and 80,000 groups of real data on a 1.8GHz Intel Core i5. Thanks to Arun Srinivasan for helpful comments.

The fastest function to run through the data.frame benchmark is data.table, which runs twice faster than dplyr, which runs ten times faster than base R.

For a benchmark that includes plyr, see this earlier Gist for a computationally more intensive test on half a million rows, where dplyr still runs 1.5 times faster than aggregate in base R.

Both tests confirm what W. Andrew Barr blogged on dplyr:

the 2 most important improvements in dplyr are >

briatte

Improving access to panel series data for social scientists: the psData package

GitHub repository: https://github.com/rOpenGov/psData

Social scientists have access to many electronically available panel series datasets. However, downloading, cleaning, and merging them together is time-consuming and error-prone: for example, using Reinhart and Rogoff's data on the fiscal costs of the financial crisis involves downloading, cleaning, and merging 4 Excel files with over 70 individual sheets, one for each country’s data. Furthermore, because such datasets are not bundled in a format that is easy to manipulate, many of them are not updated on a regular basis.

In this talk, we introduce the psData package for the R statistical software. This package is being developed under the rOpenGov framework to solve two problems:

1. Time wasted by social scientists downloading, cleaning, and transforming commo

briatte

# parse XPath syntax from well-formed HTML
library(XML)

# complete archive will take ~ 1.4 GB on disk
dir.create("declarations", showWarnings = FALSE)

# finds 941 MPs on 2014-07-24 at website launch
h = htmlParse("http://www.hatvp.fr/consulter-les-declarations-rechercher.html")
h = paste0("http://www.hatvp.fr/", xpathSApply(h, "//div[@id='annuaire']/*/*/*/a/@href"))

briatte

system("defaults write org.R-project.R force.LANG en_US.UTF-8")

briatte

briatte

# A demo of R + ggplot2, using Guardian/ICM polling data.
# 2014-10-02

# Load packages.
pkgs = c("httr", "ggplot2", "lubridate", "RColorBrewer", "reshape2")
pkgs = lapply(pkgs, FUN = function(x) {

  if(!require(x, character.only = TRUE)) {

    install.packages(x, quiet = TRUE)

briatte

briatte

# An overview of the Reinhart & Rogoff data, from an exercise by Cosma Shalizi.
# 2014-10-09

#
# package
#

library(ggplot2)

#