Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
RealClearPolitics XML Scraping
doInstall <- TRUE # Change to FALSE if you don't want packages installed.
toInstall <- c("XML", "ggplot2", "lubridate", "reshape2", "scales")
if(doInstall){install.packages(toInstall, repos = "")}
lapply(toInstall, library, character.only = TRUE)
# Find your XML file from those listed at
URL <- ""
parsedXML <- xmlParse(URL) # First pass
dateSeries <- xpathSApply(parsedXML, path = "//series") # Check the structure
Date <- sapply(xmlChildren(dateSeries[[1]]), xmlValue) # of the XML. <series>
names(Date) <- sapply(xmlChildren(dateSeries[[1]]), # is the first child node.
xmlGetAttr, "xid")
graphSeries <- xpathSApply(parsedXML, path = "//graph") # <graph> is the
obamaSeries <- sapply(xmlChildren(graphSeries[[1]]), xmlValue) # second major
names(obamaSeries) <- sapply(xmlChildren(graphSeries[[1]]), # child node.
xmlGetAttr, "xid") # ^ The first graph line
romneySeries <- sapply(xmlChildren(graphSeries[[2]]), xmlValue) # is "Obama"
names(romneySeries) <- sapply(xmlChildren(graphSeries[[2]]), # The second
xmlGetAttr, "xid") # ^ is "Romney."
# Put all of these series into a data.frame
rcpData <- data.frame(xid = names(Date), stringsAsFactors = FALSE)
rcpData$Date <- mdy(Date[rcpData$xid], tz = "EST") # lubridate!
rcpData$Romney <- as.numeric(romneySeries[rcpData$xid])
rcpData$Obama <- as.numeric(obamaSeries[rcpData$xid])
write.csv(rcpData, "RealClearPolitics Polling Average.csv", row.names = F)

This comment has been minimized.

Copy link

@ideamonger ideamonger commented Sep 13, 2016

RCP does not make the XMLs visible in the source anymore. Any idea how this can be updated to reflect this, including a new way of sniffing the XLMs? Or did they switch to other storage solution (json?)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment