Skip to content

Instantly share code, notes, and snippets.

@jebyrnes
Created November 13, 2011 00:23
Show Gist options
  • Save jebyrnes/1361353 to your computer and use it in GitHub Desktop.
Save jebyrnes/1361353 to your computer and use it in GitHub Desktop.
Code to scrape RocketHub's xml feeds for the #SciFund projects, add it to a log file, then plot it. Run the data scrape without appending the first time to generate a log file.
ID Title Name Project.Link Goal
1 Why is this Dolphin's Fin on Backwards!? Matthew S. Leslie http://www.rockethub.com/projects/3754-why-is-this-dolphin-s-fin-on-backwards 5000
2 Tracking the migration of the Atlantic Puffin Robin Freeman http://www.rockethub.com/projects/3818-tracking-the-migration-of-the-atlantic-puffin 5000
3 Cancer? Yeast has the answers Marisa Alonso-N??ez http://www.rockethub.com/projects/3753-cancer-yeast-has-the-answers 2500
4 Athlete's Foot in Worms? Rebecca Rashid Achterman http://www.rockethub.com/projects/3703-athlete-s-foot-in-worms 1000
5 Support Zombie Research! Kelly Weinersmith http://www.rockethub.com/projects/3737-support-zombie-research 3500
6 Doctor Zen and the Amazon Crayfish Zen Faulkes http://www.rockethub.com/projects/3695-doctor-zen-and-the-amazon-crayfish 1000
7 C-Cilia in Motion!! Aditya Rao http://www.rockethub.com/projects/3792-c-cilia-in-motion 5000
8 Force of Duck: Measuring explosive erection Diane Kelly http://www.rockethub.com/projects/3769-force-of-duck-measuring-explosive-erection 3125
9 Mysteries of a Prehistoric Affair Marisa Tellez http://www.rockethub.com/projects/3823-mysteries-of-a-prehistoric-affair 3000
10 Pennies instead of petroleum! Jeffrey Bodwin http://www.rockethub.com/projects/3775-pennies-instead-of-petroleum 3000
11 Dolphinpalooza Erin http://www.rockethub.com/projects/3828-dolphinpalooza 4500
12 Does the act of looking change what we see? Eric Abelson http://www.rockethub.com/projects/3777-does-the-act-of-looking-change-what-we-see 1500
13 Domesticating algae for the 21st century Steve Herbert http://www.rockethub.com/projects/3811-domesticating-algae-for-the-21st-century 3300
14 #Chlamystress Luis Valledor http://www.rockethub.com/projects/3768-chlamystress 5000
15 Can we save Collserola Natural Park? Jorge Mederos http://www.rockethub.com/projects/3788-biodiversidad-insecta-collserola 2700
16 Species in peril Hadlylab http://www.rockethub.com/projects/3767-species-in-peril 2000
17 Cats Nails: A parasitic plant of South Africa Andi Wolfe http://www.rockethub.com/projects/3761-cats-nails-a-parasitic-plant-of-south-africa 5000
18 The yin yang world of venom. Chip Cochran http://www.rockethub.com/projects/3798-the-yin-yang-world-of-venom 12000
19 Preserving wildlife to benefit farmers Center for Conservation Biology http://www.rockethub.com/projects/3847-preserving-wildlife-to-benefit-farmers 2500
20 Mapping a Bornean Soundscape Alison Styring http://www.rockethub.com/projects/3831-mapping-a-bornean-soundscape 5500
21 STEMulate Learning! Kalani Hausman http://www.rockethub.com/projects/3814-stemulate-learning 20000
22 Methods of Artificially Aging Red Wine Matthew Hutchins http://www.rockethub.com/projects/3751-methods-of-artificially-aging-red-wine 2000
23 The adventures of Merlin the Tortoise Dog Melia Nafus http://www.rockethub.com/projects/3802-the-adventures-of-merlin-the-tortoise-dog 5800
24 Hey! Did you miss that fish? Jarrett Byrnes http://www.rockethub.com/projects/3745-hey-did-you-miss-that-fish 6700
25 Corals and Climate Change Jessica Carilli http://www.rockethub.com/projects/3806-what-is-the-future-of-marshall-islands-corals 3500
26 Culture of Climate Change in French Polynesia Lopez-Carr Walker Davies Murphy Bambridge http://www.rockethub.com/projects/3809-culture-of-climate-change-in-french-polynesia 5000
27 Beethoven's repository of Open Research Daniel Mietchen http://www.rockethub.com/projects/3755-beethoven-s-repository-of-open-research 12000
28 Depression - An Illness of the Whole Body Kevin Fomalont http://www.rockethub.com/projects/3799-depression-an-illness-of-the-whole-body 5700
29 Train An Elephant Researcher Shermin de Silva http://www.rockethub.com/projects/3707-train-an-elephant-researcher 7000
30 Ancient Roman DNA Project Kristina Killgrove http://www.rockethub.com/projects/3709-ancient-roman-dna-project 6000
31 Send John to the Jungle! John Gust http://www.rockethub.com/projects/3844-send-john-to-the-jungle 1000
32 Learner Control in Online Training Programs Katelyn Cavanaugh http://www.rockethub.com/projects/3800-learner-control-in-online-training-programs 1200
33 Squirrel-Snake Face Off! Bree Putman http://www.rockethub.com/projects/3804-squirrel-snake-face-off 3500
34 Mathematics of Direct Democracy Lee Worden http://www.rockethub.com/projects/3773-mathematics-of-consensus-decision-making 3260
35 Artificial Photosynthesis at NCSU Walter Weare http://www.rockethub.com/projects/3705-artificial-photosynthesis-at-ncsu 500
36 Evolution in Agriculture Scott Chamberlain http://www.rockethub.com/projects/3790-evolution-in-agriculture 3500
37 What's that Weed?! Kelly Lyons http://www.rockethub.com/projects/3840-what-s-that-weed 1715
38 Urban Butterfly Blues Timothy Bonebrake http://www.rockethub.com/projects/3819-urban-butterfly-blues 1500
39 Saving Hawaii's Coral Reefs Levi Lewis http://www.rockethub.com/projects/3793-saving-hawaii-s-coral-reefs 5000
40 The Wild Life of Our Homes The Wild Life Team http://www.rockethub.com/projects/3789-the-wild-life-of-our-homes 15000
41 Serengeti Live Serengeti Lion Project http://www.rockethub.com/projects/3725-serengeti-live 14000
42 Bats in peril: flying foxes past and present Susan http://www.rockethub.com/projects/3787-bats-in-peril-flying-foxes-past-and-present 4000
43 Every Blip Counts _ Low Cost Seismic Sensors Debi Kilb http://www.rockethub.com/projects/3744-every-blip-counts-low-cost-seismic-sensors 4000
44 Smart Delivery Jennifer Schmitt http://www.rockethub.com/projects/3808-smart-delivery 5000
45 Magnetic Nerve Stimulator Prototype Eric Basham http://www.rockethub.com/projects/3812-magnetic-nerve-stimulator-prototype 1250
46 Turtles in the Deep Lindsey Peavey http://www.rockethub.com/projects/3810-turtles-in-the-deep 2500
47 Behold, the Power of Seagrass! Ross Whippo http://www.rockethub.com/projects/3795-behold-the-power-of-seagrass 2000
48 School of Ants School of Ants http://www.rockethub.com/projects/3827-school-of-ants 6000
49 The Evolution of Stress-Induced Hypermutation Yoav Ram http://www.rockethub.com/projects/3716-the-evolution-of-stress-induced-hypermutation 1111
library(XML)
library(plyr)
##Rscript scifundDataScrape.r
#setwd("~/scifund_analysis")
thisDate<-Sys.time()
projects<-read.csv("./projects.csv")
#data frame of what's going on now
ndf<-ddply(projects, .(ID, Project.Link), function(adf){
prj<-adf$Project.Link[1]
dollaz <- xmlValue(xmlRoot(xmlTreeParse(paste(prj, "xml", sep="."), getDTD = F))[["raised_dollars"]])
return(c(Funding=dollaz))
})
ndf$DateTime<-rep(thisDate, nrow(ndf))
write.table(ndf, "./log.csv", sep=",", row.names=F, append=TRUE, col.names = FALSE) #appends to pre-existing log file
#####write.table(ndf, "./log.csv", sep=",", row.names=F) #creates new log file, comment out after running once
library(plyr)
library(ggplot2)
##Rscript scifundGraphMake.r
#setwd("~/scifund_analysis")
fullTab<-read.csv("./log.csv")
projects<-read.csv("./projects.csv")
maxGoal<-sum(projects$Goal)
####generate graphs
mdf<-merge(projects, fullTab) #all
#mdf$Date<-strsplit(as.character(mdf$DateTime)," ")[[1]][1]
dateTemp<-strsplit(as.character(mdf$DateTime)," ")
mdf$Date<-sapply(dateTemp, function(x) x[1])
mdf_daily<-ddply(mdf, .(ID, Title, Name, Date), summarize, Funding=max(Funding, na.rm=T))
#mdf$DateTime<-as.POSIXct(as.character(mdf$DateTime), format="%y/%m/%d %H:%M:%S")
mdf$DateTime<-as.POSIXct(as.character(mdf$DateTime))
tdf<-ddply(mdf, .(DateTime), summarize, Date=Date[1], Funding=sum(Funding), Group=1) #total
tdf_daily<-ddply(tdf, .(Date), summarize, Funding=max(Funding, na.rm=T), Group=1) #total
##GENERATE GRAPHS
#setwd("~/webdocs/public_html/scifundGraphs")
#hourlies
png("1_all_projects_hourly.png")
qplot(DateTime, Funding, colour=Name, group=Name, data=mdf, geom=c("point", "line"),
main="All Projects", xlab="Date")+theme_bw(base_size=14)+opts(legend.position = "none") +
scale_x_datetime(limits=c(as.POSIXct('2011-11-07 18:00:02 PST'), Sys.time()), format = "%m-%d")
dev.off()
png("1a_all_projects_percent_hourly.png")
qplot(DateTime, Funding/Goal*100, colour=Name, group=Name, ylab="% of Goal", data=mdf, geom=c("point", "line"),
main="All Projects", xlab="")+theme_bw(base_size=14)+opts(legend.position = "none") +
scale_x_datetime(limits=c(as.POSIXct('2011-11-07 18:00:02 PST'), Sys.time()), format = "%Y-%m-%d")
dev.off()
png("3_total.png")
qplot(DateTime, Funding, main="Total #SciFund Donations\n", group=Group, data=tdf,
geom=c("point", "line"), lwd=I(1.5))+theme_bw(base_size=16)+xlab("\nDate")+ylab("Dollars Contributed\n")+
scale_x_datetime(limits=c(as.POSIXct('2011-11-07 18:00:02 PST'), Sys.time()), format = "%m-%d")
dev.off()
###all projects
d_ply(mdf, .(Name), function(adf) {
PRJ<-adf$Title[1]
png(paste(PRJ, "_hourly.png", sep=""))
a<-qplot(DateTime, Funding, data=adf, geom=c("point", "line"), group=Name, main=PRJ)+
theme_bw(base_size=14)+
scale_x_datetime(limits=c(as.POSIXct('2011-11-07 18:00:02 PST'), Sys.time()), format = "%Y-%m-%d")
print(a)
dev.off()
})
#daylies
png("2_all_projects_daily.png")
qplot(Date, Funding, colour=Name, group=Name, data=mdf_daily, geom=c("point", "line"), main="All Projects")+
theme_bw(base_size=14)+xlab("\nDate")+ylab("Dollars Contributed\n")
opts(legend.position = "none")
dev.off()
png("4_total_daily.png")
qplot(Date, Funding, main="Total #SciFund Donations", group=Group, data=tdf_daily, geom=c("point", "line"))+theme_bw(base_size=16)+
xlab("\nDate")+ylab("Dollars Contributed\n")
dev.off()
###all projects
d_ply(mdf_daily, .(Name), function(adf) {
PRJ<-adf$Title[1]
png(paste(PRJ, "_daily.png", sep=""))
a<-qplot(Date, Funding, data=adf, geom=c("point", "line"), group=Name, main=PRJ)+theme_bw(base_size=14)+xlab("\nDate")+ylab("Dollars
Contributed\n")
print(a)
dev.off()
})
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment