Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save andrewbtran/8507f79e0ded054ac2b38878f94063dc to your computer and use it in GitHub Desktop.
Save andrewbtran/8507f79e0ded054ac2b38878f94063dc to your computer and use it in GitHub Desktop.
new orleans schools for loop
schools_list <- fromJSON("")
schools_list <- schools_list$data
# for loop
# for i in 1 through number of rows of the dataframe schools_list
for (i in 1:nrow(schools_list)) {
slug <- schools_list$slug[i]
# paste0() is the base R equivalent to stringr's string_c
# It concatenates strings together with nothing separating it
# Rebuild the string to the school data json using the url we know
school_url <- paste0("", slug, ".json")
school_data <- fromJSON(school_url)
# extracting the data only
school_data <- school_data$data
# transforming data from wide to tall
# this is necesary because some schools don't have as many variables in the json as other schools
# so this way it can still be joined with incomplete data
school_data <- gather(school_data, "type", "data", 2:ncol(school_data))
# building a big data frame of all the these pulled data frames
if (i==1) { # if this is the first time this is looping...
school_df <- school_data
# then make the first pulled data frame the official data frame
} else { # otherwise if it's not the first time this has looped
# then rbind the new scraped dat frame to the older scraped data frame
school_df <- rbind(school_df, school_data)
# this is essentially appending the new data to the building data
# This prints out the status of the loop so I know how far it's progressed
print(paste0(i, " of ", nrow(schools_list)))
# Spreads out the data wide like you'd see in a spreadsheet
# By turning the wide data to tall, combining, and then spreading it wide again
# We can get the wide data we had originally but won't be hindered by the missing data this way
school_df_wide <- spread(school_df, type, data)
# This is a cheat to deal with lists in the dataframe into normal arrays
df <- data.frame(apply(school_df_wide,2,as.character))
# This only happens when you're dealing with converting json into data frames
# saving the dataframe as a spreadsheet
write.csv(df, "school_df.csv", row.names=F)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment