Skip to content

Instantly share code, notes, and snippets.

View tomschenkjr's full-sized avatar

Tom Schenk Jr tomschenkjr

View GitHub Profile
@tomschenkjr
tomschenkjr / tidy-api-downloader.R
Last active May 15, 2020 19:55
This is a conceptual study to leverage the tidyverse syntax and lazy evaluation concepts (influenced by sparklyr) to approach API wrapper packages.
# This is a conceptual study to leverage the tidyverse syntax and lazy
# evaluation concepts (influenced by sparklyr) to approach API wrapper packages.
#
# Problem statement: API wrappers often require users to download *all* data
# to manipulate and filter the information. Some packages support custom
# queries but often complicate syntax by adding many parameters within the
# function that may be hard for data scientists to formulate.
#
# Potential solution: API queries can be separated into three principal parts.
# First, to query the API metadata to understand column names and data types.
@tomschenkjr
tomschenkjr / export-socrata.R
Last active May 5, 2017 22:34
An alpha of a export.socrata() function for the RSocrata package. See https://github.com/Chicago/RSocrata/issues/126
library(devtools)
install_github("Chicago/RSocrata" ref = "issue124") # RSocrata 1.7.2-7 or above
library(RSocrata)
#' Exports CSVs from Socrata data portals
#'
#' Input the URL of a data portal (e.g., "data.cityofchicago.org") and
#' will download all CSV files (no other files supported) and saved in
#' a single directory named after the root URL (e.g., "data.cityofchicago.org/").
#' Downloaded files are compressed to GZip format and timestamped so the download
Public Function BASE64SHA1(ByVal sTextToHash As String)
Dim asc As Object
Dim enc As Object
Dim TextToHash() As Byte
Dim SharedSecretKey() As Byte
Dim bytes() As Byte
Const cutoff As Integer = 5
Set asc = CreateObject("System.Text.UTF8Encoding")
@tomschenkjr
tomschenkjr / geocode.R
Created January 10, 2016 23:10
A note on how to geocode using R, since I always forget
# A note on how to geocode using R, since I always forget
library(tigris) # also loads 'sp'
library(RSocrata)
# Obtain data
d <- read.socrata("https://data.cityofchicago.org/Education/Connect-Chicago-Locations/bmus-hp7e")
ccgeo <- tracts(state = '17', county = c('031'), cb = T) ## cb=T means smaller file
# Data organization
@tomschenkjr
tomschenkjr / Julia Workshop notes 20141115.jl
Created November 18, 2014 00:39
Notes from Julia Workshop: From Installed to Productive
# Notes from Julia Workshop: From Installed to Productive
# 2014-11-15
# http://www.meetup.com/JuliaChicago/events/216950712/
println("hello world") # => hello world
x = 5
## IF statements
@tomschenkjr
tomschenkjr / CPS_HS_Progress_Report_2014_Transformations.txt
Created December 18, 2013 19:57
This gist contains the original pre-converted data file for CPS High School Progress Report 2013-2014. The second file is a JSON layout that contains the transformations applied within OpenRefine to transform the data file. The file transforms the original file to the published copy at https://data.cityofchicago.org/id/2m8w-izji. These files acc…
[
{
"op": "core/text-transform",
"description": "Text transform on cells in column Phone Number using expression grel:value.replace(\" -\",\"\")",
"engineConfig": {
"facets": [],
"mode": "row-based"
},
"columnName": "Phone Number",
"expression": "grel:value.replace(\" -\",\"\")",
@tomschenkjr
tomschenkjr / chicago-cla-v1
Last active December 18, 2015 11:39
Draft of Chicago's Contributor Licensing Agreement
City of Chicago
Individual Contributor License Agreement ("Agreement")
======================================================
Thank you for your interest in [Project Name] by the City of Chicago (the "City"),
a municipal corporation and home rule unit of local government existing under the
Constitution of the State of Illinois. In order to clarify the intellectual property
license granted with Contributions from any person or entity, the City must have a
Contributor License Agreement ("CLA") on file that has been signed by each Contributor,
indicating agreement to the license terms below. This license is for your protection
CoefficientPlot <- function(models, alpha = 0.05, modelnames = ""){
# models must be a list()
Multiplier <- qnorm(1 - alpha / 2)
CoefficientTables <- lapply(models, function(x){summary(x)$coef})
TableRows <- unlist(lapply(CoefficientTables, nrow))
if(modelnames[1] == ""){
ModelNameLabels <- rep(paste("Model", 1:length(TableRows)), TableRows)
} else {
@tomschenkjr
tomschenkjr / reshape-tutorial.R
Created November 19, 2012 01:56
Chicago Data Visualization reshape Tutorial Script
# Install reshape package. You will be asked to pick a server.
install.packages("reshape")
# We're going to also use ggplot2, so let's install that as well:
install.packages(c("reshape","ggplot2")) # In R, c() concatenates inputs as a vector
# You will always need to load the libraries after installing the package.
library(reshape)
library(ggplot2)
@tomschenkjr
tomschenkjr / ggplot2-tutorial-3.R
Created November 12, 2012 20:45
Chicago Data visualization ggplot2 Tutorial 3 Script
# Let's customize our plot outputs. First, load the library.
library(ggplot2)
# Plot something simple
p <- ggplot(mtcars, aes(x=cyl, y=mpg)) + geom_point(aes(color=qsec, size=wt))
# Change axis
p + scale_y_continuous(limits=c(0,40))