Skip to content

Instantly share code, notes, and snippets.

View extract-all-file-git-commits.sh
# this shell script extracts all prior versions of a file in a git repository
# make sure to enable execute permissions on this file if your operating system
# requires it: chmod +x extract-all-file-git-commits.sh
# parameters:
# - path-to-file (from root of repository)
# - output directory
# - output file extention
# eg. ./extract-all-file-commits.sh "./public/data/vaccinations/locations.csv" "./output" "csv"
@technickle
technickle / ValidateOpen311GeoReportBulk.r
Last active Apr 27, 2017
R validator script for Open311 GeoReport Bulk specification compatibility
View ValidateOpen311GeoReportBulk.r
# this R script evaluates a data file for compatibility with the Open311 GeoReport Bulk specification.
# see here for the most recent version of the specification:
# http://wiki.open311.org/GeoReport/bulk
#
# it implements nearly all of the checks identified in this document
# https://docs.google.com/document/d/1GLRniiT3xvmG-i6PPeZPZDK_FhBDGCpuVh5fCexEiys/preview
# however, it is very bare bones and the results need to be interpreted.
#
# written by Andrew Nicklin (@technickle) with contributions from the Open311 community.
#
@technickle
technickle / StreamGraph-Open311-Bulk-Data.r
Last active Dec 13, 2016
Processes and visualizes compatible government service request data
View StreamGraph-Open311-Bulk-Data.r
# Process and streamgraph-visualize a service request data file compatible with the Open311 GeoReport bulk specification.
# by @technickle.
# This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
# (see https://creativecommons.org/licenses/by-sa/4.0/ for more)
# note: you may have issues installing the streamgraph library.
# For help, see comments at https://gist.github.com/technickle/67c3cebb687a3b370d0ea3435012b941
library(readr)
library(dplyr)
library(streamgraph)
View process-local-website-uris
# this script parses a CSV download from
# https://data.ny.gov/Government-Finance/New-York-State-Locality-Hierarchy-with-Websites/55k6-h6qq
# and parses each row's URI to add path, hostname, and global top-level domain columns
# the resulting file is written back to the current path
require 'uri'
require 'CSV'
# load an array of (row) arrays from "locals.csv"
localgovs = CSV.read("locals.csv")
@technickle
technickle / generate_flight_details.r
Last active Aug 29, 2015
Builds a consolidated CSV of all known world flights based upon data at http://www.openflights.org/data.html
View generate_flight_details.r
# usage: download routes.dat, airports.dat, and airlines.dat from
# http://openflights.org/data.html
# replace this location with the path where you downloaded the files
mypath = "~/"
# run the script!
# set working directory
setwd(mypath)
# load the openflights data files
@technickle
technickle / state-ope-security-data.r
Last active Aug 29, 2015
Import and process Federal OPE Campus Security Data
View state-ope-security-data.r
# This R script grabs 5 years of OPE security data for a specified state
#
# How to use:
# 1) Go to http://www.ope.ed.gov/security/GetDownloadFile.aspx
# 2) download SPSS version of 3 most recent sets (past 5 years of data)
# 3) from each zip file, copy oncampuscrime*.sav, noncampuscrime*.sav,
# Residencehallcrime*.sav, and Publicpropertycrime*.sav
# out to a folder of your choosing. It should have 12 files when done
# 4) use the one of the most recent year's .sav files to geocode addresses;
# geocoding should be in a CSV with at least UNITID_P, x, y, and county;