dannguyen

if !database: wget + grep

The Federal Aviation Administration is posting PDFs of the Section 333 exemptions that it grants, i.e. the exemptions for operators who want to fly drones commercially before the FAA finishes its rulemaking. A journalist wanted to look for exemptions granted to operators in a given U.S. state. But the FAA doesn't appear to have an easy-to-read data file to use and doesn't otherwise list exemptions by location of operator.

However, since their exemptions page is just one giant HTML table for listing the PDFs, we can just use wget to fetch all the PDFs, run pdftotext on each file, and then [grep](https://medium.com/@rualthanzauva/grep-was-a-private-command-of-m

dannguyen

The shell script was used to clean the data and upload it to BigQuery. You can access the data yourself there:

https://bigquery.cloud.google.com/table/nypd-data:nypd.stop_and_frisk

Or if you simply want a giant zip of CSV files, go here (note: You'll have to do the work of reconciling the columns in order to get all the files into one database):

http://stash.compciv.org.s3.amazonaws.com/nypd-stop-and-frisk/2003-2013.zip

The R script in this gist produces the image below:

mnutt

Instrument Anything in Rails 3

With Rails 3.0 released a few weeks ago I've migrated a few apps and I'm constantly finding useful new improvements. One such improvement is the ability to log anything in the same way that Rails internally logs ActiveRecord and ActionView. By default Rails 3 logs look slightly spiffier than those produced by Rails 2.3: (notice the second line has been cleaned up)

Started GET "/" for 127.0.0.1 at Mon Sep 06 01:07:11 -0400 2010
  Processing by HomeController#index as HTML
  User Load (0.2ms)  SELECT `users`.* FROM `users` WHERE (`users`.`id` = 3) LIMIT 1
  CACHE (0.0ms)  SELECT `users`.* FROM `users` WHERE (`users`.`id` = 3) LIMIT 1

Rendered layouts/_nav.html.erb (363.4ms)