- Datasets first, APIs second - Doing any kind of aggregate analysis usually requires working with complete datasets. REST APIs aren't ideal for this use case. APIs are not data, they are a means of exposing it.
- Machine-friendly retrieval of raw datasets - Avoid the assumption that there's a human, using a web browser, manually clicking a link. For example, scripts that fetch new daily crime data via curl would be a likely scenario. Make it easy for machines by removing authentication, unnecessary redirects, JavaScript-based retrieval or POST-style retrieval.
- Document long column names - Shapefile attributes are limited to 10 characters. This makes many attributes difficult to decipher without associated metadata. For example, here are a few attributes from a Garbage Collection dataset. Include a file with the long column name mappings and include both the long and short name in the metada
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
select array_to_string(array_agg(i), '') from | |
(select (regexp_matches('Letter-1-2', '[A-Z0-9]', 'g'))[1] i) t; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
location | |
---|---|
Luverne, Alabama | |
Madison, Alabama | |
Theodore, Alabama | |
Oneonta, Alabama | |
Odenville, Alabama | |
Heflin, Alabama | |
Jasper, Alabama | |
Midfield, Alabama | |
Greenville, Alabama |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<!DOCTYPE html> | |
<meta charset="utf-8"> | |
<style> | |
.axis path, | |
.axis line { | |
fill: none; | |
stroke: #000; | |
shape-rendering: crispEdges; | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
drop table if exists inspection_point_buffers; | |
-- Group identical overlapping points and count how many occupy | |
-- the space. | |
create temporary table inspection_overlappoing_points as | |
select a.geom as geom, | |
count(*) | |
from latest_inspections a, | |
latest_inspections b | |
where st_equals(a.geom, b.geom) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<!doctype html> | |
<head> | |
<meta charset="utf-8"> | |
<style> | |
body { | |
font-family: OpenSans, Helvetica; | |
} | |
.title { | |
margin: 0; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
column_name | |
-------------------------------------------------------- | |
crash_id | |
record_type | |
vehicle_id | |
participant_id | |
participant_display_seq | |
vehicle_coded_seq | |
participant_vehicle_seq | |
serial_ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
drop table if exists combined_geometries; | |
with boston_area_geometries as | |
( select name, | |
msa_code, | |
geom | |
from divisions | |
inner join | |
( select distinct on (msa_code) msa_code | |
from area_definitions ) ad on ad.msa_code = nctadvfp ), |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#= require d3 | |
# Draw timeseries graphs to the screen. Each element can contain a set of | |
# data-* attributes used to configure the graph. The graph should always include | |
# a data-url attribute pointing to an endpoint for time series JSON data. | |
# | |
# Any graph that includes a data-realtime attribute will update automatically. | |
# | |
# Examples: | |
# <div class="js-graph" data-url="/graphite?target=github.unicorn.{browser,api}.cpu_time.mean&from=-1hour" data-realtime></div> |
We can make this file beautiful and searchable if this error is corrected: It looks like row 6 should actually have 40 columns, instead of 10. in line 5.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
date C Io JavaScript Lua Ruby Arc Erlang Objective-C VimL Ragel in Ruby Host Scala Emacs Lisp Python Shell Assembly Java Perl C++ Objective-J ActionScript OCaml Nu PHP Common Lisp CSS Clojure Matlab CoffeeScript ooc Fancy Racket Rust TeX Haskell TypeScript Elixir wisp Go C# | |
2008-02 4.545454545454546 4.545454545454546 9.090909090909092 2.272727272727273 79.54545454545455 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 | |
2008-03 7.4074074074074066 0 7.4074074074074066 7.4074074074074066 62.96296296296296 3.7037037037037033 3.7037037037037033 3.7037037037037033 3.7037037037037033 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 | |
2008-04 6.25 0 9.375 0 78.125 0 0 0 0 3.125 3.125 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 | |
2008-05 3.225806451612903 0 6.451612903225806 0 83.87096774193549 0 0 0 0 0 0 3.225806451612903 3.225806451612903 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 | |
2008-06 3.0303030303030303 0 9.090909090909092 0 78.78787878787878 0 0 3.0303030303030303 |