MozFest: Streamtools for Open Data

_blurb.txt

Want to explore the world of open government & civic data without having to write parsers? Or perhaps you don't know how to code? Bring your laptop to this session and learn how to work with data in realtime with the Streamtools GUI! 

sources.md

Streamtools for Open Data

Data + pattern ideas.

Repo: https://github.com/jacqui/mozfest_open_data

Phone

This is just fun.

• vote

Crisis.net

• Endpoint http://api.crisis.net/item
• Api Key 535a9fa06a2165d01fabc678 (jacqui's)
• Examples
  • ebola
  • syria
• Patterns
  • ebola
  • syria

Wikipedia Edits

Connects to Wikipedia's websocket endpoint and monitors edits. Concepts: websocket, filtering, histogram/bucketed counts, masking/keeping what you want in the data.

• Endpoint ws://wikimon.hatnote.com:80/en
• API Key n/a
• Pattern builds histogram of non-robot/anon uses

US Gov Data

• Endpoint http://data.gov

OpenFDA Food Recalls

• data https://api.fda.gov/food/enforcement.json?api_key=JwWMMFxvcd95fmcGmahxmUbis3te4cYPEozY81eP&limit=25
• pattern

FCC Responses to Net Neutrality Debate

The FCC opened the data from a callout for comments on the net neutrality debate. The archives are huge. Thsi shows how streamtools can help make sense of data like that.

• pattern

CDC/Public Health

Vaccine symptoms/reactions

• data
• pattern

UK Gov Data

Tons of data available by the British government at http://data.gov.uk/data/search

• restaurant ratings - Greenwich
  • data:
    • xml
    • json api
  • [pattern:
    • xml
    • json api
  • location list
• traffic accidents
  • data
  • pattern
• gov't spending (csv)
• approved wedding places - csv
• empty residential places json

There's a 3rd-party API for the TfL Cycle Hire here: http://bike-stats.co.uk/
And detailed crime info from the UK police: http://data.police.uk/docs/ (e.g. all crime at the lat/long near Mozfest in August 2014 here)

Here's a pattern based on the bike-stats.co.uk data. There's no bike share station right near Ravensbourne, so this one is from Lancaster Drive, Blackwall (ID #526).

{"Blocks":[{"Id":"4","Type":"tolog","Rule":null,"Position":{"X":515.9886169433594,"Y":473.99147033691406}},{"Id":"1","Type":"ticker","Rule":{"Interval":"1m0s"},"Position":{"X":198.99998474121094,"Y":23.99715805053711}},{"Id":"6","Type":"bang","Rule":null,"Position":{"X":285.9886169433594,"Y":48}},{"Id":"17","Type":"filter","Rule":{"Filter":".@ID == '526'"},"Position":{"X":380.991455078125,"Y":316.9886169433594}},{"Id":"25","Type":"mask","Rule":{"Mask":{"bikesAvailable":{},"emptySlots":{}}},"Position":{"X":423.991455078125,"Y":389.9886169433594}},{"Id":"10","Type":"unpack","Rule":{"Path":".dockStation"},"Position":{"X":330.9942932128906,"Y":231.98863220214844}},{"Id":"13","Type":"gethttp","Rule":{"Path":"'http://api.bike-stats.co.uk/service/rest/bikestats?format=json'"},"Position":{"X":302.99432373046875,"Y":162}}],"Connections":[{"Id":"14","FromId":"6","ToId":"13","ToRoute":"in"},{"Id":"16","FromId":"1","ToId":"13","ToRoute":"in"},{"Id":"18","FromId":"10","ToId":"17","ToRoute":"in"},{"Id":"15","FromId":"13","ToId":"10","ToRoute":"in"},{"Id":"26","FromId":"17","ToId":"25","ToRoute":"in"},{"Id":"27","FromId":"25","ToId":"4","ToRoute":"in"}]}