invisiblefunnel

require 'spec_helper'
require 'geo_ruby/geojson'

describe "GeoJSON serialization" do
  before do
    @geometry = {"type"=>"Point", "coordinates"=>[102.0, 0.5]}
    Georelevent::App.database.create_table? :birds do
      primary_key :id
      column :geom, :geometry
    end

invisiblefunnel

[DRAFT] ETL micro-services as a stepping stone towards robust government data infrastructure

This is a proposal for the creation of a set of simple, distributed ETL web services with the goal of increasing accessibility and openness of data publicly available on the web. With these tools, the civic data community can (mostly) bypass existing bottlenecks and create an immediately useful ecosystem to build upon. It is possible (indeed, likely) that ETL micro-services will be superseded by tools like dat that will address a much wider range of data management needs, but that does not negate the need for a stopgap tool.

Use cases

• Data is accessible over HTTP, but in a non-standard format
• Systems can't directly depend on a data source because of slow response times
• Public data is only accessible through an authenticated API

invisiblefunnel

Factories
  builds a valid publisher object
  builds a valid subscription object
  builds a valid event object

Georelevent::Routes::Home
  GET /
    returns a non-empty body
    responds with 200 OK

invisiblefunnel

$ heroku run console -a whats-happ
Running `console` attached to terminal... up, run.9945
Loading production environment (Rails 4.0.5)
irb(main):001:0> Date.today
=> Wed, 04 Jun 2014
irb(main):002:0> Subscription.count
=> 33
irb(main):003:0> Notification.count
=> 2695

invisiblefunnel

require 'dotenv'
Dotenv.load

require 'json'
require 'sequel'
require 'sinatra'

DB = Sequel.connect(ENV.fetch('DATABASE_URL'))

DB.create_table?(:hooks) do

invisiblefunnel

invisiblefunnel

invisiblefunnel

georelevent_development=# EXPLAIN SELECT
georelevent_development-#   subscriptions.id as subscriber_id,
georelevent_development-#   events.id AS event_id
georelevent_development-# FROM subscriptions
georelevent_development-# INNER JOIN events
georelevent_development-#   ON ST_Intersects(subscriptions.geom, events.geom)
georelevent_development-#   AND subscriptions.publisher_id = events.publisher_id
georelevent_development-# WHERE events.id in (1,2,3,4);
                                                QUERY PLAN

invisiblefunnel

invisiblefunnel