Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
Script to download all layers from SimpleGeo storage to CSV files
# The following script allows to export a SimpleGeo Storage layer to a CSV
# You have to change the credentials and the layer that you want to export
# It will generate a CSV file with the name of the layer
#
# Author: Javier de la Torre (jatorre@vizzuality.com @jatorre)
#Adapt the following to your SimpleGeo credentials. Get it from the UI.
oauth_token="token"
oauth_secret="secret"
#-----------------
require 'rubygems'
require 'simplegeo'
require 'csv'
SimpleGeo::Client.set_credentials(oauth_token, oauth_secret)
#get all layers for user
layers = SimpleGeo::Client.get_layers()[:layers]
layers.each do |layer|
CSV.open("#{layer[:name].gsub(".","_")}.csv", "wb") do |csv|
next_cursor=""
header=false
#loop until next_cursor is nil (pagination)
#the query basically look for all records within the whole world as I couldnt find a way to download a full layer
while next_cursor do
puts "."
res = SimpleGeo::Client.get_nearby_records(layer[:name],
:lat => 0,:lon => 0, :bbox=>'-90,-180,90,180', :limit=>500,:cursor=>next_cursor)
next_cursor = res[:next_cursor]
res[:records].each do |row|
#add headers
if !header
csv << ["id", "lat", "lon","created"] + row[:record].properties.keys.map(&:to_s) - ["layer"]
header=true
else
#add records
csv << [row[:record].id,row[:record].lat,row[:record].lon, row[:record].created.strftime("%Y-%m-%d %H:%M:%S")] +
row[:record].properties.values - [layer[:name]]
end
end
end
end
end

kwylez commented Jan 18, 2012

Javier...thanks so much for writing this script. Myself and a friend made Python port of this script. I gave you credit for the original 'design' https://gist.github.com/1632842

Owner

jatorre commented Jan 18, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment