Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Script to download all layers from SimpleGeo storage to CSV files
# The following script allows to export a SimpleGeo Storage layer to a CSV
# You have to change the credentials and the layer that you want to export
# It will generate a CSV file with the name of the layer
#
# Author: Javier de la Torre (jatorre@vizzuality.com @jatorre)
#Adapt the following to your SimpleGeo credentials. Get it from the UI.
oauth_token="token"
oauth_secret="secret"
#-----------------
require 'rubygems'
require 'simplegeo'
require 'csv'
SimpleGeo::Client.set_credentials(oauth_token, oauth_secret)
#get all layers for user
layers = SimpleGeo::Client.get_layers()[:layers]
layers.each do |layer|
CSV.open("#{layer[:name].gsub(".","_")}.csv", "wb") do |csv|
next_cursor=""
header=false
#loop until next_cursor is nil (pagination)
#the query basically look for all records within the whole world as I couldnt find a way to download a full layer
while next_cursor do
puts "."
res = SimpleGeo::Client.get_nearby_records(layer[:name],
:lat => 0,:lon => 0, :bbox=>'-90,-180,90,180', :limit=>500,:cursor=>next_cursor)
next_cursor = res[:next_cursor]
res[:records].each do |row|
#add headers
if !header
csv << ["id", "lat", "lon","created"] + row[:record].properties.keys.map(&:to_s) - ["layer"]
header=true
else
#add records
csv << [row[:record].id,row[:record].lat,row[:record].lon, row[:record].created.strftime("%Y-%m-%d %H:%M:%S")] +
row[:record].properties.values - [layer[:name]]
end
end
end
end
end
@kwylez

This comment has been minimized.

Copy link

commented Jan 18, 2012

Javier...thanks so much for writing this script. Myself and a friend made Python port of this script. I gave you credit for the original 'design' https://gist.github.com/1632842

@jatorre

This comment has been minimized.

Copy link
Owner Author

commented Jan 18, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.