Skip to content

Instantly share code, notes, and snippets.

@freerobby
Last active December 10, 2015 11:59
Show Gist options
  • Save freerobby/4431324 to your computer and use it in GitHub Desktop.
Save freerobby/4431324 to your computer and use it in GitHub Desktop.
Exports all keys in a riak bucket into a newline-delimited file.
# This function uses net::http streaming and the Riak HTTP endpoint
# to export all keys in a riak bucket into a newline-delimited file.
require 'json'
require 'net/http'
RIAK_ROOT = 'http://localhost:8098'
# bucket: The bucket to export
# path: The folder into which to put the output file
# update_per: Console output every this many keys
def export_bucket_keys(bucket, path = '/mnt', update_per = 1000)
uri = URI("#{RIAK_ROOT}/buckets/#{bucket}/keys?keys=stream")
count = 0
File.open("#{path}/#{bucket}_dump.dat", 'w') do |f|
Net::HTTP.get_response(uri) do |chunk|
json_blob = ''
chunk.read_body do |body|
json_blob += body if body
if json_blob && json_blob.length > 1
begin
JSON.parse(json_blob)['keys'].each do |k|
f.puts k
count += 1
puts "Wrote #{count} keys (last: #{k})" if count % update_per == 0
end
json_blob = ''
rescue JSON::ParserError # This happens when a JSON object is streamed over multiple chunks
end
end
end
end
end
end
export_bucket_keys('links', '/mnt', 100000) # Generates /mnt/links_dump.dat with console updates every 100k lines
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment