Last active
August 29, 2015 14:00
-
-
Save mmmries/11193064 to your computer and use it in GitHub Desktop.
Avro Gem Broken??? Or I don't understand it?
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
source "https://rubygems.org" | |
gem "avro", "1.7.5" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
GEM | |
remote: https://rubygems.org/ | |
specs: | |
avro (1.7.5) | |
yajl-ruby | |
yajl-ruby (1.2.0) | |
PLATFORMS | |
ruby | |
DEPENDENCIES | |
avro (= 1.7.5) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
$ rvm use 2.1.1 | |
Using /Users/michael.ries/.rvm/gems/ruby-2.1.1 | |
$ ruby test.rb | |
~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:46:in `byte!': undefined method `unpack' for nil:NilClass (NoMethodError) | |
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:67:in `read_long' | |
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:301:in `read_data' | |
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:384:in `block in read_record' | |
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:382:in `each' | |
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:382:in `read_record' | |
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:310:in `read_data' | |
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:275:in `read' | |
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/data_file.rb:243:in `block in each' | |
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/data_file.rb:231:in `loop' | |
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/data_file.rb:231:in `each' | |
from test.rb:41:in `block in <main>' | |
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/data_file.rb:45:in `open' | |
from test.rb:40:in `<main>' | |
$ rvm use 2.0.0 | |
Using /Users/michael.ries/.rvm/gems/ruby-2.0.0-p451 | |
$ ruby test.rb | |
~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:46:in `byte!': undefined method `unpack' for nil:NilClass (NoMethodError) | |
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:67:in `read_long' | |
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:301:in `read_data' | |
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:384:in `block in read_record' | |
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:382:in `each' | |
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:382:in `read_record' | |
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:310:in `read_data' | |
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:275:in `read' | |
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/data_file.rb:243:in `block in each' | |
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/data_file.rb:231:in `loop' | |
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/data_file.rb:231:in `each' | |
from test.rb:41:in `block in <main>' | |
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/data_file.rb:45:in `open' | |
from test.rb:40:in `<main>' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
require 'bundler/setup' | |
require 'avro' | |
data = [ | |
{"guid"=>"144045de-eb44-dd1b-d9af-6c8b5d41a96e", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617818, "updated_at"=>1398180288, "deleted_at"=>nil}, | |
{"guid"=>"51e06057-14d2-7527-81fa-b07dba0a263b", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"Student Loans R' Us", "created_at"=>1386178342, "updated_at"=>1398180286, "deleted_at"=>nil}, | |
{"guid"=>"b4d1d99f-4351-d0e7-221c-a3fae08716bc", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617026, "updated_at"=>1398180288, "deleted_at"=>nil}, | |
{"guid"=>"084638fa-a78d-bbdd-e075-7c9c957a9b46", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617138, "updated_at"=>1398180288, "deleted_at"=>nil}, | |
{"guid"=>"79287c76-4e8f-0a21-7569-a2bcdc2b2f4d", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617135, "updated_at"=>1398180288, "deleted_at"=>nil}, | |
{"guid"=>"3bcc26b2-7d3b-6c4d-cb27-4eb1574b3c20", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"Cayman Islands Bank", "created_at"=>1386902345, "updated_at"=>1398180288, "deleted_at"=>nil}, | |
{"guid"=>"75e1e56c-7611-4030-d002-afa2af70e5a1", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617427, "updated_at"=>1398180288, "deleted_at"=>nil}, | |
] | |
member_schema = <<-SCHEMA | |
{"namespace": "md.data_logs", | |
"type": "record", | |
"name": "Member", | |
"fields": [ | |
{"name": "guid", "type": "string"}, | |
{"name": "user_guid", "type": "string"}, | |
{"name": "name", "type": ["string","null"]}, | |
{"name": "created_at", "type":"long"}, | |
{"name": "updated_at", "type":"long"}, | |
{"name": "deleted_at", "type":["long","null"]} | |
] | |
} | |
SCHEMA | |
filepath = "./members.avro" | |
File.unlink(filepath) if File.exists?(filepath) | |
Avro::DataFile.open(filepath, "w", member_schema) do |dw| | |
data.each do |entry| | |
dw << entry | |
end | |
end | |
entries = [] | |
Avro::DataFile.open(filepath, "r") do |reader| | |
reader.each do |entry| | |
entries << entry | |
end | |
end | |
puts "Here is the data I wrote into the file:" | |
data.each{|e| p e } | |
print "\n\n\n\n" | |
puts "Here is the data I read from the file:" | |
entries.each{|e| p e } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
require 'bundler/setup' | |
require 'avro' | |
entries = [] | |
Avro::DataFile.open("./members.from_java_client.avro", "r") do |reader| | |
reader.each do |entry| | |
entries << entry | |
end | |
end | |
puts "Here is the data I read from the file:" | |
entries.each{|e| p e } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm getting this error on and off as well, it seems to be a problem with integers and long types. I tried to trace it to a specific range of values and I found that ints greater than 6 digits would throw the error. However I've been testing successfully with ints above 6 digits for some time so the error may not be consistent with this condition.