Skip to content

Instantly share code, notes, and snippets.

@mmmries
Last active August 29, 2015 14:00
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mmmries/11193064 to your computer and use it in GitHub Desktop.
Save mmmries/11193064 to your computer and use it in GitHub Desktop.
Avro Gem Broken??? Or I don't understand it?
source "https://rubygems.org"
gem "avro", "1.7.5"
GEM
remote: https://rubygems.org/
specs:
avro (1.7.5)
yajl-ruby
yajl-ruby (1.2.0)
PLATFORMS
ruby
DEPENDENCIES
avro (= 1.7.5)
$ rvm use 2.1.1
Using /Users/michael.ries/.rvm/gems/ruby-2.1.1
$ ruby test.rb
~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:46:in `byte!': undefined method `unpack' for nil:NilClass (NoMethodError)
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:67:in `read_long'
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:301:in `read_data'
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:384:in `block in read_record'
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:382:in `each'
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:382:in `read_record'
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:310:in `read_data'
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/io.rb:275:in `read'
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/data_file.rb:243:in `block in each'
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/data_file.rb:231:in `loop'
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/data_file.rb:231:in `each'
from test.rb:41:in `block in <main>'
from ~/.rvm/gems/ruby-2.1.1/gems/avro-1.7.5/lib/avro/data_file.rb:45:in `open'
from test.rb:40:in `<main>'
$ rvm use 2.0.0
Using /Users/michael.ries/.rvm/gems/ruby-2.0.0-p451
$ ruby test.rb
~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:46:in `byte!': undefined method `unpack' for nil:NilClass (NoMethodError)
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:67:in `read_long'
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:301:in `read_data'
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:384:in `block in read_record'
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:382:in `each'
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:382:in `read_record'
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:310:in `read_data'
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/io.rb:275:in `read'
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/data_file.rb:243:in `block in each'
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/data_file.rb:231:in `loop'
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/data_file.rb:231:in `each'
from test.rb:41:in `block in <main>'
from ~/.rvm/gems/ruby-2.0.0-p451/gems/avro-1.7.5/lib/avro/data_file.rb:45:in `open'
from test.rb:40:in `<main>'
require 'bundler/setup'
require 'avro'
data = [
{"guid"=>"144045de-eb44-dd1b-d9af-6c8b5d41a96e", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617818, "updated_at"=>1398180288, "deleted_at"=>nil},
{"guid"=>"51e06057-14d2-7527-81fa-b07dba0a263b", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"Student Loans R' Us", "created_at"=>1386178342, "updated_at"=>1398180286, "deleted_at"=>nil},
{"guid"=>"b4d1d99f-4351-d0e7-221c-a3fae08716bc", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617026, "updated_at"=>1398180288, "deleted_at"=>nil},
{"guid"=>"084638fa-a78d-bbdd-e075-7c9c957a9b46", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617138, "updated_at"=>1398180288, "deleted_at"=>nil},
{"guid"=>"79287c76-4e8f-0a21-7569-a2bcdc2b2f4d", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617135, "updated_at"=>1398180288, "deleted_at"=>nil},
{"guid"=>"3bcc26b2-7d3b-6c4d-cb27-4eb1574b3c20", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"Cayman Islands Bank", "created_at"=>1386902345, "updated_at"=>1398180288, "deleted_at"=>nil},
{"guid"=>"75e1e56c-7611-4030-d002-afa2af70e5a1", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617427, "updated_at"=>1398180288, "deleted_at"=>nil},
]
member_schema = <<-SCHEMA
{"namespace": "md.data_logs",
"type": "record",
"name": "Member",
"fields": [
{"name": "guid", "type": "string"},
{"name": "user_guid", "type": "string"},
{"name": "name", "type": ["string","null"]},
{"name": "created_at", "type":"long"},
{"name": "updated_at", "type":"long"},
{"name": "deleted_at", "type":["long","null"]}
]
}
SCHEMA
filepath = "./members.avro"
File.unlink(filepath) if File.exists?(filepath)
Avro::DataFile.open(filepath, "w", member_schema) do |dw|
data.each do |entry|
dw << entry
end
end
entries = []
Avro::DataFile.open(filepath, "r") do |reader|
reader.each do |entry|
entries << entry
end
end
puts "Here is the data I wrote into the file:"
data.each{|e| p e }
print "\n\n\n\n"
puts "Here is the data I read from the file:"
entries.each{|e| p e }
require 'bundler/setup'
require 'avro'
entries = []
Avro::DataFile.open("./members.from_java_client.avro", "r") do |reader|
reader.each do |entry|
entries << entry
end
end
puts "Here is the data I read from the file:"
entries.each{|e| p e }
@benforeva
Copy link

I'm getting this error on and off as well, it seems to be a problem with integers and long types. I tried to trace it to a specific range of values and I found that ints greater than 6 digits would throw the error. However I've been testing successfully with ints above 6 digits for some time so the error may not be consistent with this condition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment