Weather Underground API Debugging

1_9_3_result.md

Ruby MRI 1.9.3 Results

[gpetrie] $ ruby -v
ruby 1.9.3p429 (2013-05-15 revision 40747) [x86_64-darwin11.4.2]

[gpetrie] $ ruby wunderground.rb
Current temperature in Cedar Rapids is: 77.0

2_0_0_result.md

Ruby MRI 2.0.0 Results

This is a direct paste from a clean Vagrant Precise32 VM build using RVM as the Ruby version manager.

vagrant@precise32:~$ ruby -v
ruby 2.0.0p195 (2013-05-14 revision 40734) [i686-linux]
vagrant@precise32:~$ ruby /vagrant/wunderground.rb
/home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/net/http/response.rb:357:in `finish': incorrect header check (Zlib::DataError)
  from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/net/http/response.rb:357:in `finish'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/net/http/response.rb:262:in `ensure in inflater'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/net/http/response.rb:262:in `inflater'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/net/http/response.rb:274:in `read_body_0'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/net/http/response.rb:201:in `read_body'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/open-uri.rb:328:in `block (2 levels) in open_http'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/net/http.rb:1413:in `block (2 levels) in transport_request'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/net/http/response.rb:162:in `reading_body'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/net/http.rb:1412:in `block in transport_request'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/net/http.rb:1403:in `catch'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/net/http.rb:1403:in `transport_request'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/net/http.rb:1376:in `request'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/open-uri.rb:319:in `block in open_http'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/net/http.rb:852:in `start'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/open-uri.rb:313:in `open_http'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/open-uri.rb:708:in `buffer_open'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/open-uri.rb:210:in `block in open_loop'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/open-uri.rb:208:in `catch'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/open-uri.rb:208:in `open_loop'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/open-uri.rb:149:in `open_uri'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/open-uri.rb:688:in `open'
	from /home/vagrant/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/2.0.0/open-uri.rb:34:in `open'
	from /vagrant/wunderground.rb:3:in `<main>'

README.md

README

The Issue

This api call to the Weather Underground service works as expected using MRI ruby-1.9.3-p429 but not when using MRI ruby-2.0.0-p195.

This gist shows the code and the response when using 1.9.3 and 2.0.0.

The code sample is taken directly from the Weather Underground's documentation

Update

After working with @zph and @injekt there have been some interesting findings. I am including wunderground-working-2_0.rb as an example of a working version of the Weather Underground API using Ruby MRI 2.0.0.

The interesting piece was when @injekt showed that the rest-client gem could call the api without error. He then found that the gem was manually rescuing the Zlib::DataError (https://github.com/rest-client/rest-client/blob/master/lib/restclient/request.rb#L258).

So, for now, if I want to use the Weather Underground API and MRI 2.0.0, I'll be doing it with the rest-client gem.

Additionally interesting reading can be found with this Ruby core commit, where a significant update was made to the Zlib libraries.

wunderground-working-2_0.rb

require 'rest-client'
require 'json'

api_key = '' # required by producing your own API key from http://www.wunderground.com/weather/api/ 
url = "http://api.wunderground.com/api/#{api_key}/geolookup/conditions/q/IA/Cedar_Rapids.json"

res = RestClient.get url
parsed_json = JSON.parse(res)
location = parsed_json['location']['city']
temp_f = parsed_json['current_observation']['temp_f']
print "Current temperature in #{location} is: #{temp_f}\n"

wunderground.rb

require 'open-uri'
require 'json'

api_key = '' # required by producing your own API key from http://www.wunderground.com/weather/api/ 

open("http://api.wunderground.com/api/#{api_key}/geolookup/conditions/q/IA/Cedar_Rapids.json") do |f|
  json_string = f.read
  parsed_json = JSON.parse(json_string)
  location = parsed_json['location']['city']
  temp_f = parsed_json['current_observation']['temp_f']
  print "Current temperature in #{location} is: #{temp_f}\n"
end

This is where exception gets thrown:

  class Inflater # :nodoc:

    ##
    # Creates a new Inflater wrapping +socket+

    def initialize socket
      @socket = socket
      # zlib with automatic gzip detection
      @inflate = Zlib::Inflate.new(32 + Zlib::MAX_WBITS)
    end

    ##
    # Finishes the inflate stream.

    def finish
      @inflate.finish # EXCEPTION THROWN HERE!
    end

    ##
    # Returns a Net::ReadAdapter that inflates each read chunk into +dest+.
    #
    # This allows a large response body to be inflated without storing the
    # entire body in memory.

    def inflate_adapter(dest)
      block = proc do |compressed_chunk|
        @inflate.inflate(compressed_chunk) do |chunk|
          dest << chunk
        end
      end

      Net::ReadAdapter.new(block)
    end

From net/http/response.rb in 2.0.0 MRI.

New Zlib options available in 2.0.0... wonder if they're the root of the issue.

See Zlib section about chunked responses:
http://globaldev.co.uk/2013/03/ruby-2-0-0-in-detail/

Also:
http://stackoverflow.com/questions/5279068/zlib-decompress-header-check-error?rq=1

@geopet

Any further answers on this? I'm curious to see how it pans out 😄

@zph Nothing so far. I wasn't able to prove that it was poor compiling that was causing the issue either. So I'm going to chat about this in the #ruby IRC channel as soon as I have a moment.

Interesting.

Wonder if @drbrain would have a better idea than we do about why rest-client succeeds with this request in MRI 2.0.0 when open-uri fails, possibly due to the ZLIB stuff.

Very interesting about @injekt finding that rest-client rescues the error and then succeeds as you note: https://github.com/rest-client/rest-client/blob/master/lib/restclient/request.rb#L258-L262.

@geopet,

Check it out: https://github.com/zph/mri_2_0_0_zlib_troubleshooting

I've promoted it to a full repo for easier management and briefly wrote up my process and progress.

Seems like a corrupted HTTP response, ie corrupted Gzip compression.

Perhaps more avenues of exploration will appear as we mull on it.

@zph
I trigger this issue when using omniauth do oauth through proxy, so I tried to monkey patch Net::HTTPResponse::Inflater https://gist.github.com/jasl/5939460