Paginated results

README.md

I think response and collection are the two classes where the bulk of the stuff is happenening, the importer is just an example of the external api. Actually looking back at this I think it could definitly be improved but I guess its an example of how to do it.

I recently did something similar, this time it was streaming a file from s3, but yielding the file line by line. The AWS SDK just gives you the s3 object chunk by chunk as its read from the socket, so I had to buffer and then yield as many lines as possible from each chunk. I returned an enumerator to do this rather than mixing in enumerable, I can't really share this code though, which is a shame cos I thought it was a nice implementation at the time.

collection.rb

module Ergast
  class Collection < Enumerator

    if Rails.env.test?
      COOL_OFF_PERIOD = 0
    else
      COOL_OFF_PERIOD = 2
    end

    def self.data_path(*parts)
      @data_path_parts = parts
    end

    def self.query(name, path)
      define_singleton_method name, ->(*args) {
        self.new(Ergast::Client.get(path.call(*args)))
      }
    end

    def initialize(response)
      @response = response
      @request_count = 0
      extract_data_from_response

      super() do |yielder|
        loop do
          yielder.yield item
        end
      end
    end

    private

    attr_accessor :response, :data, :request_count

    def item
      data.next
    rescue StopIteration
      raise StopIteration if response.last_page?

      fetch_next_page
      extract_data_from_response

      data.next
    end

    def extract_data_from_response
      path = self.class.instance_variable_get("@data_path_parts")
      self.data = response.data(*path).to_enum
    end

    def fetch_next_page
      enforce_cool_off_period do
        self.response = response.next_page
      end
    end

    def enforce_cool_off_period(&block)
      if request_count >= 3
        self.request_count = 0
        sleep COOL_OFF_PERIOD
      end

      block.call

      self.request_count = request_count + 1
    end
  end
end

response.rb

module Ergast
  module Client
    class Response

      class MissingData < StandardError ; end
      class NoMorePages < StandardError ; end

      def self.parse(raw)
        self.new(
          url: raw['MRData']['url'],
          limit: raw['MRData']['limit'].to_i,
          offset: raw['MRData']['offset'].to_i,
          total: raw['MRData']['total'].to_i,
          payload: raw['MRData']
        )
      end

      attr_reader :url, :limit, :offset, :total, :payload

      def initialize(attrs = {})
        @url = attrs[:url]
        @limit = attrs[:limit]
        @offset = attrs[:offset]
        @total = attrs[:total]
        @payload = attrs[:payload]
      end

      def data(*keys)
        keys.inject(payload) { |memo, key| memo.fetch(key) }
      rescue IndexError
        raise MissingData
      end

      def next_page
        raise NoMorePages if last_page?
        Ergast::Client.get(path, limit: Ergast::Client::LIMIT, offset: offset + Ergast::Client::LIMIT)
      end

      def last_page?
        offset + Ergast::Client::LIMIT >= total
      end

      private

      def path
        url.gsub(Ergast::Client::HOST, '').gsub(Ergast::Client::PATH, '').gsub(Ergast::Client::FORMAT, '')
      end
    end
  end
end

result_importer.rb

require 'ergast'

class ResultsImporter
  def self.import(round)
    self.new(round).import!
  end

  def initialize(round)
    @round = round
    @season = round.season
    @drivers = {}
  end

  def import!
    ActiveRecord::Base.transaction do
      results_data.map(&ResultMapper).each do |attrs|
        grid_position = attrs.delete(:grid_position)
        driver = find_driver(attrs.delete(:driver_code))

        Lap.create_zeroth!(position: grid_position, driver: driver, round: round)
        Result.create!(attrs.merge(driver: driver, round: round))
      end
    end
  end

  private

  attr_reader :drivers, :season, :round

  def find_driver(code)
    drivers[code] ||= Driver.find_by!(code: code)
  end

  def results_data
    @results_data ||= Ergast::Results.season_and_round(season.name, round.number)
  end

end

results.rb

module Ergast
  class Results < Ergast::Collection
    data_path 'RaceTable', 'Races', 0, 'Results'

    query :season_and_round, ->(season, round) { "/#{season}/#{round}/results" }
  end
end