Skip to content

Instantly share code, notes, and snippets.

@olivernn olivernn/README.md
Last active Aug 29, 2015

Embed
What would you like to do?
Paginated results

I think response and collection are the two classes where the bulk of the stuff is happenening, the importer is just an example of the external api. Actually looking back at this I think it could definitly be improved but I guess its an example of how to do it.

I recently did something similar, this time it was streaming a file from s3, but yielding the file line by line. The AWS SDK just gives you the s3 object chunk by chunk as its read from the socket, so I had to buffer and then yield as many lines as possible from each chunk. I returned an enumerator to do this rather than mixing in enumerable, I can't really share this code though, which is a shame cos I thought it was a nice implementation at the time.

module Ergast
class Collection < Enumerator
if Rails.env.test?
COOL_OFF_PERIOD = 0
else
COOL_OFF_PERIOD = 2
end
def self.data_path(*parts)
@data_path_parts = parts
end
def self.query(name, path)
define_singleton_method name, ->(*args) {
self.new(Ergast::Client.get(path.call(*args)))
}
end
def initialize(response)
@response = response
@request_count = 0
extract_data_from_response
super() do |yielder|
loop do
yielder.yield item
end
end
end
private
attr_accessor :response, :data, :request_count
def item
data.next
rescue StopIteration
raise StopIteration if response.last_page?
fetch_next_page
extract_data_from_response
data.next
end
def extract_data_from_response
path = self.class.instance_variable_get("@data_path_parts")
self.data = response.data(*path).to_enum
end
def fetch_next_page
enforce_cool_off_period do
self.response = response.next_page
end
end
def enforce_cool_off_period(&block)
if request_count >= 3
self.request_count = 0
sleep COOL_OFF_PERIOD
end
block.call
self.request_count = request_count + 1
end
end
end
module Ergast
module Client
class Response
class MissingData < StandardError ; end
class NoMorePages < StandardError ; end
def self.parse(raw)
self.new(
url: raw['MRData']['url'],
limit: raw['MRData']['limit'].to_i,
offset: raw['MRData']['offset'].to_i,
total: raw['MRData']['total'].to_i,
payload: raw['MRData']
)
end
attr_reader :url, :limit, :offset, :total, :payload
def initialize(attrs = {})
@url = attrs[:url]
@limit = attrs[:limit]
@offset = attrs[:offset]
@total = attrs[:total]
@payload = attrs[:payload]
end
def data(*keys)
keys.inject(payload) { |memo, key| memo.fetch(key) }
rescue IndexError
raise MissingData
end
def next_page
raise NoMorePages if last_page?
Ergast::Client.get(path, limit: Ergast::Client::LIMIT, offset: offset + Ergast::Client::LIMIT)
end
def last_page?
offset + Ergast::Client::LIMIT >= total
end
private
def path
url.gsub(Ergast::Client::HOST, '').gsub(Ergast::Client::PATH, '').gsub(Ergast::Client::FORMAT, '')
end
end
end
end
require 'ergast'
class ResultsImporter
def self.import(round)
self.new(round).import!
end
def initialize(round)
@round = round
@season = round.season
@drivers = {}
end
def import!
ActiveRecord::Base.transaction do
results_data.map(&ResultMapper).each do |attrs|
grid_position = attrs.delete(:grid_position)
driver = find_driver(attrs.delete(:driver_code))
Lap.create_zeroth!(position: grid_position, driver: driver, round: round)
Result.create!(attrs.merge(driver: driver, round: round))
end
end
end
private
attr_reader :drivers, :season, :round
def find_driver(code)
drivers[code] ||= Driver.find_by!(code: code)
end
def results_data
@results_data ||= Ergast::Results.season_and_round(season.name, round.number)
end
end
module Ergast
class Results < Ergast::Collection
data_path 'RaceTable', 'Races', 0, 'Results'
query :season_and_round, ->(season, round) { "/#{season}/#{round}/results" }
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.