Skip to content

Instantly share code, notes, and snippets.

@keymone keymone/gist:1827669
Created Feb 14, 2012

Embed
What would you like to do?
File.each_lines_batch
class IO
def each_lines_batch(batch_size=1000, buffer_size=4096)
lines = []
chunk = ""
while data = file.read(buffer_size)
chunk << data
lines += chunk.lines.to_a
# last line might not be read until it's end
chunk = lines.pop
while lines.size > batch_size
batch = lines.shift(batch_size)
yield batch
end
end
lines += chunk.lines.to_a
yield lines if lines.any?
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.