Skip to content

Instantly share code, notes, and snippets.

@smtlaissezfaire
Created November 15, 2019 01:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save smtlaissezfaire/21df72c619aaceeb500169f94639bd5e to your computer and use it in GitHub Desktop.
Save smtlaissezfaire/21df72c619aaceeb500169f94639bd5e to your computer and use it in GitHub Desktop.
# partially taken from https://dalibornasevic.com/posts/68-processing-large-csv-files-with-ruby
# thanks!
require 'csv'
require 'benchmark'
require 'tempfile'
def print_memory_usage
memory_before = `ps -o rss= -p #{Process.pid}`.to_i
yield
memory_after = `ps -o rss= -p #{Process.pid}`.to_i
puts "Memory: #{((memory_after - memory_before) / 1024.0).round(2)} MB"
end
def print_time_spent
time = Benchmark.realtime do
yield
end
puts "Time: #{time.round(2)}"
end
headers = ['id', 'name', 'email', 'city', 'street', 'country']
print_memory_usage do
Tempfile.open("tmp.csv") do |tempfile|
# tempfile ||= "tmp.csv"
CSV.open(tempfile, 'wb') do |csv|
print_time_spent do
line = [
"Pink Panther",
"pink.panther@example.com",
"Pink City",
"Pink Road",
"Pink Country",
]
1_000_000.times do |i|
csv << line
end
end
end
# puts File.read(tempfile)
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment