Skip to content

Instantly share code, notes, and snippets.

@snuggs
Forked from mlt/some_controller.rb
Created February 15, 2023 12:50
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save snuggs/285b7a07467e6bb297e57273de0aec51 to your computer and use it in GitHub Desktop.
Save snuggs/285b7a07467e6bb297e57273de0aec51 to your computer and use it in GitHub Desktop.
Stream PostgreSQL query with potentially huge result set as CSV using gzip compression in Rails and low memory overhead
headers['X-Accel-Buffering'] = 'no'
headers['Cache-Control'] = 'no-cache'
headers['Content-Type'] = 'text/csv; charset=utf-8'
headers['Content-Disposition'] = 'inline; filename="data.csv"'
headers['Content-Encoding'] = 'gzip'
sql = "select * from something;"
self.response_body = SqlToCsvStreamer.new(sql)
class SqlToCsvStreamer
def initialize(sql)
@sql = sql
end
def each(&block)
@block = block
begin
gz = Zlib::GzipWriter.new(self)
conn = ActiveRecord::Base.connection.raw_connection
conn.copy_data("COPY (#{@sql.chomp(';')}) TO STDOUT WITH (FORMAT CSV, HEADER TRUE, FORCE_QUOTE *, ESCAPE E'\\\\');") do
while row = conn.get_copy_data
gz.write row
end
end
ensure
gz.close
end
end
def write(row)
@block.call row
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment