Skip to content

Instantly share code, notes, and snippets.

@chriswhong
Created July 10, 2018 05:02
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save chriswhong/1718c5595078953a1ea2315ac567e9ce to your computer and use it in GitHub Desktop.
Save chriswhong/1718c5595078953a1ea2315ac567e9ce to your computer and use it in GitHub Desktop.
Streaming from ps-promise to json2csv to express response
// This code queries postgresql and streams the results into a csv
// Works like a charm for a download button, no hangup for large files!
const Json2csvTransform = require('json2csv').Transform;
const QueryStream = require('pg-query-stream');
const JSONStream = require('JSONStream');
router.get('/download.csv', async (req, res) => {
const { query } = req;
const SQL = 'SELECT * FROM table';
const qs = new QueryStream(SQL);
const transformOpts = { highWaterMark: 16384, encoding: 'utf-8' };
const json2csv = new Json2csvTransform({}, transformOpts);
// Set approrpiate download headers
res.setHeader('Content-disposition', 'attachment; filename=projects.csv');
res.writeHead(200, { 'Content-Type': 'text/csv' });
// Flush the headers before we start pushing the CSV content
res.flushHeaders();
db.stream(qs, (s) => {
// initiate streaming into the console:
s.pipe(JSONStream.stringify()).pipe(json2csv).pipe(res);
})
.then((data) => {
console.log(
'Total rows processed:', data.processed,
'Duration in milliseconds:', data.duration,
);
})
.catch((error) => {
console.log('ERROR:', error);
});
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment