Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Batch transform stream javascript
const { Transform } = require('stream');
const batchStream = (batchSize = 5) => {
let batch = [];
return new Transform({
objectMode: true,
transform(data, _, done) {
batch.push(data);
if (batch.length >= batchSize) {
done(null, batch);
batch = [];
return;
}
done();
},
flush(done) {
done(null, batch);
},
});
};
// Call with .pipe(batchStream(20))
// In the next pipe, you will get an array of 20 (or the
// remainder in the last step) of "stuff" instead of one "stuff" at a time.
// loosely based on https://www.bennadel.com/blog/3236-using-transform-streams-to-manage-backpressure-for-asynchronous-tasks-in-node-js.htm
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment