Skip to content

Instantly share code, notes, and snippets.

@framp
Created August 20, 2014 21:52
Show Gist options
  • Save framp/9103a7f1c87b0e87a4d1 to your computer and use it in GitHub Desktop.
Save framp/9103a7f1c87b0e87a4d1 to your computer and use it in GitHub Desktop.
Working on data inside compressed archives using node.js and pipes
var es = require('event-stream')
process.stdin
.pipe(es.split())
.pipe(es.map(function (data, callback) {
var line = data.toString().split(',');
if (line.length>3){
var filename = line[0];
var date = [line[1].substr(0,4), line[1].substr(4,2), line[1].substr(6)].join('-');
date = + new Date(date);
var bid = line[2];
var offer = line[3];
data = [date, bid, offer].join(',') + '\n'
}
callback(null, data);
}))
.pipe(process.stdout);
//time unzip -p input.zip | node ../fix.js | gzip -cf > output.gz | tail -n1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment