Skip to content

Instantly share code, notes, and snippets.

@Echooff3
Created July 25, 2017 23:01
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Echooff3/9eea537a569c3b2daff2640b7bd758cd to your computer and use it in GitHub Desktop.
Save Echooff3/9eea537a569c3b2daff2640b7bd758cd to your computer and use it in GitHub Desktop.
Parsing CSV with writeable buffer
const request = require('request')
const stream = require('stream')
const buffer = require('buffer')
class CSVStream extends stream.Writable {
constructor() {
super()
this._count = 0
this._buff = ""
this._push = (x) => {
//TODO figure out whether to dedup here or on DB Side
//TODO Push to DB or Kinesis
console.log(x)
}
}
_write(chunk, enc, next) {
var b = buffer.Buffer.from(chunk.toString(),'utf8')
b.map((x) => {
if(x == 10) { //nl
this._push(this._buff)
this._buff = ""
} else {
this._buff += String.fromCharCode(x)
}
})
next()
}
}
var test = () => {
request('http://localhost:8888/test.csv').pipe(new CSVStream())
}
test()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment