Skip to content

Instantly share code, notes, and snippets.

@janl
Last active August 29, 2015 14:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save janl/41d97adb43f0430c094e to your computer and use it in GitHub Desktop.
Save janl/41d97adb43f0430c094e to your computer and use it in GitHub Desktop.
> echo '{"a":1}' | ../node_query_server/bin/qs2
net.js:614
throw new TypeError('invalid data');
^
TypeError: invalid data
at WriteStream.Socket.write (net.js:614:11)
at null._onTimeout (/Users/jan/Work/qs2/node_query_server/index.js:17:20)
at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
var ndj = require('ndjson');
process.stdin.setEncoding('utf8');
function getRandomInt(min, max) {
return Math.floor(Math.random() * (max - min)) + min;
}
// var nd_out = process
// .stdout
// .pipe(ndj.serialize());
var send = function(data) {
// console.error('data: ', data);
setTimeout(function() {
// nd_out.write(data);
process.stdout.write(data);
}, getRandomInt(1, 100));
};
process
.stdin
.pipe(ndj.parse())
.on('data', send);
@janl
Copy link
Author

janl commented Dec 28, 2014

The old code has an issue where the part of a chunk after the last \n is not a full JSON object, but rather than coding that logic myself, I assume there is something in the streams universe that does that for me, but it is not obvious where to find it. e.g.https://github.com/substack/stream-handbook#consuming-a-readable-stream shows an example (way down), that looks like it does what I want, but it is not clear, whether this is available as an abstraction anywhere.

var offset = 0;

process.stdin.on('readable', function () {
    var buf = process.stdin.read();
    if (!buf) return;
    for (; offset < buf.length; offset++) {
        if (buf[offset] === 0x0a) {
            console.dir(buf.slice(0, offset).toString());
            buf = buf.slice(offset + 1);
            offset = 0;
            process.stdin.unshift(buf);
            return;
        }
    }
    process.stdin.unshift(buf);
});

@janl
Copy link
Author

janl commented Dec 28, 2014

Updated the gist and added a new call log.

@mafintosh
Copy link

@janl process.stdout.write(data) fails because data is a object (ndjson.parse() returns an object stream). since process.stdout is a binary stream it only accepts strings or buffers.

@mafintosh
Copy link

simply doing process.stdout.write(JSON.stringify(data)) would fix this 😄

@dominictarr
Copy link

the problem here is you are attempting to write a live javascript object to stdout.
this: stdout.write({a: 1}) you need to stringify it first.
one way would be to pipe it to ndjson.stringify() before piping to stdout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment