Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save ibqn/a5c1323fa7553e4d996975d8a3b029f9 to your computer and use it in GitHub Desktop.
Save ibqn/a5c1323fa7553e4d996975d8a3b029f9 to your computer and use it in GitHub Desktop.
Read a ND-JSON stream from the browser or from nodejs
/* FOR THE BROWSER
Utility function to read a ND-JSON HTTP stream.
`processLine` is a function taking a JSON object. It will be called with each element of the stream.
`response` is the result of a `fetch` request.
See usage example in the next file.
*/
const readStream = processLine => response => {
const stream = response.body.getReader();
const matcher = /\r?\n/;
const decoder = new TextDecoder();
let buf = '';
const loop = () =>
stream.read().then(({ done, value }) => {
if (done) {
if (buf.length > 0) processLine(JSON.parse(buf));
} else {
const chunk = decoder.decode(value, {
stream: true
});
buf += chunk;
const parts = buf.split(matcher);
buf = parts.pop();
for (const i of parts.filter(p => p)) processLine(JSON.parse(i));
return loop();
}
});
return loop();
}
/* FOR NODEJS
Utility function to read a ND-JSON HTTP stream.
`processLine` is a function taking a JSON object. It will be called with each element of the stream.
`response` is the result of a `fetch` request.
See usage example in the next file.
*/
const readStream = processLine => response => {
const matcher = /\r?\n/;
const decoder = new TextDecoder();
let buf = '';
return new Promise((resolve, fail) => {
response.body.on('data', v => {
const chunk = decoder.decode(v, { stream: true });
buf += chunk;
const parts = buf.split(matcher);
buf = parts.pop();
for (const i of parts.filter(p => p)) processLine(JSON.parse(i));
});
response.body.on('end', () => {
if (buf.length > 0) processLine(JSON.parse(buf));
resolve();
});
response.body.on('error', fail);
});
};
/* From browser or nodejs alike */
const stream = fetch('https://lichess.org/api/tv/feed');
// or any other ND-JSON endpoint such as:
// const stream = fetch('https://lichess.org/api/games/user/neio',{headers:{Accept:'application/x-ndjson'}});
const onMessage = obj => console.log(obj);
const onComplete = () => console.log('The stream has completed');
stream
.then(readStream(onMessage))
.then(onComplete);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment