*"If you stream it, you can do it" -- Walt Disney1 *
Streams are trickling into the scene as we search for ways to improve performance. What if instead of waiting for our entire ajax response to complete, we could start showing the data as it arrives?
Streams allow us to do this. They are a data source that can be created and processed incrementally. This means as chunks of data become available, we are able to do work on them right away.
Using the Fetch API with a data format called NDJSON that breaks larger JSON objects into smaller JSON objects delimitated by newline characters, we are able to receive a stream of smaller chunks of JSON data as a stream. As our NDJSON data is streaming in, we can start processing and rendering right away. This makes users happy because they are able to see things sooner and developers happy because it increases overall performance. If you tie all of this in with service workers, then you can really see the improvements in performance.
As seen below, you can use fetch()
with an endpoint that sends NDJSON to start manipulating and rendering that data line-by-line as you receive it.
import ndjsonStream from 'can-ndjson-stream'; const fetchNdjson = async () => { const response = await fetch('/api'); const exampleStream = ndjsonStream(response.body); let result; while (!result || !result.done) { result = await exampleStream.getReader().read(); console.log(result.done, result.value); } }
Sounds like a win-win, but what is the catch? Well, there are packages out there on npm such as ndjson that serializes JSON to NDJSON, but we don’t have an easy way to deserialize NDJSON to a stream of JavaScript objects… until now!
can-ndjson-stream is a simple JavaScript module that does the heavy lifting of serializing your NDJSON stream into a ReadableStream of JavaScript objects. Use this just as you would JSON.parse with a JSON object.
Follow these simple steps to use the can-ndjson-stream
module.
//1. require the can-ndjson-stream module import ndjsonStream from 'can-ndjson-stream'; //2. pass the NDJSON stream response body to the ndjsonStream function. Typically, this would be used with the fetch() method. const readableStream = ndjsonStream(response.body); //3. use readableStream, which is now a ReadableStream of JS objects, however you like. The ReadableStream API exposes a .getReader() and .cancel() method. //https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream
npm i can-ndjson-stream --save
Require
var ndjsonStream = require('can-ndjson-stream');
-- OR --
ES6 import
import ndjsonStream from 'can-ndjson-stream';
Once you parse your response.body
, you can interact with your ReadableStream by read
'ing it like this: exampleStream.getReader().read()
. That will return a promise that resolves to one line of your NDJSON.
import ndjsonStream from 'can-ndjson-stream'; const fetchNdjson = async () => { const response = await fetch("/api"); const exampleStream = ndjsonStream(response.body); let result; while (!result || !result.done) { result = await exampleStream.getReader().read(); console.log(result.done, result.value); //result.value is one line of your NDJSON data } }
import ndjsonStream from 'can-ndjson-stream'; fetch('/api') // make a fetch request to a NDJSON stream service .then((response) => { return ndjsonStream(response.body); //ndjsonStream parses the response.body }).then((exampleStream) => { let read; exampleStream.getReader().read().then(read = (result) => { if (result.done) return; console.log(result.value); exampleStream.getReader().read().then(read); //recurse through the stream }); });
Follow this tutorial on the Bitvoi blog that gets you started or take a look at the demo in the can-ndjson-stream repo.
If you enjoyed this article, tweet to us how you plan to use this module! Also check out the rest of the CanJS library. If you need any help, please don’t be afraid to leave a comment below or ask questions in the CanJS Gitter or forums!
Inspirational Quotes about Streams Through History1
If not us, who? If not now, when? -- John F. Kennedy
Hope is a waking stream. -- Aristotle
Life is trying things to see if they work. -- Ray Bradbury
gifs: https://github.com/bgando/ndjson-stream-gifs