Skip to content

Instantly share code, notes, and snippets.

@jimmywarting
Last active January 25, 2024 13:15
Show Gist options
  • Save jimmywarting/c0c72451240964409998388dce192861 to your computer and use it in GitHub Desktop.
Save jimmywarting/c0c72451240964409998388dce192861 to your computer and use it in GitHub Desktop.
Memory efficient block reader of fetch- `Response.body` a.k.a `RedableStream`
/**
* Read a stream into same underlying ArrayBuffer of a fixed size.
* And yield a new Uint8Array view of the same underlying buffer.
* @param {ReadableStreamBYOBReader} reader
* @param {number} chunkSize
*/
async function* blockReader(reader, chunkSize) {
let offset = 0;
let buffer = new ArrayBuffer(chunkSize)
let done, view
while (!done) {
({value: view, done} = await reader.read(new Uint8Array(buffer, offset, chunkSize - offset)))
buffer = view.buffer
if (done) break
offset += view.byteLength
if (offset === chunkSize) {
yield view
offset = 0
// if you want to reuse the same allocated buffer for efficiency,
// comment the following line:
// buffer = new ArrayBuffer(chunkSize)
// another alternative is that you `.slice()` it when you need a copy of it.
}
}
if (offset > 0) {
yield view.buffer.slice(0, offset)
}
}
const chunkSize = 65536
const response = await fetch(url)
const reader = response.body.getReader({ mode: 'byte' })
const iterator = blockReader(reader, chunkMaxSize)
// `sameUnderlyingBuffer` is an Uint8Array of the same underlying ArrayBuffer
// It means that the Uint8Array is detached in every loop.
// and not reusable in the next loop. (so don't try to concat them all)
// However the ArrayBuffer is reused / recycled and updated in every loop.
// This is the most efficient way to read a stream.
for await (const sameUnderlyingBuffer of iterator) {
// do something with the Uint8Array
// copy it if you have to, (again, it will be detached in next loop)
// const copy = sameUnderlyingBuffer.slice()
}
@DavidBuchanan314
Copy link

🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment