Skip to content

Instantly share code, notes, and snippets.

@jozefcipa
Last active April 15, 2023 01:52
Show Gist options
  • Save jozefcipa/79d1383dc265d142e73a7f29119bea6d to your computer and use it in GitHub Desktop.
Save jozefcipa/79d1383dc265d142e73a7f29119bea6d to your computer and use it in GitHub Desktop.
Process large JSON files with streams
import fs from 'fs'
import { Transform } from 'stream'
import JSONStream from 'JSONStream'
// Custom transform stream
const transformer = new Transform({
objectMode: true,
transform(jsonItem, encoding, callback) {
// your logic goes here...
// const updatedItem = {}
callback(null, updatedItem)
}
})
const input = fs.createReadStream('./raw-input.json', 'utf8')
const output = fs.createWriteStream('processed-output.json', 'utf8')
input
.pipe(JSONStream.parse('*')) // JSON structure parser pattern
.pipe(transformer) // Process data
.pipe(JSONStream.stringify()) // Convert it back to JSON
.pipe(output) // save the result
@Sidney011100
Copy link

node:internal/streams/legacy:47
dest.end();
^

TypeError: dest.end is not a function
at Stream.onend (node:internal/streams/legacy:47:10)
at Stream.emit (node:events:402:35)
at drain (/Users/sidneychong/Desktop/steamTest/node_modules/through/index.js:34:23)
at Stream.stream.queue.stream.push (/Users/sidneychong/Desktop/steamTest/node_modules/through/index.js:45:5)
at Stream. (/Users/sidneychong/Desktop/steamTest/node_modules/JSONStream/index.js:32:12)
at _end (/Users/sidneychong/Desktop/steamTest/node_modules/through/index.js:65:9)
at Stream.stream.end (/Users/sidneychong/Desktop/steamTest/node_modules/through/index.js:74:5)
at ReadStream.onend (node:internal/streams/readable:693:10)
at Object.onceWrapper (node:events:509:28)
at ReadStream.emit (node:events:390:28)

What do you do with this error?

@jozefcipa
Copy link
Author

jozefcipa commented Nov 3, 2022

Can you provide the whole code? This is just an example of how to process a JSON file, and it seems to be working fine to me, so the issue is probably somewhere else. @Sidney011100

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment