Skip to content

Instantly share code, notes, and snippets.

@jozefcipa
Last active April 15, 2023 01:52
Show Gist options
  • Save jozefcipa/79d1383dc265d142e73a7f29119bea6d to your computer and use it in GitHub Desktop.
Save jozefcipa/79d1383dc265d142e73a7f29119bea6d to your computer and use it in GitHub Desktop.
Process large JSON files with streams
import fs from 'fs'
import { Transform } from 'stream'
import JSONStream from 'JSONStream'
// Custom transform stream
const transformer = new Transform({
objectMode: true,
transform(jsonItem, encoding, callback) {
// your logic goes here...
// const updatedItem = {}
callback(null, updatedItem)
}
})
const input = fs.createReadStream('./raw-input.json', 'utf8')
const output = fs.createWriteStream('processed-output.json', 'utf8')
input
.pipe(JSONStream.parse('*')) // JSON structure parser pattern
.pipe(transformer) // Process data
.pipe(JSONStream.stringify()) // Convert it back to JSON
.pipe(output) // save the result
@jozefcipa
Copy link
Author

jozefcipa commented Nov 3, 2022

Can you provide the whole code? This is just an example of how to process a JSON file, and it seems to be working fine to me, so the issue is probably somewhere else. @Sidney011100

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment