Skip to content

Instantly share code, notes, and snippets.

@renatoathaydes
Created August 23, 2022 11:11
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save renatoathaydes/1aafa00255137e3220a8c4707fcb4b5d to your computer and use it in GitHub Desktop.
Save renatoathaydes/1aafa00255137e3220a8c4707fcb4b5d to your computer and use it in GitHub Desktop.
Dart - processing stream of JSON documents
import 'dart:convert';
main() async {
const jsons = '{"foo": "Ø"}\n{"bar": false}';
final bytes = utf8.encode(jsons);
// partitions breaking the bytes between invalid UTF-8 boundaries
final partitions =
Stream.fromIterable([bytes.sublist(0, 10), bytes.sublist(10)]);
// naive approach doesn't work
// (Unfinished UTF-8 octet sequence (at offset 10))
// await for (final partition in partitions) {
// final json = utf8.decode(partition);
// print(jsonDecode(json));
// }
// correct approach using chunked conversion
final decodedJsons = partitions
.transform(const Utf8Decoder())
.transform(const LineSplitter())
.map((json) => jsonDecode(json));
await for (final json in decodedJsons) {
print(json);
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment