Skip to content

Instantly share code, notes, and snippets.

@rianjs
Created October 11, 2021 13:21
Show Gist options
  • Save rianjs/2f549abf0be2d1db641649c596560408 to your computer and use it in GitHub Desktop.
Save rianjs/2f549abf0be2d1db641649c596560408 to your computer and use it in GitHub Desktop.
Read a portion of a gzipped text file without decompressing the whole thing using GZipStream
var i = 0;
const int limit = 100_000;
using (var fStream = fs.FileOpenRead(editionsPath))
using (var gzipStream = new GZipStream(fStream, CompressionMode.Decompress))
using (var textReader = new StreamReader(gzipStream))
{
var line = await textReader.ReadLineAsync();
while (!string.IsNullOrWhiteSpace(line) && i < limit)
{
var pieces = line.Split("\t");
var last = pieces.Last();
if (!last.StartsWith("{"))
{
Console.WriteLine(last);
}
jsonBlobs.Add(last);
i++;
line = await textReader.ReadLineAsync();
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment