Example Showing how CSV Provider will use chunks of a stream while iterating

CsvProviderStreaming.fs

#r "nuget:FSharp.Data"

open System.IO
open System.Text
open FSharp.Data

type Csv = CsvProvider<"A,B,C\n1,2,3", CacheRows=false>

let csvStream = 
    let stream = new MemoryStream()

    stream.Write(Encoding.UTF8.GetBytes("A,B,C\n"))
    for i in 1..500 do
        stream.Write(Encoding.UTF8.GetBytes($"{i},{i+1},{i+2}\n"))

    stream.Position <- 0
    stream

let mutable position: int64 = 0
for row in Csv.Load(csvStream).Rows do
    if position <> csvStream.Position then
        position <- csvStream.Position
        printfn $"A = {row.A}; B = {row.B}; C = {row.C}; StreamPosition = {csvStream.Position}"

OutputConsole.txt

A = 1; B = 2; C = 3; StreamPosition = 1024
A = 112; B = 113; C = 114; StreamPosition = 2048
A = 197; B = 198; C = 199; StreamPosition = 3072
A = 283; B = 284; C = 285; StreamPosition = 4096
A = 368; B = 369; C = 370; StreamPosition = 5120
A = 453; B = 454; C = 455; StreamPosition = 5688

Yes, my example does have the whole CSV in a MemoryStream just to show how the file is being loaded in chunks without having to actually write to my file system (by printing the Stream's position), but you can replace with a StreamReader like you have since Load has an overload for a TextReader.

It turns out I'm the dummy today and forgot that you also need to set CacheRows to false since it actually defaults to true:

type Csv = CsvProvider<"A,B,C\n1,2,3", CacheRows=false>

I updated the example above. Hopefully that works, that's what I get for not creating a 20GB file for myself to make extra sure it does what I think it does. 😆

https://fsprojects.github.io/FSharp.Data/library/CsvProvider.html#Handling-big-datasets