Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Load data in parallel
private static async Task StreamDataInParallel(StreamReader fileSource, HttpClient httpClient, string path, int maxParallel)
{
var block = new ActionBlock<string>(
async json =>
{
await httpClient.PostAsync(path, new StringContent(json, Encoding.UTF8, "application/json"));
}, new ExecutionDataflowBlockOptions
{
// Tells the action block how many we want to run at once.
MaxDegreeOfParallelism = maxParallel,
// 'Buffer' the same number of lines as there are parallel requests.
BoundedCapacity = maxParallel
});
string line;
while ((line = await fileSource.ReadLineAsync()) != null)
{
// This will not continue until there is space in the buffer.
await block.SendAsync(GetDataJson(line));
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.