Skip to content

Instantly share code, notes, and snippets.

@thanos

thanos/batch_process.ex

Created September 4, 2022 14:41
Show Gist options
  • Save thanos/605593eb6c68ddb8c0295278bced7420 to your computer and use it in GitHub Desktop.
Save thanos/605593eb6c68ddb8c0295278bced7420 to your computer and use it in GitHub Desktop.
Download ZIP
Processing a stream in batches
Raw
batch_process.ex
batch_size = 100
# 8 tasks running at the same time and we don't care about the results order
async_options = [max_concurrency: 8, ordered: false]
csv_rows
|> Stream.chunk(batch_size)
|> Task.async_stream(fn batch ->
batch
|> Enum.map(&CSV.generate_xml/1)
|> CSV.save_to_batch_rails_database()
end, async_options)
|> Stream.run()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment