Skip to content

Instantly share code, notes, and snippets.

@maxim
Last active September 9, 2022 18:15
Show Gist options
  • Save maxim/150cc4e320a7b347e540 to your computer and use it in GitHub Desktop.
Save maxim/150cc4e320a7b347e540 to your computer and use it in GitHub Desktop.
Similar to Rails `find_each`, but for Elixir's Ecto, using Stream
defmodule EctoBatchStream do
import Ecto.Query, only: [from: 1, from: 2]
@batch_size 1000
# Example:
#
# query = from u in MyApp.User, select: u.email
# stream = EctoBatchStream.stream(MyApp.Repo, query)
# stream |> Stream.take(3) |> Enum.to_list # => […]
def stream(repo, query, batch_size \\ @batch_size) do
batches_stream = Stream.unfold(0, fn
:done ->
nil
offset ->
results = repo.all(from _ in query, offset: ^offset, limit: ^batch_size)
if length(results) < batch_size,
do: { results, :done },
else: { results, offset + batch_size }
end)
batches_stream |> Stream.concat
end
end
@vyorkin
Copy link

vyorkin commented Aug 20, 2017

thanks)

@andrewsheelan
Copy link

wow, exactly what I want.. thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment