Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Similar to Rails `find_each`, but for Elixir's Ecto, using Stream
defmodule EctoBatchStream do
import Ecto.Query, only: [from: 1, from: 2]
@batch_size 1000
# Example:
#
# query = from u in MyApp.User, select: u.email
# stream = EctoBatchStream.stream(MyApp.Repo, query)
# stream |> Stream.take(3) |> Enum.to_list # => […]
def stream(repo, query, batch_size \\ @batch_size) do
batches_stream = Stream.unfold(0, fn
:done ->
nil
offset ->
results = repo.all(from _ in query, offset: ^offset, limit: ^batch_size)
if length(results) < batch_size,
do: { results, :done },
else: { results, offset + batch_size }
end)
batches_stream |> Stream.concat
end
end
@vyorkin

This comment has been minimized.

Copy link

vyorkin commented Aug 20, 2017

thanks)

@andrewsheelan

This comment has been minimized.

Copy link

andrewsheelan commented Jan 6, 2018

wow, exactly what I want.. thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.