Skip to content

Instantly share code, notes, and snippets.

@dcramer

dcramer/gist:5556161

Last active December 17, 2015 05:09
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dcramer/5556161 to your computer and use it in GitHub Desktop.
Save dcramer/5556161 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
cursor = (shard 0, id 0, group id 0)
> get 10k items from shard 0
> group id set to max(id of results), say 20000
> get 10k items from shard 1 < group id (20000)
> adjust id if we hit 10k+ items, and repeat on same shard
> once group id is hit, go to next shard, and repeat same logic used on shard 1
> once last shard is done, go back to shard 0, get the next 10k results, and reset group id
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment