Skip to content

Instantly share code, notes, and snippets.

@twillis
Created December 29, 2012 12:43
Show Gist options
  • Save twillis/4406666 to your computer and use it in GitHub Desktop.
Save twillis/4406666 to your computer and use it in GitHub Desktop.
how to get a cursor beyond the first 1250 so that paging patterns continue to work
def cursor_from_offset(q, offset, cursor=None):
"""
the maximum cursor position you can get in one shot is
1250 (page_size=250, offset=1000), so to get a cursor
beyond that requires a bit of work.
will return None if not results and more
"""
MAX_OFFSET = 1000
if offset <= MAX_OFFSET:
kw = dict(page_size=offset,
keys_only=True)
if cursor:
kw["start_cursor"] = cursor
result, c, more = q.fetch_page(**kw)
return c
else:
continue_ = True
offset_ = offset
c = None
while continue_:
if offset_ > MAX_OFFSET:
c = cursor_from_offset(q, MAX_OFFSET, cursor=c)
offset_ -= MAX_OFFSET
else:
c = cursor_from_offset(q, offset_, cursor=c)
offset_ = 0
continue_ = (offset_ > 0) and c is not None
else:
return c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment