Example of how to query a large number of entities when paging is required.

sg_rest_api_pagination_example.py

import requests
import pprint
import json
import os

site_url = os.environ["SHOTGUN_SITE_URL"] # ie: https://foobar.shotgunstudio.com

headers = {
    "Accept": "application/json",
    "Content-Type": "application/x-www-form-urlencoded"
}

resp = requests.post(
    "{0}{1}".format(site_url, "/api/v1/auth/access_token"),
    data=dict(
        username=os.environ["SHOTGUN_SITE_USERNAME"], # ie: jeff
        password=os.environ["SHOTGUN_SITE_PASSWORD"], # your user's Shotgun password...
        grant_type="password",
    ),
    headers=headers,
)

resp_data = resp.json()
token = resp_data["access_token"]

headers = {
    "Accept": "application/json",
    "Authorization": "Bearer {0}".format(token),
}

resp = requests.get(
    "{0}{1}".format(site_url, "/api/v1/entity/tasks"),
    params=dict(
        fields=["id"],
    ),
    headers=headers,
)

raw_resp = resp.json()
tasks = raw_resp["data"]

while raw_resp["data"]:
    resp = requests.get(
        "{0}{1}".format(site_url, raw_resp["links"]["next"]),
        headers=headers,
    )
    raw_resp = resp.json()
    tasks.extend(raw_resp["data"])

pprint.pprint(tasks)
print "Num Assets: %s" % len(tasks)

I'd like to point out to anyone else who might be confused trying to figure out how the pagination exactly works:

The pagination is happening on line 45:
"{0}{1}".format(site_url, raw_resp["links"]["next"]),

That line should print out to this:
/api/v1/entity/tasks?page%5Bnumber%5D=2&page%5Bsize%5D=500

Meaning if you can use https://INSERT_STUDIO_NAME.shotgunstudio.com/api/v1/entity/tasks?page%5Bnumber%5D=2&page%5Bsize%5D=500 as your url link and switch out tasks for whatever entity type you need, 2 for the page you want, and 500 for the number of items you want to return.

This is all probably obvious to others but wasn't clear to me.

Thank you @thebeeland for posting this example