Asynchronously Fetch All Elements Of A REST API with Asks and Trio

Async Fetch All

The following is an example of how to asynchronously fetch all objects at a given REST API endpoint by ID, with rate limiting, using the awesome asks library
https://asks.readthedocs.io/en/latest/

fetch.py

import trio
import asks

api_endpoint = 'https://jsonplaceholder.typicode.com' # Fake REST api used for testing

# example header payload. the fake REST API doesn't use it, but yours probably will
headers = {'Authorization': "Bearer xxxxxxxxx"}

max_concurrent_connections = 20

session = asks.Session(base_location=api_endpoint, headers=headers, connections=max_concurrent_connections)

# first request made against base endpoint will get us a list of IDs to asynchronously fetch
async def get_id_list():
    r = await session.get(path='/posts')
    data = r.json()
    return [d['id'] for d in data]
id_list = trio.run(get_id_list)

# Now comes the magic. We asynchronously iterate over the list of ids and fetch each one's full object from the API

# We need a container to collect all the responses into. It needs to be in a scope that all the individual async fetches
# can find and access it. It could be a list if you don't care about the order of the responses. Here we'll use a dict
# whose keys are the IDs we're fetching and whose values are the fetched data objects.
fetched_data = {}

async def grabber(id):
    r = await session.get(path=f'/posts/{id}')
    fetched_data[id] = r.json()

async def main():
    async with trio.open_nursery() as n:
        for id in id_list:
            n.start_soon(grabber, id)

trio.run(main)