Skip to content

Instantly share code, notes, and snippets.

@jonluca
Last active July 17, 2024 10:58
Show Gist options
  • Save jonluca/14fe99be6204f34cbd61c950b0faf3b1 to your computer and use it in GitHub Desktop.
Save jonluca/14fe99be6204f34cbd61c950b0faf3b1 to your computer and use it in GitHub Desktop.
Fast asyncio HTTP requests
import sys
import os
import json
import asyncio
import aiohttp
# Initialize connection pool
conn = aiohttp.TCPConnector(limit_per_host=100, limit=0, ttl_dns_cache=300)
PARALLEL_REQUESTS = 100
results = []
urls = ['https://jsonplaceholder.typicode.com/todos/1' for i in range(10)] #array of urls
async def gather_with_concurrency(n):
semaphore = asyncio.Semaphore(n)
session = aiohttp.ClientSession(connector=conn)
# heres the logic for the generator
async def get(url):
async with semaphore:
async with session.get(url, ssl=False) as response:
obj = json.loads(await response.read())
results.append(obj)
await asyncio.gather(*(get(url) for url in urls))
await session.close()
loop = asyncio.get_event_loop()
loop.run_until_complete(gather_with_concurrency(PARALLEL_REQUESTS))
conn.close()
print(f"Completed {len(urls)} requests with {len(results)} results")
@007-JB
Copy link

007-JB commented Aug 20, 2022

Strangely this works on my spare PC, but not my main one - despite having reinstalled pycharm, and the various libraries - any ideas why? (o should I literallly remove python and pycharm altogethr, then reinstall with new configuarations?)

None
None
None
None
None
None
None
None
None
None
Traceback (most recent call last):
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\site-packages\IPython\core\interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 28, in
loop.run_until_complete(gather_with_concurrency(PARALLEL_REQUESTS))
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 642, in run_until_complete
return future.result()
File "", line 24, in gather_with_concurrency
await asyncio.gather(*(get(url) for url in urls))
File "", line 21, in get
async with session.get(url, ssl=False) as response:
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\site-packages\aiohttp\client.py", line 1117, in aenter
self._resp = await self._coro
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\site-packages\aiohttp\client.py", line 520, in _request
conn = await self._connector.connect(
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\site-packages\aiohttp\connector.py", line 535, in connect
proto = await self._create_connection(req, traces, timeout)
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\site-packages\aiohttp\connector.py", line 892, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\site-packages\aiohttp\connector.py", line 999, in _create_direct_connection
hosts = await asyncio.shield(host_resolved)
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\site-packages\aiohttp\connector.py", line 865, in _resolve_host
addrs = await self._resolver.resolve(host, port, family=self._family)
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\site-packages\aiohttp\resolver.py", line 31, in resolve
infos = await self._loop.getaddrinfo(
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 856, in getaddrinfo
return await self.run_in_executor(
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 809, in run_in_executor
executor = concurrent.futures.ThreadPoolExecutor(
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\concurrent\futures\thread.py", line 140, in init
if max_workers <= 0:
TypeError: '<=' not supported between instances of 'NoneType' and 'int'

@jonluca
Copy link
Author

jonluca commented Aug 21, 2022

Is PARALLEL_REQUESTS set to None in your code?

File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\concurrent\futures\thread.py", line 140, in init
if max_workers <= 0:
TypeError: '<=' not supported between instances of 'NoneType' and 'int'

max_workers is None - you need to pass a valid number to loop.run_until_complete(gather_with_concurrency(PARALLEL_REQUESTS))

@007-JB
Copy link

007-JB commented Aug 21, 2022

Ta- I think I did something funny with one of the threading;/future or some other module - to allow me to increase max threads - because I'm running exact same code on other machine at it works beautifully - I'll get there - thanks so much for swift response. your code is awesome!!! thumbs up - take care, J

@eerbing
Copy link

eerbing commented Jul 17, 2024

Why I run this slower than solution 2 & 3? This costs over 1 sec, but solution 2&3 cost 0.8sec, it looks like async is always slower than multi-thread

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment