Skip to content

Instantly share code, notes, and snippets.

@theSage21
Created March 12, 2018 08:55
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save theSage21/7af2cb887167ed7bd8d3cb93a958a453 to your computer and use it in GitHub Desktop.
import asyncio
import aiohttp
import async_timeout
headers = {}
links = [] # TODO: Populate this somehow with urls
done = asyncio.Queue()
async def handle_link(url):
async with aiohttp.ClientSession() as session:
with async_timeout.timeout(10):
async with session.get(url, headers=headers) as response:
if response.status == 200:
html = await response.text()
return {'error': '', 'html': html}
else:
return {'error': response.status, 'html': ''}
done.put_nowait((url, html))
tasks = [handle_link(l) for l in links]
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment