Skip to content

Instantly share code, notes, and snippets.

@messa
Created December 4, 2019 13:51
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save messa/804178c51317b2f6b722d618723e1270 to your computer and use it in GitHub Desktop.
Save messa/804178c51317b2f6b722d618723e1270 to your computer and use it in GitHub Desktop.
from aiohttp import ClientSession
from asyncio import run, gather, create_task
url_template = 'https://jsonplaceholder.typicode.com/posts/{post_id}'
worker_count = 10
async def main():
post_ids = list(range(1, 101))
titles = {}
async with ClientSession() as session:
tasks = []
for _ in range(worker_count):
tasks.append(create_task(retrieve_worker(session, post_ids, titles)))
await gather(*tasks)
print(titles)
async def retrieve_worker(session, post_ids, titles):
while post_ids:
post_id = post_ids.pop(0)
titles[post_id] = await retrieve_title(session, post_id)
async def retrieve_title(session, post_id):
url = url_template.format(post_id=post_id)
print('Downloading', url)
async with session.get(url) as resp:
resp.raise_for_status()
data = await resp.json()
print('Downloaded', url)
return data['title']
if __name__ == '__main__':
run(main())
Downloading https://jsonplaceholder.typicode.com/posts/1
Downloading https://jsonplaceholder.typicode.com/posts/2
Downloading https://jsonplaceholder.typicode.com/posts/3
Downloading https://jsonplaceholder.typicode.com/posts/4
Downloading https://jsonplaceholder.typicode.com/posts/5
Downloading https://jsonplaceholder.typicode.com/posts/6
Downloading https://jsonplaceholder.typicode.com/posts/7
Downloading https://jsonplaceholder.typicode.com/posts/8
Downloading https://jsonplaceholder.typicode.com/posts/9
Downloading https://jsonplaceholder.typicode.com/posts/10
Downloaded https://jsonplaceholder.typicode.com/posts/2
Downloading https://jsonplaceholder.typicode.com/posts/11
Downloaded https://jsonplaceholder.typicode.com/posts/7
Downloading https://jsonplaceholder.typicode.com/posts/12
Downloaded https://jsonplaceholder.typicode.com/posts/3
Downloading https://jsonplaceholder.typicode.com/posts/13
Downloaded https://jsonplaceholder.typicode.com/posts/5
Downloading https://jsonplaceholder.typicode.com/posts/14
Downloaded https://jsonplaceholder.typicode.com/posts/10
Downloading https://jsonplaceholder.typicode.com/posts/15
Downloaded https://jsonplaceholder.typicode.com/posts/4
Downloading https://jsonplaceholder.typicode.com/posts/16
Downloaded https://jsonplaceholder.typicode.com/posts/6
Downloading https://jsonplaceholder.typicode.com/posts/17
Downloaded https://jsonplaceholder.typicode.com/posts/1
Downloading https://jsonplaceholder.typicode.com/posts/18
Downloaded https://jsonplaceholder.typicode.com/posts/8
Downloading https://jsonplaceholder.typicode.com/posts/19
Downloaded https://jsonplaceholder.typicode.com/posts/11
Downloading https://jsonplaceholder.typicode.com/posts/20
Downloaded https://jsonplaceholder.typicode.com/posts/12
Downloading https://jsonplaceholder.typicode.com/posts/21
Downloaded https://jsonplaceholder.typicode.com/posts/13
Downloading https://jsonplaceholder.typicode.com/posts/22
Downloaded https://jsonplaceholder.typicode.com/posts/14
Downloading https://jsonplaceholder.typicode.com/posts/23
Downloaded https://jsonplaceholder.typicode.com/posts/15
Downloading https://jsonplaceholder.typicode.com/posts/24
Downloaded https://jsonplaceholder.typicode.com/posts/16
Downloading https://jsonplaceholder.typicode.com/posts/25
Downloaded https://jsonplaceholder.typicode.com/posts/17
Downloading https://jsonplaceholder.typicode.com/posts/26
Downloaded https://jsonplaceholder.typicode.com/posts/18
Downloading https://jsonplaceholder.typicode.com/posts/27
Downloaded https://jsonplaceholder.typicode.com/posts/19
Downloading https://jsonplaceholder.typicode.com/posts/28
Downloaded https://jsonplaceholder.typicode.com/posts/20
Downloading https://jsonplaceholder.typicode.com/posts/29
Downloaded https://jsonplaceholder.typicode.com/posts/21
Downloading https://jsonplaceholder.typicode.com/posts/30
Downloaded https://jsonplaceholder.typicode.com/posts/22
Downloading https://jsonplaceholder.typicode.com/posts/31
Downloaded https://jsonplaceholder.typicode.com/posts/23
Downloading https://jsonplaceholder.typicode.com/posts/32
Downloaded https://jsonplaceholder.typicode.com/posts/24
Downloading https://jsonplaceholder.typicode.com/posts/33
Downloaded https://jsonplaceholder.typicode.com/posts/25
Downloading https://jsonplaceholder.typicode.com/posts/34
Downloaded https://jsonplaceholder.typicode.com/posts/28
Downloading https://jsonplaceholder.typicode.com/posts/35
Downloaded https://jsonplaceholder.typicode.com/posts/27
Downloading https://jsonplaceholder.typicode.com/posts/36
Downloaded https://jsonplaceholder.typicode.com/posts/26
Downloading https://jsonplaceholder.typicode.com/posts/37
Downloaded https://jsonplaceholder.typicode.com/posts/29
Downloading https://jsonplaceholder.typicode.com/posts/38
Downloaded https://jsonplaceholder.typicode.com/posts/30
Downloading https://jsonplaceholder.typicode.com/posts/39
Downloaded https://jsonplaceholder.typicode.com/posts/31
Downloading https://jsonplaceholder.typicode.com/posts/40
Downloaded https://jsonplaceholder.typicode.com/posts/32
Downloading https://jsonplaceholder.typicode.com/posts/41
Downloaded https://jsonplaceholder.typicode.com/posts/33
Downloading https://jsonplaceholder.typicode.com/posts/42
Downloaded https://jsonplaceholder.typicode.com/posts/34
Downloading https://jsonplaceholder.typicode.com/posts/43
Downloaded https://jsonplaceholder.typicode.com/posts/35
Downloading https://jsonplaceholder.typicode.com/posts/44
Downloaded https://jsonplaceholder.typicode.com/posts/36
Downloading https://jsonplaceholder.typicode.com/posts/45
Downloaded https://jsonplaceholder.typicode.com/posts/37
Downloading https://jsonplaceholder.typicode.com/posts/46
Downloaded https://jsonplaceholder.typicode.com/posts/38
Downloading https://jsonplaceholder.typicode.com/posts/47
Downloaded https://jsonplaceholder.typicode.com/posts/40
Downloading https://jsonplaceholder.typicode.com/posts/48
Downloaded https://jsonplaceholder.typicode.com/posts/39
Downloading https://jsonplaceholder.typicode.com/posts/49
Downloaded https://jsonplaceholder.typicode.com/posts/42
Downloading https://jsonplaceholder.typicode.com/posts/50
Downloaded https://jsonplaceholder.typicode.com/posts/41
Downloading https://jsonplaceholder.typicode.com/posts/51
Downloaded https://jsonplaceholder.typicode.com/posts/44
Downloading https://jsonplaceholder.typicode.com/posts/52
Downloaded https://jsonplaceholder.typicode.com/posts/45
Downloading https://jsonplaceholder.typicode.com/posts/53
Downloaded https://jsonplaceholder.typicode.com/posts/46
Downloading https://jsonplaceholder.typicode.com/posts/54
Downloaded https://jsonplaceholder.typicode.com/posts/43
Downloading https://jsonplaceholder.typicode.com/posts/55
Downloaded https://jsonplaceholder.typicode.com/posts/47
Downloading https://jsonplaceholder.typicode.com/posts/56
Downloaded https://jsonplaceholder.typicode.com/posts/48
Downloading https://jsonplaceholder.typicode.com/posts/57
Downloaded https://jsonplaceholder.typicode.com/posts/49
Downloading https://jsonplaceholder.typicode.com/posts/58
Downloaded https://jsonplaceholder.typicode.com/posts/50
Downloading https://jsonplaceholder.typicode.com/posts/59
Downloaded https://jsonplaceholder.typicode.com/posts/52
Downloading https://jsonplaceholder.typicode.com/posts/60
Downloaded https://jsonplaceholder.typicode.com/posts/51
Downloading https://jsonplaceholder.typicode.com/posts/61
Downloaded https://jsonplaceholder.typicode.com/posts/53
Downloading https://jsonplaceholder.typicode.com/posts/62
Downloaded https://jsonplaceholder.typicode.com/posts/54
Downloading https://jsonplaceholder.typicode.com/posts/63
Downloaded https://jsonplaceholder.typicode.com/posts/55
Downloading https://jsonplaceholder.typicode.com/posts/64
Downloaded https://jsonplaceholder.typicode.com/posts/9
Downloading https://jsonplaceholder.typicode.com/posts/65
Downloaded https://jsonplaceholder.typicode.com/posts/56
Downloading https://jsonplaceholder.typicode.com/posts/66
Downloaded https://jsonplaceholder.typicode.com/posts/58
Downloading https://jsonplaceholder.typicode.com/posts/67
Downloaded https://jsonplaceholder.typicode.com/posts/57
Downloading https://jsonplaceholder.typicode.com/posts/68
Downloaded https://jsonplaceholder.typicode.com/posts/59
Downloading https://jsonplaceholder.typicode.com/posts/69
Downloaded https://jsonplaceholder.typicode.com/posts/61
Downloading https://jsonplaceholder.typicode.com/posts/70
Downloaded https://jsonplaceholder.typicode.com/posts/60
Downloading https://jsonplaceholder.typicode.com/posts/71
Downloaded https://jsonplaceholder.typicode.com/posts/63
Downloading https://jsonplaceholder.typicode.com/posts/72
Downloaded https://jsonplaceholder.typicode.com/posts/64
Downloading https://jsonplaceholder.typicode.com/posts/73
Downloaded https://jsonplaceholder.typicode.com/posts/62
Downloading https://jsonplaceholder.typicode.com/posts/74
Downloaded https://jsonplaceholder.typicode.com/posts/65
Downloading https://jsonplaceholder.typicode.com/posts/75
Downloaded https://jsonplaceholder.typicode.com/posts/66
Downloading https://jsonplaceholder.typicode.com/posts/76
Downloaded https://jsonplaceholder.typicode.com/posts/67
Downloading https://jsonplaceholder.typicode.com/posts/77
Downloaded https://jsonplaceholder.typicode.com/posts/69
Downloading https://jsonplaceholder.typicode.com/posts/78
Downloaded https://jsonplaceholder.typicode.com/posts/70
Downloading https://jsonplaceholder.typicode.com/posts/79
Downloaded https://jsonplaceholder.typicode.com/posts/71
Downloading https://jsonplaceholder.typicode.com/posts/80
Downloaded https://jsonplaceholder.typicode.com/posts/68
Downloading https://jsonplaceholder.typicode.com/posts/81
Downloaded https://jsonplaceholder.typicode.com/posts/73
Downloading https://jsonplaceholder.typicode.com/posts/82
Downloaded https://jsonplaceholder.typicode.com/posts/75
Downloading https://jsonplaceholder.typicode.com/posts/83
Downloaded https://jsonplaceholder.typicode.com/posts/76
Downloading https://jsonplaceholder.typicode.com/posts/84
Downloaded https://jsonplaceholder.typicode.com/posts/72
Downloading https://jsonplaceholder.typicode.com/posts/85
Downloaded https://jsonplaceholder.typicode.com/posts/74
Downloading https://jsonplaceholder.typicode.com/posts/86
Downloaded https://jsonplaceholder.typicode.com/posts/77
Downloading https://jsonplaceholder.typicode.com/posts/87
Downloaded https://jsonplaceholder.typicode.com/posts/78
Downloading https://jsonplaceholder.typicode.com/posts/88
Downloaded https://jsonplaceholder.typicode.com/posts/80
Downloading https://jsonplaceholder.typicode.com/posts/89
Downloaded https://jsonplaceholder.typicode.com/posts/79
Downloading https://jsonplaceholder.typicode.com/posts/90
Downloaded https://jsonplaceholder.typicode.com/posts/81
Downloading https://jsonplaceholder.typicode.com/posts/91
Downloaded https://jsonplaceholder.typicode.com/posts/82
Downloading https://jsonplaceholder.typicode.com/posts/92
Downloaded https://jsonplaceholder.typicode.com/posts/83
Downloading https://jsonplaceholder.typicode.com/posts/93
Downloaded https://jsonplaceholder.typicode.com/posts/84
Downloading https://jsonplaceholder.typicode.com/posts/94
Downloaded https://jsonplaceholder.typicode.com/posts/85
Downloading https://jsonplaceholder.typicode.com/posts/95
Downloaded https://jsonplaceholder.typicode.com/posts/86
Downloading https://jsonplaceholder.typicode.com/posts/96
Downloaded https://jsonplaceholder.typicode.com/posts/87
Downloading https://jsonplaceholder.typicode.com/posts/97
Downloaded https://jsonplaceholder.typicode.com/posts/88
Downloading https://jsonplaceholder.typicode.com/posts/98
Downloaded https://jsonplaceholder.typicode.com/posts/90
Downloading https://jsonplaceholder.typicode.com/posts/99
Downloaded https://jsonplaceholder.typicode.com/posts/91
Downloading https://jsonplaceholder.typicode.com/posts/100
Downloaded https://jsonplaceholder.typicode.com/posts/89
Downloaded https://jsonplaceholder.typicode.com/posts/92
Downloaded https://jsonplaceholder.typicode.com/posts/93
Downloaded https://jsonplaceholder.typicode.com/posts/95
Downloaded https://jsonplaceholder.typicode.com/posts/94
Downloaded https://jsonplaceholder.typicode.com/posts/97
Downloaded https://jsonplaceholder.typicode.com/posts/96
Downloaded https://jsonplaceholder.typicode.com/posts/98
Downloaded https://jsonplaceholder.typicode.com/posts/99
Downloaded https://jsonplaceholder.typicode.com/posts/100
{2: 'qui est esse', 7: 'magnam facilis autem', 3: 'ea molestias quasi exercitationem repellat qui ipsa sit aut', 5: 'nesciunt quas odio', 10: 'optio molestias id quia eum', 4: 'eum et est occaecati', 6: 'dolorem eum magni eos aperiam quia', 1: 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit', 8: 'dolorem dolore est ipsam', 11: 'et ea vero quia laudantium autem', 12: 'in quibusdam tempore odit est dolorem', 13: 'dolorum ut in voluptas mollitia et saepe quo animi', 14: 'voluptatem eligendi optio', 15: 'eveniet quod temporibus', 16: 'sint suscipit perspiciatis velit dolorum rerum ipsa laboriosam odio', 17: 'fugit voluptas sed molestias voluptatem provident', 18: 'voluptate et itaque vero tempora molestiae', 19: 'adipisci placeat illum aut reiciendis qui', 20: 'doloribus ad provident suscipit at', 21: 'asperiores ea ipsam voluptatibus modi minima quia sint', 22: 'dolor sint quo a velit explicabo quia nam', 23: 'maxime id vitae nihil numquam', 24: 'autem hic labore sunt dolores incidunt', 25: 'rem alias distinctio quo quis', 28: 'delectus ullam et corporis nulla voluptas sequi', 27: 'quasi id et eos tenetur aut quo autem', 26: 'est et quae odit qui non', 29: 'iusto eius quod necessitatibus culpa ea', 30: 'a quo magni similique perferendis', 31: 'ullam ut quidem id aut vel consequuntur', 32: 'doloremque illum aliquid sunt', 33: 'qui explicabo molestiae dolorem', 34: 'magnam ut rerum iure', 35: 'id nihil consequatur molestias animi provident', 36: 'fuga nam accusamus voluptas reiciendis itaque', 37: 'provident vel ut sit ratione est', 38: 'explicabo et eos deleniti nostrum ab id repellendus', 40: 'enim quo cumque', 39: 'eos dolorem iste accusantium est eaque quam', 42: 'commodi ullam sint et excepturi error explicabo praesentium voluptas', 41: 'non est facere', 44: 'optio dolor molestias sit', 45: 'ut numquam possimus omnis eius suscipit laudantium iure', 46: 'aut quo modi neque nostrum ducimus', 43: 'eligendi iste nostrum consequuntur adipisci praesentium sit beatae perferendis', 47: 'quibusdam cumque rem aut deserunt', 48: 'ut voluptatem illum ea doloribus itaque eos', 49: 'laborum non sunt aut ut assumenda perspiciatis voluptas', 50: 'repellendus qui recusandae incidunt voluptates tenetur qui omnis exercitationem', 52: 'qui enim et consequuntur quia animi quis voluptate quibusdam', 51: 'soluta aliquam aperiam consequatur illo quis voluptas', 53: 'ut quo aut ducimus alias', 54: 'sit asperiores ipsam eveniet odio non quia', 55: 'sit vel voluptatem et non libero', 9: 'nesciunt iure omnis dolorem tempora et accusantium', 56: 'qui et at rerum necessitatibus', 58: 'voluptatum itaque dolores nisi et quasi', 57: 'sed ab est est', 59: 'qui commodi dolor at maiores et quis id accusantium', 61: 'voluptatem doloribus consectetur est ut ducimus', 60: 'consequatur placeat omnis quisquam quia reprehenderit fugit veritatis facere', 63: 'voluptas blanditiis repellendus animi ducimus error sapiente et suscipit', 64: 'et fugit quas eum in in aperiam quod', 62: 'beatae enim quia vel', 65: 'consequatur id enim sunt et et', 66: 'repudiandae ea animi iusto', 67: 'aliquid eos sed fuga est maxime repellendus', 69: 'fugiat quod pariatur odit minima', 70: 'voluptatem laborum magni', 71: 'et iusto veniam et illum aut fuga', 68: 'odio quis facere architecto reiciendis optio', 73: 'consequuntur deleniti eos quia temporibus ab aliquid at', 75: 'dignissimos eum dolor ut enim et delectus in', 76: 'doloremque officiis ad et non perferendis', 72: 'sint hic doloribus consequatur eos non id', 74: 'enim unde ratione doloribus quas enim ut sit sapiente', 77: 'necessitatibus quasi exercitationem odio', 78: 'quam voluptatibus rerum veritatis', 80: 'labore in ex et explicabo corporis aut quas', 79: 'pariatur consequatur quia magnam autem omnis non amet', 81: 'tempora rem veritatis voluptas quo dolores vero', 82: 'laudantium voluptate suscipit sunt enim enim', 83: 'odit et voluptates doloribus alias odio et', 84: 'optio ipsam molestias necessitatibus occaecati facilis veritatis dolores aut', 85: 'dolore veritatis porro provident adipisci blanditiis et sunt', 86: 'placeat quia et porro iste', 87: 'nostrum quis quasi placeat', 88: 'sapiente omnis fugit eos', 90: 'ad iusto omnis odit dolor voluptatibus', 91: 'aut amet sed', 89: 'sint soluta et vel magnam aut ut sed qui', 92: 'ratione ex tenetur perferendis', 93: 'beatae soluta recusandae', 95: 'id minus libero illum nam ad officiis', 94: 'qui qui voluptates illo iste minima', 97: 'quas fugiat ut perspiciatis vero provident', 96: 'quaerat velit veniam amet cupiditate aut numquam ut sequi', 98: 'laboriosam dolor voluptates', 99: 'temporibus sit alias delectus eligendi possimus magni', 100: 'at nam consequatur ea labore ea harum'}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment