Skip to content

Instantly share code, notes, and snippets.

@linxlunx
Created October 28, 2022 03:18
Show Gist options
  • Save linxlunx/b90e7b8db9df5468b23c51e3a0f03643 to your computer and use it in GitHub Desktop.
Save linxlunx/b90e7b8db9df5468b23c51e3a0f03643 to your computer and use it in GitHub Desktop.
Multithread using concurrent.futures
import requests
from time import time
from concurrent.futures import ThreadPoolExecutor, as_completed
now = time()
urls = ['https://linggar.asia', 'https://detik.com', 'https://kompas.com']
def download_page(url):
return requests.get(url).links
def download_all(urls):
threads = []
thread_map = {}
with ThreadPoolExecutor(max_workers=len(urls)) as executor:
for url in urls:
future = executor.submit(download_page, url)
thread_map[future] = url
threads.append(future)
for task in as_completed(threads):
url = thread_map[task]
print(url, task.result())
download_all(urls)
print(time() - now
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment