Skip to content

Instantly share code, notes, and snippets.

@AnderRV
Created August 11, 2021 15:10
Show Gist options
  • Save AnderRV/a04d4757990c412bb82a05fedef5e8a2 to your computer and use it in GitHub Desktop.
Save AnderRV/a04d4757990c412bb82a05fedef5e8a2 to your computer and use it in GitHub Desktop.
from threading import Thread
def queue_worker(i, q):
while True:
url = q.get() # Get an item from the queue, blocks until one is available
print('to process:', url)
q.task_done() # Notifies the queue that the item has been processed
q = queue.Queue()
Thread(target=queue_worker, args=(0, q), daemon=True).start()
q.put('https://scrapeme.live/shop/page/1/')
q.join() # Blocks until all items in the queue are processed and marked as done
print('Done')
# to process: https://scrapeme.live/shop/page/1/
# Done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment