Skip to content

Instantly share code, notes, and snippets.

@AnderRV

AnderRV/main.partial.py Secret

Created August 20, 2021 10:13
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save AnderRV/ca76b6733c381c0949fdc2be45900881 to your computer and use it in GitHub Desktop.
Save AnderRV/ca76b6733c381c0949fdc2be45900881 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
main.partial.py
maximum_items = 5
while True:
visited = connection.scard('crawling:visited') # count URLs in visited
queued = connection.scard('crawling:queued')
if queued + visited > maximum_items:
print('Exiting! Over maximum')
break
# ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment