Skip to content

Instantly share code, notes, and snippets.

@adi928

adi928/crawler

Created September 28, 2019 16:01
Show Gist options
  • Save adi928/cb6830bd66b31561159de547f850d9c9 to your computer and use it in GitHub Desktop.
Save adi928/cb6830bd66b31561159de547f850d9c9 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
crawler
from http_crawler import crawl
if __name__ == '__main__':
for rsp in crawl('https://admissions.iusb.edu/', exceptions='https://students.iusb.edu/about/big-list.html'):
if rsp.status_code == 200:
print('Got {}'.format(rsp.url))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment