Skip to content

Instantly share code, notes, and snippets.

@AnderRV

AnderRV/parserlist.py Secret

Created August 20, 2021 10:16
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save AnderRV/004d891f58cb88e52d6f4048209bb24e to your computer and use it in GitHub Desktop.
Save AnderRV/004d891f58cb88e52d6f4048209bb24e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
parserlist.py
from urllib.parse import urlparse
from parsers import defaults
parsers = {
'scrapeme.live': defaults,
'quotes.toscrape.com': defaults,
}
def get_parser(url):
hostname = urlparse(url).hostname # extract domain from URL
if hostname in parsers:
# use the dict above to return the custom parser if present
return parsers[hostname]
return defaults
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment