Skip to content

Instantly share code, notes, and snippets.

@AnderRV
Created August 20, 2021 10:14
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save AnderRV/4c0dfe8a584334a49eef09cd30287028 to your computer and use it in GitHub Desktop.
Save AnderRV/4c0dfe8a584334a49eef09cd30287028 to your computer and use it in GitHub Desktop.
import repo
def extract_content(url, soup):
return soup.title.string # extract page's title
def store_content(url, content):
# store in a hash with the URL as the key and the title as the content
repo.set_content(url, content)
def allow_url_filter(url):
return True # allow all by default
def get_html(url):
# ... same as before
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment