Skip to content

Instantly share code, notes, and snippets.

@aljiwala
Created July 26, 2018 12:09
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save aljiwala/138311f81b2e1e5247c9dc53aec7ef5b to your computer and use it in GitHub Desktop.
Save aljiwala/138311f81b2e1e5247c9dc53aec7ef5b to your computer and use it in GitHub Desktop.
Get meta keywords and description from website.
def get_soup(url):
# Func which should return `BeautifulSoup` object.
pass
def get_meta_from_website(url, attrs=['description', 'keywords']):
"""
Should return meta data from given website URL.
* Receives:
- url (string)
- attrs (list)
* Returns
- d (dict)
- err (string)
"""
d = dict()
soup, err = get_soup(url)
if err:
return dict(), err
if soup.title:
d['title'] = soup.title.text.strip()
meta_list = soup.find_all('meta')
for meta in meta_list:
m_attrs = meta.attrs
attr_namev = m_attrs.get('name', '').strip().lower()
if 'name' in m_attrs.keys() and attr_namev in attrs:
name = m_attrs.get('name', '').lower()
content = m_attrs.get('content', '')
d[name] = content
return d, None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment