Skip to content

Instantly share code, notes, and snippets.

@bennylope
Created May 9, 2013 20:13
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bennylope/5550220 to your computer and use it in GitHub Desktop.
Save bennylope/5550220 to your computer and use it in GitHub Desktop.
Super simple scraper that uses `requests` to fetch a single page and then uses BeautifulSoup to parse the meta description.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import re
import requests
from bs4 import BeautifulSoup
url = "http://www.wellfireinteractive.com"
response = requests.get(url, headers={'User-Agent': 'TotallyWellMeaningPageScraper'})
content = BeautifulSoup(response.content)
tag = content.find("meta", attrs={'name': re.compile('description', re.I)})
description = '' if not tag else tag.get('content', '')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment