Skip to content

Instantly share code, notes, and snippets.

@baskaufs
Last active January 21, 2019 21:36
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save baskaufs/0d01635d30f89044fe3153f3d4302a2c to your computer and use it in GitHub Desktop.
Save baskaufs/0d01635d30f89044fe3153f3d4302a2c to your computer and use it in GitHub Desktop.
Python script to scrape chang.org
import requests # best library to manage HTTP transactions
import csv # library to read/write/parse CSV files
from bs4 import BeautifulSoup # web-scraping library
acceptMime = 'text/html'
searchTerm = 'pollution'
baseUri = 'https://www.change.org/search?q='+searchTerm
r = requests.get(baseUri, headers={'Accept' : acceptMime})
soup = BeautifulSoup(r.text,features="html5lib")
# this search string limits results to only div elements with the attribute that's exactly equal to"search-result"
# the select function returns a list of soup objects that can each be searched
for cat in soup.select('div[class="search-result"]'):
#for cat in soup.select('div[class="media-profile-body txt-m"]'):
print(cat)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment