Skip to content

Instantly share code, notes, and snippets.

@KameronKales
Created March 9, 2018 00:52
Show Gist options
  • Save KameronKales/a314df91bc942445c2c296f9bd49bb34 to your computer and use it in GitHub Desktop.
Save KameronKales/a314df91bc942445c2c296f9bd49bb34 to your computer and use it in GitHub Desktop.
Python For Sourcers Web-scraping Data
import requests
from bs4 import BeautifulSoup
import csv
leads = []
rates = []
for i in range(3):
url = "https://www.greatschools.org/virginia/manassas/prince-william-county-public-schools/schools/?page={}".format(i)
r = requests.get(url, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36'})
soup = BeautifulSoup(r.text, 'lxml')
print url
for sub_heading in soup.find_all("a", {"class":"open-sans_sb mbs font-size-medium rs-schoolName"}):
lead = sub_heading.text
leads.append(lead)
for sub_headings in soup.find_all("span", {"class":"gs-rating"}):
rate = sub_headings.text
rates.append(rate)
print leads, rates
with open('filename.csv', 'wb') as myfile:
wr = csv.writer(myfile, quoting=csv.QUOTE_ALL)
wr.writerow(leads)
wr.writerow(rates)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment