Skip to content

Instantly share code, notes, and snippets.

@jiobu1
Created April 26, 2021 19:36
Show Gist options
  • Save jiobu1/2fbc003ce688d58b384206d2842d7511 to your computer and use it in GitHub Desktop.
Save jiobu1/2fbc003ce688d58b384206d2842d7511 to your computer and use it in GitHub Desktop.
looping through city amounts
for i in cities['city']:
fetching = True
page = 0
while fetching:
page += 1
url = url_pre + urllib.parse.quote(i) + '/schools/?page={}&tableView=Overview&view=table'.format(page)
print("Fetching ", url)
driver.get(url)
html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')
# check if last page
page_status = soup.find('div', {'class': 'pagination-summary'})
# create list of pagination summary text [Showing, 1, to, 25, of, 1,109, schools, found, in, New, York, NY]
page_status_list = page_status.text.strip().split()
ending = (page_status_list[3]).replace(',', '') # postion 3 is the number of the last item on page
total = (page_status_list[5]).replace(',' , '') # position 5 is teh total number of schools
if int(ending) >= int(total):
fetching = False # stops the loop
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment