Skip to content

Instantly share code, notes, and snippets.

@umangahuja1
Created October 19, 2018 14:46
Show Gist options
  • Save umangahuja1/eb0641a3458a1e91e6e71c662a912735 to your computer and use it in GitHub Desktop.
Save umangahuja1/eb0641a3458a1e91e6e71c662a912735 to your computer and use it in GitHub Desktop.
import requests
from bs4 import BeautifulSoup
from time import sleep
base_url = 'http://quotes.toscrape.com/page/'
all_urls = list()
def generate_urls():
for i in range(1,11):
all_urls.append(base_url + str(i))
def scrape(url):
res = requests.get(url)
print(res.status_code, res.url)
generate_urls()
for url in all_urls:
scrape(url)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment