Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
Scraping suru product IDs
This script will print the IDs of all the products returned by a given search query.
Substitue url (line 17) string with any jewgaya query. The script will stop after reaching the last page.
The URL must be properly encoded according to RFC 3986.
from bs4 import BeautifulSoup
import requests
def get_soup(url):
content = requests.get(url)
return BeautifulSoup(content.text, 'html.parser')
address = ''
url = ''
soup = get_soup(url)
while True:
titles = soup.find_all('p', class_='title')
for item in titles:
a = item.find('a').attrs['href']
link = a.rsplit('/')[-1]
next_page = soup.find('li', class_='next')
if next_page is not None:
url_next = address + next_page.find('a').attrs['href']
soup = get_soup(url_next)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.