Skip to content

Instantly share code, notes, and snippets.

@iampawan
Created April 21, 2019 06:00
Show Gist options
  • Save iampawan/f7c084f8ba8889a87b3e8d148fc78341 to your computer and use it in GitHub Desktop.
Save iampawan/f7c084f8ba8889a87b3e8d148fc78341 to your computer and use it in GitHub Desktop.
Stack Overflow Questions Scrapping
import requests
from bs4 import BeautifulSoup
import json
res = requests.get("https://stackoverflow.com/questions")
soup = BeautifulSoup(res.text, "html.parser")
questions_data = {
"questions": []
}
questions = soup.select(".question-summary")
for que in questions:
q = que.select_one('.question-hyperlink').getText()
vote_count = que.select_one('.vote-count-post').getText()
views = que.select_one('.views').attrs['title']
questions_data['questions'].append({
"question": q,
"views": views,
"vote_count": vote_count
})
json_data = json.dumps(questions_data)
print(json_data)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment