Skip to content

Instantly share code, notes, and snippets.

@YiLi225

YiLi225/scrapTags.py

Created October 25, 2022 15:59
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save YiLi225/6ad914a6715d368ffe93770fb9821723 to your computer and use it in GitHub Desktop.
Save YiLi225/6ad914a6715d368ffe93770fb9821723 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
scrapTags.py
from bs4 import BeautifulSoup
from urllib.request import urlopen, Request
from collections import defaultdict
tagInfo = defaultdict(list)
for tag in tags:
startLink = "https://medium.com/tag/"+tag
response = requests.get(startLink, allow_redirects=True)
page = response.content
soup = BeautifulSoup(page, 'html.parser')
articleAuthor = soup.find_all('p', {'class':"bm b dm dn fv"})
article, author = articleAuthor[0].contents[0], articleAuthor[1].contents[0]
tagInfo[tag].append([article, author])
print(f'**** {tag} = {[article, author]} ***')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment