Skip to content

Instantly share code, notes, and snippets.

Created May 14, 2019 18:37
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
What would you like to do?
A python script to scrape today's trending Twitter hashtags from
from bs4 import BeautifulSoup
import urllib
URL = ""
# Open the URL
page = urllib.request.Request(URL)
result = urllib.request.urlopen(page)
# Store the HTML page in a variable
resulttext =
# Creates a nested data structure
soup = BeautifulSoup(resulttext, 'html.parser')
# Since we are interested only in an element with class "list-group"
# We will search for all elements with that class in the soup
soup = soup.find_all(class_= "list-group")
# Soup now contains an array of all elements with the class "list-group"
# Since the Trending today list is the first on the page, it's index is 0
trending_list = soup[0]
# Now we will iterate through the elements of the <ul>
# <ul> has <li> tags nested inside
trending_tags = []
for li in trending_list.contents:
# There is an <a> tag nested in each <li>
a = li.contents[0]
# The contents of 'a' is just the text inside the tag
tag = a.contents[0].strip()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment