Skip to content

Instantly share code, notes, and snippets.

@shubhamcodez
Last active May 19, 2023 09:48
Show Gist options
  • Save shubhamcodez/0e51a5ee9a8d3115ddad96edaca7e1ac to your computer and use it in GitHub Desktop.
Save shubhamcodez/0e51a5ee9a8d3115ddad96edaca7e1ac to your computer and use it in GitHub Desktop.
# Make sure you pip install snscrape pandas
import snscrape.modules.twitter as sntwitter
import pandas as pd
#Enter usersnames of the users whose tweets you need in the list
users = ['elonmusk']
#Enter the number of tweets per user you'd like. Note: In case user has less than limit, all the tweets get downloaded.
limit = 10000
for user in users:
until = "2022-12-15" #Enter the date until you want the tweet for
since = "2022-01-01" #Enter the date from the tweets start getting queried
query = f"(from:{user}) until:{until} since:{since}"
tweets = []
for tweet in sntwitter.TwitterSearchScraper(query).get_items():
#print(vars(tweet)) #gets all the attributes retrieved.
if len(tweets) == limit:
break
else:
contents = [tweet.user.displayname, tweet.user.username, tweet.url, tweet.date, tweet.content, tweet.retweetCount, tweet.likeCount] #Instruction: Add more attributes here
tweets.append(contents)
df = pd.DataFrame(tweets, columns= ["Username","User handle", "Tweet", "Date of posting" ,"Text","Retweet count","Like count"]) #Instruction: add a new name for column added
df.to_csv(user+'tweets.csv')
@schatekar
Copy link

I think my problem is installing re

ERROR: Could not find a version that satisfies the requirement re (from versions: none)
ERROR: No matching distribution found for re

I am new to python so bear with me.
I tried installing re2 but that did not work either.

@shubhamcodez
Copy link
Author

shubhamcodez commented Jan 31, 2023

I think my problem is installing re

ERROR: Could not find a version that satisfies the requirement re (from versions: none)
ERROR: No matching distribution found for re

I am new to python so bear with me. I tried installing re2 but that did not work either.

Oh, I see. No worries.
To build re2, you need to install Visual C++14.
Download this: https://visualstudio.microsoft.com/visual-cpp-build-tools/
Then installed, the selected packages in this image and also, Windows 10 SDK package:
image
Once you have installed it, restart the pc and try installing re2 again.
Let me know if it works.

@schatekar
Copy link

Actually, it turned out my problem was entirely different. I was using py to run the script. I used python and it is working fine.

@shubhamcodez
Copy link
Author

shubhamcodez commented Jan 31, 2023 via email

@Christopher-Hayes
Copy link

Christopher-Hayes commented Mar 7, 2023

Some modifications I made on MacOS:

  • (I don't write a lot of python) I had trouble with python trying to run as Python2. But, creating a venv fixed this issue for me - python3 -m venv py3 to create, then source py3/bin/activate to activate (py3 is the environment name, you can call it anything)
  • It seems only snscrape and pandas are actually used. So, installing numpy, re, and os aren't needed. The import lines for those packages can be removed.
  • The new install command was pip3 install snscrape pandas

Btw - thank you for this script. There's some giant repos on GitHub with dozens of dependencies. This is much lighter to run.

@shubhamcodez
Copy link
Author

@Christopher-Hayes Thank you for making these changes. I am cursed with the habit of importing libraries. :p

@smyja
Copy link

smyja commented May 18, 2023

script isn't working again

@shubhamcodez
Copy link
Author

script isn't working again

I am sorry to inform you that twitter has updated its APIs, and none of the third-party scrapers will work.

@smyja
Copy link

smyja commented May 19, 2023

script isn't working again

I am sorry to inform you that twitter has updated its APIs, and none of the third-party scrapers will work.

Yeah, snscraper still works though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment