Skip to content

Instantly share code, notes, and snippets.

@vickyqian
Last active May 11, 2024 16:19
Show Gist options
  • Save vickyqian/f70e9ab3910c7c290d9d715491cde44c to your computer and use it in GitHub Desktop.
Save vickyqian/f70e9ab3910c7c290d9d715491cde44c to your computer and use it in GitHub Desktop.
A Python script to download all the tweets of a hashtag into a csv
import tweepy
import csv
import pandas as pd
####input your credentials here
consumer_key = ''
consumer_secret = ''
access_token = ''
access_token_secret = ''
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth,wait_on_rate_limit=True)
#####United Airlines
# Open/Create a file to append data
csvFile = open('ua.csv', 'a')
#Use csv Writer
csvWriter = csv.writer(csvFile)
for tweet in tweepy.Cursor(api.search,q="#unitedAIRLINES",count=100,
lang="en",
since="2017-04-03").items():
print (tweet.created_at, tweet.text)
csvWriter.writerow([tweet.created_at, tweet.text.encode('utf-8')])
@eshraongithub
Copy link

eshraongithub commented Nov 30, 2020

This code works. Thank you!
How can I increase the character limit that is currently forced on the tweets extracted?

Thank you!

@McHarold404
Copy link

What is the max value to which I can set the count to?

@mindyng
Copy link

mindyng commented Jan 5, 2021

Thanks! This is short and sweet. Worked for me using: Python 3.7.6

also, for those wondering how to get output into a Pandas DF, I did this (added two lists):

created_at = []
text = []
for tweet in tweepy.Cursor(api.search,q=["#covidnurse"],count=100,
                           lang="en",
                           since="2020-03-15").items():
    print (tweet.created_at, tweet.text)
    csvWriter.writerow([tweet.created_at, tweet.text.encode('utf-8')])
    created_at.append(tweet.created_at)
    text.append(tweet.text)

df =  pd.DataFrame(
    {'created_at': created_at,
     'tweet': text
    })
df #RT (Retweets) are duplicates since Twitter is an echo chamber; will need to get rid of duplicates

df_clean = df[~df.tweet.str.contains("RT")]
df_clean

@lcontini
Copy link

lcontini commented Feb 5, 2021

Great work!!
I have a question- will it return any response if there is one only tweet which is dated Feb 19, 2015? For hashtag- LexusEnformRemote
I have read Tweepy only return tweets for the past 7 days.
One more thing, how to search for multiple hashtags in a single loop?

Answering my own question of getting tweets beyond the past 7 days. I used GetOldTweets3 library by @Jefferson-Henrique and it's working fine for me. We can get tweets by multiple user accounts in one go but search doesn't take two hashtags.
Link

@abhishek-negi to search for multiple hashtags you could perform a loop from a list.

Something like this:

list_tweets_all_hashtags = []
hashtags_list = #tag1, #tag2, #tagN

for hashtag in hashtags_list:
    list_by_tag = [for tweet in tweepy.Cursor(api.search, q=hashtag, result_type="recent").items(100)]
    list_tweets_all_hashtags.extend(list_by_tag)
    
print(list_tweets_all_hashtags)

@ashex08
Copy link

ashex08 commented Feb 17, 2021

I am getting the below error at this line: csvFile = open('askgretawhy.csv', 'a')

Traceback (most recent call last):
File "", line 1, in
PermissionError: [Errno 13] Permission denied: 'askgretawhy.csv'

Please help me understand where is the issue.

@lcontini
Copy link

I am getting the below error at this line: csvFile = open('askgretawhy.csv', 'a')

Traceback (most recent call last):
File "", line 1, in
PermissionError: [Errno 13] Permission denied: 'askgretawhy.csv'

Please help me understand where is the issue.

It’s most likely you don’t have the appropriate permissions to write/overwrite file askgretawhy.csv or to write on the current folder where the program runs.

If the file already exists, I suggest to run chmod 777 askgretawhy.csv and try again. If not, try giving less strict permissions to the current folder, if you can.

@ashex08
Copy link

ashex08 commented Feb 17, 2021

I am getting the below error at this line: csvFile = open('askgretawhy.csv', 'a')
Traceback (most recent call last):
File "", line 1, in
PermissionError: [Errno 13] Permission denied: 'askgretawhy.csv'
Please help me understand where is the issue.

It’s most likely you don’t have the appropriate permissions to write/overwrite file askgretawhy.csv or to write on the current folder where the program runs.

If the file already exists, I suggest to run chmod 777 askgretawhy.csv and try again. If not, try giving less strict permissions to the current folder, if you can.

How do I change the permissions? I am working on basic Python editor and do not know where can I create the file for Python to read it.

@riiyasharmaa
Copy link

File "C:/Users/hp/Documents/1.py", line 5
consumer_key ='249QYJU8UqoS364VqIOoWmpJQ'
^
IndentationError: unexpected indent

Getting this error please help
ASAP!

@mindyng
Copy link

mindyng commented May 21, 2021

@riiyasharmaa you might need to unindent (tab to the left)

@ftirador
Copy link

Many thanks for the crawler mate, I was looking for it and didn´t have time to spend on it.

cheers

@Nathalie91
Copy link

I get This error from Python :
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 22-23: truncated \UXXXXXXXX escape
Any help please !!

@Nathalie91
Copy link

Hello,
how we can add the geolocalisation like : France to this script and more filters in the search ???

@coder-nooby
Copy link

AttributeError Traceback (most recent call last)
C:\Users\ADMINI~1\AppData\Local\Temp/ipykernel_11192/3895188609.py in
19 #Use csv Writer
20 csvWriter = csv.writer(csvFile)
---> 21 for tweet in tweepy.Cursor(api.search, q="#unitedAIRLINES",count=100, lang="en", since="2021-09-14").items():
22 print (tweet.created_at, tweet.text)
23 csvWriter.writerow([tweet.created_at, tweet.text.encode('utf-8')])

AttributeError: 'API' object has no attribute 'search'

I am using the latest version of Anaconda Navigator 2.10 and jupyter Notebook 6.4.4.

I have regenerated API tokens. How do should I fix this?

@ansegura7
Copy link

AttributeError Traceback (most recent call last) C:\Users\ADMINI~1\AppData\Local\Temp/ipykernel_11192/3895188609.py in 19 #Use csv Writer 20 csvWriter = csv.writer(csvFile) ---> 21 for tweet in tweepy.Cursor(api.search, q="#unitedAIRLINES",count=100, lang="en", since="2021-09-14").items(): 22 print (tweet.created_at, tweet.text) 23 csvWriter.writerow([tweet.created_at, tweet.text.encode('utf-8')])

AttributeError: 'API' object has no attribute 'search'

I am using the latest version of Anaconda Navigator 2.10 and jupyter Notebook 6.4.4.

I have regenerated API tokens. How do should I fix this?

I am getting the exact error reported above.

I had to upgrade my version of Anaconda and then installed tweepy version 4.1.0 and from there the following command is not working for me:

tweepy.Cursor(api.search, q=ht, since=since_date_str).items()

Any idea how to solve it? do I have to go back to a previous version of tweepy? if so, which version?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment