Skip to content

Instantly share code, notes, and snippets.

@davisvictorns
Last active September 15, 2021 01:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save davisvictorns/6208745ea398bb196fde1f9a40e2632b to your computer and use it in GitHub Desktop.
Save davisvictorns/6208745ea398bb196fde1f9a40e2632b to your computer and use it in GitHub Desktop.
c = twitter.cursor(twitter.search, q='#7deSetPelaLiberdade', count=100, result_type='mixed', tweet_mode='extended')
tweets = []
count = 0
n_tweets = 30000
for tweet in c:
count += 1
tweets.append(tweet)
# each 100 tweets requires 1 request
# sleep after 180 requests
if count > 0 and count % (180 * 100) == 0:
print(f'running save iteration {count}')
df_result = tweets_to_df(tweets)
print(df_result.shape)
save_tweets(df_result)
tweets = []
print('sleep for 15 min')
time.sleep(60 * 15)
if (count > n_tweets):
df_result = tweets_to_df(tweets)
print(df_result.shape)
save_tweets(df_result)
break
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment