Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
NOTE: this code is for a previous version of the Twitter API and I will not be updating in the near future. If someone else would like to, I'd welcome that! Feel free to ping me. END NOTE. Here I define a Tweet listener that creates a file called 'tweets.txt', collects streaming tweets as .jsons and writes them to the file 'tweets.txt'; once 100…
class MyStreamListener(tweepy.StreamListener):
def __init__(self, api=None):
super(MyStreamListener, self).__init__()
self.num_tweets = 0
self.file = open("tweets.txt", "w")
def on_status(self, status):
tweet = status._json
self.file.write( json.dumps(tweet) + '\n' )
self.num_tweets += 1
if self.num_tweets < 100:
return True
else:
return False
self.file.close()
def on_error(self, status):
print(status)
@wkkim-se

This comment has been minimized.

Copy link

wkkim-se commented Aug 26, 2016

How does 'self.file.close()' get called when the prior if-else blocks returns?

@lincht

This comment has been minimized.

Copy link

lincht commented Sep 25, 2016

Does the line tweet_list.append(status) serve any specific purpose here?
The first time I ran the code, it threw a NameError saying that tweet_list doesn't exist. After removing that line, the code seems to run just fine.

@jakemorse93

This comment has been minimized.

Copy link

jakemorse93 commented Nov 9, 2016

I'm getting a NameError as well: "NameError: name 'tweet_list' is not defined". Anybody have a solution for this? I don't see where tweet_list is originally defined in the code.

@hugobowne

This comment has been minimized.

Copy link
Owner Author

hugobowne commented Nov 9, 2016

@LincT @jakemore93 @wkkim-se excuse delay: i wasn't alerted of these messages until jake pinged me personally: thanks, jake!

  1. tweet_list.append(status) does nothing and doesn't belong here; it is a relic from a previous version in which i saved to a list, rather than a file
  2. @wwkim-se : you're right -- it doesn't close; one really should close that! A while loop would suffice, i think
@kamrankausar

This comment has been minimized.

Copy link

kamrankausar commented Jan 7, 2017

Can you please tell me how to access past data from twitter. or send me link where i get all the details .print('Thanks'*10, 'in', "advance")

@plumps

This comment has been minimized.

Copy link

plumps commented Jun 13, 2017

@wkkim-se of course not :-) Other than that, for the sake of correctness and memory leaks, self.file.write should be surrounded by a with context manager like,

# __init__()
self.file_name = "tweets.txt"

# on_status()
with open(self.file_name, 'w') as file:
    file.write(json.dumps(tweet) + '\n')
@Divkar94

This comment has been minimized.

Copy link

Divkar94 commented Nov 20, 2017

Hi! Can you explain what 'self' does? If you could explain the workings of this code in detail, it would be great! I'm a beginner at this. I'm not sure i understand the first part of the code or when the class gets called.Thanks!!

@ovihentea

This comment has been minimized.

Copy link

ovihentea commented Nov 22, 2017

@Divkar94, Hugo explains it in the context of DataCamp's Importing Data in Python

@JagDecoded

This comment has been minimized.

Copy link

JagDecoded commented Dec 14, 2017

@Divkar94 It's bit advance of python (Object Oriented Par / Class Part).

In simple self here is like a variable which will hold the object address when we will call any method of that class using the object (for that instance)

for example,

class X:
    def adrs(self):
        print(type(self))
        print(id(self))
x1= X()
x1.adrs()

x2=X()
x2.adrs()

adding to that using self we can allocate value to that particular instance.

Hope you understand.

@hiliev

This comment has been minimized.

Copy link

hiliev commented Dec 19, 2017

The file open mode in @plumps' code should be 'a' for appending, otherwise the previous contents will be overwritten with each new tweet.

@eric-ahlgren

This comment has been minimized.

Copy link

eric-ahlgren commented Jan 23, 2018

Can't we just put the close() statement above the return statement in the else block? As this is written now we never hit the close statement, so I am confused how the solution is a while loop. Is there anything wrong with this?

        else:
            self.file.close()
            return False
@Iqlaas

This comment has been minimized.

Copy link

Iqlaas commented Jul 19, 2018

datacamp rules- especially hugo.

@ajaigovindg

This comment has been minimized.

Copy link

ajaigovindg commented Sep 24, 2018

@hugobowne

As per the comments from @wkkim-se @plumps and @hiliev I believe there are corrections required in this class for downloading streaming Twitter data. I am new to Python so not yet into creating classes. I have made the corrections as per my understanding and it appends the text file. Hope it is correct. Could you review below code and possibly correct the code here in Github since there will be many like me who will be confused and hunting for the solution!

class MyStreamListener (tweepy.StreamListener):
    def __init__(self, api = None):
        super(MyStreamListener, self).__init__()
        self.num_tweets = 0
        self.file_name = "tweets.txt"
        #self.file = open("tweets.txt", "w")

    def on_status(self, status):
        tweet = status._json
        with open(self.file_name, 'a') as file:
            file.write(json.dumps(tweet) + '\n')
        self.num_tweets += 1
        if self.num_tweets < 100:
            return True
        else:
            return False

    def on_error(self, status):
        print(status)
@nipun-goyal

This comment has been minimized.

Copy link

nipun-goyal commented Oct 10, 2019

@hugobowne Thanks for sharing this code. I had a query related to Loading and Exploring the twitter data.

I was trying the below code on my laptop. However, it returned an error. Error message has been shown below:

Read in tweets and store in list: tweets_data

for line in tweets_file:
tweet = json.loads(line)
tweets_data.append(tweet)

Error Message: JSONDecodeError: Extra data: line 1 column 5703 (char 5702)

Could you please help me with this issue?

Nipun

@gislipals

This comment has been minimized.

Copy link

gislipals commented Oct 28, 2019

Anyone else getting a '401' response from Twitter when you replace the mock access and consumer keys with your own?

@absonob

This comment has been minimized.

Copy link

absonob commented Dec 23, 2019

This code is the child class, where is the parent class? Post it please

@IbraM1993

This comment has been minimized.

Copy link

IbraM1993 commented Dec 31, 2019

@eric-ahlgren I think it will work just fine

@plumps If it was on_error and not on_status, and the file was opened in on_status not in init, wouldn't the file close? And if it did close, the file was opened in "w" mode but not "a" , wouldn't the content be lost every time the file is reopened?

@strashynskyi

This comment has been minimized.

Copy link

strashynskyi commented Mar 27, 2020

@hugobowne can you please add the correct code?

@hugobowne

This comment has been minimized.

Copy link
Owner Author

hugobowne commented Mar 27, 2020

@strashynskyi thanks for pinging me. it looks like this the twitter API has changed so that this code doesn't run now. I don't have the bandwidth to go in and figure out what the correct code looks like. If someone else wants to, that would be great. I've made the following note in the description of this gist:

NOTE: this code is for a previous version of the Twitter API and I will not be updating in the near future. If someone else would like to, I'd welcome that! Feel free to ping me. END NOTE.

@VictorOmondi1997

This comment has been minimized.

Copy link

VictorOmondi1997 commented Jun 2, 2020

class MyStreamListener (tweepy.StreamListener):
    def __init__(self, api = None):
        super(MyStreamListener, self).__init__()
        self.num_tweets = 0
        self.file_name = "tweets.txt"
        #self.file = open("tweets.txt", "w")

    def on_status(self, status):
        tweet = status._json
        with open(self.file_name, 'a') as file:
            file.write(json.dumps(tweet) + '\n')
        self.num_tweets += 1
        if self.num_tweets < 100:
            return True
        else:
            return False

    def on_error(self, status):
        print(status)
@tab1tha

This comment has been minimized.

Copy link

tab1tha commented Jun 25, 2020

class MyStreamListener (tweepy.StreamListener):
    def __init__(self, api = None):
        super(MyStreamListener, self).__init__()
        self.num_tweets = 0
        self.file_name = "tweets.txt"
        #self.file = open("tweets.txt", "w")

    def on_status(self, status):
        tweet = status._json
        with open(self.file_name, 'a') as file:
            file.write(json.dumps(tweet) + '\n')
        self.num_tweets += 1
        if self.num_tweets < 100:
            return True
        else:
            return False

    def on_error(self, status):
        print(status)

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.