Skip to content

Instantly share code, notes, and snippets.

@moxak
Last active March 15, 2024 11:05
Show Gist options
  • Star 7 You must be signed in to star a gist
  • Fork 3 You must be signed in to fork a gist
  • Save moxak/ed83dd4169112a0b1669500fe855101a to your computer and use it in GitHub Desktop.
Save moxak/ed83dd4169112a0b1669500fe855101a to your computer and use it in GitHub Desktop.
This is one to modify twint which is python module to scrape twitter without API token. In order to use, you must put this script on `twint/` and replace defalt one.
import re
import time
import requests
import logging as logme
class TokenExpiryException(Exception):
def __init__(self, msg):
super().__init__(msg)
class RefreshTokenException(Exception):
def __init__(self, msg):
super().__init__(msg)
class Token:
def __init__(self, config):
self._session = requests.Session()
self._session.headers.update({'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0'})
self.config = config
self._retries = 5
self._timeout = 10
self._session.headers.update({'authorization': 'Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA'})
self.url = 'https://api.twitter.com/1.1/guest/activate.json'
def _request(self):
for attempt in range(self._retries + 1):
# The request is newly prepared on each retry because of potential cookie updates.
req = self._session.prepare_request(requests.Request('POST', self.url))
logme.debug(f'Retrieving {req.url}')
try:
r = self._session.send(req, allow_redirects=True, timeout=self._timeout)
except requests.exceptions.RequestException as exc:
if attempt < self._retries:
retrying = ', retrying'
level = logme.WARNING
else:
retrying = ''
level = logme.ERROR
logme.log(level, f'Error retrieving {req.url}: {exc!r}{retrying}')
else:
success, msg = (True, None)
msg = f': {msg}' if msg else ''
if success:
logme.debug(f'{req.url} retrieved successfully{msg}')
return r
if attempt < self._retries:
# TODO : might wanna tweak this back-off timer
sleep_time = 2.0 * 2 ** attempt
logme.info(f'Waiting {sleep_time:.0f} seconds')
time.sleep(sleep_time)
else:
msg = f'{self._retries + 1} requests to {self.url} failed, giving up.'
logme.fatal(msg)
self.config.Guest_token = None
raise RefreshTokenException(msg)
def refresh(self):
logme.debug('Retrieving guest token')
res = self._request()
res_json = res.json()
if "guest_token" in res_json.keys():
logme.debug('Found guest token in HTML')
self.config.Guest_token = res_json["guest_token"]
else:
self.config.Guest_token = None
raise RefreshTokenException('Could not find the Guest token in HTML')
@moxak
Copy link
Author

moxak commented Jan 12, 2022

modified refresh function to return RefreshTokenException.

@daqing613
Copy link

Dear Moxak,

By using your token.py, the twint works for a few days. Now I got an error "CRITICAL:root:twint.run:Twint:Feed:noData'globalObjects'". And I find that the request to twitter is forbidden. May I know how to get the bearer token? Have you encountered the same issue?

Really appreciated your help, and thanks in advance.

Daniel.

@moxak
Copy link
Author

moxak commented Apr 25, 2023

Hi daqing613,

Thank you for using it.
When did it stop working? Recently, there have been significant changes to the Twitter API, which might be the cause of the issue.

@ElenaAEGeo
Copy link

Hi Moxak ,

I have the same error from Friday.
Twitter is changing access to scraping: https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api

@moxak
Copy link
Author

moxak commented Apr 27, 2023

Thanks for providing the info, ElenaAEGeo.

In this script, it specifies the use of Twitter API version 1.1 (Standard Plan), but it seems that the old plans such as 1.1 and 2.0 will be discontinued by April 29th.
https://twittercommunity.com/t/deprecating-the-premium-v1-1-api/191092

To continue using the Twitter API, we'll need to switch to a different plan, such as the Free, Basic, or Enterprise plans. The Free plan appears to be similar to version 2.0 api, and since the authentication methods and endpoints used have changed significantly, we will need to make fundamental modifications to the script mentioned above in order to get it working.

Therefore, it would be difficult to make the necessary modifications within the next couple of days. I'm sorry.

@daqing613
Copy link

Hi daqing613,

Thank you for using it. When did it stop working? Recently, there have been significant changes to the Twitter API, which might be the cause of the issue.

Thanks for your reply. I figured out that the original API had been deprecated. Looks like Nitter still works fine. I hope that makes some help.

@moxak
Copy link
Author

moxak commented Apr 28, 2023

Looks like Nitter still works fine. I hope that makes some help.

I'll check it out :)

@AlexUXrm
Copy link

Thanks for providing the info, ElenaAEGeo.

In this script, it specifies the use of Twitter API version 1.1 (Standard Plan), but it seems that the old plans such as 1.1 and 2.0 will be discontinued by April 29th. https://twittercommunity.com/t/deprecating-the-premium-v1-1-api/191092

To continue using the Twitter API, we'll need to switch to a different plan, such as the Free, Basic, or Enterprise plans. The Free plan appears to be similar to version 2.0 api, and since the authentication methods and endpoints used have changed significantly, we will need to make fundamental modifications to the script mentioned above in order to get it working.

Therefore, it would be difficult to make the necessary modifications within the next couple of days. I'm sorry.

Yes it is definitely an API issue. At my level, I tried to fix it, but unfortunately I couldn't do it. I would like to hope that your efforts in the correction will be much more successful. I want to believe that twint will still work.

@shadowLiar
Copy link

嗨大庆613,
感谢您使用它。它什么时候停止工作的?最近,Twitter API 发生了重大变化,这可能是问题的原因。

感谢您的回复。我发现原来的 API 已经被弃用了。看起来Nitter仍然可以正常工作。我希望这能有所帮助。

您好,但是怎么像twint那样使用呀

@xinmans
Copy link

xinmans commented Jun 20, 2023

root@2a06eacc8a18:# ls
twint
root@2a06eacc8a18:
# twint -u watarubun -o output.json --json���������������������������������CRITICAL:root:twint.run:Twint:Feed:noData'globalObjects'
sleeping for 15 secs
CRITICAL:root:twint.run:Twint:Feed:noData'globalObjects'
sleeping for 15 secs
CRITICAL:root:twint.run:Twint:Feed:noData'globalObjects'
sleeping for 27.0 secs
CRITICAL:root:twint.run:Twint:Feed:noData'globalObjects'
sleeping for 64.0 secs

still failure

@apambalik1337
Copy link

root@2a06eacc8a18:# ls twint root@2a06eacc8a18:# twint -u watarubun -o output.json --json���������������������������������CRITICAL:root:twint.run:Twint:Feed:noData'globalObjects' sleeping for 15 secs CRITICAL:root:twint.run:Twint:Feed:noData'globalObjects' sleeping for 15 secs CRITICAL:root:twint.run:Twint:Feed:noData'globalObjects' sleeping for 27.0 secs CRITICAL:root:twint.run:Twint:Feed:noData'globalObjects' sleeping for 64.0 secs

still failure

same problem here i think its maybe because the twitter api changed

@xinmans
Copy link

xinmans commented Jun 29, 2023

hope have a update

@darkzbaron
Copy link

+1

@IIIIIIIop
Copy link

dear @moxak,thank for your code,but now it doesn't work, I noticed the Nitter you mentioned before, but I am not good at using Nitter. Have you tried it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment