Skip to content

Instantly share code, notes, and snippets.

@stepney141

stepney141/BookmarkAPI_en.md

Last active April 4, 2023 01:45
Show Gist options
  • Star 74 You must be signed in to star a gist
  • Fork 3 You must be signed in to fork a gist
  • Save stepney141/c161a83f02c42e161c905249733b9225 to your computer and use it in GitHub Desktop.
Save stepney141/c161a83f02c42e161c905249733b9225 to your computer and use it in GitHub Desktop.
Download ZIP
(DEPRECATED) Twitter Undocumented Endpoints for Bookmark
Raw
BookmarkAPI_en.md

Twitter Undocumented Bookmark API (DEPRECATED)

Update: 2022-03-25

Twitter has released the official API v2 endpoint for the bookmark feature. https://twittercommunity.com/t/build-with-bookmarks-on-the-twitter-api-v2/168804/

The following descriptions are or will soon be no longer useful; I suggest using the new official API.

I found out the endpoints for bookmark with Chrome Developer Tools: GET timeline/bookmark, POST bookmark/entries/remove, POST bookmark/entries/remove. The rate limits below are values returned by an official endpoint GET application/rate_limit_status.

This document is still a work in progress because I got stuck in GET timeline/bookmark. Please let me know if you find how to use it.

Notes

  • It is necessary that x-csrf-token in a request header and ct0 in a cookie are the same value. Twitter uses them to avoid CSRF attacks. I recommend that you extract the values from your browsers.
  • All of the endpoints requires OAuth2 Authorizations. Note that they refuse OAuth2 Bearer tokens obtained from POST oauth2/token.
  • You can easily reach the rate limit and get HTTP 429 Error (too many requests), so you should be careful about how many requests you send.
  • I have heard that someone said that "GET timeline/bookmark" returned HTTP 403 Error even though OAuth authentication succeeded. Maybe the endpoint refuses mechanical accesses.
  • In some cases, perhaps it is better to use the official TweetDeck Collection API instead of the undocumented and uncertain API.
  • cf: https://github.com/geekodour/twitmarks/ / https://github.com/acorn/twitter-bookmarks-search (It seems the developers understand how to use the endpoints)
Raw
BookmarkAPI_ja.md

Twitterブックマークの非公開API（調査未完了）

Update: 2022-03-25

Twitter API v2にて、新たにブックマーク操作用の公式APIがリリースされました。 https://twittercommunity.com/t/build-with-bookmarks-on-the-twitter-api-v2/168804/

下記の調査内容は既に使えなくなっているか、または近い将来使えなくなる可能性があります。公式のAPIを使用することをオススメいたします。

Chromeのデベロッパーツールをこねくり回して見つけたエンドポイント（利用方法要調査）

ドキュメントの日本語版は"GET timeline/bookmark"の解析が完了したら書きます。
※リクエスト制限は公式エンドポイントの"GET application/rate_limit_status"で返された値です。

その他情報

  • リクエスト制限を超えて叩くとHTTP 429 Error(Too many requests)を返す模様
  • 「"GET timeline/bookmark"が403 Errorを返してきた」「認証が通った上で弾かれているように見える」という趣旨の情報あり
  • わざわざこれ使わなくてもTweetDeckのcollection機能で代用可能かも（TD以外では使用できないが）
  • cf: https://github.com/geekodour/twitmarks/ （先駆者による解析内容が使われたOSS、要ソース確認）
Raw
EntriesAdd_en.md

POST bookmark/entries/add

Adds the Tweet specified by the tweet_id parameter to the users' bookmarks.

The API requires an OAuth2 Authorization. Note that it refuses Bearer tokens obtained from POST oauth2/token.

Resource URL

https://api.twitter.com/1.1/bookmark/entries/add.json

Resource Information
Response formats JSON
Requires authentication? Yes
Rate limited? Yes
Requests / 15-min window 180

Parameters

Name Required Description Default Value Example
tweet_id required The numerical ID of the Tweet to add to bookmark. 1234567890
tweet_mode optional Valid request values are compat and extended, which give compatibility mode and extended mode, respectively for Tweets that contain over 140 characters. extended

Example Request

POST /1.1/bookmark/entries/remove.json HTTP/1.1
Host: api.twitter.com
x-csrf-token: c4e034 ... ff317c
Authorization: Bearer AAAA%2FAAA%3DAAAAAAAA
cookie: auth_token=2ff2dfe3 ... 57bbd4ca; ct0=c4e034 ... ff317c
Content-Type: application/x-www-form-urlencoded

tweet_id=1234567890&tweet_mode=extended

Note that x-csrf-token in a request header and ct0 in a cookie have to be the same value. Twitter uses them to avoid CSRF attacks. I recommend you extract the values from your browsers.

Example Response

{
    "objects": {},
    "response": {
        "errors": []
    }
}
Raw
EntriesRemove_en.md

POST bookmark/entries/remove

Removes the Tweet specified by the tweet_id parameter from the users' bookmarks.

The API requires an OAuth2 Authorization. Note that it refuses Bearer tokens obtained from POST oauth2/token.

Resource URL

https://api.twitter.com/1.1/bookmark/entries/remove.json

Resource Information
Response formats JSON
Requires authentication? Yes
Rate limited? Yes
Requests / 15-min window 180

Parameters

Name Required Description Default Value Example
tweet_id required The numerical ID of the Tweet to remove from bookmark. 1234567890
tweet_mode optional Valid request values are compat and extended, which give compatibility mode and extended mode, respectively for Tweets that contain over 140 characters. extended

Example Request

POST /1.1/bookmark/entries/remove.json HTTP/1.1
Host: api.twitter.com
x-csrf-token: c4e034 ... ff317c
Authorization: Bearer AAAA%2FAAA%3DAAAAAAAA
cookie: auth_token=2ff2dfe3 ... 57bbd4ca; ct0=c4e034 ... ff317c
Content-Type: application/x-www-form-urlencoded

tweet_id=1234567890&tweet_mode=extended

Note that x-csrf-token in a request header and ct0 in a cookie have to be the same value. Twitter uses them to avoid CSRF attacks. I recommend you extract the values from your browsers.

Example Response

{
    "objects": {},
    "response": {
        "errors": []
    }
}
Raw
TimelineBookmark_en.md

GET timeline/bookmark (WIP)

Resource URL

https://api.twitter.com/2/timeline/bookmark.json

Resource Information
Response formats JSON
Requires authentication? Yes
Rate limited? Yes
Requests / 15-min window 1000

Parameters

Name Required Description Default Value Example

Example Request

Example Response

@jborichevskiy
Copy link

Hi!

I'm trying to extract the data out of https://api.twitter.com/2/timeline/bookmark.json and am making some headway. To start, I'm extracting my cookies and headers from a selenium session elsewhere, and then following along with how the browser makes the requests:

url = "https://api.twitter.com/2/timeline/bookmark.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_composer_source=true&include_ext_alt_text=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweets=true&count=20&ext=mediaStats%2ChighlightedLabel%2CcameraMoment"

(there's a lot of crap happening here that I haven't experimented removing/altering yet -- I just lifted the URL straight from dev tools)

A successful response to this looks like so:

{'globalObjects': {'broadcasts': {},
                   'cards': {},
                   'lists': {},
                   'media': {},
                   'moments': {},
                   'places': {},
                   'topics': {},
                   'tweets': { # our actual tweet dictionaries },
                   'users': { # our user dictionaries },
 'timeline': {'id': 'Custom-xyz',
              'instructions': [
                {
                    [
                        {'addEntries': 
                            
                            ...
                                   # typically the last element
                                    {'content': 
                                        {'operation': 
                                            {'cursor': 
                                                {'cursorType': 'Bottom',
                                                    'stopOnEmptyResponse': True,
                                                    'value': 'our_cursor_value=='
                                                }
                                            }
                                        },
                              'entryId': 'cursor-bottom-xyz',
                              'sortIndex': 'xyz'
                                    }
                                ]
                            }
                        }
                    ]
                }
            ],
              'responseObjects': {'feedbackActions': {}
            }
        }
    }

Within that response we can find the 'value': 'our_cursor_value==' which is used for the next request. I build this request by appending &cursor=our_cursor_value== to the end of our first URL.

This also returns a 200 response with the next 20 bookmarked tweets. This process can be repeated a few more times (extracting the latest cursor and passing it in) until, after 3-6 requests it starts returning a 401 all of sudden:

ipdb> response
<Response [401]>
ipdb> response.json()
{'errors': [{'code': 32, 'message': 'Could not authenticate you.'}]}

My ideas so far for why I'm hitting that 401 status:

  • I'm not sending a valid heartbeat to Twitter and it invalidates the cursor value
    • If I had to guess it would be the POST:client.json request happening every few seconds/minutes.
  • I'm hitting 1000 requests/15 minutes. hard to believe as my requests sent total out to ~5/min. But it's possible it's counting them different internally.
  • Some other thing I'm missing entirely.

If I get anywhere further I'll update here.

@stepney141
Copy link
Author

stepney141 commented Apr 10, 2020 •
edited

@jborichevskiy, Thanks for your comment and awesome information!
I'm confused at the complicated responses of /2/timeline/bookmark.json so I really appreciate your helpful analysis! :-)

Now I'm investigating how /1.1/bookmark/entries/add.json and /1.1/bookmark/entries/remove.json works and trying to resolve some issues.
First, I extracted my requests' data from my browser, and got the following request with chrome devtools:

:authority: api.twitter.com
:method: POST
:path: /1.1/bookmark/entries/add.json
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: XXX
authorization: XXX
cache-control: no-cache
content-length: XXX
content-type: application/x-www-form-urlencoded
cookie: personalization_id="XXX"; guest_id=XXX; _ga=XXX; ads_prefs="XXX"; kdt=XXX; remember_checked_on=1; _twitter_sess=XXX; auth_token=XXX; csrf_same_site_set=1; rweb_optin=side_no_out; csrf_same_site=1; twid=XXX; tfw_exp=0; lang=ja; night_mode=1; des_opt_in=Y; _gid=XXX; external_referer=XXX; ct0=XXX 
origin: https://twitter.com
pragma: no-cache
referer: https://twitter.com/i/bookmarks
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-site
user-agent: XXX
x-csrf-token: XXX
x-twitter-active-user: yes
x-twitter-auth-type: OAuth2Session
x-twitter-client-language: ja

tweet_id=AAAAA&tweet_mode=extended

(I replaced some personal data with "XXX" instead)
In original request AAAAA was a tweet ID integer that every tweet has its own.

Then I tried to POST the same request with Postman, but I failed; it returned {'errors': [{'code': 32, 'message': 'Could not authenticate you.'}]}.

I suppose that's because I sent the wrong OAuth2 token, but I'm not sure.
I'll update here too if I make progress.

@stepney141
Copy link
Author

stepney141 commented Apr 12, 2020 •
edited

I succeeded in requesting POST /1.1/bookmark/entries/add.json and POST /1.1/bookmark/entries/remove.json with Postman.

A successful HTTP request is like this:

POST /1.1/bookmark/entries/add.json HTTP/1.1
Host: api.twitter.com
x-csrf-token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Authorization: Bearer AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%2FAAAAAAAAAAAA
                      AAAAAAAA%3DAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
cookie: auth_token=BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB; ct0=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Content-Type: application/x-www-form-urlencoded

tweet_id=1234567891011121314&tweet_mode=extended

(The tokens and tweet_id in the above are example values)
x-csrf-token and ct0 have the same value.
A request for remove.json also successfully works with the same header values.
Note that we have to extract x-csrf-token, auth_token, ct0, and Bearer token in Authorizationout of browsers each time when we write requests.
I'm planning to develop a browser extension that makes the process easier.

A successful response body is like this:

{
    "objects": {},
    "response": {
        "errors": []
    }
}

Anyway, I finished my work for the add/remove endpoints, so I'll concentrate on /2/timeline/bookmark.json.

@jborichevskiy
Copy link

jborichevskiy commented Apr 13, 2020 •
edited

Glad to hear it was helpful!

As far as 2/timeline/bookmark.json goes, here is the code that was working for me last weekend (where working = successfully looping through 7 times to fetch all my bookmarks). It is no longer working today (second request fails with a 400 and {'errors': [{'code': 214, 'message': 'Bad request.'}]}). Here's my testing Python script:

def pull_bookmarks(headers, cookies):
    url = "https://api.twitter.com/2/timeline/bookmark.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_composer_source=true&include_ext_alt_text=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweets=true&count=20&ext=mediaStats%2ChighlightedLabel%2CcameraMoment"

    def extract_cursor(response):
        try:
            return response.json()['timeline']['instructions'][0]['addEntries']['entries'][-1]['content']['operation']['cursor']['value']
        except KeyError as e: 
            return None

    initial = True
    while True:
        if initial:
            response = api_request(url, headers, cookies)
            for tweet in response.json()['globalObjects']['tweets'].values():
                print(tweet['created_at'], tweet['id_str'], tweet['full_text'][:50])

            initial = False
        else:
            sleep(5)  # Apparently 4 is too fast and 6 is too slow... idk wtf is going on
            
            cursor_value = extract_cursor(response) 
            if cursor_value is None:
                break

            response = api_request(url + f'&cursor={cursor_value}', headers, cookies)
            print(response.status_code)

            if response.status_code != 200:
                break

            for tweet in response.json()['globalObjects']['tweets'].values():
                print(tweet['created_at'], tweet['id_str'], tweet['full_text'][:50])

I highly doubt they changed their API this week, so I suspect the problem is with interaction between my cookies and the cursor. Or earth's magnetic field. Who knows, really.

Hoping to continue poking at this next weekend, so will update if I get it working!

@thadk
Copy link

thadk commented Jul 24, 2020

I think this also supports the bookmarks.json API https://github.com/acorn/twitter-bookmarks-search

@stepney141
Copy link
Author

@thadk I didn't know the extension supports the bookmark api. I'll add it to my documents and try to examine it. Thank you very much!

@thadk
Copy link

thadk commented Jul 25, 2020 •
edited

I did npm install/npm run build the latest version, added it to Chrome to try the latest version on GitHub. Maybe it only gets and searches the latest page of bookmarks, but it could give an idea. It seemed to go back about 3 months through about 500 tweets.

@Kenan7
Copy link

Kenan7 commented May 2, 2021

does this work right now?

@stepney141

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment