Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
How to scrape Historical Tweet Data from Twitter using Web Scraper Extension
{
"_id":"twitter_feed",
"startUrl":[
"https://twitter.com/search?l=&q=web%20scraping%20since%3A2018-10-01%20until%3A2018-10-05&src=typd&lang=en"
],
"selectors":[
{
"id":"tweet",
"type":"SelectorElementScroll",
"parentSelectors":[
"_root"
],
"selector":"div.tweet",
"multiple":true,
"delay":"2000"
},
{
"id":"handle",
"type":"SelectorText",
"parentSelectors":[
"tweet"
],
"selector":"span.username",
"multiple":false,
"regex":"",
"delay":0
},
{
"id":"name",
"type":"SelectorText",
"parentSelectors":[
"tweet"
],
"selector":"strong.fullname",
"multiple":false,
"regex":"",
"delay":0
},
{
"id":"content",
"type":"SelectorText",
"parentSelectors":[
"tweet"
],
"selector":".tweet-text",
"multiple":false,
"regex":"",
"delay":0
},
{
"id":"replies",
"type":"SelectorText",
"parentSelectors":[
"tweet"
],
"selector":"div.ProfileTweet-action.ProfileTweet-action--reply span.ProfileTweet-actionCountForPresentation",
"multiple":false,
"regex":"",
"delay":0
},
{
"id":"retweets",
"type":"SelectorText",
"parentSelectors":[
"tweet"
],
"selector":"div.ProfileTweet-action.ProfileTweet-action--retweet button.ProfileTweet-actionButton span.ProfileTweet-actionCountForPresentation",
"multiple":false,
"regex":"",
"delay":0
},
{
"id":"favorites",
"type":"SelectorText",
"parentSelectors":[
"tweet"
],
"selector":"div.ProfileTweet-action.ProfileTweet-action--favorite button.ProfileTweet-actionButton span.ProfileTweet-actionCountForPresentation",
"multiple":false,
"regex":"",
"delay":0
},
{
"id":"unix_timestamp",
"type":"SelectorElementAttribute",
"parentSelectors":[
"tweet"
],
"selector":"span._timestamp",
"multiple":false,
"extractAttribute":"data-time-ms",
"delay":0
},
{
"id":"published_date",
"type":"SelectorElementAttribute",
"parentSelectors":[
"tweet"
],
"selector":".time a.tweet-timestamp",
"multiple":false,
"extractAttribute":"title",
"delay":0
},
{
"id":"url",
"type":"SelectorElementAttribute",
"parentSelectors":[
"tweet"
],
"selector":"a.tweet-timestamp",
"multiple":false,
"extractAttribute":"href",
"delay":0
}
]
}
@JagosRadovic

This comment has been minimized.

Copy link

JagosRadovic commented Jul 28, 2019

doesn't work with the new twitter layout, please update! thanks

@adit-negi

This comment has been minimized.

Copy link

adit-negi commented Aug 29, 2019

Could you please update this.

@silviamosquera

This comment has been minimized.

Copy link

silviamosquera commented Sep 15, 2019

it's not working, could you please update? thanks!

@afghanpasha

This comment has been minimized.

Copy link

afghanpasha commented Apr 23, 2020

Hi,

It's not working, please update the code.

Thanks

@lsciutto

This comment has been minimized.

Copy link

lsciutto commented Apr 30, 2020

Hello. I have tried the code and it is not working. A scrape file is not been generated.

@codeBlooded1997

This comment has been minimized.

Copy link

codeBlooded1997 commented May 6, 2020

Hi,

It is not working, please updarte it.

Thanks.

@serselimys

This comment has been minimized.

Copy link

serselimys commented May 25, 2020

Has anybody found an updated version??, if so can you guys please share

@afghanpasha

This comment has been minimized.

Copy link

afghanpasha commented May 25, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.