Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Getting the Alexa top 1 million sites directly from the server, unzipping it, parsing the csv and getting each line as an array.
var request = require('request');
var unzip = require('unzip');
var csv2 = require('csv2');
request.get('http://s3.amazonaws.com/alexa-static/top-1m.csv.zip')
.pipe(unzip.Parse())
.on('entry', function (entry) {
entry.pipe(csv2()).on('data', console.log);
})
;
@fiestabonita

This comment has been minimized.

Copy link

fiestabonita commented Jan 31, 2015

Plop this into a terminal to get the aforementioned nodejs script working on Ubuntu:

sudo apt-get install nodejs node-request npm
npm install unzip
npm install csv2

chilts: i'm diggin' it. thank you for sharing this.

@steakknife

This comment has been minimized.

Copy link

steakknife commented Feb 2, 2016

+🍺

@vionemc

This comment has been minimized.

Copy link

vionemc commented Sep 30, 2016

May I ask?
Will it always give the latest data or is it the data from 2013, the time when you make this script?

@ao

This comment has been minimized.

Copy link

ao commented Oct 29, 2016

@vionemc The hosted zip file is automatically updated daily by alexa.

@BastienL

This comment has been minimized.

Copy link

BastienL commented Nov 22, 2016

Unfortunately the file no longer exists.

@ao

This comment has been minimized.

Copy link

ao commented Nov 22, 2016

Alexa has stopped offering this file (top-1m.csv.zip), you can now get the alternative free from Statvoo:
https://siteinfo.statvoo.com/dl/top-1million-sites.csv.zip (Ref: https://siteinfo.statvoo.com/top/sites)

@thomas-dorka

This comment has been minimized.

Copy link

thomas-dorka commented Nov 23, 2016

Interestingly, the download works again today. I contacted the Alexa support and they said that the service is discontinued. I added a follow up question on why the service is available again and for how long. Let's see what happens there. The page that originally referenced the file (https://support.alexa.com/hc/en-us/articles/200461990-Can-I-get-a-list-of-top-sites-from-an-API-) does not link to the file anymore.

@thomas-dorka

This comment has been minimized.

Copy link

thomas-dorka commented Nov 24, 2016

Reply from Alexa:

"The file is temporarily available again, yes. We'll post updates concerning the file to our FAQ and Twitter. We do not have any additional updates to share at this time."

So need to monitor e.g. this: https://twitter.com/Alexa_Support

@fizerkhan

This comment has been minimized.

Copy link

fizerkhan commented Nov 27, 2016

The link http://s3.amazonaws.com/alexa-static/top-1m.csv.zip is working now. But I don't know whether the file is upto-date. Any ideas?

@ao Does the link https://statvoo.com/dl/top-1million-sites.csv.zip from statvoo upto-date? (Or) Is it just a copy from Alaxa csv?

@karan1149

This comment has been minimized.

Copy link

karan1149 commented Dec 1, 2016

Is there a smaller CSV anywhere? Maybe top 50,000 or top 100,000 sites?

@Ayesh

This comment has been minimized.

Copy link

Ayesh commented Dec 5, 2016

CSV file is working again! Nice!
The data is not exactly up to date. I would say about 2 months. I have a site in the current the 67,000 positions today, and is in the lists 78,000s.

@djcornell

This comment has been minimized.

Copy link

djcornell commented Dec 15, 2016

OpenDNS has published a new top 1 million list here: http://s3-us-west-1.amazonaws.com/umbrella-static/index.html While the list is not composed in the same way we hope that it will be useful to some. Read up on the details here: https://blog.opendns.com/2016/12/14/cisco-umbrella-1-million/

@xdanx

This comment has been minimized.

Copy link

xdanx commented Mar 6, 2017

Can confirm http://s3.amazonaws.com/alexa-static/top-1m.csv.zip still available

@titus-shoats

This comment has been minimized.

Copy link

titus-shoats commented Mar 25, 2017

Thanks for this!

@elhardoum

This comment has been minimized.

Copy link

elhardoum commented Mar 29, 2017

@karan1149 head -10000 top-1million-sites.csv will display top 10,000. it will be faster than iterating through all the list.
To find a specific domain cat top-1million-sites.csv | grep github.com.

@linussjo

This comment has been minimized.

Copy link

linussjo commented Apr 12, 2017

@xdanx is that file still updated daily or is it from a specific date?

@alexlehm

This comment has been minimized.

Copy link

alexlehm commented May 7, 2017

the zip file has a timestamp of 5/7/2017 today, so it is at least generated each day.

@doctorhy

This comment has been minimized.

Copy link

doctorhy commented May 21, 2017

I need to generate a long list of URL-like names (>10million). Any idea?

@liesislukas

This comment has been minimized.

Copy link

liesislukas commented Jun 16, 2017

@doctorhy it's definitely not a place to ask it, go to stackoverflow.com and ask there. Not sure why not to google around how to generate a random string using a language of your choice in a first place.

@laravelish

This comment has been minimized.

Copy link

laravelish commented Aug 20, 2017

why do people need those domains ?

@hossamhossny

This comment has been minimized.

Copy link

hossamhossny commented Sep 19, 2017

It is definitely outdated. I have a site in the top million and it is not in the list. I would agree that the file is a few months old, 2-3 perhaps.

@AdityaAnand1

This comment has been minimized.

Copy link

AdityaAnand1 commented Nov 4, 2017

Anybody know of a working solution? I'd prefer a script that I can run on my own to update my database.

@alaa-abdelsamie

This comment has been minimized.

Copy link

alaa-abdelsamie commented Jan 2, 2018

hello
Can you explain to me how I can get this rule by mysql file
thank you.

@saoirs3

This comment has been minimized.

Copy link

saoirs3 commented Feb 3, 2018

You might want to try this alternative from Cisco/OpenDNS

http://s3-us-west-1.amazonaws.com/umbrella-static/index.html

@chilts

This comment has been minimized.

Copy link
Owner Author

chilts commented Apr 4, 2019

It seems that this link http://s3.amazonaws.com/alexa-static/top-1m.csv.zip is still there (2019-04-04), though I agree with many observations above that the list looks stale. I know Alexa does have a delay in the public list, but this looks fairly old to me and much older than 3 months or so.

Why? Because https://www.alexa.com/siteinfo/cssminifier.com (a site I run) tells me it's 95,425 rank in the world, but the spreadsheet today is more like 50,000. I remember that the site was up there at some point but that was a few years ago! So, all in all, I dunno.

And as an update for the code above, I created a dir in /tmp/top100 and did the following and voila, it still works with no changes and no errors:

$ node --version
v8.14.0
$ npm --version
6.9.0
$ npm install csv2 request unzip
+ csv2@0.1.1
+ request@2.88.0
+ unzip@0.1.11
updated 3 packages and audited 366 packages in 1.52s

$ node alexa.ja
[ '1', 'google.com' ]
[ '2', 'youtube.com' ]
[ '3', 'facebook.com' ]
...etc...
[ '999998', 'gaest.com' ]
[ '999999', 'gazetehaberler.com' ]
[ '1000000', 'gehring-group.com' ]

Thanks to everyone else for also suggesting other alternatives that are kept up to date, even if they are slightly different.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.