Skip to content

Instantly share code, notes, and snippets.

@aowongster
Forked from chilts/alexa.js
Last active November 21, 2022 18:08
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save aowongster/a69c84b66c74ca037e7094bed61e48b0 to your computer and use it in GitHub Desktop.
Save aowongster/a69c84b66c74ca037e7094bed61e48b0 to your computer and use it in GitHub Desktop.
Getting the Alexa top 1 million sites directly from the server, unzipping it, parsing the csv and getting each line as an array. Now with majestic
var request = require('request');
var unzip = require('unzip');
var csv2 = require('csv2');
const alexa = 'http://s3.amazonaws.com/alexa-static/top-1m.csv.zip'
const majestic = 'http://downloads.majesticseo.com/majestic_million.csv'
const sources = [majestic];
const zSources = [alexa];
source.forEach(getFile);
zSources.forEach(getZFile);
const getFile = (url) => {
request.get(url)
.on('entry', function (entry) {
entry.pipe(csv2()).on('data', console.log);
});
}
const getZFile = (url) => {
request.get(url)
.pipe(unzip.Parse())
.on('entry', function (entry) {
entry.pipe(csv2()).on('data', console.log);
});
}
@lillois59
Copy link

lillois59 commented Nov 21, 2022

Hi @aowongster , im trying to be better in Cloud bug bounty can you help me to fix this please ?

node dns.js
fs.js:47
} = primordials;
^

ReferenceError: primordials is not defined
at fs.js:47:5
at req_ (/usr/local/lib/node_modules/unzip/node_modules/natives/index.js:143:24)
at Object.req [as require] (/usr/local/lib/node_modules/unzip/node_modules/natives/index.js:55:10)
at Object. (/usr/local/lib/node_modules/unzip/node_modules/graceful-fs/fs.js:1:37)
at Module._compile (node:internal/modules/cjs/loader:1097:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1149:10)
at Module.load (node:internal/modules/cjs/loader:975:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Module.require (node:internal/modules/cjs/loader:999:19)
at require (node:internal/modules/cjs/helpers:102:18)

Node.js v17.1.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment