Skip to content

Instantly share code, notes, and snippets.

@MattSandy
Last active January 20, 2024 15:57
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save MattSandy/91c28069291b86f9a617ab0f608c9bdc to your computer and use it in GitHub Desktop.
Save MattSandy/91c28069291b86f9a617ab0f608c9bdc to your computer and use it in GitHub Desktop.
Tweet Scraper in Node.JS
var Twitter = require('node-tweet-stream')
, t = new Twitter({
consumer_key: '',
consumer_secret: '',
token: '',
token_secret: ''
});
var watch = [
"SB52",
"SBLII",
"SuperBowl",
"sb52",
"Superbowl",
"SuperBowlLII",
"SuperBowlLll",
"sblii",
"SuperBowl2018",
"superbowl",
"sb2018"];
var fs = require('fs');
var ndjson = require('ndjson');
var tweet_count = 0;
var serialize = ndjson.serialize();
serialize.on('data', function(line) {
fs.appendFile('data.json', line, function (err) {
tweet_count = tweet_count + 1;
console.log(tweet_count)
});
});
t.on('tweet', function (tweet) {
serialize.write(tweet);
});
watch.forEach(function(tag) {
t.track(tag);
console.log(tag);
});
{
"name": "stream-tweets-to-ndjson",
"version": "1.0.0",
"description": "Stream tweets matching an array for hashtags to an ndjson file.",
"main": "app.js",
"dependencies": {
"fs": "^0.0.1-security",
"ndjson": "^1.5.0",
"node-tweet-stream": "^2.0.3"
},
"keywords": [
"ndjson",
"twitter",
"tweets"
],
"author": "Matt Sandy",
"license": "ISC"
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment