Weird Twitter Search behavior

README.md

Description: with a search query containing the word "abdullah" (not for any different word so far), the Twitter Search API stops returning results after a few elements.

You can test this behavior with the attached NodeJS script (edit the config.json file and set your OAuth bearer token there). Try to launch the following command and observe the output:

$ node twitter-search.js
Retrieved 100 elements so far. Going on, last date was Tue Aug 18 23:55:12 +0000 2015 .
Retrieved 200 elements so far. Going on, last date was Tue Aug 18 23:51:16 +0000 2015 .
Retrieved 300 elements so far. Going on, last date was Tue Aug 18 23:47:42 +0000 2015 .
Retrieved 400 elements so far. Going on, last date was Tue Aug 18 23:43:50 +0000 2015 .
Retrieved 500 elements so far. Going on, last date was Tue Aug 18 23:40:12 +0000 2015 .
Retrieved 600 elements so far. Going on, last date was Tue Aug 18 23:36:48 +0000 2015 .
Retrieved 700 elements so far. Going on, last date was Tue Aug 18 23:32:52 +0000 2015 .
Retrieved 800 elements so far. Going on, last date was Tue Aug 18 23:30:08 +0000 2015 .
Retrieved 900 elements so far. Going on, last date was Tue Aug 18 23:27:23 +0000 2015 .
Done with 980 elements. Last date is Tue Aug 18 23:24:58 +0000 2015 .

The search parameters are since:2015-08-12, until:2015-08-19 and query:abdullah (you can change those in config.json).

Why does the next_results element stops at Tue Aug 18 23:24:58 +0000 2015? The same thing happens by lowering the until date, e.g. for until:2015-08-18 only 68 elements are returned:

$ node twitter-abdullah.js 
Done with 68 elements. Last date is Mon Aug 17 23:55:52 +0000 2015 .

This does NOT happen (or at least, I've never found the issue) for different query parameters.

The behavior was noticed when a more complex search query contained the term abdullah in it (as an "OR" term), as removing it made the search "complete" (that is, the since value was reached by the latest returned tweet).

config.json

{
    "bearer": "valid bearer token here",
    "query": "abdullah",
    "since": "2015-08-12",
    "until": "2015-08-19"
}

twitter-search.js

#!/usr/bin/env node

'use strict';

var https = require('https');
var config = require('./config.json');

var count = 0;
var BASE = 'api.twitter.com';
var PATH = '/1.1/search/tweets.json';
var AUTH = 'Bearer ' + config.bearer;
var nextUrl = '?q=' + config.query + '%20since%3A' + config.since + '%20until%3A' + config.until + '&count=100&include_entities=1&result_type=recent';

var makeRequest = function() {
    // make a request
    var req = https.request({
        hostname: BASE,
        path: PATH + nextUrl,
        headers: {
            'Authorization': AUTH
        }
    }, function(res) {
        // read the body
        var body = '';
        res.on('data', function(chunk) {
            body += chunk;
        });
        // when the request ends
        res.on('end', function() {
            var data = JSON.parse(body);
            // update the found elements count
            count += data.statuses.length;
            nextUrl = data.search_metadata.next_results;
            var lastDate = data.statuses[data.statuses.length - 1].created_at;
            // if there is a next url (next_results), make another request
            if (nextUrl) {
                console.log(
                    'Retrieved', count, 'elements so far.',
                    'Going on, last date was', lastDate, '.');
                makeRequest();
            } else {
                console.log('Done with', count, 'elements.',
                'Last date is', lastDate, '.');
                process.exit(0);
            }
        });
    });
    req.end();

    req.on('error', function(e) {
        console.error(e);
    });
};

makeRequest();