daha/README.md

## README.md

      
    Raw
  

              README.md
            
          
    Twitter user tweets downloader

This gist is basically a snapshot of my project
twitter-viz on GitHub. My goals
with this project is to:

Play with the twitter API in javascript.
Play a little with jQuery, css-selectors and effects.
Try twitter bootstrap for the layout/formatting of the page.
Write code to download all the tweets for a user.
Extract the code for downloading the tweets into an object.
Create visualizations of the data
Analyze the data in web-workers.

This gist contains the first 5 points.
Play with the twitter API in javascript.

It seems like the REST resources support JSONP. At least the resources
I have tried. Only the the search API explicitly listed the callback
parameter.
Play a little with jQuery, css-selectors and effects.

I tried the getJSON function from jQuery but I did not manage to get
notification about errors. I only tried the error-function. I found
jQuery-JSONP on google code
which did what I wanted it to do.
I first used an class invisible from Twitter bootstrap to hide the
progress bar when it wasn't used. Then I started to use fadeout from
jQuery to hide it after it had been shown. Now I had two ways of
hiding it, the invisible class and the jQuery way with display
none. This caused me some confusion at first, when I wanted to show
the progress bar a second time. I dropped the invisible class, I did
not need both the hiding mechanisms.
When I understood the fadeout used a display none to hide the element I
tried to unhide it with the code below. But that code did not work if
the element did not have a style attribute. I ended up using the
function show() instead, much shorter.
$('#element').removeStyle('display');

Try twitter bootstrap for the layout/formatting of the page.

I did not manage to work with Twitter bootstrap to place my progress
bar and warning note at the position I wanted so I ended up writing
some custom css to get OK placement vertically.
#alert_container {
position:absolute;
top:15px;
}

I first tried to write the div for the alert in html, but that did not
work very well since when the error was closed the element was removed
from the dom and then I could not show another error. Instead I had to
write a new div with the alert and activate it with the alert()
function.
Write code to download all the tweets for a user.

Twitter has a good article about how to work with timelines,
Working with Timelines.
Which I followed. But I ran into some problems. I did get duplicate
entries even though I followed the instructions. When I debugged the
code I found that a big number in javascript plus or minus one is
usually the same big number. The problem is that the tweet ID:s is 64
bit and javascript only support numbers up to 53 bit then they loose
precision. I had previously noticed that the the id field and id_str
did not match. Now I knew why. I ended up using a BigInteger
implementation to handle my max_id and since_id variables to avoid
getting duplicate entries.
I also added some code to store the tweets in localStorage so they do
not have to be downloaded each time the same user is queried. The
localStorage seems to be able to fit 2000-2500 tweets on Chrome. When
they do not fit a exception is thrown, which I did not catch at
first. I did not find the problem until I enabled the debug feature in
Chrome "stop on all unhandled exceptions", the "pause" button in the
bottom in the scripts view in the developer tools.
Extract the code for downloading the tweets into an object.

I found a good article from Douglas Crockford
Private Members in JavaScript
on how to do this. I wanted to have a few public methods accessing
private content and a bunch of private methods, not accessible from
outside the object. For the public methods I ended up with
"privileged" methods as described in the article.

  
## index.html
<!DOCTYPE html>
<html lang="en">
<head>
<link rel="stylesheet" href="http://twitter.github.com/bootstrap/assets/css/bootstrap.css">
<style type="text/css">
#alert_container {
position:absolute;
top:15px;
}

#progressbar {
position:absolute;
top:25px;
}
</style>

<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js" type="text/javascript"></script>
<script src="http://jquery-jsonp.googlecode.com/files/jquery.jsonp-2.3.0.min.js" type="text/javascript"></script>
<script src="https://raw.github.com/jtobey/javascript-bignum/master/biginteger.js" type="text/javascript"></script>
<script src="http://twitter.github.com/bootstrap/assets/js/bootstrap.min.js" type="text/javascript"></script>
<script src="TwitterUserTimeline.js" type="text/javascript"></script>
<script src="twitterclient.js" type="text/javascript"></script>

</head>
<body>
	<form id="userSearch" class="well form-search">
		<div class="control-group">
			<i class="icon-search"></i> <input type="text"
				placeholder="Twitter username" class="input-medium search-query"
				id="twitter_username_query">
		</div>
	</form>
	<div class="row">
		<dl id="user_info" class="dl-horizontal span5"></dl>
		<div class="span5">
			<div id="user_image"></div>
			<div id="user_description"></div>
		</div>
		<div id="tweets" class="span12"></div>
	</div>
	<div id="alert_container"></div>
	<div id="progressbar"
		class="span4 offset4 progress progress-striped active" style="display: none; ">
		<div class="bar" style="width: 0%;"></div>
	</div>
</body>
</html>

## twitterclient.js
// Copyright (c) 2012, David Haglund
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions
// are met:
//
//     * Redistributions of source code must retain the above
//       copyright notice, this list of conditions and the following
//       disclaimer.
//
//     * Redistributions in binary form must reproduce the above
//       copyright notice, this list of conditions and the following
//       disclaimer in the documentation and/or other materials
//       provided with the distribution.
//
//     * Neither the name of the copyright holder nor the names of its
//       contributors may be used to endorse or promote products
//       derived from this software without specific prior written
//       permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
// FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
// COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
// INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
// (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
// SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
// OF THE POSSIBILITY OF SUCH DAMAGE.

/*globals $,document,TwitterUserTimeline */
"use strict";

$(document).ready(function () {
    var userInfoUrl = 'https://api.twitter.com/1/users/show.json?callback=?&include_entities=true&screen_name=',
        twitterUserTimeline = null;

    function hideProgressBar(duration) {
        $('#progressbar').fadeOut(duration, function () {
            $('.bar').width('0%');
        });
    }

    function postError(msg) {
        $('#alert_container').html((' <div class="span4 offset4 alert alert-error">' +
            '  <button class="close" data-dismiss="alert">x</button>' +
            '  <div id="alert_message">' + msg + '</div>' +
            ' </div>'));
        $(".alert").alert();
    }

    function error(d, msg) {
        hideProgressBar('fast');
        postError('Failed to fetch user timeline: ' + msg);
    }

    function clearErrors() {
        $(".alert").alert('close');
    }

    function insertLabeledText(selector, label, text) {
        if (text !== '') {
            selector.append('<dt>' + label + '</dt><dd>' + text + '</dd>');
        }
    }

    function updateProgressBar(percent) {
        $('.bar').width(percent + '%');
        $('#progressbar').show();
    }

    function showTweets(tweets) {
        $.each(tweets, function (i, tweet) {
            $('#tweets').append('<p>' + tweet.text + '</p>');
        });
        return false;
    }

    function complete(tweets) {
        showTweets(tweets);
        hideProgressBar('slow');
    }

    twitterUserTimeline = new TwitterUserTimeline();
    twitterUserTimeline.updateProgress = updateProgressBar;
    twitterUserTimeline.complete = complete;
    twitterUserTimeline.error = error;
    console.log(twitterUserTimeline);

    $('#userSearch').submit(function () {
        var userInfo,
            twitterUsername = $('#twitter_username_query').val().toLowerCase();

        clearErrors();

        $.jsonp({
            url: userInfoUrl + twitterUsername,
            success: function (response) {
                userInfo = $('#user_info');
                userInfo.html('');
                insertLabeledText(userInfo, 'Name', response.name);
                insertLabeledText(userInfo, 'Screen name', '@' + response.screen_name);
                insertLabeledText(userInfo, 'Id', response.id);
                insertLabeledText(userInfo, 'Location', response.location);
                insertLabeledText(userInfo, 'Followers', response.followers_count);
                insertLabeledText(userInfo, 'Following', response.friends_count);
                insertLabeledText(userInfo, 'Favorites', response.favourites_count);
                insertLabeledText(userInfo, 'Tweets', response.statuses_count);
                insertLabeledText(userInfo, 'Created', response.created_at);

                $('#user_image').html('<img src="' + response.profile_image_url + '">');
                $('#user_description').html('<p>' + response.description + '</p>');

                $('#tweets').html('');
                twitterUserTimeline.fetchUserTweets(twitterUsername, response.statuses_count);
            },
            error: function (d, msg) {
                hideProgressBar('fast');
                postError('Failed to find user ' + twitterUsername);
            }
        });
        return false;
    });
});

## TwitterUserTimeline.js
// Copyright (c) 2012, David Haglund
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions
// are met:
//
//     * Redistributions of source code must retain the above
//       copyright notice, this list of conditions and the following
//       disclaimer.
//
//     * Redistributions in binary form must reproduce the above
//       copyright notice, this list of conditions and the following
//       disclaimer in the documentation and/or other materials
//       provided with the distribution.
//
//     * Neither the name of the copyright holder nor the names of its
//       contributors may be used to endorse or promote products
//       derived from this software without specific prior written
//       permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
// FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
// COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
// INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
// (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
// SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
// OF THE POSSIBILITY OF SUCH DAMAGE.

/*globals $,BigInteger,localStorage */
"use strict";

function TwitterUserTimeline() {
    var that = this,
        userTimelineUrlBase = ('https://api.twitter.com/1/statuses/user_timeline.json'
            + '?callback=?'
            + '&count=100'
            + '&trim_user=1'
            + '&include_entities=1'
            + '&include_rts=1'
            + '&contributor_details=1'
            + '&screen_name='),
        requestNumber = 0,
        maxId = BigInteger(0),
        sinceId = BigInteger(0),
        expectedTweetCount = 0;

    this.tweets = [];

    this.complete = function (tweets) {
        // A callback function called when all the available tweets has been downloaded.
    };

    this.updateProgress = function (percent) {
        // A callback function called when the progress is updated.
    };

    this.error = function (d, msg) {
        // A callback function called in case of a error.
    };

    function updateProgress() {
        var percent = Math.floor(that.tweets.length / expectedTweetCount * 100);
        that.updateProgress(percent);
    }

    function getLocalStorageKey(twitterUsername) {
        return 'user=' + twitterUsername.toLowerCase();
    }

    function saveTweetsToLocalStorage(twitterUsername) {
        try {
            localStorage[getLocalStorageKey(twitterUsername)] = JSON.stringify(that.tweets);
        } catch (e) {
            console.log("Local storage is full, failed to store tweets for " + twitterUsername);
        }
    }

    function addMaxIdIfPresent(currentUrl) {
        var result = currentUrl;
        if (!maxId.isZero()) {
            result += '&max_id=' + maxId.prev().toString();
        }
        return result;
    }

    function addSinceIdIfPresent(currentUrl) {
        var result = currentUrl;
        if (!sinceId.isZero()) {
            result += "&since_id=" + sinceId.next().toString();
        }
        return result;
    }

    function makeSuccessFunction(currentRequestNumber, twitterUsername, baseUrl) {
        return function (response) {
            if (currentRequestNumber === requestNumber) {
                if (response.length > 0) { // did get new data
                    if (sinceId.isZero()) { // a response with older tweets
                        that.tweets = that.tweets.concat(response);
                    } else if (maxId.isZero()) { // a response with newer tweets
                        that.tweets = response.concat(that.tweets);
                    } else { // need to sort!
                        that.tweets = that.tweets.concat(response);
                        that.tweets.sort(function (a, b) {
                            return BigInteger(a.id_str).compare(BigInteger(b.id_str));
                        });
                    }
                    updateProgress();
                    maxId = BigInteger(response[response.length - 1].id_str); // The oldest is last
                    fetchUserTimeline(currentRequestNumber, twitterUsername, baseUrl);
                } else {
                    that.updateProgress(100);
                    saveTweetsToLocalStorage(twitterUsername);
                    that.complete(that.tweets);
                }
            } else {
                console.log('Received response from old search!', currentRequestNumber, requestNumber);
            }
        };
    }

    function makeRequest(currentRequestNumber, twitterUsername, baseUrl, requestUrl) {
        $.jsonp({
            url: requestUrl,
            success: makeSuccessFunction(currentRequestNumber, twitterUsername, baseUrl),
            error: that.error
        });
    }

    function fetchUserTimeline(currentRequestNumber, twitterUsername, baseUrl) {
        var currentUrl = baseUrl;

        // https://dev.twitter.com/docs/working-with-timelines
        currentUrl = addMaxIdIfPresent(currentUrl);
        currentUrl = addSinceIdIfPresent(currentUrl);
        makeRequest(currentRequestNumber, twitterUsername, baseUrl, currentUrl);
    }

    this.fetchUserTweets = function (twitterUsername,  tweetCount) {
        var currentRequestNumber,
            localStorageKey = getLocalStorageKey(twitterUsername),
            currentUrl = userTimelineUrlBase + twitterUsername;
        requestNumber += 1;
        currentRequestNumber = requestNumber;
        expectedTweetCount = tweetCount;
        that.tweets = [];
        maxId = BigInteger(0);
        sinceId = BigInteger(0);
        updateProgress();

        if (localStorage[localStorageKey]) {
            that.tweets = JSON.parse(localStorage[localStorageKey]);
            updateProgress();
            console.log('Found ' + that.tweets.length + ' tweets in localStorage!');
            if (that.tweets.length > 0) {
                // set sinceId to only fetch new feeds
                sinceId = BigInteger(that.tweets[0].id_str);
            }
        }

        fetchUserTimeline(currentRequestNumber, twitterUsername, currentUrl);
    };
}
	<!DOCTYPE html>
	<html lang="en">
	<head>
	<link rel="stylesheet" href="http://twitter.github.com/bootstrap/assets/css/bootstrap.css">
	<style type="text/css">
	#alert_container {
	position:absolute;
	top:15px;
	}

	#progressbar {
	position:absolute;
	top:25px;
	}
	</style>

	<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js" type="text/javascript"></script>
	<script src="http://jquery-jsonp.googlecode.com/files/jquery.jsonp-2.3.0.min.js" type="text/javascript"></script>
	<script src="https://raw.github.com/jtobey/javascript-bignum/master/biginteger.js" type="text/javascript"></script>
	<script src="http://twitter.github.com/bootstrap/assets/js/bootstrap.min.js" type="text/javascript"></script>
	<script src="TwitterUserTimeline.js" type="text/javascript"></script>
	<script src="twitterclient.js" type="text/javascript"></script>

	</head>
	<body>
	<form id="userSearch" class="well form-search">
	<div class="control-group">
	<i class="icon-search"></i> <input type="text"
	placeholder="Twitter username" class="input-medium search-query"
	id="twitter_username_query">
	</div>
	</form>
	<div class="row">
	<dl id="user_info" class="dl-horizontal span5"></dl>
	<div class="span5">
	<div id="user_image"></div>
	<div id="user_description"></div>
	</div>
	<div id="tweets" class="span12"></div>
	</div>
	<div id="alert_container"></div>
	<div id="progressbar"
	class="span4 offset4 progress progress-striped active" style="display: none; ">
	<div class="bar" style="width: 0%;"></div>
	</div>
	</body>
	</html>
	// Copyright (c) 2012, David Haglund
	// All rights reserved.
	//
	// Redistribution and use in source and binary forms, with or without
	// modification, are permitted provided that the following conditions
	// are met:
	//
	// * Redistributions of source code must retain the above
	// copyright notice, this list of conditions and the following
	// disclaimer.
	//
	// * Redistributions in binary form must reproduce the above
	// copyright notice, this list of conditions and the following
	// disclaimer in the documentation and/or other materials
	// provided with the distribution.
	//
	// * Neither the name of the copyright holder nor the names of its
	// contributors may be used to endorse or promote products
	// derived from this software without specific prior written
	// permission.
	//
	// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
	// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
	// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
	// FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
	// COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
	// INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
	// (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
	// SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
	// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
	// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
	// OF THE POSSIBILITY OF SUCH DAMAGE.

	/globals $,document,TwitterUserTimeline /
	"use strict";

	$(document).ready(function () {
	var userInfoUrl = 'https://api.twitter.com/1/users/show.json?callback=?&include_entities=true&screen_name=',
	twitterUserTimeline = null;

	function hideProgressBar(duration) {
	$('#progressbar').fadeOut(duration, function () {
	$('.bar').width('0%');
	});
	}

	function postError(msg) {
	$('#alert_container').html((' <div class="span4 offset4 alert alert-error">' +
	' <button class="close" data-dismiss="alert">x</button>' +
	' <div id="alert_message">' + msg + '</div>' +
	' </div>'));
	$(".alert").alert();
	}

	function error(d, msg) {
	hideProgressBar('fast');
	postError('Failed to fetch user timeline: ' + msg);
	}

	function clearErrors() {
	$(".alert").alert('close');
	}

	function insertLabeledText(selector, label, text) {
	if (text !== '') {
	selector.append('<dt>' + label + '</dt><dd>' + text + '</dd>');
	}
	}

	function updateProgressBar(percent) {
	$('.bar').width(percent + '%');
	$('#progressbar').show();
	}

	function showTweets(tweets) {
	$.each(tweets, function (i, tweet) {
	$('#tweets').append('<p>' + tweet.text + '</p>');
	});
	return false;
	}

	function complete(tweets) {
	showTweets(tweets);
	hideProgressBar('slow');
	}

	twitterUserTimeline = new TwitterUserTimeline();
	twitterUserTimeline.updateProgress = updateProgressBar;
	twitterUserTimeline.complete = complete;
	twitterUserTimeline.error = error;
	console.log(twitterUserTimeline);

	$('#userSearch').submit(function () {
	var userInfo,
	twitterUsername = $('#twitter_username_query').val().toLowerCase();

	clearErrors();

	$.jsonp({
	url: userInfoUrl + twitterUsername,
	success: function (response) {
	userInfo = $('#user_info');
	userInfo.html('');
	insertLabeledText(userInfo, 'Name', response.name);
	insertLabeledText(userInfo, 'Screen name', '@' + response.screen_name);
	insertLabeledText(userInfo, 'Id', response.id);
	insertLabeledText(userInfo, 'Location', response.location);
	insertLabeledText(userInfo, 'Followers', response.followers_count);
	insertLabeledText(userInfo, 'Following', response.friends_count);
	insertLabeledText(userInfo, 'Favorites', response.favourites_count);
	insertLabeledText(userInfo, 'Tweets', response.statuses_count);
	insertLabeledText(userInfo, 'Created', response.created_at);

	$('#user_image').html('<img src="' + response.profile_image_url + '">');
	$('#user_description').html('<p>' + response.description + '</p>');

	$('#tweets').html('');
	twitterUserTimeline.fetchUserTweets(twitterUsername, response.statuses_count);
	},
	error: function (d, msg) {
	hideProgressBar('fast');
	postError('Failed to find user ' + twitterUsername);
	}
	});
	return false;
	});
	});