Skip to content

Instantly share code, notes, and snippets.

@mhawksey
Created February 1, 2012 15:56
Show Gist options
  • Save mhawksey/1717690 to your computer and use it in GitHub Desktop.
Save mhawksey/1717690 to your computer and use it in GitHub Desktop.
TAGS hack to get tweets from Topsy's Otter API
// this function was inspired by a suggestion by @CoBPEZ last year. Originally coded for TAGSv2 I've dusted it off
// for something I was doing with TAGSv3 http://mashe.hawksey.info/2012/01/twitter-archive-tagsv3/
// extra column headings you can use with this are topsy_trackback_url score hits trackback_total
function getTopsyTweets(searchTerm, maxResults, maxtime, languageCode) {
// maxtime is a unix-timestamp format
//Based on Mikael Thuneberg getTweets - mod by mhawksey to convert to json
// if you include setRowsData this can be used to output chosen entries
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sumSheet = ss.getSheetByName("Readme/Settings");
var REFERER = sumSheet.getRange("B9").getValue();
try {
var data =[];
var idx = 0;
var last_offset = 0;
var page = 1;
var max_page = 10
if (typeof maxResults == "undefined") {
maxResults = 100;
}
if (typeof maxtime == "undefined") {
maxtime = Math.round(new Date().getTime()/1000.0)+"";
}
if (maxResults > 100) {
resultsPerPage = 100;
} else {
resultsPerPage = maxResults;
}
searchTerm = encodeURIComponent(searchTerm);
while (page <= max_page) {
Logger.log(page);
var URL = "http://otter.topsy.com/search.json"
URL = URL + "?q=" + searchTerm;
URL = URL + "&maxtime=" +maxtime;
URL = URL + "&nohidden=0";
URL = URL + "&perpage=" + resultsPerPage;
URL = URL + "&offset=" + last_offset;
URL = URL + "&page=" + page;
URL = URL + "&order=-date";
URL = URL + "&type=tweet";
if (typeof languageCode != "undefined") {
if (languageCode.length > 0) {
URL = URL + "&lang=" + languageCode;
}
}
var response = UrlFetchApp.fetch(URL, {method:'get', headers: { "User-Agent": REFERER}});
if (response.getResponseCode() == 200) {
var parsed_response = Utilities.jsonParse(response.getContentText()).response;
last_offset = parsed_response.last_offset;
var total_results = parsed_response.total;
if (total_results < maxResults) {
maxResults = total_results;
}
var objects = parsed_response.list;
if (!objects.length){
return data;
}
for (i in objects){
objects[i]["id_str"] = objects[i].trackback_permalink.match(/(\d)+$/)[0];
objects[i]["status_url"] = objects[i].trackback_permalink;
objects[i]["from_user"] = objects[i].trackback_author_nick;
objects[i]["from_user_id"] = objects[i].trackback_author_nick;
objects[i]["from_user_id_str"] = objects[i].trackback_author_nick;
objects[i]["text"] = objects[i].content;
objects[i]["created_at"] = new Date(objects[i].trackback_date*1000);
objects[i]["profile_image_url"] = objects[i].topsy_author_img;
data[idx] = objects[i];
idx++;
}
}
page++;
}
return data;
} catch (e) {
Browser.msgBox(e.message);
return data;
}
}
@reesd
Copy link

reesd commented Feb 13, 2012

First of all, thank you for all the great work you have done on the TAGS stuff and in general. Its a treasure trove of information on twitter mining.

I don't suppose you have a copy of the modified TAGS3 spreadsheet around that is using getTopsyTweets do you? I've got a few questions about how to best use this that a live example could probably answer for me like:

How do you suggest using this getTopsyTweets method with TAGS3? Would you just replace the getTweets call with this?

Is there some way they are automatically combined that I am missing?

Also, do I need to manually add the different topsy columns?

Thanks,
dave

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment