A quick overview of the Twitter Search API.

twitter-search-api.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Twitter's Search API\n",
    "\n",
    "Twitter provides an API endpoint to find tweets whose content meets particular criteria. We can use that functionality to build a \"corpus\" of tweets for data analysis or creative re-use.\n",
    "\n",
    "The full documentation for the Search API endpoint is [available on Twitter's developer site](https://dev.twitter.com/rest/reference/get/search/tweets). I'll take you through a few quick examples below using the `node-twitter-api` library for NodeJS.\n",
    "\n",
    "First, we need to load the library and create a Twitter API object. ([See the previous tutorial if you don't have the needed keys, tokens and secrets](https://gist.github.com/aparrish/110612483c2aab5c78a6cbd1e0d6403f).)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "undefined"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var twitterAPI = require('node-twitter-api')\n",
    "var consumerKey = \"...\";\n",
    "var consumerSecret = \"...\";\n",
    "var accessToken = \"...\";\n",
    "var tokenSecret = \"...\";\n",
    "var twitter = new twitterAPI({\n",
    "    consumerKey: consumerKey,\n",
    "    consumerSecret: consumerSecret});"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.search()` method performs the request to the Twitter Search API. The first parameter of the method is an object whose keys correspond to the parameters listed in the \"parameters\" section of the [documentation](https://dev.twitter.com/rest/reference/get/search/tweets). The second and third parameters are the access token and token secret, respectively, and the fourth parameter is a function that will be called when results are returned. Here's an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "undefined"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{ statuses: \n",
      "   [ { created_at: 'Tue Feb 14 20:50:26 +0000 2017',\n",
      "       id: 831606548300517400,\n",
      "       id_str: '831606548300517376',\n",
      "       text: 'RT @robbietilton: When people ask me what @ITP_NYU is... https://t.co/XnuWPssRgJ',\n",
      "       truncated: false,\n",
      "       entities: [Object],\n",
      "       extended_entities: [Object],\n",
      "       metadata: [Object],\n",
      "       source: '<a href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for Android</a>',\n",
      "       in_reply_to_status_id: null,\n",
      "       in_reply_to_status_id_str: null,\n",
      "       in_reply_to_user_id: null,\n",
      "       in_reply_to_user_id_str: null,\n",
      "       in_reply_to_screen_name: null,\n",
      "       user: [Object],\n",
      "       geo: null,\n",
      "       coordinates: null,\n",
      "       place: null,\n",
      "       contributors: null,\n",
      "       retweeted_status: [Object],\n",
      "       is_quote_status: false,\n",
      "       retweet_count: 2,\n",
      "       favorite_count: 0,\n",
      "       favorited: false,\n",
      "       retweeted: false,\n",
      "       possibly_sensitive: false,\n",
      "       lang: 'en' },\n",
      "     { created_at: 'Tue Feb 14 18:54:34 +0000 2017',\n",
      "       id: 831577388525957100,\n",
      "       id_str: '831577388525957120',\n",
      "       text: 'RT @ITP_NYU: Congrats to @ITP_NYU or \"Dellin Betances With Wolves\" for winning the grand prize @nycmedialab @mlbam hackathon in the VR/AR t…',\n",
      "       truncated: false,\n",
      "       entities: [Object],\n",
      "       metadata: [Object],\n",
      "       source: '<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>',\n",
      "       in_reply_to_status_id: null,\n",
      "       in_reply_to_status_id_str: null,\n",
      "       in_reply_to_user_id: null,\n",
      "       in_reply_to_user_id_str: null,\n",
      "       in_reply_to_screen_name: null,\n",
      "       user: [Object],\n",
      "       geo: null,\n",
      "       coordinates: null,\n",
      "       place: null,\n",
      "       contributors: null,\n",
      "       retweeted_status: [Object],\n",
      "       is_quote_status: true,\n",
      "       quoted_status_id: 831507587619242000,\n",
      "       quoted_status_id_str: '831507587619241985',\n",
      "       retweet_count: 1,\n",
      "       favorite_count: 0,\n",
      "       favorited: false,\n",
      "       retweeted: false,\n",
      "       lang: 'en' },\n",
      "     { created_at: 'Tue Feb 14 18:28:58 +0000 2017',\n",
      "       id: 831570947958247400,\n",
      "       id_str: '831570947958247426',\n",
      "       text: 'RT @robbietilton: When people ask me what @ITP_NYU is... https://t.co/XnuWPssRgJ',\n",
      "       truncated: false,\n",
      "       entities: [Object],\n",
      "       extended_entities: [Object],\n",
      "       metadata: [Object],\n",
      "       source: '<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>',\n",
      "       in_reply_to_status_id: null,\n",
      "       in_reply_to_status_id_str: null,\n",
      "       in_reply_to_user_id: null,\n",
      "       in_reply_to_user_id_str: null,\n",
      "       in_reply_to_screen_name: null,\n",
      "       user: [Object],\n",
      "       geo: null,\n",
      "       coordinates: null,\n",
      "       place: null,\n",
      "       contributors: null,\n",
      "       retweeted_status: [Object],\n",
      "       is_quote_status: false,\n",
      "       retweet_count: 2,\n",
      "       favorite_count: 0,\n",
      "       favorited: false,\n",
      "       retweeted: false,\n",
      "       possibly_sensitive: false,\n",
      "       lang: 'en' } ],\n",
      "  search_metadata: \n",
      "   { completed_in: 0.077,\n",
      "     max_id: 831606548300517400,\n",
      "     max_id_str: '831606548300517376',\n",
      "     next_results: '?max_id=831570947958247425&q=nyu%20itp&count=3&include_entities=1&result_type=recent',\n",
      "     query: 'nyu+itp',\n",
      "     refresh_url: '?since_id=831606548300517376&q=nyu%20itp&result_type=recent&include_entities=1',\n",
      "     count: 3,\n",
      "     since_id: 0,\n",
      "     since_id_str: '0' } }\n"
     ]
    }
   ],
   "source": [
    "twitter.search(\n",
    "    {\"q\": \"nyu itp\", \"result_type\": \"recent\", \"count\": 3},\n",
    "    accessToken,\n",
    "    tokenSecret,\n",
    "    function(error, data, response) {\n",
    "        if (error) {\n",
    "            console.log(\"something went wrong: \", error);\n",
    "        }\n",
    "        console.log(data);\n",
    "    }\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The object that comes to the callback function as the second parameter has two top-level keys: `statuses`, which is a list of [tweet objects](https://dev.twitter.com/overview/api/tweets), and `search_metadata`, which is an object that gives metadata about the search (like the URL to retrieve the next page of search results).\n",
    "\n",
    "## Search parameters and query operators\n",
    "\n",
    "The documentation has a full list, but here are a few of the most important query parameters that you can include in the call to `.search()`:\n",
    "\n",
    "* `q`: The search query. At the most basic level, this specifies the string that tweets should match in order to be included in the results. You can include special metacharacters and keywords to perform even more powerful searches; see below.\n",
    "* `result_type`: By default, the Search API will return a mixture of \"popular\" tweets and recent tweets. Passing the value `recent` for this parameter directs the API to only return the most recent results.\n",
    "* `count`: The number of tweets to return in the results, at maximum. Defaults to 15, with a maximum of 100. (If you need even more results than this, you can page through the search results using the `next_results` attribute of the `search_metadata` returned in the API response.)\n",
    "* `since_id`: If specified, only tweets with an `id` greater than this value will be returned. Use this parameter if you're querying the Search API on an interval (e.g., every five minutes) to only return tweets that have been posted since you last checked.\n",
    "\n",
    "The Search API allows you to perform queries more sophisticated than simple keyword matches using query operators. These operators are also supported in Twitter's [search interface for the web](http://twitter.com/search). [You can find a full list of operators here](https://dev.twitter.com/rest/public/search). Query operators allow you to find tweets from a particular user, tweets than mention a particular user, tweets that have attached media, etc. For example, here's a search that finds tweets by [Dan Shiffman](http://twitter.com/shiffman) that contain the word \"amazing\" (and prints out the text of the tweet):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "undefined"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@HaydnGy yes, that would be amazing! Marking dups and other organization tasks would also be very useful. @meiamsome has done some of this!\n",
      "RT @creativewang: @shiffman My animated version, thanks the amazing tutorial. 🙌 https://t.co/wwUoHXTozq\n",
      "@dandersod that's amazing.  (Collector's item T-shirt)\n",
      "Coming soon to @thecodingtrain: guest tutorial by the amazing @emilyxxie! https://t.co/JWqUT0z3Cx\n",
      "RT @Jinjolux: @shiffman First look at my map tile generator with p5.js, it is amazing what you can do with Perlin Noise :D https://t.co/pcm…\n"
     ]
    }
   ],
   "source": [
    "twitter.search(\n",
    "    {\"q\": \"from:shiffman amazing\", \"result_type\": \"recent\", \"count\": 100},\n",
    "    accessToken,\n",
    "    tokenSecret,\n",
    "    function(error, data, response) {\n",
    "        if (error) {\n",
    "            console.log(\"something went wrong: \", error);\n",
    "        }\n",
    "        for (var i = 0; i < data['statuses'].length; i++) {\n",
    "            console.log(data['statuses'][i]['text']);\n",
    "        }\n",
    "    }\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bot example\n",
    "\n",
    "[Here's a simple bot](https://github.com/aparrish/example-twitter-bot-node/blob/master/bots_are/bot.js) that finds instances of the phrase \"bots are.\" It then extracts the remaining portion of the tweet and recombines them into a new tweet."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Javascript (Node.js)",
   "language": "javascript",
   "name": "javascript"
  },
  "language_info": {
   "file_extension": ".js",
   "mimetype": "application/javascript",
   "name": "javascript",
   "version": "4.2.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}