WIkimedia Hackathon - Bits Pilani Hyderabad Campus

00 - Intro to Jupyter.ipynb

01 - Python Tutorial.ipynb

02 - Intro to pywikibot.ipynb

03 - Wikidata.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Wikidata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wikidata is one of the newer families added to the wikimedia projects. It acts as a central storage for structured data for all the wikimedia projects. It solves 2 major problems that wikimedia projects used to face:\n",
    "\n",
    " - Copied information (If the information changes it has to be manually edited in each website)\n",
    " - Unstructured data (A better method of storing data which can be queried using SQL or similar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pywikibot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "wikidata = pywikibot.Site('wikidata', 'wikidata')\n",
    "wikidata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "testwikidata = pywikibot.Site('test', 'wikidata')\n",
    "testwikidata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Items and Properties"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In wikidata, every page is either an item or a property.\n",
    "\n",
    "**Items** are used to represent all the things in human knowledge: including topics, concepts, and objects. For example; [color (Q1075)](https://www.wikidata.org/wiki/Q1075), [Albert Einstein (Q937)](https://www.wikidata.org/wiki/Q937), [Earth (Q2)](https://www.wikidata.org/wiki/Q2), and [cat (Q146)](https://www.wikidata.org/wiki/Q146) are all considered as items in Wikidata.\n",
    "\n",
    "**Properties** are the things that describe and define a item. Each data bit related to an item is a type of property. Properties are different for different types of items. Examples of properties for [Python (Q28865)](https://www.wikidata.org/wiki/Q28865) are: [license (P275)](https://www.wikidata.org/wiki/Property:P275), [bug tracking system (P1401)](https://www.wikidata.org/wiki/Property:P1401), [official website (P856)](https://www.wikidata.org/wiki/Property:P856), [Stack exchange tag (P1482)](https://www.wikidata.org/wiki/Property:P1482).\n",
    "\n",
    "The wikidata API helps to query data form wikidata using SparkQL to filter properties. So, for example you can find all countries in the world which have a population between 10 million to 300 million with just 1 query. In the earlier category interface, there would have to be a category for this information or you would have to parse evry country's page to find the population using Natural language processing!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise - Contribute to wikidata\n",
    "Find an item on wikidata and edit it to add some additional information. Some tips on finding an item:\n",
    " - Your favourite book or author\n",
    " - Your city or state, or a popular place near your hometown\n",
    " - A software, tool, or programming language that you like using\n",
    " - If you can type in a native language, try translating the label/title of an item"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Examples of bots with wikidata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The wikidata game\n",
    "\n",
    "The wikidata game is an example of a bot which helps users contribute better to wikidata. You can check out the wikidata game at <https://tools.wmflabs.org/wikidata-game>\n",
    "\n",
    "The wikidata game finds possible pages which do not have a certain type of information using the structured queries (For example human items which have no gender) and shows the wikipedia page related to that item. Then the user is expected to identify a specific property of the item (For example male or female for the property gender)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The wikidata resonator\n",
    "\n",
    "Wikidata resonator is a script which pulls data from wikidata and joins all the property data of an item to form a descriptive paragraph about the item. You can check it out at https://tools.wmflabs.org/reasonator/\n",
    "\n",
    "Other than forming a descriptive paragraph of the item, it also groups similar properites like \"Relative\" group, \"External sources\" etc. based on some simple conditions. It also creates a timeline of the item if possible based on the properties that would have a datetime data type. It also generates a QR Code to use for the related wikipedia page and shows related images pulled from the related commons.wikimedia page!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Fetching data from wikidata using pywikibot\n",
    "The first thing we're going to do is figure out how to get data from wikidata using pywikibot. Here, it's not like a generic mediawiki website, where the text of the page is pulled. Here, the data is structured. We use a `ItemPage` class in pywikibot which can handle these items in a better way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "itempage = pywikibot.ItemPage(wikidata, \"Q42\")  # Q42 is Douglas Adams\n",
    "itempage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In wikidata, the `page.text` won't work like other mediawiki websites where it gives a string of the whole content in the page. The data and properties are stored in Python dictionary structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "itempage.get()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to get the data using the title of the page rather than the item ID, we can get the wikidata article associated to a wikipedia page using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "itempage == pywikibot.ItemPage.fromPage(pywikibot.Page(pywikibot.Site('en', 'wikipedia'), 'Douglas Adams'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a closer look at the data items given by an item page. It is a dictionary with the following keys:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "itemdata = itempage.get()\n",
    "itemdata.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Labels, Descriptions and Aliases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**Labels** are the name or title of the wikidata item.\n",
    "\n",
    "**Aliases** are alternate labels for the same item in the same lanugage. For example, \"[Python (Q28865)](<https://www.wikidata.org/wiki/Q28865>)\" The programming language has the alias \"Python language\", \"Python programming language\", \"/usr/bin/python\", etc.\n",
    "\n",
    "**Descriptions** are useful statements which can help distinguish items with similar labels. Wikidata items are unique only by their item ID (Qxx) hence the description helps differentiate behind \"[Python (Q271218)](<https://www.wikidata.org/wiki/Q271218>)\" the genus of reptiles, and \"[Python (Q28865)](<https://www.wikidata.org/wiki/Q28865>)\" the programming language, and \"[Python (Q15728)](<https://www.wikidata.org/wiki/Q15728>)\" the family of missiles!\n",
    "\n",
    "As wikidata does not have a specific code/language (the code we use is \"wikidata\") it has data for all languages. Hence, the same item can have a different label in Engligh, Arabic, or French. So, these fields in the data are dictionaries with the key as the language code and the value as the label in that language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "itemdata['labels']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For convenience, after the `itempage.get()` is called, the data is stored in the page variable also:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "itemdata['labels'] == itempage.labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise - Check whether a given Item has a label in your native language\n",
    "Find the language code for your native language and write a function to check if a given item has a label in that language, and what other aliases it has in that language."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Claims"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Claims** Are other wikidtaa pages that are linked to the given item using properties. The 'claims' are stored as another dictionary with the keys as property IDs (P1003, P1005, P1006, etc.) and the value is a list of objects `pywikibot.page.Claim`.\n",
    "\n",
    "Hence, the claim is the value of all the properties that have been set for the given item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "itemdata['claims']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Similarly, this is available in the page object using:\n",
    "itemdata['claims'] == itempage.claims"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the [P800 (notable work)](<https://www.wikidata.org/wiki/Property:P800>) for the item [Q42 (Douglas Adams)](<https://www.wikidata.org/wiki/Q42>):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "itempage.claims['P800']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are multiple claims for the property \"P800\" and we can ask pywikibot to resolve the claim and fetch the data about the claim:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "itempage.claims['P800'][0].getTarget()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we notice that the claim for the first \"notable work (P800)\" of \"Douglas Adams (Q42)\" is the item Q25169. As this is another `ItemPage` we can fetch the english label for this item by doing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "p800_claim_target = itempage.claims['P800'][0].getTarget()\n",
    "p800_claim_target.get()\n",
    "p800_claim_target.labels['en']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, finally we were able to find one of the most notable work of Douglas Adams using the wikidata API exposed by pywikibot. Imagine doing the same in the english wikipedia !\n",
    "\n",
    "**Thought exercise**: How would you figure out the most notable work of the author using the chunk of text given by an English wikipedia Page ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise - Check whether item is in India\n",
    "Given a item ID, check whether the Item is in India by checking the value of the \"country\" property of the item. Write a function that checks this.\n",
    "\n",
    "Hence, when the function is run on Q987 ([New Delhi](<https://www.wikidata.org/wiki/Q987>)), it should give `True` but on Q62 ([San Francisco](<https://www.wikidata.org/wiki/Q62>)) it should give `False`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Property Pages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, it is important to be able to fetch data about the property we find itself. For example, if we want to list the english label of the property and the value in a tabular form.\n",
    "\n",
    "To do this, we use a `PropertyPage` object to deal with properties:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "propertypage = pywikibot.PropertyPage(wikidata, 'P512')\n",
    "propertypage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the `PropertyPage`, we again can access the data similar to how it was accessed in the `ItemPage`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "propertypage.get()\n",
    "propertypage.labels['en']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Wikidata data types\n",
    "On Wikidata, we've already seen ItemPages and PropertyPages. But sometimes, a Claim's value need not be another Itempage, and can be some other data type like text, number, datetime, etc. Pywikibot provides a class for each of these data-types for easier accesibility to the value and resolve the claim.\n",
    "\n",
    "The Data types available in wikidata can be seen at: https://www.wikidata.org/wiki/Special:ListDatatypes\n",
    "\n",
    "The wikidata datatypes provided and the corresponding name of the wikidata data-type are:\n",
    " - Item - `pywikibot.page.ItemPage` - Link to other items at the project.\n",
    " - Property - `pywikibot.page.PropertyPage` - Link to properties at the project.\n",
    " - Global Coordinate - `pywikibot.Coordinate` - Literal data for a geographical position given as a latitude-longitude pair in gms or decimal degrees.\n",
    " - Time - `pywikibot.WbTime` - Literal data field for a point in time.\n",
    " - Quantity - `pywikibot.WbQuantity` - Literal data field for a quantity that relates to some kind of well-defined unit.\n",
    " - Monolingual Text - `pywikibot.MonoLingualText` - Literal data field for a string that is not translated into other languages.\n",
    "\n",
    "#### Data types mapping to str\n",
    "Some types of wikidata have are made specially to show them using a different method for example, showing it as a link, etc. But they all map to the python `str`. They are:\n",
    " - String - `str` - Literal data field for a string of glyphs. Generally do not depend on language of reader.\n",
    " - URL - `str` - Literal data field for a URL.\n",
    " - External Identifier - `str` - Literal data field for an external identifier. External identifiers may automatically be linked to an authoritative resource for display.\n",
    " - Mathematical formula - `str` - Literal data field for mathematical expressions, formula, equations and such, expressed in a variant of LaTeX."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Item\n",
    "item = pywikibot.ItemPage(wikidata, \"Q42\").get()['claims']['P31'][0].getTarget()\n",
    "print(\"Type:\", type(item))\n",
    "print(\"Instance of Douglas Adams:\", item, '(', item.get()['labels']['en'], ')')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Property\n",
    "_property = pywikibot.PropertyPage(wikidata, \"Property:P31\")\n",
    "_property.get()\n",
    "print(\"Type:\", type(_property))\n",
    "print(\"Property 'instance of':\", _property, '(', _property.labels['en'], ')')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Global Coordinate\n",
    "coord = pywikibot.ItemPage(wikidata, \"Q668\").get()['claims']['P625'][0].getTarget()\n",
    "print(\"Type:\", type(coord))\n",
    "print(\"Coordinate location of India:\", coord)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Time\n",
    "_time = pywikibot.ItemPage(wikidata, \"Q28865\").get()['claims']['P571'][0].getTarget()\n",
    "print(\"Type:\", type(_time))\n",
    "print(\"Inception of Python (programming language):\", _time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Quantity\n",
    "qty = pywikibot.ItemPage(wikidata, \"Q668\").get()['claims']['P1082'][0].getTarget()\n",
    "print(\"Type:\", type(qty))\n",
    "print(\"Population in India:\", qty)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Monolingual text\n",
    "monolingual_text = pywikibot.ItemPage(wikidata, \"Q42\").get()['claims']['P1477'][0].getTarget()\n",
    "print(\"Type:\", type(monolingual_text))\n",
    "print(\"Birth name of Douglas Adams:\", monolingual_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# String\n",
    "_string = pywikibot.ItemPage(wikidata, \"Q28865\").get()['claims']['P348'][0].getTarget()\n",
    "print(\"Type:\", type(_string))\n",
    "print(\"Version of Python:\", _string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Mathematical Formula\n",
    "formula = pywikibot.ItemPage(wikidata, \"Q11518\").get()['claims']['P2534'][0].getTarget()\n",
    "print(\"Type:\", type(formula))\n",
    "print(\"Formula of Pythagorean theorem:\", formula)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise - Find URL and External identifier\n",
    "Using the API, find the type and value of the [Official website (P856)](<https://www.wikidata.org/wiki/Property:P856>) and the [Freebase identifier (P646)](<https://www.wikidata.org/wiki/Property:P646>) of [Python (Q28865)](<https://www.wikidata.org/wiki/Q28865>)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Adding more meaning to Properties"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Frequently, a property value may require additional data. For example, consider the property [educated at (P69)](<https://www.wikidata.org/wiki/Property:P69>) in the earlier data fetched from [Douglas Adams (Q42)](<https://www.wikidata.org/wiki/Q42>). It can be seen on WikiData that \"St John's College\" is mentioned as one of his education schools from \"[start time](<https://www.wikidata.org/wiki/Property:P580>)\" of 1971 to \"[end time](<https://www.wikidata.org/wiki/Property:P582>)\" of 1974. And it also says his \"[academic major](<https://www.wikidata.org/wiki/Property:P812>)\" is English literature and \"[academic degree](<https://www.wikidata.org/wiki/Property:P512>)\" was Bachelor of Arts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Qualifiers\n",
    "All this information would not be found if Wikidata was restricted to a (property, value) storage structure. Hence, Wikidata also allows **Qualifiers**. Qualifiers expand on the dta provided by a (property, value) pair by giving it context. Qualifiers also consist of a (property, value) !\n",
    "\n",
    "In the above example, the properties which are being used as qualifiers are:\n",
    " - start time - P580 - https://www.wikidata.org/wiki/Property:P580\n",
    " - end time - P582 - https://www.wikidata.org/wiki/Property:P580\n",
    " - academic major - P812 - https://www.wikidata.org/wiki/Property:P812\n",
    " - academic degree - P512 - https://www.wikidata.org/wiki/Property:P512"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "itempage.claims['P69']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "itempage.claims['P69'][0].getTarget().get()\n",
    "itempage.claims['P69'][0].getTarget().labels['en']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "itempage.claims['P69'][0].qualifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The qualifiers are again claims, as they are similar to the (property, value) pair for item pages. Let us see what the value of the qualifier is by resolving the claim:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Fetch the label of the P512 (academic degree) property\n",
    "claim = itempage.claims['P69'][0]\n",
    "claim.qualifiers['P512'][0].getTarget().get()\n",
    "claim.qualifiers['P512'][0].getTarget().labels['en']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some qualifiers may have a value which is not another item, for example the \"start date\" qualifier. In such cases, we need to check the type of the item:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "claim = itempage.claims['P69'][0]\n",
    "claim.qualifiers['P580'][0].getTarget()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, `WBTime` is a pywikibot class which handles the format of WikiBase Time. Wiki base is the underlying technology that powers the structured editing and so on of Wikidata."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other functions to modify qualifiers are:\n",
    " - `claim.removeQualifier()`\n",
    " - `claim.addQualifier()`\n",
    " - `claim.has_qualifier()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise - Find time studied at school\n",
    "In the case of [Douglas Adams](https://www.wikidata.org/wiki/Q42) as we saw, there are 2 schools mentioned. Using the [start time](https://www.wikidata.org/wiki/Property:P580) and [end time](https://www.wikidata.org/wiki/Property:P580), find the number of years that he studdied in any school and print it out in the format:\n",
    "\n",
    "    <school name>: <start year> to <end year> => <n> years"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reference (Sources)\n",
    "Other than the qualifier, we would also want the source of the data. Hence, the **reference** or **source** field helps in adding, removing, editing these:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "itempage.claims['P69'][0].getSources()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, the source is a (property, value) where the property describes what type of source it is. Some popular properties are:\n",
    " - Imported from - https://www.wikidata.org/wiki/Property:P143 - Used when the data is being imported from another wikimedia website.\n",
    " - stated in - https://www.wikidata.org/wiki/Property:P248 - Used when refereing to a book or similar.\n",
    " - reference URL - https://www.wikidata.org/wiki/Property:P854 - Used when linking to a blog or similar\n",
    " \n",
    "A source is again a list of (property, value) tuples. It can have additional properties like: \"original language of work\", \"publisher\", \"author\", \"title\", \"retrieved\", etc. if necessary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us take a look at a source here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "source = itempage.claims['P69'][0].getSources()[0]\n",
    "source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Get the value of the first tuple in the source:\n",
    "source['P248']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "source['P248'][0].getTarget().get()['labels']['en']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise - Check number of values that have a source as English Wikipedia\n",
    "A large number of data in wikidata was pulled from the \"English Wikipedia\". Go through all (property, value) pairs and check how many of them were taken from the [English Wikipedia (Q328)](https://www.wikidata.org/wiki/Q328)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Search wikidata using python\n",
    "The Search on wikidata can be triggered using the python api too. To do this, we use the `pywikibot.data.api.Request` class which provides with helper functions to query the wikidata website and fetch results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "india_search = pywikibot.data.api.Request(\n",
    "    site=wikidata,\n",
    "    parameters={\"action\": \"wbsearchentities\",\n",
    "                \"format\": \"json\",\n",
    "                \"type\": \"item\",\n",
    "                \"language\": \"en\",\n",
    "                \"search\": \"India\"})\n",
    "\n",
    "india_search.submit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, this function simply returns the dictionary search results directly. This needs to be parsed to be more useful. By modifying the `parameters` we can also search for the item using other languages and types."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Create an ItemPage for every search result\n",
    "The search result given by the API is a python dictionary. But it has a lot of data which may not be very useful to us.\n",
    " 1. Loop over every search item and create a `ItemPage` object and store these in a list.\n",
    " 1. Finally, loop over the ItemPage list and print the english label for each item in the search using the ItemPage class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8. Further Reading\n",
    "\n",
    "To read more ways of using pywikibot to access wikidata, go to https://www.wikidata.org/wiki/Wikidata:Pywikibot_-_Python_3_Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SparQL queries are SQL like queries that can be run on Wikidata to fetch data from it. To try out SparkQL queries and visualize the data using nice plots, you can use <https://query.wikidata.org>. It has a lot of example SparQL queries which can be useful to learn SparQL."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}

04 - SparQL as a Wikidata pywikibot generator.ipynb

05 - Getting census data.ipynb