Named Entity Recognition with Stanford NER and NLTK

NER_Stanford_NLTK.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Purpose\n",
    "This notebook is a proof of concept to implement named entity recognition on unstructured texts. It intends to depict the named entity recognition model that will bring satisfactory results for identifying persons' names.\n",
    "\n",
    "### Method\n",
    "\n",
    "This notebook uses the Stanford Named Entity Recognition framework (Finkel et al 2005) to recognise persons' names. Three Stanford classifiers were used to assess the accuracy of persons' names recognition: english.all.3class.distsim.crf.ser.gz (3class), english.conll.4class.distsim.crf.ser.gz (4class), and english.muc.7class.distsim.crf.ser.gz (7class).\n",
    "\n",
    "The kind of entities that can be recognised by the models are described below (Stanford Named Entity Recognizer, 2017)\n",
    "\n",
    "|Classifier|Recognisable Entities\n",
    "|--|---------------------|\n",
    "|3class|Location, Person, Organization\n",
    "|4class|Location, Person, Organization, Misc\n",
    "|7class|Location, Person, Organization, Money, Percent, Date, Time\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk import sent_tokenize\n",
    "from nltk import word_tokenize\n",
    "from nltk.tag import StanfordNERTagger\n",
    "from itertools import groupby"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simple test text contains the name of the football player Franz Beckenbauer written in four different ways to assess the capacity of the Stanford classifiers to identify a common name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text = \"\"\"Gary Winston Lineker was an excellent football player.\n",
    "GARY WINSTON LINEKER was a striker.\n",
    "gary winston lineker was born in England.\n",
    "gARY WiNsTon lInEker is married to Danielle Bux.\n",
    "Gary W. Lineker, Kanny Sansom and Peter Shilton played together.\n",
    "The defenders:\n",
    "    Gary Stevens\n",
    "    Kenny Sansom\n",
    "    Terry Butcher\n",
    "The midfields were:\n",
    "    - Bryan Robson;\n",
    "    - Ray Wilkins;\n",
    "    - Chris Waddle.\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Finding the words by tokenising the text into sentences and then the sentences into words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = sent_tokenize(text)\n",
    "tokenized_sentences = [word_tokenize(sentence) for sentence in sentences]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preforming named entity recognition with the **english.all.3class.distsim.crf.ser.gz** classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sn_3class = StanfordNERTagger('/Library/stanford-ner-2017-06-09/classifiers/english.all.3class.distsim.crf.ser.gz',\n",
    "                       path_to_jar='/Library/stanford-ner-2017-06-09/stanford-ner.jar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ne_annot_sent_3c = [sn_3class.tag(sent) for sent in tokenized_sentences]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "persons_sn_3class = []\n",
    "for annot_sent in ne_annot_sent_3c:\n",
    "    for annot_token in annot_sent:\n",
    "        if annot_token[1] == 'PERSON':\n",
    "            persons_sn_3class.append(annot_token[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Performing named entity recognition with the **english.conll.4class.distsim.crf.ser.gz** classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sn_4class = StanfordNERTagger('/Library/stanford-ner-2017-06-09/classifiers/english.conll.4class.distsim.crf.ser.gz',\n",
    "                       path_to_jar='/Library/stanford-ner-2017-06-09/stanford-ner.jar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ne_annot_sent_4c = [sn_4class.tag(sent) for sent in tokenized_sentences]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "persons_sn_4class = []\n",
    "for annot_sent in ne_annot_sent_4c:\n",
    "#     print (annot_sent)\n",
    "    for annot_token in annot_sent:\n",
    "        if annot_token[1] == 'PERSON':\n",
    "            persons_sn_4class.append(annot_token[0])\n",
    "# print(\"Persons' names found:\", persons_sn_4class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Performing named entity recognition with the **english.muc.7class.distsim.crf.ser.gz** classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sn_7class = StanfordNERTagger('/Library/stanford-ner-2017-06-09/classifiers/english.muc.7class.distsim.crf.ser.gz',\n",
    "                       path_to_jar='/Library/stanford-ner-2017-06-09/stanford-ner.jar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ne_annot_sent_7c = [sn_7class.tag(sent) for sent in tokenized_sentences]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "persons_sn_7class = []\n",
    "for annot_sent in ne_annot_sent_7c:\n",
    "    for annot_token in annot_sent:\n",
    "        if annot_token[1] == 'PERSON':\n",
    "            persons_sn_7class.append(annot_token[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identifying Individuals out of Names\n",
    "\n",
    "The Stanford Classifier identifies and tags persons' names, but does not identify individuals. The best approach to overcome this limitation would be to train an IOB (Inside, Outside, Beginning) named entity chunker for the domain of the corpus. This approach requires a large amount of text on the target domain.\n",
    "\n",
    "This proof of concept used a simple solution, which was to consider the continuous occurrence of PERSON entities as an individual. The function *get_individuals* below implements this functionality. However, this solution also has a drawback. The PERSON entities that represent different individuals appear in the text without any character between them will be interpreted as a unique individual. This issue is shown below where the three defenders that appear as a list with no bullets in the test text are considered as just one individual. We can also see that this problem does not happen for the list of midfielders, which uses bullets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_individuals(ne_annot_sent):\n",
    "    individuals = []\n",
    "    for annot_sent in ne_annot_sent:\n",
    "        #print(annot_sent)\n",
    "        for tag, chunk in groupby(annot_sent, lambda x:x[1]):\n",
    "            if (tag == \"PERSON\"):\n",
    "                individuals.append(\" \".join(w for w, t in chunk))                \n",
    "    return individuals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing the Classifiers' Outcomes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Gary', 'Winston', 'Lineker', 'GARY', 'WINSTON', 'LINEKER', 'gARY', 'WiNsTon', 'lInEker', 'Danielle', 'Bux', 'Gary', 'W.', 'Lineker', 'Kanny', 'Sansom', 'Peter', 'Shilton', 'Gary', 'Stevens', 'Kenny', 'Sansom', 'Terry', 'Bryan', 'Robson', 'Ray', 'Wilkins', 'Chris', 'Waddle'] \n",
      "\n",
      " ['Gary Winston Lineker', 'GARY WINSTON LINEKER', 'gARY WiNsTon lInEker', 'Danielle Bux', 'Gary W. Lineker', 'Kanny Sansom', 'Peter Shilton', 'Gary Stevens Kenny Sansom Terry', 'Bryan Robson', 'Ray Wilkins', 'Chris Waddle']\n"
     ]
    }
   ],
   "source": [
    "print(persons_sn_3class, '\\n\\n', get_individuals(ne_annot_sent_3c))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Gary', 'Winston', 'Lineker', 'GARY', 'WINSTON', 'LINEKER', 'Danielle', 'Bux', 'Gary', 'W.', 'Lineker', 'Kanny', 'Sansom', 'Peter', 'Shilton', 'Gary', 'Stevens', 'Kenny', 'Sansom', 'Terry', 'Butcher', 'Bryan', 'Robson', 'Ray', 'Wilkins', 'Chris', 'Waddle'] \n",
      "\n",
      " ['Gary Winston Lineker', 'GARY WINSTON LINEKER', 'Danielle Bux', 'Gary W. Lineker', 'Kanny Sansom', 'Peter Shilton', 'Gary Stevens Kenny Sansom Terry Butcher', 'Bryan Robson', 'Ray Wilkins', 'Chris Waddle']\n"
     ]
    }
   ],
   "source": [
    "print(persons_sn_4class, '\\n\\n', get_individuals(ne_annot_sent_4c))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Gary', 'Winston', 'Lineker', 'GARY', 'WINSTON', 'LINEKER', 'Danielle', 'Bux', 'Gary', 'W.', 'Lineker', 'Kanny', 'Sansom', 'Peter', 'Shilton', 'Gary', 'Stevens', 'Kenny', 'Sansom', 'Terry', 'Butcher', 'Bryan', 'Robson', 'Ray', 'Wilkins', 'Chris', 'Waddle'] \n",
      "\n",
      " ['Gary Winston Lineker', 'WINSTON LINEKER', 'WiNsTon lInEker', 'Gary W. Lineker', 'Kanny Sansom', 'Peter Shilton', 'Gary Stevens Kenny Sansom Terry Butcher', 'Bryan Robson', 'Ray Wilkins', 'Chris Waddle']\n"
     ]
    }
   ],
   "source": [
    "print(persons_sn_4class, '\\n\\n', get_individuals(ne_annot_sent_7c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|Entities on text|3class|4class|7class|\n",
    "|--------------------|---------------------|---------------------|---------------------|\n",
    "|Gary Winston Lineker|Gary Winston Lineker |Gary Winston Lineker |Gary Winston Lineker|\n",
    "|GARY WINSTON LINEKER|GARY WINSTON LINEKER |GARY WINSTON LINEKER |WINSTON LINEKER|\n",
    "|gary winston lineker|NR |NR |NR |\n",
    "|gARY WiNsTon lInEker|gARY WiNsTon lInEker |NR |WiNsTon lInEker\n",
    "|Danielle Bux|Danielle Bux |Danielle Bux |NR |\n",
    "|Gary Lineker|Gary Lineker |Gary Lineker |Gary Lineker|\n",
    "|Kanny Sansom|Kanny Sansom |Kanny Sansom |Kanny Sansom|\n",
    "|Peter Shilton|Peter Shilton |Peter Shilton |Peter Shilton|\n",
    "|Gary Stevens|*Gary Stevens Kenny Sansom Terry* |*Gary Stevens Kenny Sansom Terry Butcher* | *Gary Stevens Kenny Sansom Terry Butcher*|\n",
    "|Kenny Sansom| WG|WG | WG|\n",
    "|Terry Butcher| WG|WG | WG|\n",
    "|Bryan Robson|Bryan Robson |Bryan Robson |Bryan Robson|\n",
    "|Ray Wilkins| Ray Wilkins |Ray Wilkins |Ray Wilkins|\n",
    "|Chris Waddle|Chris Waddle |Chris Waddle | Chris Waddle|\n",
    "\n",
    "The table above presents the persons' names recognised by each classifier. The names identified are displayed under each classifier name. NR stands for Not Recognised, and WG stands for Wrongly Grouped. In this last case, the names were correctly identified but weren't correctly grouped as the name of a particular person.\n",
    "\n",
    "By the results presented above, we can see that the *english.all.3class.distsim.crf.ser.gz* classifier is capable of recognizing the hugest variety of text formattings. Thus, this classifier was used in the tests on real trust deeds.\n",
    "It is worth noting that no classifier has identified names typed entirely in lowercase. This was unexpected since lower casing the entire text is a very usual preprocessing technique."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "The Stanford's Named Entity Recognition (NER) feature functions reasonably good to identifying persons in unstructured texts. A huge drawback of this NER engine is that it does not join the names of a single entity. In this way, it is the duty of the programmer to implement an approach to find the entities out of the persons' names tagged by the engine. The naive approach used in the notebook for demonstration purpose is not good enough for a production environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "* Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. Proceedings of the 43nd Annual Meeting of the Association for Computational Linguistics (ACL 2005), pp. 363-370. http://nlp.stanford.edu/~manning/papers/gibbscrf3.pdf\n",
    "\n",
    "* Stanford Named Entity Recognizer (NER), 2017 https://nlp.stanford.edu/software/CRF-NER.shtml"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}