Creating a First Look at Your Text Collection.ipynb

{
 "worksheets": [
  {
   "cells": [
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Goal** In this assignment, you'll make a first pass look at your newly adopted text collection similar to the Wolfram Alpha's view.\n\n"
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Title, author, and other metadata**.  First, print out some summary information that gives the background explaining what this collection is and where it comes from:"
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "print 'Title: Descriptions of clinical trial protocols'\nprint 'Author: investigators who have submitted trial protocol information'",
     "prompt_number": 1,
     "outputs": [
      {
       "output_type": "stream",
       "text": "Title: Descriptions of clinical trial protocols\nAuthor: investigators who have submitted trial protocol information\n",
       "stream": "stdout"
      }
     ],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "import nltk\nimport re\nsent_tokenizer=nltk.data.load('tokenizers/punkt/english.pickle')",
     "prompt_number": 2,
     "outputs": [],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**First, load in the file or files below.**  First, take a look at your text.  An easy way to get started is to first read it in, and then run it through the sentence tokenizer to divide it up, even if this division is not fully accurate.  You may have to do a bit of work to figure out which will be the \"opening phrase\" that Wolfram Alpha shows.  Below, write the code to read in the text and split it into sentences, and then print out the **opening phrase**."
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "import codecs\nraw = codecs.open('ct_desc_jason.txt', 'r', 'utf-8').read()\nsents = sent_tokenizer.tokenize(raw)\nprint sents[0]",
     "prompt_number": 3,
     "outputs": [
      {
       "output_type": "stream",
       "text": "OBJECTIVES:  Primary  - To determine the maximum tolerated dose (MTD) of stereotactic body radiotherapy (SBRT) in medically inoperable patients with centrally located stage I non-small cell lung cancer.\n",
       "stream": "stdout"
      }
     ],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Next, tokenize.**  Look at the several dozen sentences to see what kind of tokenization issues you'll have.  Write a regular expression tokenizer, using the nltk.regexp_tokenize() as seen in class, to do a nice job of breaking your text up into words.  You may need to make changes to the regex pattern that is given in the book to make it work well for your text collection. \n\n*Note that this is the key part of the assignment.  How you break up the words will have effects down the line for how you can manipulate your text collection.  You may want to refine this code later.*"
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "pattern = r'''(?x)    # set flag to allow verbose regexps\n        ([A-Za-z]\\.)+      # abbreviations, e.g. U.S.A., i.e.\n      | \\(([A-Za-z]\\.?)+\\)  # abbreviations in parentheses\n      | \\w+(-\\w+)*          # words with optional internal hyphens\n      | \\$?\\d+(\\.\\d+)?%?    # currency and percentages, e.g. $12.40, 82%\n      | \\.\\.\\.              # ellipsis\n      | [\\.,;\"'?():\\-_`]+ # these are separate tokens\n    '''\ntokens = nltk.regexp_tokenize(raw, pattern)",
     "prompt_number": 4,
     "outputs": [],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Compute word counts.** Now compute your frequency distribution using a FreqDist over the words. Let's not do lowercasing or stemming yet.  You can run this over the whole collection together, or sentence by sentence. Write the code for computing the FreqDist below."
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "from string import punctuation\nnormtext = [t for t in tokens if t not in punctuation \n                                 and t.lower() not in nltk.corpus.stopwords.words('english') \n                                 and not re.search(r'''^[\\.,;\"'?():\\-_`]+$''',t)]\nwordfreq = nltk.FreqDist(normtext)",
     "prompt_number": 5,
     "outputs": [],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Creating a table.**\nPython provides an easy way to line columns up in a table.  You can specify a width for a string such as %6s, producing a string that is padded to width 6. It is right-justified by default, but a minus sign in front of it switches it to left-justified, so -3d% means left justify an integer with width 3.  *AND* if you don't know the width in advance, you can make it a variable by using an asterisk rather than a number before the '\\*s%' or the '-\\*d%'.  Check out this example (this is just fyi):"
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "print '%-16s' % 'Info type', '%-16s' % 'Value'\nprint '%-16s' % 'number of words', '%-16d' % 100000\n",
     "prompt_number": 6,
     "outputs": [
      {
       "output_type": "stream",
       "text": "Info type        Value           \nnumber of words  100000          \n",
       "stream": "stdout"
      }
     ],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Word Properties Table** Next there is a table of word properties, which you should compute (skip unique word stems, since we will do stemming in class on Wed).  Make a table that prints out:\n1. number of words\n2. number of unique words\n3. average word length\n4. longest word\n\nYou can make your table look prettier than the example I showed above if you like!\n\nYou can decide for yourself if you want to eliminate punctuation and function words (stop words) or not.  It's your collection!  \n"
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "print '%-25s' % 'Info type', '%s' % 'Value'\nprint '%-25s' % 'number of words', '{0:,}'.format(len(normtext)), 'words'\nprint '%-25s' % 'number of unique words', '{0:,}'.format(len(set(normtext))), 'words'\nprint '%-25s' % 'average word length', '%0.2f characters' % (sum([len(w) for w in normtext]) / float(len(normtext)))\nprint '%-25s' % 'longest word', '%s' % sorted([(w,len(w)) for w in normtext], key=lambda x: x[1], reverse=True)[0][0]",
     "prompt_number": 7,
     "outputs": [
      {
       "output_type": "stream",
       "text": "Info type                 Value\nnumber of words           355,119 words\nnumber of unique words    ",
       "stream": "stdout"
      },
      {
       "output_type": "stream",
       "text": "30,970 words\naverage word length       ",
       "stream": "stdout"
      },
      {
       "output_type": "stream",
       "text": "7.00 characters\nlongest word              ",
       "stream": "stdout"
      },
      {
       "output_type": "stream",
       "text": "pressure-controlled-inverted-ratio-ventilation\n",
       "stream": "stdout"
      }
     ],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Most Frequent Words List.** Next is the most frequent words list.  This table shows the percent of the total as well as the most frequent words, so compute this number as well.  "
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "print ' | '.join(['%s (%0.2f%%)' % (word, cnt / float(len(normtext)) * 100) for word, cnt in wordfreq.items()][:50])",
     "prompt_number": 8,
     "outputs": [
      {
       "output_type": "stream",
       "text": "patients (1.32%) | study (1.27%) | treatment (0.74%) | 1 (0.63%) | 2 (0.57%) | 3 (0.46%) | Patients (0.45%) | receive (0.41%) | disease (0.37%) | days (0.33%) | months (0.33%) | blood (0.32%) | clinical (0.31%) | may (0.30%) | therapy (0.30%) | 6 (0.30%) | weeks (0.30%) | dose (0.29%) | 4 (0.27%) | group (0.27%) | randomized (0.24%) | 5 (0.24%) | patient (0.24%) | also (0.24%) | subjects (0.24%) | day (0.23%) | response (0.23%) | one (0.22%) | time (0.22%) | cancer (0.21%) | two (0.21%) | every (0.21%) | using (0.21%) | used (0.21%) | mg (0.20%) | use (0.20%) | studies (0.20%) | effects (0.20%) | years (0.20%) | risk (0.20%) | data (0.19%) | first (0.19%) | treated (0.19%) | care (0.18%) | trial (0.17%) | IV (0.17%) | determine (0.17%) | drug (0.17%) | daily (0.17%) | levels (0.17%)\n",
       "stream": "stdout"
      }
     ],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Most Frequent Capitalized Words List** We haven't lower-cased the text so you should be able to compute this. Don't worry about whether capitalization comes from proper nouns, start of sentences, or elsewhere. You need to make a different FreqDist to do this one.  Write the code here for the new FreqDist and the List itself.  Show the list here."
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "wordfreq_cap = nltk.FreqDist([n for n in normtext if re.search('^[A-Z]', n)])\n\nprint ' | '.join(['%s (%0.2f%%)' % (word, cnt / float(len(normtext)) * 100) for word, cnt in wordfreq_cap.items()][:50])",
     "prompt_number": 9,
     "outputs": [
      {
       "output_type": "stream",
       "text": "Patients (0.45%) | IV (0.17%) | II (0.14%) | Determine (0.12%) | OBJECTIVES (0.09%) | Study (0.09%) | However (0.08%) | Phase (0.08%) | Treatment (0.08%) | Participants (0.08%) | OUTLINE (0.08%) | Subjects (0.07%) | HIV (0.07%) | Arm (0.07%) | Day (0.06%) | Group (0.06%) | Secondary (0.06%) | III (0.06%) | B (0.06%) | I. (0.05%) | Primary (0.05%) | CT (0.05%) | Compare (0.05%) | C (0.05%) | ACCRUAL (0.05%) | Blood (0.04%) | PROJECTED (0.04%) | Hospital (0.04%) | Health (0.03%) | University (0.03%) | Data (0.03%) | D (0.03%) | Therefore (0.03%) | MRI (0.03%) | Research (0.03%) | Clinical (0.03%) | One (0.03%) | Center (0.03%) | TB (0.03%) | PET (0.03%) | PTSD (0.03%) | Scale (0.03%) | DNA (0.03%) | Medical (0.03%) | Although (0.02%) | MTD (0.02%) | National (0.02%) | Furthermore (0.02%) | United (0.02%) | HCV (0.02%)\n",
       "stream": "stdout"
      }
     ],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Sentence Properties Table** This summarizes number of sentences and average sentence length in words and characters (you decide if you want to include stopwords/punctuation or not).  Print those out in a table here."
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "print '%-25s' % 'Info type', '%s' % 'Value'\nprint '%-25s' % 'number of sentences', '{0:,}'.format(len(sents)), 'sentences'\nprint '%-25s' % 'average sentence length*', '%0.2f characters' % (sum([len(n) for n in normtext]) / float(len(sents)))\nprint '%-25s' % '', '%0.2f words' % (len(normtext) / float(len(sents)))\nprint\nprint '* average number of words and characters do not include punctuation, spaces, or stopwords'",
     "prompt_number": 10,
     "outputs": [
      {
       "output_type": "stream",
       "text": "Info type                 Value\nnumber of sentences       25,278 sentences\naverage sentence length*  ",
       "stream": "stdout"
      },
      {
       "output_type": "stream",
       "text": "98.27 characters\n                          14.05 words\n\n* average number of words and characters do not include punctuation, spaces, or stopwords\n",
       "stream": "stdout"
      }
     ],
     "language": "python",
     "trusted": true,
     "collapsed": false
    }
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "name": "",
  "signature": "sha256:db390eb24a491058e7f1b3e2c84666c2eab193c215aaf94aa6b28db4b3cc3b6e",
  "gist_id": "2d0aa63756f350f0ac1a"
 },
 "nbformat": 3
}