NLTK Tokenization, Tagging, Chunking, Treebank

nltk_tokenize_tag_chunk.rst

Sentence Tokenization

>>> from nltk import tokenize >>> para = "Hello. My name is Jacob. Today you'll be learning NLTK." >>> sents = tokenize.sent_tokenize(para) >>> sents ['Hello.', 'My name is Jacob.', "Today you'll be learning NLTK."]

Word Tokenization

>>> sent = tokenize.word_tokenize(sents[2]) >>> sent ['Today', 'you', "'ll", 'be', 'learning', 'NLTK', '.']

go to http://text-processing.com/demo/tokenize/ to see how all NLTK word tokenizers work

POS Tagging

>>> from nltk import tag >>> tagged_sent = tag.pos_tag(sent) >>> tagged_sent [('Today', 'NN'), ('you', 'PRP'), ("'ll", 'MD'), ('be', 'VB'), ('learning', 'VBG'), ('NLTK', 'NNP'), ('.', '.')]

see http://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html for a list of treebank POS tags

NE Chunking

>>> from nltk import chunk >>> tree = chunk.ne_chunk(tagged_sent) >>> tree Tree('S', [('Today', 'NN'), ('you', 'PRP'), ("'ll", 'MD'), ('be', 'VB'), ('learning', 'VBG'), Tree('ORGANIZATION', [('NLTK', 'NNP')]), ('.', '.')]) >>> tree.draw()

Treebank Corpus

>>> from nltk.corpus import treebank_chunk >>> treebank_chunk.tagged_sents()[0] [('Pierre', 'NNP'), ('Vinken', 'NNP'), (',', ','), ('61', 'CD'), ('years', 'NNS'), ('old', 'JJ'), (',', ','), ('will', 'MD'), ('join', 'VB'), ('the', 'DT'), ('board', 'NN'), ('as', 'IN'), ('a', 'DT'), ('nonexecutive', 'JJ'), ('director', 'NN'), ('Nov.', 'NNP'), ('29', 'CD'), ('.', '.')] >>> treebank_chunk.chunked_sents()[0] Tree('S', [Tree('NP', [('Pierre', 'NNP'), ('Vinken', 'NNP')]), (',', ','), Tree('NP', [('61', 'CD'), ('years', 'NNS')]), ('old', 'JJ'), (',', ','), ('will', 'MD'), ('join', 'VB'), Tree('NP', [('the', 'DT'), ('board', 'NN')]), ('as', 'IN'), Tree('NP', [('a', 'DT'), ('nonexecutive', 'JJ'), ('director', 'NN'), ('Nov.', 'NNP'), ('29', 'CD')]), ('.', '.')]) >>> treebank_chunk.chunked_sents()[0].draw()