Skip to content

Instantly share code, notes, and snippets.

@j2labs

j2labs/gist:218198

Created October 25, 2009 19:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save j2labs/218198 to your computer and use it in GitHub Desktop.
Save j2labs/218198 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.rb
>>> import nltk.data
>>> splitter = nltk.data.load('tokenizers/punkt/english.pickle')
>>> splitter.tokenize('I think Washington D.C. is neato')
['I think Washington D.C.', 'is neato']
>>> splitter.tokenize('I think Washington D. C. is neato')
['I think Washington D. C. is neato']
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment