Skip to content

Instantly share code, notes, and snippets.

@kylebgorman

kylebgorman/word_tokenize.py

Last active June 18, 2023 05:35
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kylebgorman/d32ce28f938239a4d63302c46d08420c to your computer and use it in GitHub Desktop.
Save kylebgorman/d32ce28f938239a4d63302c46d08420c to your computer and use it in GitHub Desktop.
Download ZIP
Applies NLTK PTB tokenizer to input text
Raw
word_tokenize.py
#!/usr/bin/env python
import fileinput
import nltk
if __name__ == "__main__":
for line in fileinput.input():
print(" ".join(nltk.word_tokenize(line)))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment