Skip to content

Instantly share code, notes, and snippets.

@shubham769

shubham769/tokenization.py Secret

Created April 28, 2020 13:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save shubham769/9b5230b3379087f6a41d097f6142cce8 to your computer and use it in GitHub Desktop.
Save shubham769/9b5230b3379087f6a41d097f6142cce8 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
tokenization.py
import nltk
nltk.download('punkt')
from nltk.tokenize import word_tokenize
text = "hello, how are you?"
tokens = word_tokenize(text)
print(tokens)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment