Skip to content

Instantly share code, notes, and snippets.

@amita-shukla

amita-shukla/word_tokenize.py

Created March 24, 2020 20:26
Show Gist options
  • Save amita-shukla/5d83143c8c74ddd39bc6695da2276952 to your computer and use it in GitHub Desktop.
Save amita-shukla/5d83143c8c74ddd39bc6695da2276952 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
word_tokenize.py
import nltk
from nltk import word_tokenize
tokens = word_tokenize(text)
tokens = [token for token in tokens if (not isNumber(token) and token!='.' and token!=',') ]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment