Skip to content

Instantly share code, notes, and snippets.

@megha444
Created October 13, 2020 16:28
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save megha444/5ea336b8201fd8b536d0754b43bc7472 to your computer and use it in GitHub Desktop.
import nltk
from nltk.tokenize import word_tokenize
nltk.download ('punkt')
text = "This is a sample text to show word tokenization"
print(word_tokenize(text))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment