Skip to content

Instantly share code, notes, and snippets.

@fsndzomga

fsndzomga/nltk-tokenize.py

Created September 7, 2023 11:46
Show Gist options
  • Save fsndzomga/87d0ac6a342e1b3ec68a4e85ed0a5f5d to your computer and use it in GitHub Desktop.
Save fsndzomga/87d0ac6a342e1b3ec68a4e85ed0a5f5d to your computer and use it in GitHub Desktop.
Download ZIP
Tokenisation using NLTK
Raw
nltk-tokenize.py
from nltk.tokenize import word_tokenize
sentence = "Natural Language Processing is fascinating."
tokens = word_tokenize(sentence)
print(tokens)
# Output: ['Natural', 'Language', 'Processing', 'is', 'fascinating', '.']
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment