Skip to content

Instantly share code, notes, and snippets.

@EnkrateiaLucca

EnkrateiaLucca/tokens.py

Created July 24, 2020 17:25
Show Gist options
  • Save EnkrateiaLucca/74c6583f4c45e2f2f67040d321db9f4d to your computer and use it in GitHub Desktop.
Save EnkrateiaLucca/74c6583f4c45e2f2f67040d321db9f4d to your computer and use it in GitHub Desktop.
Download ZIP
tokenization
Raw
tokens.py
from nltk import word_tokenize
# Loading my article in the article.txt file
with open("article.txt", "r", encoding="utf8") as intro:
text = intro.readlines()
# Selecting a sentence as a string
sentence = text[3]
print(word_tokenize(sentence))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment