Skip to content

Instantly share code, notes, and snippets.

@BURG3R5

BURG3R5/tokenization.py

Created June 6, 2022 12:20
Show Gist options
  • Save BURG3R5/0b2c95e0e3f3afac43298547bc291dba to your computer and use it in GitHub Desktop.
Save BURG3R5/0b2c95e0e3f3afac43298547bc291dba to your computer and use it in GitHub Desktop.
Download ZIP
Raw
tokenization.py
# IN: string
for punctuation in "!()-[]{};:, <>./?@#$%^&*_~'\"\\":
string = string.replace(punctuation, " ")
string = string.lower()
tokens = word_tokenize(string)
tokens = [token for token in tokens if token not in stopwords.words()]
# OUT: tokens
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment