Skip to content

Instantly share code, notes, and snippets.

@makispl

makispl/tokenization.py

Created April 11, 2020 22:10
Show Gist options
  • Save makispl/eaa6e0883cea7762d5abf129abdd1f89 to your computer and use it in GitHub Desktop.
Save makispl/eaa6e0883cea7762d5abf129abdd1f89 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
tokenization.py
training_set['SMS'] = training_set['SMS'].apply(lambda sms: nltk.word_tokenize(sms))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment