Skip to content

Instantly share code, notes, and snippets.

@amankharwal
Created December 7, 2020 13:05
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save amankharwal/ae6b4e6e1e52c2a666c7d3f739f49905 to your computer and use it in GitHub Desktop.
Save amankharwal/ae6b4e6e1e52c2a666c7d3f739f49905 to your computer and use it in GitHub Desktop.
from nltk.stem.porter import PorterStemmer
porter=PorterStemmer()
def tokenizer(text):
return text.split()
def tokenizer_porter(text):
return [porter.stem(word) for word in text.split()]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment