Skip to content

Instantly share code, notes, and snippets.

@himlohiya

himlohiya/stemmer_lemmatize_text.py

Created June 29, 2018 18:42
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save himlohiya/cf99ed03a1ba72f068aa8abcea27c8fb to your computer and use it in GitHub Desktop.
Save himlohiya/cf99ed03a1ba72f068aa8abcea27c8fb to your computer and use it in GitHub Desktop.
Download ZIP
Raw
stemmer_lemmatize_text.py
def simple_stemmer(text):
ps = nltk.porter.PorterStemmer()
text = ' '.join([ps.stem(word) for word in text.split()])
return text
def lemmatize_text(text):
text = nlp(text)
text = ' '.join([word.lemma_ if word.lemma_ != '-PRON-' else word.text for word in text])
return text
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment