Skip to content

Instantly share code, notes, and snippets.

@mlai-demo

mlai-demo/lemma_covid.py

Created April 18, 2020 17:45
Show Gist options
  • Save mlai-demo/60579d552e45f5ec34a241356f4c42e1 to your computer and use it in GitHub Desktop.
Save mlai-demo/60579d552e45f5ec34a241356f4c42e1 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
lemma_covid.py
with open('all_abstracts_tokens.txt', encoding = "utf-8") as f, open('all_abstracts_lemmas.txt', 'w', encoding = "utf-8") as out_f:
text = f.read()
tokens = word_tokenize(text)
lemma = WordNetLemmatizer()
lemmed = [lemma.lemmatize(word) for word in tokens]
new_lem_text = ' '.join(lemmed)
out_f.write(new_lem_text)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment