Skip to content

Instantly share code, notes, and snippets.

@CyrilRJK

CyrilRJK/artm_tokenizer.py Secret

Created June 7, 2022 14:45
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save CyrilRJK/68b7b81f505c06fc8b3ecc8e346d48d2 to your computer and use it in GitHub Desktop.
Save CyrilRJK/68b7b81f505c06fc8b3ecc8e346d48d2 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
artm_tokenizer.py
tokenizer = cv.build_tokenizer()
tokenized_docs = [tokenizer(doc) for doc in df['processed docs']]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment