Skip to content

Instantly share code, notes, and snippets.

@Shriram016

Shriram016/roberta_tokenizer.txt Secret

Created November 2, 2020 05:02
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Shriram016/60bbc46238692bb331edf977a436f5a0 to your computer and use it in GitHub Desktop.
Save Shriram016/60bbc46238692bb331edf977a436f5a0 to your computer and use it in GitHub Desktop.
Download ZIP
Pretrained Roberta tokenizer
Raw
roberta_tokenizer.txt
from transformers import RobertaTokenizer
tokenizer = RobertaTokenizer.from_pretrained('roberta-base',add_prefix_space=True)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment