Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
from tokenizers import Tokenizer
# save tokenizer
tokenizer.save("my_tokenizer.json")
# load tokenizer
tokenizer = Tokenizer.from_file("my_tokenizer.json")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.