Skip to content

Instantly share code, notes, and snippets.

@gamingflexer

gamingflexer/main.py

Created July 19, 2023 09:02
Show Gist options
  • Star 10 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save gamingflexer/3364999976db4f8ba8df7829d7dfe384 to your computer and use it in GitHub Desktop.
Save gamingflexer/3364999976db4f8ba8df7829d7dfe384 to your computer and use it in GitHub Desktop.
Download ZIP
Anthropic's tokenizer for Claude
Raw
main.py
from transformers import PreTrainedTokenizerFast
fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="/home/ubuntu/LLM/module/claude-v1-tokenization.json")
text = "Hello, this is a test input."
tokens = fast_tokenizer.tokenize(text)
tokens
@danikhan632
Copy link

wget https://public-json-tokenization-0d8763e8-0d7e-441b-a1e2-1c73b8e79dc3.storage.googleapis.com/claude-v1-tokenization.json

this is for downloading the tokenizer

@ayansengupta17
Copy link

Could you provide the link for Claude 3?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment