Skip to content

Instantly share code, notes, and snippets.

@viniciusarruda

viniciusarruda/compare_tokenization.ipynb

Last active July 27, 2023 22:19
Show Gist options
  • Save viniciusarruda/f0903aa56ee0747e13c50c271d4c4120 to your computer and use it in GitHub Desktop.
Save viniciusarruda/f0903aa56ee0747e13c50c271d4c4120 to your computer and use it in GitHub Desktop.
Download ZIP
Compare llama.cpp vs Meta tokenization for LLaMA v1.
Display the source blob
Display the rendered blob
Raw
compare_tokenization.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment