Skip to content

Instantly share code, notes, and snippets.

@Norod

Norod/tokenizer-hebrew-wiki-from-gpt-j.ipynb

Created June 4, 2023 18:15
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Norod/f77dd37666e4e3e5773324b1e6c28a0b to your computer and use it in GitHub Desktop.
Save Norod/f77dd37666e4e3e5773324b1e6c28a0b to your computer and use it in GitHub Desktop.
Download ZIP
Train a hebrew tokenizer based on GPT-J using Hebrew WiKi as dataset
Display the source blob
Display the rendered blob
Raw
tokenizer-hebrew-wiki-from-gpt-j.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment