Joshua Lochner xenova

## tiktoken-to-hf.ipynb

      
        
          
            
              
              1 file
            
          
          
            
              
              2 forks
            
          
          
            
              
              17 comments
            
          
          
            
              
              13 stars
            
          
        
        
          
              
          
          
            
                xenova
                / tiktoken-to-hf.ipynb
            
            
              Last active
              April 30, 2024 02:04
            
              
                Convert tiktoken tokenizers to the Hugging Face tokenizers format
              
          
        
      
        
  
    
    

          
    
      
      Sorry, something went wrong. Reload?
      Sorry, we cannot display this file.
      Sorry, this file is invalid so it cannot be displayed.
      
          Viewer requires iframe.