Danil Nikiforov danilNil

## README.md

      
              2 files
            
          
              0 forks
            
          
              3 comments
            
          
              12 stars
            
          
                izikeros
                / README.md
            
            
              Last active
              March 26, 2024 18:11
            
              
                [split text fixed tokens] Split text into parts with limited length in tokens #llm #tokens #python
              
          
    Text Splitter


A Python script for splitting text into parts with controlled (limited) length in tokens. This script utilizes the tiktoken library for encoding and decoding text.
Table of Contents


Introduction
Installation