Skip to content

Instantly share code, notes, and snippets.

@amrakm

amrakm/langchain_text_splitter.py

Created April 11, 2023 18:47
Show Gist options
  • Save amrakm/3faf613ff166f1f1af7b822bf2506c14 to your computer and use it in GitHub Desktop.
Save amrakm/3faf613ff166f1f1af7b822bf2506c14 to your computer and use it in GitHub Desktop.
Download ZIP
langchain text splitter
Raw
langchain_text_splitter.py
from langchain.text_splitter import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=20,
length_function=tiktoken_len,
separators=["\n\n", "\n", " ", ""]
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment