Skip to content

Instantly share code, notes, and snippets.

@jphme

jphme/test_embeddings_split.ipynb

Last active January 13, 2024 00:53
Show Gist options
  • Save jphme/598f3a6881b2c31af10ca03350458c5c to your computer and use it in GitHub Desktop.
Save jphme/598f3a6881b2c31af10ca03350458c5c to your computer and use it in GitHub Desktop.
Download ZIP
Proof of Concept Split by Embeddings
Display the source blob
Display the rendered blob
Raw
test_embeddings_split.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@jphme
Copy link
Author

jphme commented Jan 13, 2024

2 random wiki articles copied together, the correct split is found immediately.

Could be improved in various (recursively splitting more granular pieces / smaller windows and parallelize everything) but I think the concept is sound and a good implementation will be very fast and flexible..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment