Skip to content

Instantly share code, notes, and snippets.

View theNullCrown's full-sized avatar
💤

theNullCrown theNullCrown

💤
  • California Polytechnic State University
  • San Luis Obispo, CA
View GitHub Profile
@theNullCrown
theNullCrown / fastfuzzyfiltering.md
Last active August 10, 2023 19:13
Smarter filtering of location and other strings for vector search

Fast Fuzzy Filtering (F3)

Algorithm for pre-filtering locations and other similar strings during vector search.

The Problem

We built multiple vector databases of World Bank project documents with multiple location, topic and theme metadata fields using Marqo, AWS Kendra, Weaviate, Pinecone. The typical query looks like:

What are some lessons learned from rural water supply projects in Northern Africa?
What are the effects on girls' education due to sanitation projects in Asia and South America?