Conceptual example around a vector store, here Chroma

vsp-chroma.py

from pprint import pprint as pp

import chromadb

client = chromadb.Client()
collection = client.create_collection("all-my-documents")

# "document" is Chroma terminology. In VectorDB and others it is "chunk"
#   And when querying with Chroma, response "units" are "documents". So better
#   chop them finely to get good granularity, or not at all to get whole docs.
documents = [
    "Whatever this is it should at some point include the word document, and nothing else perhaps.",
    "This is document1",
    "This is document2",
    "This is not",
    "Not at all",
    "Not even remotely",
]

collection.add(
    documents=documents,
    ids=[f"d{i}" for i in range(len(documents))],
)

results = collection.query(
    query_texts=["This is a query document, and nothing else"],
    n_results=2,
)

pp(results)

Typical run:

{'data': None,
 'distances': [[1.064234972000122, 1.1793456077575684]],
 'documents': [['This is document1',
                'Whatever this is it should at some point include the word '
                'document, and nothing else perhaps.']],
 'embeddings': None,
 'ids': [['d1', 'd0']],
 'metadatas': [[None, None]],
 'uris': None}

Because of the query, tailored for the "documents", we can see what is going to be retrieved, here a glorified direct word matching.

@ic a few questions
if you do n_resultts=3 , do you get "This is document2" in the results?
what about n_results=4, do you get only 3 results, or also some noise output?

You also the mention the need for training in our call on Friday, but it looks like working out of the box without training here, is that correct?

Thank you, great questions.

if you do n_resultts=3 , do you get "This is document2" in the results?
Yes, consistently.

what about n_results=4, do you get only 3 results, or also some noise output?
We get some "noise". In fact it is about how k-neighbour (approximate) algorithms work. We ask then for the 4 closest neighbours, and it complies:

[
    "This is document1",
    "Whatever this is it should at some point include the word document, and nothing else perhaps.",
    "This is document2",
    "This is not"
  ]

You also the mention the need for training in our call on Friday, but it looks like working out of the box without training here, is that correct?
The example here works on plain English, which is the training dataset, basically. But if we work on say URDF, we're really "out of distribution" and need specific training for generating the "URDF language".