Skip to content

Instantly share code, notes, and snippets.

Scenario Corpus Size Indexing Time Search Latency Hardware
Academic Search 30M documents 24 hours (8 GPUs) ~50ms (GPU) FAISS
E-Commerce 200M products 10 hours (sharded) ~10ms per shard FAISS
Knowledge Management 500M documents ~1 minute (incremental) ~50ms (GPU) PLAID
Conversational AI 60M queries 2 hours (historical) ~15ms FAISS
Multi-Domain Search 1B pages 48 hours (distributed) ~300ms total Hierarchical
Task Action Required
Data Logging Record task parameters and results.
Insight Generation Prompt LLM to synthesize multi-scale insights.
Insight Validation Apply scoring mechanisms to ensure quality.
Retrieval Method Strength Limitation
Hashmap Indexing Fast and reliable for known tasks Limited adaptability to new tasks.
Vector Similarity Adaptable to novel scenarios May introduce irrelevant insights.
Insight Scale Description Example
General Insights Broad rules for all tasks Always place objects before picking a new one.
Environment Insights Context-specific rules Utensils are often near the sink in kitchens.
Subtask Insights Step-by-step guidance for specific actions Retrieve a knife before selecting the tomato for slicing.
Experience Type Purpose Example
Successful Experiences Identify effective actions and outcomes Completing a pick-and-place task without errors.
Paired Experiences Compare successes with failures for deeper insights Failed to open a fridge; succeeded by locating the handle.
https://ewr-htmldev-n1.mediamath.com/t1/api/v1/atomic_creatives/limit/advertiser.agency.organization=100048?full=atomic_creative%2Cvendor_pixel%2Cvendor_pixel_domain%2Cvendor_domain%2Cvendor%2Ccreative_asset&with=advertiser%2Cconcepts&with=creatives%2Cvendor_pixels%2Cvendor_pixel_domains%2Cvendor_domain%2Cvendor&with=creatives%2Ccreative_assets&sort_by=-updated_on%2Cid&order_by=descending&currentTime=1406927754061&page_limit=50&page_offset=0&q=status%3D%3D1%26file_type!%3Dvideo
<?xml version='1.0' ?>
<result called_on="2014-06-06 16:49:02.020518+00">
<form name="contract" label="Contract">
<field name="created_on" default="now()" label="Created On" readonly="1" type="datetime" />
<field name="dmp_fee_cpm" label="DMP Fee CPM" max="9999999.99" type="decimal" />
<field name="dmp_fee_cpts" label="DMP Fee Cpts" max="9999999.99" type="decimal" />
<field name="dmp_fee_pct" label="DMP Fee %" max="999.99" type="decimal" />
<field name="dmp_fee_unit" label="DMP Fee Unit" type="option">
<options>
[2010-11-08 23:03:21,615][DEBUG][action.index ] [Hamilton Slade] [hb-test][0], node[YYQu5DwHRbORPx7qgvdX2w], [P], s[STARTED]: Failed to execute [index {[hb-test][section][10389984], source[{"_id":10389984,"name":"COPY OF => fla / fan forum","itemType":"section","state":"D","createdOn":"2010-05-24T00:00:00-0400","userDate":"2010-05-24T00:00:00-0400","lastSave":"2010-05-24T00:00:00-0400","lastPublish":"2010-05-24T00:00:00-0400","namespaceId":1,"displayData":"name=COPY%20OF%20%3D%3E%20fla%20%2F%20fan%20forum","appAccountIds":[14],"fields":{"url":"/fan_forum/index.jsp?c_id=fla","short_title":"Fans","id":"fan_forum","is_web_only":"N","pages":{"XMLCascade":"Y","items":[{"rank":0,"viewId":0,"_id":9014108,"name":"Fla Mermaids, Manatees & Billy","itemType":"page","state":"A","createdOn":"2010-03-30T00:00:00-0400","userDate":"2010-03-26T00:00:00-0400","namespaceId":1,"displayData":"name=Fla%20Mermaids%2C%20Manatees%20%26%20Billy","appAccountIds":[14],"fields":{"id":"entertainment","short_title":"Mermaids, M
@diptamay
diptamay / Section
Created November 9, 2010 03:30
Section
{
"_id" : 10389984,
"name" : "COPY OF => fla / fan forum",
"itemType" : "section",
"state" : "D",
"createdOn" : "2010-05-24T00:00:00-0400",
"userDate" : "2010-05-24T00:00:00-0400",
"lastSave" : "2010-05-24T00:00:00-0400",
"lastPublish" : "2010-05-24T00:00:00-0400",
"namespaceId" : 1,
{
"_id" : 10990239,
"name" : "DAM Media: 10990239 8/14/10: A-Rod belts three home runs against Angels",
"itemType" : "video",
"state" : "A",
"userDate" : "2010-08-15T00:00:00-0400",
"appAccountIds" : [ 14,179 ],
"fields" : {
"kicker" : "Yankees: A-Rod",
"blurb" : "8/14/10: A-Rod belts three home runs against Angels",