Building off of the https://gist.github.com/seanstory/a08db2e149897da656db3a1ca72e17ac and https://gist.github.com/seanstory/a280a85d067e61bfeb5911bf2654e6e2, this prompt considers the generated indices, then generates a set of test questions and expected results. The result can then be used to evaluate a tool that tries to pick the best index t…

query-generation-prompt.jinja2

You are an expert-level data analyst and a master specialist in search systems. Your mission is to generate a highly realistic and diverse TSV dataset of approximately 20 natural language queries based on the provided Elasticsearch indices.

This dataset's primary purpose is to benchmark an AI model's ability to precisely identify the correct target index, their respective fields and generate accurate search and filter clauses. Your generated queries must therefore include a range of difficulties designed to test the model's understanding of nuance, context, and a user's true intent.

## 1. CONTEXT: INPUT ELASTICSEARCH INDICES
You will receive the index schemas as a JSON object. Note that indices may be from the same domain and share similar field names to create challenging evaluation scenarios.

## 2. PRE-GENERATION REASONING (MANDATORY INTERNAL STEPS)
Before you generate the TSV, you MUST perform the following internal reasoning steps. This analysis is critical for creating relevant and challenging queries.

### A. Domain and Persona Analysis:
* **Examine Metadata:** Carefully analyze the `_meta.description` and field descriptions for each index.
* **Identify Domain:** Determine the business domain for each index (e.g., E-commerce, IT Service Management).
* **Define Personas & Goals:** For each index, define the most likely user persona (e.g., Customer, Support Engineer, Product Manager) and their primary goal (e.g., purchase a product, resolve a technical issue, audit data).

### B. Use Case Brainstorming:
* Based on the personas, brainstorm real-world tasks. What specific questions would they ask to accomplish their goals?
* Example (Customer): "Show me what people think about the battery life on this phone."
* Example (Engineer): "Find all P1 errors from the 'payment-gateway' service in the last hour."

### C. Query Generation Strategy:
* Formulate a plan to generate ~20 queries with a deliberate mix of difficulties: ~40% easy, ~40% medium, and ~20% hard.
* **Field Selection Strategy:**
    * For approximately 60% of 'medium' and 'hard' queries, the search component MUST target a **single, objectively clear** full-text, semantic, or hybrid field. The query's wording should make it unambiguous which single field is the intended search target.
    * For the remaining ~40% of queries, you can use multi-field search, but the choice of fields must still be objectively justifiable from the query's phrasing.
* **Ambiguity Strategy (for Hard Queries):** Strategize how to use subtle wording, persona cues, and intent to create ambiguity between indices. Force the model to rely on the nuanced differences in the index or field-level descriptions to make the correct choice.

## 3. CORE TASK & QUERY GENERATION GUIDELINES
Generate the TSV dataset according to your reasoning and these strict guidelines.

* **CRITICAL RULE: Search-First Principle:** EVERY query MUST have a primary search component. The `search_fields` list in your output MUST NOT be empty, and it MUST only contain fields of a searchable type (e.g., `text`, `semantic_text`). Queries that only filter data are invalid. Identification of a search, filter and display fields for each query should be objectively possible regardiless of the query's difficulty.

* **Define Query Difficulty:** Classify each query as "easy", "medium", or "hard" and provide structured reasoning.
    * **easy:** The query uses terms that directly and uniquely map to a single index and its fields.
    * **medium:** The query requires some inference. There is a plausible "distractor" index or field that the model could incorrectly choose. Your reasoning must identify this distractor.
    * **hard:** The query is unambiguous to a human expert but is an intentional trap for an AI. It requires a deep understanding of persona, intent, or subtle clues in the metadata to resolve the ambiguity between two or more very similar indices/fields.

* **Populate Display Fields:**
    * The `display_fields` column should be populated **only when** the natural language query explicitly asks to see information that is clearly captured in one or more fields (e.g., "show me the **price** and **SKU**...").
    * If the query does not explicitly request specific output fields, the `display_fields` list MUST contain all fields from the `search_fields` and `filter_fields` lists.

## 4. OUTPUT FORMAT
Present your final output as a single TSV (Tab-Separated Values) structure. Do not write any other text or explanation outside of this structure.

### TSV Columns:
* `natural_language_query`: (string) The generated user query.
* `target_index_id`: (string) The `index_id` of the single correct index.
* `search_fields`: (list of strings) The primary field(s) for the search clause. MUST only contain fields of type `text` or `semantic_text`. MUST NOT BE EMPTY.
* `search_fields_values`: (string) A string representation of a Python dictionary. Keys are the field names from 'search_fields', and values are the corresponding search terms extracted from the query.
* `filter_fields`: (string) A string representation of a Python dictionary defining the filter criteria. Keys are the filter fields, and values specify the filter condition (e.g., exact match, range). Can be an empty dictionary string: `{}`.
* `display_fields`: (list of strings) Fields the user explicitly asked to see in the output. Defaults to `search_fields` keys + `filter_fields` keys if not specified in the query.
* `difficulty`: (string) One of: "easy", "medium", or "hard".
* `reasoning`: (string) A multi-line string with structured explanations. You MUST use the following format:
    ```
    Index_reasoning: [Short, objective explanation for choosing this index.]
    search_field_reasoning: [Short, objective explanation for choosing the search field(s) and their corresponding values.]
    Filter_reasoning: [Short, objective explanation for choosing the filter field(s) and their conditions. If empty, state "No filters were specified in the query."]
    ```

{{ Input_Indices }}