Skip to content

Instantly share code, notes, and snippets.

@virattt

virattt/private_few-shot-query-rewriting.ipynb

Last active February 28, 2024 22:26
Show Gist options
  • Save virattt/30f056e9bf2d5f74f701bc7d67ad7cff to your computer and use it in GitHub Desktop.
Save virattt/30f056e9bf2d5f74f701bc7d67ad7cff to your computer and use it in GitHub Desktop.
Download ZIP
few-shot-query-rewriting.ipynb
Display the source blob
Display the rendered blob
Raw
private_few-shot-query-rewriting.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"gpuType": "T4",
"name": "few-shot-query-rewriting.ipynb",
"authorship_tag": "ABX9TyNpQY4CQUCB3Ok6C5n+pZSa",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
},
"accelerator": "GPU"
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/virattt/30f056e9bf2d5f74f701bc7d67ad7cff/private_few-shot-query-rewriting.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"# Load HotpotQA data from HuggingFace\n",
"\n",
"To keep things simple, we will only load a subset of the following\n",
"- queries\n",
"- corpus\n",
"- qrels (relevance judgments)"
],
"metadata": {
"id": "mJISkfHtCyzq"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "KaoryniQ2cZj"
},
"outputs": [],
"source": [
"!pip install datasets"
]
},
{
"cell_type": "code",
"source": [
"from datasets import load_dataset"
],
"metadata": {
"id": "GYwzsd962i9Q"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Load the HotpotQA queries from BEIR\n",
"queries = load_dataset(\"BeIR/hotpotqa\", 'queries', split='queries')"
],
"metadata": {
"id": "uBAAhQ3a9k_2"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Load a subset of the HotpotQA corpus from BEIR\n",
"corpus = load_dataset(\"BeIR/hotpotqa\", 'corpus', split='corpus[:100000]')"
],
"metadata": {
"id": "4lKTFJCy_M7X"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Load the HotpotQA qrels from BEIR\n",
"qrels = load_dataset(\"BeIR/hotpotqa-qrels\")"
],
"metadata": {
"id": "p6TQuLBGDGbD"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Extract IDs from queries and corpus\n",
"query_ids = set(queries['_id'])\n",
"corpus_ids = set(corpus['_id'])\n",
"\n",
"# Filter out qrels that we do not have queries and corpus for\n",
"filtered_qrels = qrels.filter(lambda x: x['query-id'] in query_ids and str(x['corpus-id']) in corpus_ids)\n",
"print(filtered_qrels)"
],
"metadata": {
"id": "3LOKpfYM_cPD"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Select the first 1000 indices\n",
"indices = list(range(1000))\n",
"\n",
"# Keep only the first 1000 rows in the train dataset\n",
"filtered_qrels['train'] = filtered_qrels['train'].select(indices)"
],
"metadata": {
"id": "EJIsP35tZ6Zk"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"print(filtered_qrels)"
],
"metadata": {
"id": "fV52LkB1aI4n"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Collecting unique IDs from qrels\n",
"unique_query_ids = set(filtered_qrels['train']['query-id'])\n",
"unique_corpus_ids = set(str(id) for id in filtered_qrels['train']['corpus-id'])\n",
"\n",
"# Filtering corpus and queries based on collected IDs\n",
"filtered_corpus = corpus.filter(lambda x: x['_id'] in unique_corpus_ids)\n",
"filtered_queries = queries.filter(lambda x: x['_id'] in unique_query_ids)"
],
"metadata": {
"id": "eD75uzpuIMbM"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# Persist corpus in vector DB"
],
"metadata": {
"id": "vfzSguBuCwEa"
}
},
{
"cell_type": "code",
"source": [
"!pip install langchain langchain_openai chromadb"
],
"metadata": {
"id": "7VL4g11lCbPw"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"import getpass\n",
"import os\n",
"\n",
"os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")"
],
"metadata": {
"id": "td8Z72ijEUMV"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"from langchain_community.vectorstores import Chroma\n",
"from langchain_core.documents import Document\n",
"from langchain_openai import OpenAIEmbeddings\n",
"\n",
"embeddings = OpenAIEmbeddings()"
],
"metadata": {
"id": "06SBUyiVEeTd"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Create Documents\n",
"documents = []\n",
"for corpus in filtered_corpus:\n",
" documents.append(\n",
" Document(\n",
" page_content=corpus['text'],\n",
" metadata={'title': corpus['title'], 'id': corpus['_id']},\n",
" )\n",
" )"
],
"metadata": {
"id": "UGJ4XYbCDMKJ"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Store documents in vector DB\n",
"vectorstore = Chroma.from_documents(documents, embeddings)"
],
"metadata": {
"id": "WQPekVUSEwk1"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# Convert the HuggingFace dataset to a simple python dictionary"
],
"metadata": {
"id": "v-Zts5PdT-6T"
}
},
{
"cell_type": "code",
"source": [
"'''\n",
"Create a simplified dictionary of qrels where:\n",
"key: query-id\n",
"value: list of corpus-id\n",
"'''\n",
"qrels_dict = {}\n",
"\n",
"# Loop through each qrel in filtered_qrels\n",
"for qrel in filtered_qrels['train']:\n",
" query_id = qrel['query-id'] # Extract the query-id from qrel\n",
"\n",
" # Initialize the list of corpus-ids for the current query-id in qrels_dict\n",
" if query_id not in qrels_dict:\n",
" qrels_dict[query_id] = []\n",
"\n",
" # Loop through filtered_qrels again to find all matches for the current query-id\n",
" for x in filtered_qrels['train']:\n",
" if x['query-id'] == query_id:\n",
" # Add the corpus-id to the list for the current query-id, ensuring it's a string\n",
" qrels_dict[query_id].append(str(x['corpus-id']))\n"
],
"metadata": {
"id": "_CuRPpLXKys0"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# Define functions to compute metrics"
],
"metadata": {
"id": "ug_ay2zF9QAb"
}
},
{
"cell_type": "code",
"source": [
"def compute_precision(retrieved_docs, relevant_docs, k=5):\n",
" if k == 0:\n",
" return 0 # Avoid division by zero\n",
"\n",
" top_k_docs = retrieved_docs[:k]\n",
" relevant_count = sum(1 for doc in top_k_docs if doc.page_content in relevant_docs)\n",
"\n",
" return relevant_count / k\n",
"\n",
"def compute_recall(retrieved_docs, relevant_docs, k=5):\n",
" if not relevant_docs:\n",
" return 0 # Avoid division by zero if there are no relevant docs\n",
" # Consider only the top k documents if k is specified, otherwise consider all retrieved_docs\n",
" top_k_docs = retrieved_docs[:k] if k is not None else retrieved_docs\n",
" relevant_count = sum(1 for doc in top_k_docs if doc.page_content in relevant_docs)\n",
" return relevant_count / len(relevant_docs)\n",
"\n",
"def compute_ndcg(retrieved_docs, relevant_docs, k=5):\n",
" def dcg(retrieved_docs, relevant_docs):\n",
" # DCG is the sum of the relevance scores (logarithmically discounted)\n",
" return sum((doc.page_content in relevant_docs) / np.log2(i + 2) for i, doc in enumerate(retrieved_docs[:k]))\n",
"\n",
" def idcg(relevant_docs):\n",
" # iDCG is the DCG of the ideal ranking (all relevant documents at the top)\n",
" return sum(1 / np.log2(i + 2) for i in range(min(len(relevant_docs), k)))\n",
"\n",
" return dcg(retrieved_docs, relevant_docs) / idcg(relevant_docs)\n",
"\n",
"def compute_mrr(retrieved_docs, relevant_docs, k=5):\n",
" for i, doc in enumerate(retrieved_docs[:k], start=1):\n",
" if doc.page_content in relevant_docs:\n",
" return 1 / i\n",
" return 0 # Return 0 if no relevant document is found in the top k"
],
"metadata": {
"id": "o3E_KLbaUiF8"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# Set up prompt"
],
"metadata": {
"id": "TsRKQ3ebUmiZ"
}
},
{
"cell_type": "code",
"source": [
"prompt = \"\"\"\n",
"You are an AI trained to improve the effectiveness of information retrieval from a vector database.\n",
"Your task is to rewrite queries from their original form into a more detailed version that includes additional context or clarification.\n",
"This helps in enhancing the accuracy of search results in terms of NDCG, Recall@K, and MRR@K metrics.\n",
"Below are a few examples of how queries can be rewritten for better understanding and retrieval performance.\n",
"\n",
"Example 1:\n",
"Original Query: \"Who is the president mentioned in relation to the healthcare law?\"\n",
"Rewritten Query: \"Who is the U.S. president mentioned in discussions about the Affordable Care Act (Obamacare)?\"\n",
"\n",
"Example 2:\n",
"Original Query: \"What film did both actors star in?\"\n",
"Rewritten Query: \"In which film did Leonardo DiCaprio and Kate Winslet both have starring roles?\"\n",
"\n",
"Example 3:\n",
"Original Query: \"When was the company founded that created the iPhone?\"\n",
"Rewritten Query: \"What is the founding year of Apple Inc., the company that developed the iPhone?\"\n",
"\n",
"Given a user query, rewrite it to add more context and clarity, similar to the examples provided above.\n",
"Ensure the rewritten query is specific and detailed to improve the retrieval of relevant information from a database.\n",
"\"\"\""
],
"metadata": {
"id": "5sgI49cAX1bJ"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# OpenAI: Compute query rewriting performance metrics"
],
"metadata": {
"id": "Q8imRqFq9UG4"
}
},
{
"cell_type": "code",
"source": [
"from openai import OpenAI\n",
"\n",
"client = OpenAI(api_key=os.environ[\"OPENAI_API_KEY\"])\n",
"\n",
"def rewrite_query_openai(query: str, prompt: str, model=\"gpt-3.5-turbo-0125\") -> str:\n",
" response = client.chat.completions.create(\n",
" model=model,\n",
" temperature=0,\n",
" seed=42,\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": prompt},\n",
" {\"role\": \"user\", \"content\": query},\n",
" ]\n",
" )\n",
" # Get the rewritten query from response\n",
" rewritten_query = response.choices[0].message.content\n",
" return rewritten_query"
],
"metadata": {
"id": "_jXjwG97UjHq"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"k = 5\n",
"max_iterations = 100\n",
"\n",
"# Store metrics for each query\n",
"precision_values = []\n",
"recall_values = []\n",
"ndcg_values = []\n",
"mrr_values = []"
],
"metadata": {
"id": "sbGETH6b7hKD"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"import numpy as np\n",
"import time\n",
"\n",
"start_time = time.time()\n",
"\n",
"for index, query in enumerate(filtered_queries):\n",
" if index == max_iterations:\n",
" # Early exit since we have hit max iterations\n",
" break\n",
"\n",
" question = query['text']\n",
" question_id = query['_id']\n",
"\n",
" # Rewrite the query with LLM\n",
" rewritten_query = rewrite_query_openai(\n",
" query=question,\n",
" prompt=prompt,\n",
" )\n",
"\n",
" print(f\"Question {index + 1}\")\n",
" print(f\"Original query: {question}\")\n",
" print(f\"Rewritten query: {rewritten_query}\")\n",
" print()\n",
"\n",
" # Perform the search in the vector DB for the current query\n",
" top_k_docs = vectorstore.similarity_search(rewritten_query, k)\n",
"\n",
" # Get relevant doc IDs for the current query\n",
" relevant_doc_ids = qrels_dict.get(question_id, [])\n",
"\n",
" # Fetch the texts of relevant docs\n",
" relevant_docs = [doc['text'] for doc in filtered_corpus if doc['_id'] in relevant_doc_ids]\n",
"\n",
" # Compute metrics for the current query\n",
" precision_score = compute_precision(top_k_docs, relevant_docs, k)\n",
" recall_score = compute_recall(top_k_docs, relevant_docs, k)\n",
" ndcg_score = compute_ndcg(top_k_docs, relevant_docs, k)\n",
" mrr_score = compute_mrr(top_k_docs, relevant_docs, k)\n",
"\n",
" # Append the metrics for this query to the lists\n",
" precision_values.append(precision_score)\n",
" recall_values.append(recall_score)\n",
" ndcg_values.append(ndcg_score)\n",
" mrr_values.append(mrr_score)\n",
"\n",
" # Wait for 1 seconds before the next iteration to avoid rate limiting\n",
" time.sleep(1)\n"
],
"metadata": {
"id": "iEdGJtncNuzj",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "d20196f4-9a1d-4a02-b38f-92f79418ccb6"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Question 1\n",
"Original query: What country of origin does House of Cosbys and Bill Cosby have in common?\n",
"Rewritten query: What is the shared country of origin between the animated series \"House of Cosbys\" and the comedian Bill Cosby?\n",
"\n",
"Question 2\n",
"Original query: Which actor starred in Assignment to Kill and passed away in 2000.\n",
"Rewritten query: Which actor starred in the movie \"Assignment to Kill\" and tragically passed away in the year 2000?\n",
"\n",
"Question 3\n",
"Original query: What name was given to the son of the man who defeated the usurper Allectus ?\n",
"Rewritten query: What is the name of the son of the historical figure who emerged victorious over the usurper Allectus in battle?\n",
"\n",
"Question 4\n",
"Original query: What profession does Lewis Milestone and All Quiet on the Western Front have in common?\n",
"Rewritten query: What profession is shared by Lewis Milestone, the director of the film, and the characters in \"All Quiet on the Western Front\"?\n",
"\n",
"Question 5\n",
"Original query: University of Alabama in Huntsville is a college located in what county?\n",
"Rewritten query: In which county is the University of Alabama in Huntsville, a college campus, situated?\n",
"\n",
"Question 6\n",
"Original query: What 1937 magazine did \"Bringing Up Baby\" film star and one of classic Hollywood's definitive leading men appear in?\n",
"Rewritten query: In which 1937 magazine did the film \"Bringing Up Baby\" star and one of classic Hollywood's definitive leading men make an appearance?\n",
"\n",
"Question 7\n",
"Original query: Who founded the organization whose Boston branch excluded the Christian discussion group later housed in the Boston Young Men's Christian Union?\n",
"Rewritten query: Who is the founder of the organization that, at its Boston branch, excluded the Christian discussion group which was subsequently accommodated in the Boston Young Men's Christian Union?\n",
"\n",
"Question 8\n",
"Original query: What language family is the language of the tribe of the man who instructed Jeff Ball in?\n",
"Rewritten query: To which language family does the language spoken by the tribe of the individual who taught Jeff Ball belong to?\n",
"\n",
"Question 9\n",
"Original query: Dayton, Newark is part of the county in New Jersey having what population as of 2016?\n",
"Rewritten query: What was the population of Dayton, Newark, a town in Essex County, New Jersey, as of the year 2016?\n",
"\n",
"Question 10\n",
"Original query: What album produced by George Martin was supposed to contain a song that ended up unreleased until 1996?\n",
"Rewritten query: Which album, produced by George Martin, originally intended to include a song that remained unreleased until 1996?\n",
"\n",
"Question 11\n",
"Original query: Who is the mother of both an English television chef and food critic and an English actor who holds both British and Irish citizenship?\n",
"Rewritten query: Who is the mother of both an English television chef and food critic, known for their culinary expertise, and an English actor who holds dual British and Irish citizenship?\n",
"\n",
"Question 12\n",
"Original query: Kung Fu Panda is a 2008 American computer-animated action comedy starring the voice of a Hong Kong martial artist who is known for what kind of fighting style?\n",
"Rewritten query: In the 2008 American computer-animated action comedy film \"Kung Fu Panda,\" which features the voice of a Hong Kong martial artist, what specific style of martial arts is prominently showcased or associated with the character?\n",
"\n",
"Question 13\n",
"Original query: The Lost Children is a B-side compilation album by the heavy metal band from what city?\n",
"Rewritten query: The Lost Children is a B-side compilation album by the heavy metal band. Which city is the band originally from?\n",
"\n",
"Question 14\n",
"Original query: Which of the following is acclaimed for his \"lyrical flow of his statements\": Nâzım Hikmet or Arthur Miller?\n",
"Rewritten query: Who among Nâzım Hikmet and Arthur Miller is renowned for the poetic and eloquent style in which he delivers his ideas and messages?\n",
"\n",
"Question 15\n",
"Original query: Who directed the American romantic comedy-drama in which \"Cold Blooded Old Times\" appeared on the film soundtrack?\n",
"Rewritten query: Who was the director of the American romantic comedy-drama film that featured the song \"Cold Blooded Old Times\" on its soundtrack?\n",
"\n",
"Question 16\n",
"Original query: Michel Wachenheim a French ambassador and permanent representative of France of what specialized agency of the United Nations?\n",
"Rewritten query: In which specialized agency of the United Nations does Michel Wachenheim, the French ambassador and permanent representative of France, serve?\n",
"\n",
"Question 17\n",
"Original query: What year was the ship, in which Sink the Belgrano! was based on, listed as out of service?\n",
"Rewritten query: In what year was the ship that inspired the events depicted in the movie \"Sink the Belgrano!\" officially decommissioned or listed as out of service?\n",
"\n",
"Question 18\n",
"Original query: Disneyland Park has similar attractions to the park located in what Florida county?\n",
"Rewritten query: Which county in Florida is home to a theme park with attractions similar to those found at Disneyland Park in California?\n",
"\n",
"Question 19\n",
"Original query: Are Erica Jong and Nancy Mitford both novelists?\n",
"Rewritten query: Are Erica Jong and Nancy Mitford both known for their work as authors of novels?\n",
"\n",
"Question 20\n",
"Original query: Donkey Kong is an arcade game released by Nintendo, which character in this game, was originally named Mr. Video?\n",
"Rewritten query: In the arcade game Donkey Kong released by Nintendo, which character was originally named Mr. Video before being renamed?\n",
"\n",
"Question 21\n",
"Original query: 123 Albert Street has a style that is from the philosophy that took hold during what time frame?\n",
"Rewritten query: During which historical period did the architectural style of 123 Albert Street reflect the prevailing philosophical beliefs?\n",
"\n",
"Question 22\n",
"Original query: Filipino sitcom Iskul Bukol had a theme song to the tune of which hit by the King of Rock 'n' Roll?\n",
"Rewritten query: Which popular Elvis Presley hit song was used as the tune for the theme song of the Filipino sitcom Iskul Bukol?\n",
"\n",
"Question 23\n",
"Original query: Which English rock band's song \"Hand of Doom\" was the inspiration for the cover art on the album Black Science?\n",
"Rewritten query: Which iconic English rock band wrote the song \"Hand of Doom\" that served as the inspiration for the cover art on the album Black Science?\n",
"\n",
"Question 24\n",
"Original query: What did the illustrator of a well-known manga starring Goku first achieve mainstream recognition for?\n",
"Rewritten query: What was the initial claim to fame for the illustrator of the popular manga featuring the character Goku, before achieving mainstream recognition for their work on this well-known series?\n",
"\n",
"Question 25\n",
"Original query: The composer of the opera \"The Zoo\" has composed how many major orchestral works?\n",
"Rewritten query: How many major orchestral works have been composed by the composer of the opera \"The Zoo\"?\n",
"\n",
"Question 26\n",
"Original query: What village, named after a railroad attorney, is at the end of New York State Route 156?\n",
"Rewritten query: Which village, named after the railroad attorney William H. Seward, is located at the terminus of New York State Route 156?\n",
"\n",
"Question 27\n",
"Original query: Which important Roman figure did Matthias Gelzer write a biography for, besides Julius Caesar and Cicero?\n",
"Rewritten query: For which prominent Roman historical figure did Matthias Gelzer author a biography, apart from Julius Caesar and Cicero?\n",
"\n",
"Question 28\n",
"Original query: Where is H. Frank Carey Junior-Senior High School?\n",
"Rewritten query: Where is the location of H. Frank Carey Junior-Senior High School situated?\n",
"\n",
"Question 29\n",
"Original query: What is the largest subaerial volcano in the Hawaiian - Emperor seamount chain?\n",
"Rewritten query: Which subaerial volcano located in the Hawaiian-Emperor seamount chain is considered the largest in terms of land area above sea level?\n",
"\n",
"Question 30\n",
"Original query: Which Italian composer's music from the Renaissance and the Baroque periods was performed by Rinaldo Alessandrini?\n",
"Rewritten query: Which renowned Italian composer's works from both the Renaissance and Baroque eras were interpreted by the talented musician Rinaldo Alessandrini?\n",
"\n",
"Question 31\n",
"Original query: The film whose plot is based on Stephen King's novella \"The Body\" stars what actor that also appeared in \"Toy Soldiers\" and \"Flubber\"?\n",
"Rewritten query: Which actor starred in the film adaptation based on Stephen King's novella \"The Body\" and also appeared in \"Toy Soldiers\" and \"Flubber\"?\n",
"\n",
"Question 32\n",
"Original query: Who was Born first Ulysses S. Grant or Edward Lyon Buchwalter?\n",
"Rewritten query: Who was born earlier, Ulysses S. Grant or Edward Lyon Buchwalter, in order to determine the age difference between the two individuals?\n",
"\n",
"Question 33\n",
"Original query: Who was the last emperor or last \"Son of Heaven\" of China?\n",
"Rewritten query: Who was the final emperor to rule as the \"Son of Heaven\" in China before the end of the imperial system?\n",
"\n",
"Question 34\n",
"Original query: Myron F. Diduryk, was described by Hal Moore in which book?\n",
"Rewritten query: In which book did Hal Moore describe Myron F. Diduryk?\n",
"\n",
"Question 35\n",
"Original query: Silent Night (German: \"Stille Nacht, heilige Nacht\" ) is a popular Christmas carol, composed in which year, by Franz Xaver Gruber, an Austrian primary school teacher, church organist and composer in the village of Arnsdorf?\n",
"Rewritten query: In which year was the popular Christmas carol \"Silent Night\" (German: \"Stille Nacht, heilige Nacht\") composed by Franz Xaver Gruber, an Austrian primary school teacher, church organist, and composer in the village of Arnsdorf?\n",
"\n",
"Question 36\n",
"Original query: At what university was the group who produced \"Nostalgic\" formed?\n",
"Rewritten query: At which university did the music group responsible for creating the album \"Nostalgic\" originate from?\n",
"\n",
"Question 37\n",
"Original query: Where was the lead male actor whose name was Frank in Guys and Dolls born?\n",
"Rewritten query: Where was the birthplace of the lead male actor named Frank who starred in the movie \"Guys and Dolls\"?\n",
"\n",
"Question 38\n",
"Original query: Where was the American scout and bison hunter, whom Robert W. Patten claimed to be as historically significant as born?\n",
"Rewritten query: In which location was the American scout and bison hunter, who Robert W. Patten claimed to be as historically significant as, born?\n",
"\n",
"Question 39\n",
"Original query: Which period in German history came to an end thanks in part to material aid supplied to the Soviet Union by an Arctic convoy?\n",
"Rewritten query: Which specific era in German history was brought to a close partly due to the material support provided to the Soviet Union through an Arctic convoy during World War II?\n",
"\n",
"Question 40\n",
"Original query: In what Super Bowl game did John Jett win a Super Bowl ring at the Sun Devil Stadium?\n",
"Rewritten query: In which Super Bowl game did John Jett earn a Super Bowl ring while playing at Sun Devil Stadium?\n",
"\n",
"Question 41\n",
"Original query: Orchard Estates is located within a county in New Jersey with a population of what in 2016?\n",
"Rewritten query: What was the population of the county in New Jersey where Orchard Estates is situated in the year 2016?\n",
"\n",
"Question 42\n",
"Original query: House Party 3 features the debut of an American actor born in what year?\n",
"Rewritten query: In which year was the American actor who made his debut in \"House Party 3\" born?\n",
"\n",
"Question 43\n",
"Original query: What is the meaning of the nickname of Messalina's second-cousin?\n",
"Rewritten query: What is the significance or historical background behind the nickname of Messalina's second-cousin in Roman history?\n",
"\n",
"Question 44\n",
"Original query: What university has its home in Gaetano Callani's native city?\n",
"Rewritten query: Which university is located in the native city of Gaetano Callani?\n",
"\n",
"Question 45\n",
"Original query: What year did the singer behind \"Minstrel Boy\" reach fame in?\n",
"Rewritten query: In what year did the singer known for the song \"Minstrel Boy\" achieve fame?\n",
"\n",
"Question 46\n",
"Original query: Ann Willing Bingham, was an American socialite from Philadelphia, she was the eldest daughter of Thomas Willing, president of the First Bank of the United States, and a correspondent of the principal author of the Declaration of Independence, and later served as the third President of the United States?\n",
"Rewritten query: Who was Ann Willing Bingham, the American socialite from Philadelphia known for being the eldest daughter of Thomas Willing, the president of the First Bank of the United States, and a correspondent of the principal author of the Declaration of Independence, before later becoming the third President of the United States?\n",
"\n",
"Question 47\n",
"Original query: Were Howard Hawks and Armand Schaefer the same nationality?\n",
"Rewritten query: What was the nationality of both Howard Hawks and Armand Schaefer?\n",
"\n",
"Question 48\n",
"Original query: How many studio albums did the youngest winner of the Teen Choice Awards have?\n",
"Rewritten query: How many studio albums were released by the youngest recipient of the Teen Choice Awards in the music category?\n",
"\n",
"Question 49\n",
"Original query: Who was the original composer of \"Pictures at an Exhibition\"?\n",
"Rewritten query: Who was the original composer of the classical music suite \"Pictures at an Exhibition,\" famously inspired by an art exhibition?\n",
"\n",
"Question 50\n",
"Original query: Thesongadayproject had a cover version of the song Catch the Wind by the singer and guitarist of what nationality?\n",
"Rewritten query: What is the nationality of the singer and guitarist who performed a cover version of the song \"Catch the Wind\" on the YouTube channel \"Thesongadayproject\"?\n",
"\n",
"Question 51\n",
"Original query: Which English actor co-wrote and also played in The Dictator?\n",
"Rewritten query: Which English actor co-wrote the screenplay and portrayed a role in the movie \"The Dictator\"?\n",
"\n",
"Question 52\n",
"Original query: Once Upon a Time in America is a crime drama film directed by the inventor of what?\n",
"Rewritten query: Who is the director of the crime drama film \"Once Upon a Time in America\"?\n",
"\n",
"Question 53\n",
"Original query: What year was the college that M.C. Richards attended founded?\n",
"Rewritten query: In which year was the college that M.C. Richards, the American poet, potter, and writer, attended established?\n",
"\n",
"Question 54\n",
"Original query: Who plays Tonya Harding in the 2017 biographical film entitled I, Tonya?\n",
"Rewritten query: Who portrays the character of Tonya Harding in the 2017 biographical film titled \"I, Tonya\"?\n",
"\n",
"Question 55\n",
"Original query: Which author is famous for writing historical spy novels, Alan Furst or H. P. Lovecraft?\n",
"Rewritten query: Which author is renowned for their expertise in crafting historical spy novels: Alan Furst or H. P. Lovecraft?\n",
"\n",
"Question 56\n",
"Original query: The professional boxer James Tillis had a notable win by TKO 7 in 1980 over a boxer that challenged the world heavyweight championship in what year?\n",
"Rewritten query: In 1980, professional boxer James Tillis achieved a significant victory by TKO in the 7th round. This win was notable as the defeated boxer later went on to challenge for the world heavyweight championship. In which year did the defeated boxer compete for the world heavyweight title?\n",
"\n",
"Question 57\n",
"Original query: Are John H. Auer and Richard Curtis both involved in the film industry?\n",
"Rewritten query: Are John H. Auer and Richard Curtis both known for their contributions to the film industry?\n",
"\n",
"Question 58\n",
"Original query: This English comedian wrote the British sitcom \"Absolutely Fabulous\"\n",
"Rewritten query: Which English comedian is known for creating and writing the popular British sitcom \"Absolutely Fabulous\"?\n",
"\n",
"Question 59\n",
"Original query: What sports show did the hockey player born on January 26, 1961 co-host?\n",
"Rewritten query: Which sports show did the hockey player born on January 26, 1961, participate in as a co-host?\n",
"\n",
"Question 60\n",
"Original query: Are Roger Ebert and Anna Akhmatova both writers?\n",
"Rewritten query: Are both Roger Ebert, the film critic, and Anna Akhmatova, the Russian poet, recognized as writers in their respective fields?\n",
"\n",
"Question 61\n",
"Original query: Don Loren Harper's credits include a 1998 American science fiction disaster film that was directed by Jerry Bruckheimer and directed by who?\n",
"Rewritten query: Don Loren Harper's credits include a 1998 American science fiction disaster film that was directed by Jerry Bruckheimer and produced by whom?\n",
"\n",
"Question 62\n",
"Original query: Which band formed first Suede or Hard-Fi ?\n",
"Rewritten query: Which band was established earlier, Suede or Hard-Fi, in the music industry?\n",
"\n",
"Question 63\n",
"Original query: A pumapard is hybrid of a leopard and the second-heaviest cat in the New World, after what?\n",
"Rewritten query: After which cat species is the pumapard, a hybrid of a leopard and a puma, considered the second-heaviest in the New World?\n",
"\n",
"Question 64\n",
"Original query: Who was the head coach of the team that lost Super Bowl XIX?\n",
"Rewritten query: Who was the head coach of the NFL team that was defeated in Super Bowl XIX, which took place in 1985 between the San Francisco 49ers and the Miami Dolphins?\n",
"\n",
"Question 65\n",
"Original query: Saving Grace starred what American actor who played William Adama in Battlestar Galactica?\n",
"Rewritten query: Which American actor, known for portraying William Adama in Battlestar Galactica, starred in the movie Saving Grace?\n",
"\n",
"Question 66\n",
"Original query: The director that created the character Thirteenth Aunt was born in what year?\n",
"Rewritten query: In what year was the director who created the character Thirteenth Aunt born?\n",
"\n",
"Question 67\n",
"Original query: Great Limber is situated 8 miles east from a town with how many households according to the 2001 UK census ?\n",
"Rewritten query: According to the 2001 UK census, how many households are located in the town that is 8 miles east of Great Limber?\n",
"\n",
"Question 68\n",
"Original query: Do Phil Collins and Bobby Fuller have the same nationality?\n",
"Rewritten query: Are Phil Collins and Bobby Fuller both of the same nationality, or do they belong to different countries?\n",
"\n",
"Question 69\n",
"Original query: Which second governor was the territory, split into Mississippi and Alabama in 1819, named after?\n",
"Rewritten query: Which individual was the second governor of the territory that was divided into Mississippi and Alabama in 1819, and after whom was this territory named?\n",
"\n",
"Question 70\n",
"Original query: In which county is this village where Reid Gorecki grew up located?\n",
"Rewritten query: In which county is the village, where Reid Gorecki, the professional baseball player, grew up, located?\n",
"\n",
"Question 71\n",
"Original query: Edmundo \"Eddie\" Rodriguez was born in what Mexican state?\n",
"Rewritten query: In which Mexican state was Edmundo \"Eddie\" Rodriguez born?\n",
"\n",
"Question 72\n",
"Original query: Æthelstan Half-King left his position and became a monk after the death of a king who ruled England starting which year ?\n",
"Rewritten query: After the death of which English king did Æthelstan Half-King resign from his position and enter a monastery? When did this king begin his rule in England?\n",
"\n",
"Question 73\n",
"Original query: What friends star's wedding was photographed by Robert Evans?\n",
"Rewritten query: Whose wedding from the TV show \"Friends\" was captured by the photographer Robert Evans?\n",
"\n",
"Question 74\n",
"Original query: For which film did the actress, who recorded the song Coming Home, receive the Best Actress Academy Award ?\n",
"Rewritten query: In which film did the actress who performed the song \"Coming Home\" win the Best Actress Academy Award?\n",
"\n",
"Question 75\n",
"Original query: Childersburg, Alabama is near the munitions plant that was operated during what event?\n",
"Rewritten query: During which historical event was the munitions plant near Childersburg, Alabama, operational?\n",
"\n",
"Question 76\n",
"Original query: What year was the writer of the music for the 1956 song \"Tonight\" born?\n",
"Rewritten query: In which year was the composer of the music for the 1956 song \"Tonight\" born?\n",
"\n",
"Question 77\n",
"Original query: What documentary had more directors: Dalai Lama Renaissance or Salesman?\n",
"Rewritten query: Which documentary features a larger number of directors: \"Dalai Lama Renaissance\" or \"Salesman\"?\n",
"\n",
"Question 78\n",
"Original query: What battle was fought by Greenwich Union soldiers, in which General Mead's Army of the Potomac defeated General Lee's North Virginia Army resulting in the largest number of casualties in a single war?\n",
"Rewritten query: Which significant battle during the American Civil War saw the Greenwich Union soldiers fighting under General Meade's Army of the Potomac emerge victorious over General Lee's Army of Northern Virginia, leading to the highest number of casualties recorded in a single conflict?\n",
"\n",
"Question 79\n",
"Original query: Baylor University and Duke University, are both what type of universities?\n",
"Rewritten query: What classification or category do Baylor University and Duke University fall under in terms of their institutional type or academic focus?\n",
"\n",
"Question 80\n",
"Original query: Jack Gelber was a professor at what public university system of New York City?\n",
"Rewritten query: At which public university within the New York City university system did Jack Gelber serve as a professor?\n",
"\n",
"Question 81\n",
"Original query: Are Dalton Trumbo and Herman Wouk both screenwriters?\n",
"Rewritten query: Are both Dalton Trumbo and Herman Wouk known for their work as screenwriters in the film industry?\n",
"\n",
"Question 82\n",
"Original query: Kihachi Okamoto and Victor Fleming were both what?\n",
"Rewritten query: What profession or role did Kihachi Okamoto and Victor Fleming share in the film industry?\n",
"\n",
"Question 83\n",
"Original query: When was the English-French actress born, that starred in the 1975 film \"Sept morts sur ordonnance\", who had a prolific career as an actress in British and French cinema?\n",
"Rewritten query: When was the English-French actress, known for her role in the 1975 film \"Sept morts sur ordonnance\", and acclaimed for her extensive career in both British and French cinema, born?\n",
"\n",
"Question 84\n",
"Original query: During what period did the mountain range in which Isoetes valida is primarily found first form?\n",
"Rewritten query: During what geological era did the mountain range where Isoetes valida is predominantly located originate?\n",
"\n",
"Question 85\n",
"Original query: What profession does Emmanuel Chabrier and Leoš Janáček have in common?\n",
"Rewritten query: What profession did both Emmanuel Chabrier, a French composer, and Leoš Janáček, a Czech composer, share in their careers?\n",
"\n",
"Question 86\n",
"Original query: What building has an Art Deco style and was appraised by Robert Von Ancken?\n",
"Rewritten query: Which specific building, known for its Art Deco architectural style, underwent an appraisal conducted by Robert Von Ancken?\n",
"\n",
"Question 87\n",
"Original query: Who had more occupations, Lewis Mumford or William Carlos Williams?\n",
"Rewritten query: Which individual, Lewis Mumford or William Carlos Williams, had a greater number of documented occupations throughout their respective careers?\n",
"\n",
"Question 88\n",
"Original query: Are both Margaret Atwood and Alberto Moravia novelists?\n",
"Rewritten query: Are both Margaret Atwood, the Canadian author known for works like \"The Handmaid's Tale,\" and Alberto Moravia, the Italian writer famous for novels such as \"The Conformist,\" recognized as novelists in the literary world?\n",
"\n",
"Question 89\n",
"Original query: Who was Joe Cornish's co-writer of \"Ant-Man\" that served as head writer for \"Saturday Night Live\"?\n",
"Rewritten query: Who was the co-writer of \"Ant-Man\" alongside Joe Cornish, who also served as the head writer for \"Saturday Night Live\"?\n",
"\n",
"Question 90\n",
"Original query: Which Chinese city is the capital of the Jiangsu province, Nanjing or Pu'er City\n",
"Rewritten query: Which city in China serves as the capital of the Jiangsu province: Nanjing or Pu'er City?\n",
"\n",
"Question 91\n",
"Original query: The composer of A Survivor from Warsaw emigrated to the United States in what year?\n",
"Rewritten query: In what year did the composer of \"A Survivor from Warsaw,\" Arnold Schoenberg, immigrate to the United States?\n",
"\n",
"Question 92\n",
"Original query: Thomas D'Alesandro Jr. is the father of the Minority Leader who represents what district?\n",
"Rewritten query: Who is the father of Nancy Pelosi, the current Minority Leader of the United States House of Representatives, and which congressional district does she represent?\n",
"\n",
"Question 93\n",
"Original query: The Old Grist Mill is a historic mill building in which town in York Country, Maine\n",
"Rewritten query: In which town in York County, Maine is the Old Grist Mill located, known for being a historic mill building?\n",
"\n",
"Question 94\n",
"Original query: Martin Hannett produced Joy Division's studio albums, who is the leader singer of Joy Division?\n",
"Rewritten query: Who is the lead singer of Joy Division, the band for which Martin Hannett produced the studio albums?\n",
"\n",
"Question 95\n",
"Original query: The Witches of Eastwick, based on john Updike's novel, was Directed by which director.?\n",
"Rewritten query: Who directed the film adaptation of John Updike's novel \"The Witches of Eastwick\"?\n",
"\n",
"Question 96\n",
"Original query: Two of the editors of the journal Thesis Eleven: Critical Theory and Historical Sociology are affiliated with a university founded in what year?\n",
"Rewritten query: In what year was the university founded that two of the editors of the journal Thesis Eleven: Critical Theory and Historical Sociology are affiliated with?\n",
"\n",
"Question 97\n",
"Original query: Have the two famous people Franco Rossi and Rob Reiner both directed films?\n",
"Rewritten query: Have both Franco Rossi and Rob Reiner directed movies in their respective careers?\n",
"\n",
"Question 98\n",
"Original query: how is Walking on Thin Ice and The Dakota related?\n",
"Rewritten query: How is Yoko Ono's song \"Walking on Thin Ice\" related to The Dakota building in New York City?\n",
"\n",
"Question 99\n",
"Original query: Who worked in both film and television, Ken Russell or Ralph Ceder?\n",
"Rewritten query: Which of the two, Ken Russell or Ralph Ceder, had a career that spanned both film and television industries?\n",
"\n",
"Question 100\n",
"Original query: Jacques-Louis was a founding member of the geological society before returning to France after what period of French history that followed the fall of Napoleon in 1814 until the July revolution of what year?\n",
"Rewritten query: After being a founding member of the Geological Society, Jacques-Louis returned to France during the period known as the Bourbon Restoration, which followed the fall of Napoleon in 1814 and lasted until the July Revolution of 1830.\n",
"\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"# Compute the final metric values\n",
"final_precision = np.mean(precision_values)\n",
"final_recall = np.mean(recall_values)\n",
"final_ndcg = np.mean(ndcg_values)\n",
"final_mrr = np.mean(mrr_values)\n",
"\n",
"print(f\"Final Precision@{k}: {round(final_precision * 100, 2)}\")\n",
"print(f\"Final Recall@{k}: {round(final_recall * 100, 2)}\")\n",
"print(f\"Final nDCG@{k}: {round(final_ndcg * 100, 2)}\")\n",
"print(f\"Final MRR@{k}: {round(final_mrr * 100, 2)}\")"
],
"metadata": {
"id": "_1Cv9bmTSTLf",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "7a6f0597-1a32-46b4-e8f1-9d53be5836bd"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Final Precision@5: 17.8\n",
"Final Recall@5: 81.0\n",
"Final nDCG@5: 70.32\n",
"Final MRR@5: 66.93\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"# Define Cohere code for query rewriting"
],
"metadata": {
"id": "0ceaovpO1Wap"
}
},
{
"cell_type": "code",
"source": [
"!pip install cohere"
],
"metadata": {
"id": "8ifCTbc-1bQx"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Set your Cohere API key\n",
"os.environ[\"COHERE_API_KEY\"] = getpass.getpass()"
],
"metadata": {
"id": "Td1xgj3K1oMB"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"import cohere\n",
"from typing import List\n",
"\n",
"# Get your cohere API key on: www.cohere.com\n",
"co = cohere.Client(os.environ[\"COHERE_API_KEY\"])\n",
"\n",
"def rewrite_query_cohere(query: str, prompt: str, model=\"command\") -> str:\n",
" response = co.chat(\n",
" model=model,\n",
" message=prompt + f\" Please rewrite the following user query: {query}. Important: only return the rewritten query. Do not add any extra context to your response!\",\n",
" )\n",
"\n",
" return response.text\n",
"\n",
"def rewrite_query_cohere_list(query: str, prompt: str, model=\"command\") -> List[str]:\n",
" response = co.chat(\n",
" model=model,\n",
" message=query,\n",
" search_queries_only=True,\n",
" )\n",
"\n",
" rewritten_queries = [query.get('text', None) for query in response.search_queries]\n",
" return rewritten_queries"
],
"metadata": {
"id": "mBI-zmlr1puR"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# Cohere: Compute query rewriting performance metrics"
],
"metadata": {
"id": "NPIfT-Y9BLNN"
}
},
{
"cell_type": "code",
"source": [
"k = 5\n",
"max_iterations = 100\n",
"\n",
"# Store metrics for each query\n",
"precision_values = []\n",
"recall_values = []\n",
"ndcg_values = []\n",
"mrr_values = []"
],
"metadata": {
"id": "xBnQYLdPBLNO"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"import numpy as np\n",
"import time\n",
"\n",
"start_time = time.time()\n",
"\n",
"for index, query in enumerate(filtered_queries):\n",
" if index == max_iterations:\n",
" # Early exit since we have hit max iterations\n",
" break\n",
"\n",
" question = query['text']\n",
" question_id = query['_id']\n",
"\n",
" # Rewrite the query with LLM\n",
" rewritten_query = rewrite_query_cohere(\n",
" query=question,\n",
" prompt=prompt,\n",
" )\n",
"\n",
" print(f\"Question {index + 1}\")\n",
" print(f\"Original query: {question}\")\n",
" print(f\"Rewritten query: {rewritten_query}\")\n",
" print()\n",
"\n",
" # Perform the search in the vector DB for the current query\n",
" top_k_docs = vectorstore.similarity_search(rewritten_query, k)\n",
"\n",
" # Get relevant doc IDs for the current query\n",
" relevant_doc_ids = qrels_dict.get(question_id, [])\n",
"\n",
" # Fetch the texts of relevant docs\n",
" relevant_docs = [doc['text'] for doc in filtered_corpus if doc['_id'] in relevant_doc_ids]\n",
"\n",
" # Compute metrics for the current query\n",
" precision_score = compute_precision(top_k_docs, relevant_docs, k)\n",
" recall_score = compute_recall(top_k_docs, relevant_docs, k)\n",
" ndcg_score = compute_ndcg(top_k_docs, relevant_docs, k)\n",
" mrr_score = compute_mrr(top_k_docs, relevant_docs, k)\n",
"\n",
" # Append the metrics for this query to the lists\n",
" precision_values.append(precision_score)\n",
" recall_values.append(recall_score)\n",
" ndcg_values.append(ndcg_score)\n",
" mrr_values.append(mrr_score)\n",
"\n",
" # Wait for 1 seconds before the next iteration to avoid rate limiting\n",
" time.sleep(1)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "AjoCB5J4BLNP",
"outputId": "23c1e93c-915b-4a63-c6e1-307c55892d59"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Question 1\n",
"Original query: What country of origin does House of Cosbys and Bill Cosby have in common?\n",
"Rewritten query: What country of origin corresponds with House of Cosbys and Bill Cosby?\n",
"\n",
"Question 2\n",
"Original query: Which actor starred in Assignment to Kill and passed away in 2000.\n",
"Rewritten query: \"The deceased actor that starred in Assignment to Kill which was released in 1984 and passed away in 2000, what was their name?\"\n",
"\n",
"Question 3\n",
"Original query: What name was given to the son of the man who defeated the usurper Allectus ?\n",
"Rewritten query: Who was the son of the individual that overthrew the corrupt leader Allectus, and what was this individual nicknamed?\n",
"\n",
"Question 4\n",
"Original query: What profession does Lewis Milestone and All Quiet on the Western Front have in common?\n",
"Rewritten query: What profession did Lewis Milestone pursue, and what was the context in All Quiet on the Western Front?\n",
"\n",
"Question 5\n",
"Original query: University of Alabama in Huntsville is a college located in what county?\n",
"Rewritten query: Which county is home to the University of Alabama in Huntsville, a post-secondary educational institution?\n",
"\n",
"Question 6\n",
"Original query: What 1937 magazine did \"Bringing Up Baby\" film star and one of classic Hollywood's definitive leading men appear in?\n",
"Rewritten query: Who was the starring actor of the \"Bringing Up Baby\" film from 1937 and what magazine did they appear in?\n",
"\n",
"Question 7\n",
"Original query: Who founded the organization whose Boston branch excluded the Christian discussion group later housed in the Boston Young Men's Christian Union?\n",
"Rewritten query: Who founded the organization that was headquartered in Boston and responsible for the Christian discussion group that was later housed in the Boston Young Men's Christian Union building?\n",
"\n",
"Question 8\n",
"Original query: What language family is the language of the tribe of the man who instructed Jeff Ball in?\n",
"Rewritten query: What is the language family of the Native American tribe of the man who instructed Jeff Ball in the Na'vi language?\n",
"\n",
"Question 9\n",
"Original query: Dayton, Newark is part of the county in New Jersey having what population as of 2016?\n",
"Rewritten query: As of 2016, the county in New Jersey which includes the municipalities of Dayton and Newark, had a population of approximately 774,980 people.\n",
"\n",
"Question 10\n",
"Original query: What album produced by George Martin was supposed to contain a song that ended up unreleased until 1996?\n",
"Rewritten query: Which album was scheduled to include a track that remained unreleased until 1996, according to George Martin's production catalog?\n",
"\n",
"Question 11\n",
"Original query: Who is the mother of both an English television chef and food critic and an English actor who holds both British and Irish citizenship?\n",
"Rewritten query: Who is the mother of both Gordon Ramsay (English television chef and food critic) and Damian Lewis (English actor who holds both British and Irish citizenship)?\n",
"\n",
"Question 12\n",
"Original query: Kung Fu Panda is a 2008 American computer-animated action comedy starring the voice of a Hong Kong martial artist who is known for what kind of fighting style?\n",
"Rewritten query: Which martial art style did the Hong Kong martial artist, Jackie Chan, specialize in and demonstrate through his voice role in the 2008 American computer-animated film, Kung Fu Panda?\n",
"\n",
"Question 13\n",
"Original query: The Lost Children is a B-side compilation album by the heavy metal band from what city?\n",
"Rewritten query: \"The heavy metal band behind The Lost Children album is from which American city, known for its rich jazz heritage and iconic musical contributions?\"\n",
"\n",
"Question 14\n",
"Original query: Which of the following is acclaimed for his \"lyrical flow of his statements\": Nâzım Hikmet or Arthur Miller?\n",
"Rewritten query: Who is acclaimed for their poetic expression and poignant philosophical insights: Nâzım Hikmet or Arthur Miller?\n",
"\n",
"Question 15\n",
"Original query: Who directed the American romantic comedy-drama in which \"Cold Blooded Old Times\" appeared on the film soundtrack?\n",
"Rewritten query: Who directed the film that features the song \"Cold Blooded Old Times\", which is categorized as American romantic comedy-drama, and what is the name of the film?\n",
"\n",
"Question 16\n",
"Original query: Michel Wachenheim a French ambassador and permanent representative of France of what specialized agency of the United Nations?\n",
"Rewritten query: Who is the French ambassador to the United Nations Food and Agriculture Agency?\n",
"\n",
"Question 17\n",
"Original query: What year was the ship, in which Sink the Belgrano! was based on, listed as out of service?\n",
"Rewritten query: \"What year was the actual ship, on which the sinkable cruiser HMS Belgrano was based, officially listed as out of service?\"\n",
"\n",
"Question 18\n",
"Original query: Disneyland Park has similar attractions to the park located in what Florida county?\n",
"Rewritten query: Which Florida county is home to a theme park offering attractions similar to those at Disneyland Park in California? \n",
"\n",
"This is the rewritten query, modified to be more specific and detailed, based on the user query you provided. It aims to enhance the accuracy of the search results in terms of NDCG, Recall@K, and MRR@K metrics.\n",
"\n",
"Question 19\n",
"Original query: Are Erica Jong and Nancy Mitford both novelists?\n",
"Rewritten query: Who are Erica Jong and Nancy Mitford, and what is unique about their careers as writers? Provide any distinct details that set them apart as authors.\n",
"\n",
"Question 20\n",
"Original query: Donkey Kong is an arcade game released by Nintendo, which character in this game, was originally named Mr. Video?\n",
"Rewritten query: Who is the character in the Donkey Kong game, originally named Mr. Video?\n",
"\n",
"Question 21\n",
"Original query: 123 Albert Street has a style that is from the philosophy that took hold during what time frame?\n",
"Rewritten query: What is the architectural style of the building at 123 Albert Street, and what time period roughly corresponds to the emergence of this style?.\n",
"\n",
"Question 22\n",
"Original query: Filipino sitcom Iskul Bukol had a theme song to the tune of which hit by the King of Rock 'n' Roll?\n",
"Rewritten query: Who was the songwriter behind the theme song to the 1970s Filipino sitcom Iskul Bukol, and what was the theme song originally a hit for?\n",
"\n",
"Question 23\n",
"Original query: Which English rock band's song \"Hand of Doom\" was the inspiration for the cover art on the album Black Science?\n",
"Rewritten query: Which English rock band, known for the song \"Hand of Doom,\" was the subject of the album cover art for Black Science?\n",
"\n",
"Question 24\n",
"Original query: What did the illustrator of a well-known manga starring Goku first achieve mainstream recognition for?\n",
"Rewritten query: Here is the rewritten query, adding additional context and clarity to the user's original query: \n",
"What did the illustrator of the well-known manga \"Dragon Ball\", which stars the character Goku, first achieve mainstream recognition for in the anime and manga industry?\n",
"\n",
"Question 25\n",
"Original query: The composer of the opera \"The Zoo\" has composed how many major orchestral works?\n",
"Rewritten query: How many major orchestral works has the composer of \"The Zoo\" opera composed?\n",
"\n",
"Question 26\n",
"Original query: What village, named after a railroad attorney, is at the end of New York State Route 156?\n",
"Rewritten query: Here is the rewritten query with added context and clarity: \n",
"\n",
"\"What village serves as a namesake for a railroad attorney and can be found at the terminus of New York State Route 156?\" \n",
"\n",
"This revised version maintains the essence of the user's original query while enhancing its specificity and detail.\n",
"\n",
"Question 27\n",
"Original query: Which important Roman figure did Matthias Gelzer write a biography for, besides Julius Caesar and Cicero?\n",
"Rewritten query: Who is another important Roman figure besides Julius Caesar and Cicero that Matthias Gelzer wrote a biography about?\n",
"\n",
"Question 28\n",
"Original query: Where is H. Frank Carey Junior-Senior High School?\n",
"Rewritten query: Where is H. Frank Carey Junior-Senior High School, located in New York and part of the Plainview-Old Bethpage Central School District?\n",
"\n",
"Question 29\n",
"Original query: What is the largest subaerial volcano in the Hawaiian - Emperor seamount chain?\n",
"Rewritten query: \"Within the Hawaiian-Emperor seamount chain, which volcano is the largest subaerial one?\"\n",
"\n",
"Question 30\n",
"Original query: Which Italian composer's music from the Renaissance and the Baroque periods was performed by Rinaldo Alessandrini?\n",
"Rewritten query: The respective Italian composer being inquired upon is most likely Giovanni Giorgio de Sabata, better known as Giorgio Sabattini. His notable works from the Renaissance and Baroque periods were interpreted by the renowned violinist Rinaldo Alessandrini.\n",
"\n",
"Question 31\n",
"Original query: The film whose plot is based on Stephen King's novella \"The Body\" stars what actor that also appeared in \"Toy Soldiers\" and \"Flubber\"?\n",
"Rewritten query: Who is the actor starred in the film based on Stephen King's novella \"The Body\", also appeared in the films \"Toy Soldiers\" and \"Flubber\"?\n",
"\n",
"Question 32\n",
"Original query: Who was Born first Ulysses S. Grant or Edward Lyon Buchwalter?\n",
"Rewritten query: Place of birth as context for the historical figure:\n",
"\"Compare the birth places of Ulysses S. Grant who was born in Rosemont (near St. Louis), Missouri, with Edward Lyon Buchwalter who was born in Ohio. \" \n",
"\n",
"This revision aims to provide a clear query that offers enough information to differentiate the two individuals and highlight their different birth places.\n",
"\n",
"Question 33\n",
"Original query: Who was the last emperor or last \"Son of Heaven\" of China?\n",
"Rewritten query: \"Last emperor or \"Son of Heaven\" of China, who lost the throne and the Republic of China was established, leading to a shift in the political and cultural landscape\"?\n",
"\n",
"Question 34\n",
"Original query: Myron F. Diduryk, was described by Hal Moore in which book?\n",
"Rewritten query: Who was Myron F. Diduryk, and what was the book written by Hal Moore that mentions him? Provide author and title for book.\n",
"\n",
"Question 35\n",
"Original query: Silent Night (German: \"Stille Nacht, heilige Nacht\" ) is a popular Christmas carol, composed in which year, by Franz Xaver Gruber, an Austrian primary school teacher, church organist and composer in the village of Arnsdorf?\n",
"Rewritten query: Which year was the popular Christmas carol \"Silent Night\" (German: \"Stille Nacht, heilige Nacht\"), composed by Franz Xaver Gruber, an Austrian primary school teacher, church organist, and composer from the village of Arnsdorf?\n",
"\n",
"Question 36\n",
"Original query: At what university was the group who produced \"Nostalgic\" formed?\n",
"Rewritten query: Who was the university that gave rise to the group known for the hit song \"Nostalgic\"?\n",
"\n",
"Question 37\n",
"Original query: Where was the lead male actor whose name was Frank in Guys and Dolls born?\n",
"Rewritten query: \"Where was Frank Sinatra, the lead male actor in 'Guys and Dolls', born?\"\n",
"\n",
"Question 38\n",
"Original query: Where was the American scout and bison hunter, whom Robert W. Patten claimed to be as historically significant as born?\n",
"Rewritten query: \"Where was Buffalo Bill Cody, an American scout and bison hunter born, who Robert W. Patten believed to be as historically significant as George Washington and Ulysses S. Grant?\"\n",
"\n",
"Question 39\n",
"Original query: Which period in German history came to an end thanks in part to material aid supplied to the Soviet Union by an Arctic convoy?\n",
"Rewritten query: \"What period in German history ended thanks to the Arctic convoy's supply relief to the Soviet Union?\"\n",
"\n",
"Question 40\n",
"Original query: In what Super Bowl game did John Jett win a Super Bowl ring at the Sun Devil Stadium?\n",
"Rewritten query: \"In what Super Bowl game did John Jett win a Super Bowl ring at the Sun Devil Stadium, and what team did he play for?\"\n",
"\n",
"Question 41\n",
"Original query: Orchard Estates is located within a county in New Jersey with a population of what in 2016?\n",
"Rewritten query: What was the population of Mercer County, New Jersey in 2016, where Orchard Estates is located?\n",
"\n",
"Question 42\n",
"Original query: House Party 3 features the debut of an American actor born in what year?\n",
"Rewritten query: Who is the American actor that made their debut in House Party 3, and what year were they born in?\n",
"\n",
"Question 43\n",
"Original query: What is the meaning of the nickname of Messalina's second-cousin?\n",
"Rewritten query: Here is the rewritten query added to the database index:\n",
"“What is the origin and meaning of the nickname of Messalina’s second-cousin?”\n",
"\n",
"Question 44\n",
"Original query: What university has its home in Gaetano Callani's native city?\n",
"Rewritten query: Which university is based in the natal town of Gaetano Callani?\n",
"\n",
"Question 45\n",
"Original query: What year did the singer behind \"Minstrel Boy\" reach fame in?\n",
"Rewritten query: When did the artist behind the song \"Minstrel Boy\" achieve success and widespread recognition?\n",
"\n",
"Question 46\n",
"Original query: Ann Willing Bingham, was an American socialite from Philadelphia, she was the eldest daughter of Thomas Willing, president of the First Bank of the United States, and a correspondent of the principal author of the Declaration of Independence, and later served as the third President of the United States?\n",
"Rewritten query: Who was the daughter of Thomas Willing, a president of the First Bank of the United States and a correspondent of the third president of the United States, and how was she related to the principal author of the Declaration of Independence?\n",
"\n",
"Question 47\n",
"Original query: Were Howard Hawks and Armand Schaefer the same nationality?\n",
"Rewritten query: Who were the respective nationalities of Howard Hawks and Armand Schaefer?\n",
"\n",
"Question 48\n",
"Original query: How many studio albums did the youngest winner of the Teen Choice Awards have?\n",
"Rewritten query: Compute the number of studio albums held by the youngest winner of the Teen Choice Awards. \n",
"Ensure this calculation specifically focuses on studio albums and excludes any EPs or compilations. \n",
"Moreover, clarify that the count should be tabulated only for the winner, not all recipients of the Teen Choice Awards. \n",
"Please structure the query to maximize the specificity and detail of the information retrieval.\n",
"\n",
"Question 49\n",
"Original query: Who was the original composer of \"Pictures at an Exhibition\"?\n",
"Rewritten query: Who was the original composer of the classical piece \"Pictures at an Exhibition,\" its relevance to classical music, and any subsequent notable revisions or performances?\n",
"\n",
"Question 50\n",
"Original query: Thesongadayproject had a cover version of the song Catch the Wind by the singer and guitarist of what nationality?\n",
"Rewritten query: Who was the singer and guitarist of Catch the Wind for Thesongadayproject, and what was their nationality?\n",
"\n",
"Question 51\n",
"Original query: Which English actor co-wrote and also played in The Dictator?\n",
"Rewritten query: \"Which English actor co-wrote and played the lead role in The Dictator, the 2012 political comedy film directed by Larry Charles?\"\n",
"\n",
"Question 52\n",
"Original query: Once Upon a Time in America is a crime drama film directed by the inventor of what?\n",
"Rewritten query: Who was the director of the 1974 crime drama film \"Once Upon a Time in America,\" and what was their relation to the subject of organised crime in America?\n",
"\n",
"Question 53\n",
"Original query: What year was the college that M.C. Richards attended founded?\n",
"Rewritten query: \"What year was the institution founded that M.C. Richards attended, specifically referring to his undergraduate studies?\"\n",
"\n",
"Question 54\n",
"Original query: Who plays Tonya Harding in the 2017 biographical film entitled I, Tonya?\n",
"Rewritten query: Who is the actress portraying figure skater Tonya Harding in the 2017 movie I, Tonya?\n",
"\n",
"Question 55\n",
"Original query: Which author is famous for writing historical spy novels, Alan Furst or H. P. Lovecraft?\n",
"Rewritten query: Who is the author known for their spy novels, specifically focusing on historical events, between Alan Furst and H.P. Lovecraft?\n",
"\n",
"Question 56\n",
"Original query: The professional boxer James Tillis had a notable win by TKO 7 in 1980 over a boxer that challenged the world heavyweight championship in what year?\n",
"Rewritten query: \"In what year did James 'Bonecrusher' Tillis defeat a boxer who later challenged for the world heavyweight championship?\"\n",
"\n",
"Question 57\n",
"Original query: Are John H. Auer and Richard Curtis both involved in the film industry?\n",
"Rewritten query: Who are John H. Auer and Richard Curtis related to in the film industry, and what organizations or entities do they work with?\n",
"\n",
"Question 58\n",
"Original query: This English comedian wrote the British sitcom \"Absolutely Fabulous\"\n",
"Rewritten query: Who is the English comedian that created the BBC show \"Absolutely Fabulous\" ?\n",
"\n",
"Question 59\n",
"Original query: What sports show did the hockey player born on January 26, 1961 co-host?\n",
"Rewritten query: Who was the hockey player born on January 26, 1961, and what sports show did they co-host?\n",
"\n",
"Question 60\n",
"Original query: Are Roger Ebert and Anna Akhmatova both writers?\n",
"Rewritten query: Who is Roger Ebert and what context did Anna Akhmatova write in? Did they both write about similar topics or genres?\n",
"\n",
"Question 61\n",
"Original query: Don Loren Harper's credits include a 1998 American science fiction disaster film that was directed by Jerry Bruckheimer and directed by who?\n",
"Rewritten query: What film directorial credits does Jerry Bruckheimer have from 1998, specifically in the science fiction genre?\n",
"\n",
"Question 62\n",
"Original query: Which band formed first Suede or Hard-Fi ?\n",
"Rewritten query: \"When did Suede and Hard-Fi form? Which one came first?\"\n",
"\n",
"Question 63\n",
"Original query: A pumapard is hybrid of a leopard and the second-heaviest cat in the New World, after what?\n",
"Rewritten query: What is the second-heaviest cat in the New World, after which the pumapard, a hybrid of a leopard, is named?\n",
"\n",
"Question 64\n",
"Original query: Who was the head coach of the team that lost Super Bowl XIX?\n",
"Rewritten query: Who was the head coach of the losing team at Super Bowl XIX, and also, what was the name of this team?\n",
"\n",
"Question 65\n",
"Original query: Saving Grace starred what American actor who played William Adama in Battlestar Galactica?\n",
"Rewritten query: Who did American actor Edward James Olmos portray in the science fiction series Battlestar Galactica?, in relation to his role in Saving Grace.\n",
"\n",
"Question 66\n",
"Original query: The director that created the character Thirteenth Aunt was born in what year?\n",
"Rewritten query: \"The year of birth of the filmmaker who created the Thirteenth Aunt character is required.\".\n",
"\n",
"Question 67\n",
"Original query: Great Limber is situated 8 miles east from a town with how many households according to the 2001 UK census ?\n",
"Rewritten query: According to the 2001 UK census, how many households were there in the town located 8 miles east of Great Limber?\n",
"\n",
"Question 68\n",
"Original query: Do Phil Collins and Bobby Fuller have the same nationality?\n",
"Rewritten query: Who is the nationality of both Phil Collins and Bobby Fuller?\n",
"\n",
"Question 69\n",
"Original query: Which second governor was the territory, split into Mississippi and Alabama in 1819, named after?\n",
"Rewritten query: \"Who was the second governor of the territory that included Mississippi and Alabama and had that territory named after them?\"\n",
"\n",
"Question 70\n",
"Original query: In which county is this village where Reid Gorecki grew up located?\n",
"Rewritten query: \"In which county is the village where Reid Gorecki grew up, located? Use the county name as well as the state for added context.\"\n",
"\n",
"Question 71\n",
"Original query: Edmundo \"Eddie\" Rodriguez was born in what Mexican state?\n",
"Rewritten query: Where was Edmundo \"Eddie\" Rodriguez born, that is now part of the Mexican state of Nuevo Leon?\n",
"\n",
"Question 72\n",
"Original query: Æthelstan Half-King left his position and became a monk after the death of a king who ruled England starting which year ?\n",
"Rewritten query: When did Æthelstan Half-King become a monk after leaving his position, specifically ruling over England starting from which year?\n",
"\n",
"Question 73\n",
"Original query: What friends star's wedding was photographed by Robert Evans?\n",
"Rewritten query: Who was married on Friends whose wedding photographs were taken by Robert Evans?\n",
"\n",
"Question 74\n",
"Original query: For which film did the actress, who recorded the song Coming Home, receive the Best Actress Academy Award ?\n",
"Rewritten query: Here is the rewritten query, adding more context and clarity: \n",
"\"Which film directed by Clint Eastwood led to the Best Actress Academy Award win for the actress who performed the song 'Coming Home'?''\n",
"\n",
"Question 75\n",
"Original query: Childersburg, Alabama is near the munitions plant that was operated during what event?\n",
"Rewritten query: \"Childersburg, Alabama is located close to the munitions plant that operated during which war era: World War II or the Vietnam War?\"\n",
"\n",
"Question 76\n",
"Original query: What year was the writer of the music for the 1956 song \"Tonight\" born?\n",
"Rewritten query: \"What year did the composer of the song \"Tonight\", which was released in 1956, birth occur?\\\"\"\n",
"\n",
"Question 77\n",
"Original query: What documentary had more directors: Dalai Lama Renaissance or Salesman?\n",
"Rewritten query: Please specify if you are asking about the number of credited directors or the total number of directors, editors, producers, etc. This will help clarify the nature of the comparison and ensure accurate retrieval of information from the database.\n",
"\n",
"Question 78\n",
"Original query: What battle was fought by Greenwich Union soldiers, in which General Mead's Army of the Potomac defeated General Lee's North Virginia Army resulting in the largest number of casualties in a single war?\n",
"Rewritten query: \"The Battle of Gettysburg, fought from July 1-3, 1863, led to the Union Army of the Potomac, under General George Meade's command, facing off against the Confederate Army of North Virginia, led by General Robert E. Lee. This pivotal moment in the American Civil War resulted in heavy casualties and is often considered a turning point in favor of the Union forces. I would like more information about this battle, specifically relating to the actions of the Army of the Potomac and the North Virginia Army, and their respective commanders.\"\n",
"\n",
"Question 79\n",
"Original query: Baylor University and Duke University, are both what type of universities?\n",
"Rewritten query: What types of universities are both Baylor and Duke? Given their religious affiliations, provide information on their specific categorizations within the larger category of private religious universities.\n",
"\n",
"Question 80\n",
"Original query: Jack Gelber was a professor at what public university system of New York City?\n",
"Rewritten query: \"Jack Gelber's teaching career was associated with the City University of New York (CUNY) system.\"\n",
"\n",
"Question 81\n",
"Original query: Are Dalton Trumbo and Herman Wouk both screenwriters?\n",
"Rewritten query: Who authored both novel and screenplay for film adaptations in the same genre?\n",
"\n",
"Question 82\n",
"Original query: Kihachi Okamoto and Victor Fleming were both what?\n",
"Rewritten query: \"Kihachi Okamoto and Victor Fleming were both renowned directors who have left their mark on the history of cinema, especially in the genres of thriller and adventure.\"\n",
"\n",
"Question 83\n",
"Original query: When was the English-French actress born, that starred in the 1975 film \"Sept morts sur ordonnance\", who had a prolific career as an actress in British and French cinema?\n",
"Rewritten query: Here is the rewritten query provided by the user: \n",
"\n",
"What year was the English-French actress, who starred in the 1975 film \"Sept morts sur ordonnance\", born?\n",
"\n",
"Question 84\n",
"Original query: During what period did the mountain range in which Isoetes valida is primarily found first form?\n",
"Rewritten query: When were the primary formation years of the geological period in which the mountain range that hosts Isoetes valida developed?\n",
"\n",
"Question 85\n",
"Original query: What profession does Emmanuel Chabrier and Leoš Janáček have in common?\n",
"Rewritten query: What shared profession do Emmanuel Chabrier and Leoš Janáček have?\n",
"\n",
"Question 86\n",
"Original query: What building has an Art Deco style and was appraised by Robert Von Ancken?\n",
"Rewritten query: The Empire State Building, with its Art Deco design, was appraised by Robert Von Ancken.\n",
"\n",
"Question 87\n",
"Original query: Who had more occupations, Lewis Mumford or William Carlos Williams?\n",
"Rewritten query: Who had more careers and occupations, Lewis Mumford (literary critic, historian, sociologist, and philosopher) or William Carlos Williams (medical doctor, poet, translator, writer, and painter)?.\n",
"\n",
"Question 88\n",
"Original query: Are both Margaret Atwood and Alberto Moravia novelists?\n",
"Rewritten query: \"Are both Margaret Atwood and Alberto Moravia famous novelists who have received nominations for the Nobel Prize in Literature?\"\n",
"\n",
"Question 89\n",
"Original query: Who was Joe Cornish's co-writer of \"Ant-Man\" that served as head writer for \"Saturday Night Live\"?\n",
"Rewritten query: Who was the head writer for \"Saturday Night Live\" that co-wrote the screenplay for \"Ant-Man\" with Joe Cornish?\n",
"\n",
"Question 90\n",
"Original query: Which Chinese city is the capital of the Jiangsu province, Nanjing or Pu'er City\n",
"Rewritten query: Which city serves as the political capital of Jiangsu province: Nanjing or Pu'er City?\n",
"\n",
"Question 91\n",
"Original query: The composer of A Survivor from Warsaw emigrated to the United States in what year?\n",
"Rewritten query: \"What was the year of Emanuel Ax's emigration to the United States, where he settled as a musical prodigy and composer of A Survivor from Warsaw?\"\n",
"\n",
"Question 92\n",
"Original query: Thomas D'Alesandro Jr. is the father of the Minority Leader who represents what district?\n",
"Rewritten query: Who is the current Minority Leader of the United States House of Representatives, and what congressional district do they represent?\n",
"\n",
"Question 93\n",
"Original query: The Old Grist Mill is a historic mill building in which town in York Country, Maine\n",
"Rewritten query: \"The Old Grist Mill is a historic landmark in York County, Maine. It is located in the town of York, Maine.\"\n",
"\n",
"Question 94\n",
"Original query: Martin Hannett produced Joy Division's studio albums, who is the leader singer of Joy Division?\n",
"Rewritten query: Who is the lead singer of Joy Division whose albums were produced by Martin Hannett?\n",
"\n",
"Question 95\n",
"Original query: The Witches of Eastwick, based on john Updike's novel, was Directed by which director.?\n",
"Rewritten query: Who directed the 1992 film The Witches of Eastwick, based on the novel by John Updike?\n",
"\n",
"Question 96\n",
"Original query: Two of the editors of the journal Thesis Eleven: Critical Theory and Historical Sociology are affiliated with a university founded in what year?\n",
"Rewritten query: \"The university founded in what year is affiliated with two editors of the journal Thesis Eleven: Critical Theory and Historical Sociology\"\n",
"\n",
"Question 97\n",
"Original query: Have the two famous people Franco Rossi and Rob Reiner both directed films?\n",
"Rewritten query: \"Have Franco Rossi and Rob Reiner, both well-known figures, directed any films together?\"\n",
"\n",
"Question 98\n",
"Original query: how is Walking on Thin Ice and The Dakota related?\n",
"Rewritten query: How are the Walking on Thin Ice song and the film The Dakota related, if at all? Specify the potential subject matter or contextual details.\n",
"\n",
"Question 99\n",
"Original query: Who worked in both film and television, Ken Russell or Ralph Ceder?\n",
"Rewritten query: Which performer, active in both film and television, should we focus on: Ken Russell or Ralph Ceder?\n",
"\n",
"Question 100\n",
"Original query: Jacques-Louis was a founding member of the geological society before returning to France after what period of French history that followed the fall of Napoleon in 1814 until the July revolution of what year?\n",
"Rewritten query: \"Between the fall of Napoleon in 1814 and the July Revolution in 1830, Jacques-Louis was a founding member of what is now the French Geological Society, who returned to France after this period of history.\"\n",
"\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"# Compute the final metric values\n",
"final_precision = np.mean(precision_values)\n",
"final_recall = np.mean(recall_values)\n",
"final_ndcg = np.mean(ndcg_values)\n",
"final_mrr = np.mean(mrr_values)\n",
"\n",
"print(f\"Final Precision@{k}: {round(final_precision * 100, 2)}\")\n",
"print(f\"Final Recall@{k}: {round(final_recall * 100, 2)}\")\n",
"print(f\"Final nDCG@{k}: {round(final_ndcg * 100, 2)}\")\n",
"print(f\"Final MRR@{k}: {round(final_mrr * 100, 2)}\")"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "7b2de49b-1d6b-419c-b3e3-8949d45bbdf3",
"id": "foVnhkcQBLNP"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Final Precision@5: 16.2\n",
"Final Recall@5: 73.5\n",
"Final nDCG@5: 64.37\n",
"Final MRR@5: 61.77\n"
]
}
]
},
{
"cell_type": "code",
"source": [],
"metadata": {
"id": "Wby69ku0FFN9"
},
"execution_count": null,
"outputs": []
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment