Skip to content

Instantly share code, notes, and snippets.

@SlyEcho

SlyEcho/jeopardy.ipynb

Last active April 25, 2023 09:28
Show Gist options
  • Save SlyEcho/a1e6ac9e44eb48a6769b44b61050b635 to your computer and use it in GitHub Desktop.
Save SlyEcho/a1e6ac9e44eb48a6769b44b61050b635 to your computer and use it in GitHub Desktop.
Download ZIP
Jeopardy eval
Display the source blob
Display the rendered blob
Raw
jeopardy.ipynb
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Jeopardy eval\n",
"\n",
"This is evaluating the Jeopardy dataset from https://github.com/aigoopy/llm-jeopardy using llama.cpp and langchain."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"from langchain.llms.base import LLM\n",
"from typing import Optional, List, Mapping, Any\n",
"import subprocess\n",
"import tempfile\n",
"\n",
"class LlamaCppOfflineLLM(LLM):\n",
" \n",
" path: str\n",
" model: str\n",
" context: int = 2048\n",
" threads: int = 4\n",
" batch: int = 512\n",
" limit: int = 128\n",
" temp: float = 0.9\n",
"\n",
" @property\n",
" def _llm_type(self) -> str:\n",
" return \"custom\"\n",
" \n",
" def _call(self, prompt: str, stop: Optional[List[str]]) -> str:\n",
" with tempfile.NamedTemporaryFile(mode='w', encoding='UTF-8', suffix='.txt') as tmp:\n",
" tmp.write(prompt)\n",
" tmp.flush()\n",
" \n",
" cmd = [\n",
" self.path,\n",
" \"--model\", self.model,\n",
" \"--ctx_size\", str(self.context),\n",
" \"--threads\", str(self.threads),\n",
" \"--batch_size\", str(self.batch),\n",
" \"--n_predict\", str(self.limit),\n",
" \"--temp\", str(self.temp),\n",
" \"--file\", tmp.name,\n",
" ]\n",
"\n",
" if stop is not None:\n",
" for s in stop:\n",
" cmd.extend([\"--reverse-prompt\", s])\n",
" #print(f\"{cmd}\")\n",
" result = subprocess.run(\n",
" cmd,\n",
" stdin=subprocess.DEVNULL,\n",
" stdout=subprocess.PIPE,\n",
" stderr=subprocess.DEVNULL,\n",
" encoding='UTF-8'\n",
" )\n",
" \n",
" if (result.returncode != 0):\n",
" return \"error\"\n",
" result = result.stdout[len(prompt):].lstrip()\n",
" if stop is not None:\n",
" for s in stop:\n",
" i = result.find(s)\n",
" if i >= 0:\n",
" result = result[0:i]\n",
" return result.rstrip()\n",
"\n",
" @property\n",
" def _identifying_params(self) -> Mapping[str, Any]:\n",
" return {\"model\": self.model}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Modify these paths for your system:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"LLAMA_PATH=\"/home/henri/src/llama.cpp/build/bin/main\"\n",
"ALPACA_MODEL=\"/home/henri/src/llama.cpp/models/alpaca-native-7b-q4_0.bin\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This is the prompting setup. I didn't think it was necessary to transform the questions into normal questions, instead I let the LLM learn the format of the quiz.\n",
"\n",
"This turns out to be very easy, actually."
]
},
{
"cell_type": "code",
"execution_count": 88,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
"Prompt after formatting:\n",
"\u001b[32;1m\u001b[1;3mBelow is an instruction that describes a task. Write a response that appropriately completes the request. We are playing a game called Jeopardy where the question is a statement and the answer is in the form of a question.\n",
"\n",
"### Instruction:\n",
"In the category Geograpy we have: This is the largest city in Europe.\n",
"\n",
"### Response:\n",
"What is Moscow?\n",
"\n",
"### Instruction:\n",
"In the category Animals we have: This is the most popular pet in the US.\n",
"\n",
"### Response:\n",
"What is a dog?\n",
"\n",
"### Instruction:\n",
"In the category People we have: He is the richest man in the World.\n",
"\n",
"### Response:\n",
"Who is Jeff Bezos?\n",
"\n",
"### Instruction:\n",
"In the category History we have: The first leap year was on this year.\n",
"\n",
"### Response:\n",
"When was 45 BC?\n",
"\n",
"### Instruction:\n",
"In the category Technology we have: The iPhone was the first phone with this.\n",
"\n",
"### Response:\n",
"\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n"
]
},
{
"data": {
"text/plain": [
"'What is touchscreen capability?'"
]
},
"execution_count": 88,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langchain import FewShotPromptTemplate, LLMChain, PromptTemplate\n",
"\n",
"#{instruction}\n",
"\n",
"prompt = FewShotPromptTemplate(\n",
" prefix= \"Below is an instruction that describes a task. Write a response that appropriately completes the request. We are playing a game called Jeopardy where the question is a statement and the answer is in the form of a question.\",\n",
" examples= [\n",
" { \"category\": \"Geograpy\", \"answer\": \"This is the largest city in Europe.\", \"question\": \"What is Moscow?\" },\n",
" { 'category': 'Animals', 'answer': 'This is the most popular pet in the US.', 'question': 'What is a dog?' },\n",
" { 'category': 'People', 'answer': 'He is the richest man in the World.', 'question': 'Who is Jeff Bezos?' },\n",
" { 'category': 'History', 'answer': 'The first leap year was on this year.', 'question': 'When was 45 BC?' },\n",
" ],\n",
" example_prompt=PromptTemplate(\n",
" input_variables=['category', 'answer', 'question'],\n",
" template = \"### Instruction:\\nIn the category {category} we have: {answer}\\n\\n### Response:\\n{question}\"\n",
" ),\n",
" suffix= \"### Instruction:\\nIn the category {category} we have: {answer}\\n\\n### Response:\\n\",\n",
" input_variables= ['category', 'answer']\n",
")\n",
"\n",
"chat = LLMChain(\n",
" llm=LlamaCppOfflineLLM(path=LLAMA_PATH, model=ALPACA_MODEL, threads=8, limit=10),\n",
" prompt=prompt,\n",
" verbose=True,\n",
")\n",
"chat.predict(category='Technology', answer='The iPhone was the first phone with this.', stop='###')"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## The data\n",
"\n",
"This was copied from the spreadsheet and converted to JSON and pasted here."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"data = [\n",
" {\n",
" \"category\": \"The Oscars\",\n",
" \"answer\": \"Born in 1932 & the son of a percussionist in the CBS radio orchestra, he's been nominated for 53 Oscars\",\n",
" \"question\": \"Who is John Williams?\"\n",
" },\n",
" {\n",
" \"category\": \"English Literature\",\n",
" \"answer\": \"It says, \\\"The mind is its own place, & in itself can make a heaven of hell, a hell of heaven. What matter where, if I be still the same\\\"\",\n",
" \"question\": \"What is Paradise Lost?\"\n",
" },\n",
" {\n",
" \"category\": \"Writers' Lesser-Known Works\",\n",
" \"answer\": \"Known for more philosophical works, he wrote the play \\\"La Mandragola\\\", in which Florentines are rewarded for immoral actions\",\n",
" \"question\": \"Who is Niccolò Machiavelli?\"\n",
" },\n",
" {\n",
" \"category\": \"Exploration\",\n",
" \"answer\": \"James Cook's account of a 1774 visit here records an object \\\"near 27 feet long, and upwards of 8 feet over the breast or shoulders\\\"\",\n",
" \"question\": \"What is Easter Island (Rapa Nui)?\"\n",
" },\n",
" {\n",
" \"category\": \"The Bill of Rights\",\n",
" \"answer\": \"England's \\\"Bloody Assizes\\\" & a 1685 life sentence for perjury were 2 main origins of this amendment to the U.S. Constitution\",\n",
" \"question\": \"What is the Eighth Amendment?\"\n",
" },\n",
" {\n",
" \"category\": \"Nobel Peace Prize Winners\",\n",
" \"answer\": \"At times they each lived on Vilakazi St. in Soweto, so it claims to be the world's only street home to 2 Nobel Peace Prize winners\",\n",
" \"question\": \"Who are Nelson Mandela & Desmond Tutu?\"\n",
" },\n",
" {\n",
" \"category\": \"Famous Names\",\n",
" \"answer\": \"In 1966, the year of his death, he shared plans for an experimental prototype community in Florida\",\n",
" \"question\": \"Who is Walt Disney?\"\n",
" },\n",
" {\n",
" \"category\": \"Geography\",\n",
" \"answer\": \"Of the 13 nations through which the Equator passes, it's the only one whose coastline borders the Caribbean Sea\",\n",
" \"question\": \"What is Colombia?\"\n",
" },\n",
" {\n",
" \"category\": \"Fashion History\",\n",
" \"answer\": \"These decorative items get their name from their origin in the port city of Strasbourg, on the border of France & Germany\",\n",
" \"question\": \"What are rhinestones?\"\n",
" },\n",
" {\n",
" \"category\": \"Movies of the '80s\",\n",
" \"answer\": \"Based on an off-Broadway play with just 3 characters, it won the Best Picture Oscar & the actors in all 3 roles were nominated\",\n",
" \"question\": \"What is Driving Miss Daisy?\"\n",
" },\n",
" {\n",
" \"category\": \"Novelists\",\n",
" \"answer\": \"A 2012 book review noted subjects that \\\"sparked his ire\\\": capital punishment, big tobacco & \\\"the plight of the unjustly convicted\\\"\",\n",
" \"question\": \"Who is John Grisham?\"\n",
" },\n",
" {\n",
" \"category\": \"20th Century Eponyms\",\n",
" \"answer\": \"A 1940 headline about this included \\\"failure\\\", \\\"liability when it came to offense\\\" & \\\"stout hearts no match for tanks\\\"\",\n",
" \"question\": \"What is the Maginot Line?\"\n",
" },\n",
" {\n",
" \"category\": \"City History\",\n",
" \"answer\": \"Over 700 years after its traditional 1252 founding date, this port city became associated with a psychological response\",\n",
" \"question\": \"What is Stockholm?\"\n",
" },\n",
" {\n",
" \"category\": \"Brand Names\",\n",
" \"answer\": \"The success of this brand has its roots with a hydrotherapy pump its cofounder created for his son, who had arthritis\",\n",
" \"question\": \"What is Jacuzzi?\"\n",
" },\n",
" {\n",
" \"category\": \"American Authors\",\n",
" \"answer\": \"In a periodical in 1807, he called New York City \\\"Gotham, Gotham! Most enlightened of cities\\\"\",\n",
" \"question\": \"Who is Washington Irving?\"\n",
" },\n",
" {\n",
" \"category\": \"Symbols\",\n",
" \"answer\": \"In math, it's a rotated V; in society, it's a feeling of some marginalized or underrepresented people\",\n",
" \"question\": \"What is \\\"less than\\\"?\"\n",
" },\n",
" {\n",
" \"category\": \"Movie Theme Songs\",\n",
" \"answer\": \"Monty Norman, the composer of this character's theme, said the staccato riff conveyed sexiness, mystery & ruthlessness\",\n",
" \"question\": \"Who is James Bond?\"\n",
" },\n",
" {\n",
" \"category\": \"American Novelists\",\n",
" \"answer\": \"He served with an airman named Yohannan in World War II & despite what readers might think, he said he enjoyed his service\",\n",
" \"question\": \"Who is Joseph Heller?\"\n",
" },\n",
" {\n",
" \"category\": \"Medieval Places\",\n",
" \"answer\": \"One of the participants in an 1170 event at this place said, \\\"Let us away, knights; he will rise no more\\\"\",\n",
" \"question\": \"What is Canterbury, England? (Canterbury Cathedral)\"\n",
" },\n",
" {\n",
" \"category\": \"Countries of Africa\",\n",
" \"answer\": \"At one time a province of the Roman Empire, this kingdom is known to Arabic scholars as Al-Maghrib Al-Aqsa, \\\"the far west\\\"\",\n",
" \"question\": \"What is Morocco?\"\n",
" },\n",
" {\n",
" \"category\": \"Statehood\",\n",
" \"answer\": \"Congress relented in 1890 after this prospective state said it would wait 100 years rather than come in without the women\",\n",
" \"question\": \"What is Wyoming?\"\n",
" },\n",
" {\n",
" \"category\": \"1980s Movies\",\n",
" \"answer\": \"A writer & producer of this movie said he wanted it to be like a Western or James Bond film, \\\"only it takes place in the '30s\\\"\",\n",
" \"question\": \"What is Raiders of the Lost Ark?\"\n",
" },\n",
" {\n",
" \"category\": \"Art Exhibitions\",\n",
" \"answer\": \"In 1898 what's been called the first blockbuster art show was devoted to him & put on for Queen Wilhelmina's coronation\",\n",
" \"question\": \"Who is Rembrandt?\"\n",
" },\n",
" {\n",
" \"category\": \"Countries of the World\",\n",
" \"answer\": \"Part of the largest contiguous land empire during the 1200s & 1300s, today it's the world's second-largest landlocked country\",\n",
" \"question\": \"What is Mongolia?\"\n",
" },\n",
" {\n",
" \"category\": \"Literature\",\n",
" \"answer\": \"A 2006 book was titled \\\"The Poem That Changed America:\\\" this \\\"Fifty Years Later\\\"\",\n",
" \"question\": \"What is \\\"Howl\\\"?\"\n",
" },\n",
" {\n",
" \"category\": \"Invasions\",\n",
" \"answer\": \"Backed by 14,000 troops, he invaded England to restore, in his words, its \\\"religion, laws, and liberties\\\"\",\n",
" \"question\": \"Who is William of Orange?\"\n",
" },\n",
" {\n",
" \"category\": \"Landmarks\",\n",
" \"answer\": \"After its completion in the late 19th c., it was called \\\"a truly tragic street lamp\\\" & a \\\"high & skinny pyramid of iron ladders\\\"\",\n",
" \"question\": \"What is the Eiffel Tower?\"\n",
" },\n",
" {\n",
" \"category\": \"Geographic Name's the Same\",\n",
" \"answer\": \"The busiest passenger port in the U.K., it shares its name with a capital of one of the original 13 states\",\n",
" \"question\": \"What is Dover?\"\n",
" },\n",
" {\n",
" \"category\": \"Names in the Bookstore\",\n",
" \"answer\": \"This man made lists, perhaps to cope with depression; a set of lists he published in 1852 made his name synonymous with a type of book\",\n",
" \"question\": \"Who is Peter Mark Roget?\"\n",
" },\n",
" {\n",
" \"category\": \"U.S. History\",\n",
" \"answer\": \"An 1869 presidential pardon was granted to this man, due in part to a plea by the Medical Society of Harford County, Maryland\",\n",
" \"question\": \"Who is Dr. Samuel Mudd?\"\n",
" },\n",
" {\n",
" \"category\": \"American Literature\",\n",
" \"answer\": \"Letters, pocket knives, C rations & steel helmets are among the tangible items referred to in the title of this modern war classic\",\n",
" \"question\": \"What is The Things They Carried?\"\n",
" },\n",
" {\n",
" \"category\": \"Nonfiction\",\n",
" \"answer\": \"It has the line, \\\"The discovery of America…opened up fresh ground for the rising bourgeoisie\\\"\",\n",
" \"question\": \"What is The Communist Manifesto\"\n",
" },\n",
" {\n",
" \"category\": \"Laws in U.S. History\",\n",
" \"answer\": \"A radical Republican championed this 1875 act but the Supreme Court struck it down in 1883; a new version was passed 81 years later\",\n",
" \"question\": \"What is the Civil Rights Act?\"\n",
" },\n",
" {\n",
" \"category\": \"Names of Myth\",\n",
" \"answer\": \"Her brothers, Castor & Pollux, saved her after Theseus stole her away as a kid; a larger force would seek her later in life\",\n",
" \"question\": \"Who is Helen of Troy?\"\n",
" },\n",
" {\n",
" \"category\": \"African Countries\",\n",
" \"answer\": \"Once Africa's largest country in area, it dropped to third in 2011 when a portion of it declared independence\",\n",
" \"question\": \"What is Sudan?\"\n",
" },\n",
" {\n",
" \"category\": \"The Ancient World\",\n",
" \"answer\": \"The ancient writer Galen said books on ships arriving to this city's port were seized, originals kept & copies returned\",\n",
" \"question\": \"What is Alexandria?\"\n",
" },\n",
" {\n",
" \"category\": \"Famous Names\",\n",
" \"answer\": \"For a special 1970s cookbook, he provided one simple recipe -- a can of Campbell's tomato soup & 2 cans of milk\",\n",
" \"question\": \"Who is Andy Warhol?\"\n",
" },\n",
" {\n",
" \"category\": \"People & Places\",\n",
" \"answer\": \"Thought to descend from people of Southeast Asia, the Chamorro make up this U.S. territory's largest ethnic group\",\n",
" \"question\": \"What is Guam?\"\n",
" },\n",
" {\n",
" \"category\": \"Current World Leaders\",\n",
" \"answer\": \"In office from 2022, the president of this country has taken so many foreign trips a play on his name is \\\"Ferdinand Magellan Jr.\\\"\",\n",
" \"question\": \"What is the Philippines?\"\n",
" },\n",
" {\n",
" \"category\": \"Writers & The South\",\n",
" \"answer\": \"In 1939 he lived on Toulouse Street in the French Quarter & chose the professional name that bonded him to the South\",\n",
" \"question\": \"Who is Tennessee Williams?\"\n",
" },\n",
" {\n",
" \"category\": \"National Parks\",\n",
" \"answer\": \"It's named for a river indigenous people called Mi tse a-da-zi, translated by French-speaking trappers as \\\"Pierre Jaune\\\"\",\n",
" \"question\": \"What is Yellowstone?\"\n",
" },\n",
" {\n",
" \"category\": \"Sports\",\n",
" \"answer\": \"In 2010 they introduced the 4-point shot, 35 feet from the basket\",\n",
" \"question\": \"Who are the Harlem Globetrotters?\"\n",
" },\n",
" {\n",
" \"category\": \"The U.S. Military\",\n",
" \"answer\": \"Losses over Asia in the 1960s led to the establishment of the program known as this at a San Diego naval base in 1969\",\n",
" \"question\": \"What is \\\"Top Gun\\\"?\"\n",
" },\n",
" {\n",
" \"category\": \"Art & Science\",\n",
" \"answer\": \"A craft that visited it was named for Giotto, based on the story that 680 years earlier, the painter depicted it as the Star of Bethlehem\",\n",
" \"question\": \"What is Halley's Comet?\"\n",
" },\n",
" {\n",
" \"category\": \"Words From World War I\",\n",
" \"answer\": \"\\\"Cistern\\\" & \\\"reservoir\\\" were suggested names for a secret invention, but the British preferred this less clumsy monosyllable\",\n",
" \"question\": \"What is \\\"tank\\\"?\"\n",
" },\n",
" {\n",
" \"category\": \"European History\",\n",
" \"answer\": \"Until 1806, some German nobles included among their honors the title of \\\"Elector\\\" for their role in selecting this personage\",\n",
" \"question\": \"What is Holy Roman Emperor?\"\n",
" },\n",
" {\n",
" \"category\": \"Theater History\",\n",
" \"answer\": \"In 1904, wearing a harness, actress Nina Boucicault became the first to play this character onstage\",\n",
" \"question\": \"Who is Peter Pan?\"\n",
" },\n",
" {\n",
" \"category\": \"European Cities\",\n",
" \"answer\": \"Alphabetically the first German city in encyclopedias, it was also the first one taken by the Allies in World War II\",\n",
" \"question\": \"What is Aachen?\"\n",
" },\n",
" {\n",
" \"category\": \"Word Origins\",\n",
" \"answer\": \"This Sanskrit word referring to a spoken word or phrase comes from a word for \\\"to think\\\"\",\n",
" \"question\": \"What is mantra?\"\n",
" },\n",
" {\n",
" \"category\": \"Inventions\",\n",
" \"answer\": \"1917's \\\"Elements of Trench Warfare\\\" said this Old West item was \\\"difficult to destroy\\\" & \\\"difficult to get through\\\"\",\n",
" \"question\": \"What is barbed wire?\"\n",
" },\n",
" {\n",
" \"category\": \"World War II\",\n",
" \"answer\": \"Mimi Reinhard, who never learned to type using more than 2 fingers, produced this with 1,100 names, including hers\",\n",
" \"question\": \"What is Schindler's list?\"\n",
" },\n",
" {\n",
" \"category\": \"Mythology\",\n",
" \"answer\": \"Poseidon carried off the maiden Theophane & turned her into a ewe; their offspring was the source of this mythical object\",\n",
" \"question\": \"What is the Golden Fleece?\"\n",
" },\n",
" {\n",
" \"category\": \"Literature\",\n",
" \"answer\": \"Published in 2011, P.D. James' final novel, \\\"Death Comes to Pemberley\\\", was a sequel to this novel from 200 years earlier\",\n",
" \"question\": \"What is Pride and Prejudice?\"\n",
" },\n",
" {\n",
" \"category\": \"U.S. State Names\",\n",
" \"answer\": \"5 U.S. states have 6-letter names; only these 2 west of the Mississippi River border each other\",\n",
" \"question\": \"What are Oregon & Nevada?\"\n",
" },\n",
" {\n",
" \"category\": \"Word Origins\",\n",
" \"answer\": \"Originally relating to a story of suffering, this word now more commonly refers to strong emotion of any kind\",\n",
" \"question\": \"What is passion?\"\n",
" },\n",
" {\n",
" \"category\": \"World Cinema\",\n",
" \"answer\": \"The 2007 biopic called \\\"La Môme\\\" in France, meaning \\\"The Kid\\\", was released in the U.S. under this other French title\",\n",
" \"question\": \"What is La Vie en Rose?\"\n",
" },\n",
" {\n",
" \"category\": \"History\",\n",
" \"answer\": \"Returning home in 1493, Columbus stopped in the Azores at an island with this name, also something he'd lost off the Haiti coast\",\n",
" \"question\": \"What is Santa Maria?\"\n",
" },\n",
" {\n",
" \"category\": \"Landmarks\",\n",
" \"answer\": \"Pskov & Nizhny Novgorod are 2 of the cities that have a fortress called this\",\n",
" \"question\": \"What is a kremlin?\"\n",
" },\n",
" {\n",
" \"category\": \"Foreign-Born Authors\",\n",
" \"answer\": \"In the 1950s the New York Times said this author \\\"is writing about all lust\\\" & his lecherous narrator \\\"is all of us\\\"\",\n",
" \"question\": \"Who is Vladimir Nabokov?\"\n",
" },\n",
" {\n",
" \"category\": \"Astronomy & Geography\",\n",
" \"answer\": \"At the winter solstice, the sun is in Sagittarius; it once appeared in this constellation, giving a geographic feature its name\",\n",
" \"question\": \"What is Capricorn?\"\n",
" },\n",
" {\n",
" \"category\": \"Television\",\n",
" \"answer\": \"Mike Post combined the sound of a slamming jail door, an anvil & 100 men stomping on a floor for this series that debuted in 1990\",\n",
" \"question\": \"What is Law & Order?\"\n",
" },\n",
" {\n",
" \"category\": \"British Landmarks\",\n",
" \"answer\": \"Like Sir Thomas More, 3 16th century English queens are buried at this location\",\n",
" \"question\": \"What is the Tower of London?\"\n",
" },\n",
" {\n",
" \"category\": \"Early American History\",\n",
" \"answer\": \"In 1692 Increase Mather wrote, \\\"It were better that ten suspected\\\" these \\\"escape, than that one innocent person … be condemned\\\"\",\n",
" \"question\": \"What are witches?\"\n",
" },\n",
" {\n",
" \"category\": \"Geography Mnemonics\",\n",
" \"answer\": \"Mimal, sometimes said to be the silhouette of a chef or elf, stands for Minnesota, Iowa, Missouri, and these 2 states\",\n",
" \"question\": \"What are Arkansas and Louisiana?\"\n",
" },\n",
" {\n",
" \"category\": \"Business Milestones\",\n",
" \"answer\": \"These were first sold in 1908, at a price equivalent to about $27,000 today\",\n",
" \"question\": \"What is the Ford Model T?\"\n",
" },\n",
" {\n",
" \"category\": \"In The Bookstore\",\n",
" \"answer\": \"The name of this author dead since 2013 now appears on books written by a former U.S. marshal & a former Apache helicopter pilot\",\n",
" \"question\": \"Who is Tom Clancy?\"\n",
" },\n",
" {\n",
" \"category\": \"Historic Art\",\n",
" \"answer\": \"The artwork once known in France as \\\"la tapisserie de la Reine Mathilde\\\" is better known as this\",\n",
" \"question\": \"What is the Bayeux Tapestry?\"\n",
" },\n",
" {\n",
" \"category\": \"Pop Stars\",\n",
" \"answer\": \"In 2022 she became the first woman to have a Billboard Top 10 album in 5 decades starting with the 1980s\",\n",
" \"question\": \"Who is Madonna?\"\n",
" },\n",
" {\n",
" \"category\": \"Classic Tale Characters\",\n",
" \"answer\": \"In one 19th century translation, she \\\"perceived the dawn of day and ceased\\\" speaking nearly 1,000 times\",\n",
" \"question\": \"Who is Scheherazade?\"\n",
" },\n",
" {\n",
" \"category\": \"USA\",\n",
" \"answer\": \"Ironically, though this company founded in the 1860s is Moore County, Tennessee's largest employer, Moore is a dry county\",\n",
" \"question\": \"What is Jack Daniel's?\"\n",
" },\n",
" {\n",
" \"category\": \"Historic People\",\n",
" \"answer\": \"After a 1789 event, he wrote, \\\"My first determination was to seek a supply of…water at Tofoa, & afterwards to sail for Tongataboo\\\"\",\n",
" \"question\": \"Who was William Bligh?\"\n",
" },\n",
" {\n",
" \"category\": \"The Movies\",\n",
" \"answer\": \"Laurence Olivier & Ernest Borgnine were considered for the lead role & Sergio Leone to direct for this film that turned 50 in 2022\",\n",
" \"question\": \"What is The Godfather?\"\n",
" },\n",
" {\n",
" \"category\": \"Continental Geography\",\n",
" \"answer\": \"Until a 1903 secession, this country's contiguous territory spanned 2 continents\",\n",
" \"question\": \"What is Colombia?\"\n",
" },\n",
" {\n",
" \"category\": \"Foreign-Born Authors\",\n",
" \"answer\": \"Early in her career she translated romance novels into Spanish, often changing the dialogue to make the heroines smarter\",\n",
" \"question\": \"Who is Isabel Allende?\"\n",
" },\n",
" {\n",
" \"category\": \"Historic Crimes\",\n",
" \"answer\": \"Saying it was stolen by Napoleon, self-styled Italian patriot Vincenzo Peruggia took it in 1911\",\n",
" \"question\": \"What is the Mona Lisa?\"\n",
" },\n",
" {\n",
" \"category\": \"U.S. Bodies of Water\",\n",
" \"answer\": \"Continuing a downward trend, in July 2022 it was at 27% capacity, its lowest level since 1937 when it was first being filled\",\n",
" \"question\": \"What is Lake Mead?\"\n",
" },\n",
" {\n",
" \"category\": \"Gods & Goddesses\",\n",
" \"answer\": \"Each morning she began her ride in her chariot across the sky ahead of her brother Sol, or Helios\",\n",
" \"question\": \"Who is Aurora (or Eos)?\"\n",
" },\n",
" {\n",
" \"category\": \"America At War\",\n",
" \"answer\": \"Until the Civil War, the Jan. 8 date of this battle of dubious military importance but big morale value was a national holiday\",\n",
" \"question\": \"What is the Battle of New Orleans?\"\n",
" },\n",
" {\n",
" \"category\": \"Children's Books\",\n",
" \"answer\": \"Its title character is told \\\"By the time you are real, most of your hair has been loved off…your eyes drop out & you get…shabby\\\"\",\n",
" \"question\": \"What is The Velveteen Rabbit?\"\n",
" },\n",
" {\n",
" \"category\": \"TV Finales\",\n",
" \"answer\": \"In a reunion over 40 years in the making, Dolly Parton appeared as an angel named Agnes in the final episode of this comedy in 2022\",\n",
" \"question\": \"What is Grace and Frankie?\"\n",
" },\n",
" {\n",
" \"category\": \"American Poems\",\n",
" \"answer\": \"In an 1847 poem this character sees her town of Grand-Pré burned, but finally reunites with her beau for a kiss before his death\",\n",
" \"question\": \"Who is Evangeline?\"\n",
" },\n",
" {\n",
" \"category\": \"Famous Names\",\n",
" \"answer\": \"In 2001 he published a book called \\\"Banging Your Head Against a Brick Wall\\\"; in 2002, \\\"Existencilism\\\"\",\n",
" \"question\": \"Who is Banksy?\"\n",
" },\n",
" {\n",
" \"category\": \"Children's Lit\",\n",
" \"answer\": \"The title object of this book \\\"never looked more beautiful…each strand held dozens of bright drops of early morning dew\\\"\",\n",
" \"question\": \"What is Charlotte's Web?\"\n",
" },\n",
" {\n",
" \"category\": \"Classic Songs\",\n",
" \"answer\": \"The shouts of excited children at a 1946 holiday parade are said to have inspired this perennial favorite\",\n",
" \"question\": \"What is \\\"Here Comes Santa Claus\\\"?\"\n",
" },\n",
" {\n",
" \"category\": \"Brand Names\",\n",
" \"answer\": \"Unable to make these candies perfectly round, the confectioner embraced this flawed name for the product\",\n",
" \"question\": \"What are Milk Duds?\"\n",
" },\n",
" {\n",
" \"category\": \"Countries of the World\",\n",
" \"answer\": \"It's home to 58 UNESCO World Heritage Sites, more than any other country; the sites include a volcano & a lagoon\",\n",
" \"question\": \"What is Italy?\"\n",
" },\n",
" {\n",
" \"category\": \"Action Movies\",\n",
" \"answer\": \"Its last line is \\\"If this is their idea of Christmas, I gotta be here for New Year's\\\"\",\n",
" \"question\": \"What is Die Hard?\"\n",
" },\n",
" {\n",
" \"category\": \"Presidential Facts\",\n",
" \"answer\": \"Only 3 presidents have married while in office— John Tyler was the first & he was the last\",\n",
" \"question\": \"Who is Woodrow Wilson?\"\n",
" },\n",
" {\n",
" \"category\": \"19th Century Americans\",\n",
" \"answer\": \"Demonstrating the dignity & humanity of Black Americans, he sat for 160 known photographs, the most of any American in the 19th century\",\n",
" \"question\": \"Who is Frederick Douglass?\"\n",
" },\n",
" {\n",
" \"category\": \"Latin Phrases\",\n",
" \"answer\": \"Originally, this 3-word phrase referred to when a doctor or apothecary substituted one medicine for another\",\n",
" \"question\": \"What is \\\"quid pro quo\\\"?\"\n",
" },\n",
" {\n",
" \"category\": \"1970s Movies\",\n",
" \"answer\": \"A 1975 premiere of this comedy advertised free coconuts for the first thousand in the audience\",\n",
" \"question\": \"What is Monty Python and the Holy Grail?\"\n",
" },\n",
" {\n",
" \"category\": \"Name's The Same\",\n",
" \"answer\": \"A cocktail, an island & a WWII venture originally called \\\"Development of Substitute Materials\\\" all bear this name\",\n",
" \"question\": \"What is Manhattan?\"\n",
" },\n",
" {\n",
" \"category\": \"U.S. Presidents\",\n",
" \"answer\": \"He was sworn in twice as President within 2 years, first by his father & then later by a former U.S. President\",\n",
" \"question\": \"Who is Calvin Coolidge?\"\n",
" },\n",
" {\n",
" \"category\": \"Plays\",\n",
" \"answer\": \"A 1609 story in which an exiled king of Bulgaria creates a sea palace with his magic may have inspired the plot of this play\",\n",
" \"question\": \"What is The Tempest?\"\n",
" },\n",
" {\n",
" \"category\": \"Landmarks\",\n",
" \"answer\": \"In 2009, during a 20th anniversary celebration, it was called \\\"an edifice of fear. On Nov. 9, it became a place of joy\\\"\",\n",
" \"question\": \"What is the Berlin Wall?\"\n",
" },\n",
" {\n",
" \"category\": \"World Capitals\",\n",
" \"answer\": \"Among its nicknames are the \\\"City of Classical Music\\\" &, possibly in honor of a famous resident from 1860 to 1938, the \\\"City of Dreams\\\"\",\n",
" \"question\": \"What is Vienna, Austria?\"\n",
" },\n",
" {\n",
" \"category\": \"Language & Its Meanings\",\n",
" \"answer\": \"Now meaning someone with nocturnal habits, it catches a sleeping dove in Shakespeare's \\\"…Lucrece\\\"\",\n",
" \"question\": \"What is a night owl?\"\n",
" },\n",
" {\n",
" \"category\": \"Flags of Our Hemisphere\",\n",
" \"answer\": \"The stars on this country's flag represent states, 26 of them; unlike the USA's, its \\\"federal district\\\" gets its own 27th star\",\n",
" \"question\": \"What is Brazil?\"\n",
" },\n",
" {\n",
" \"category\": \"Names in U.S. History\",\n",
" \"answer\": \"This father was the only man among the 13 plaintiffs in a class-action case filed in 1951\",\n",
" \"question\": \"Who is Oliver Brown?\"\n",
" },\n",
" {\n",
" \"category\": \"Children's Authors\",\n",
" \"answer\": \"Reversing the story of this heroine she created, Patricia Maclachlan was born on the prairie but spent much of her life in New England\",\n",
" \"question\": \"Who is Sarah? (from Sarah, Plain and Tall)\"\n",
" }\n",
"]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Processing\n",
"\n",
"Then just run the progam on the items one-by-one."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tqdm.notebook import tqdm\n",
"\n",
"chat.verbose = False\n",
"\n",
"results = []\n",
"\n",
"for p in tqdm(data):\n",
" category = p[\"category\"]\n",
" answer = p[\"answer\"]\n",
" correct = p[\"question\"]\n",
" print(f\"{category}: {answer}\")\n",
" predicted = chat.predict(category=category, answer=answer, stop='###')\n",
" print(f\" {predicted} ('{correct}')\")\n",
" results.append({'category': category, 'answer': answer, 'predicted': predicted, 'correct': correct})\n"
]
},
{
"cell_type": "code",
"execution_count": 108,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"pd.set_option('display.max_rows', None)\n",
"results = pd.DataFrame(results)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Evaluation\n",
"\n",
"Looking to see if the answer is correct or not.\n",
"This could probably be done by an LLM itself..."
]
},
{
"cell_type": "code",
"execution_count": 103,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'project coronado'"
]
},
"execution_count": 103,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def clean(s):\n",
" rem = ['what','who','where','when','is','was','were','the','a']\n",
" words = [w if w not in rem else '' for w in s.lower().replace('\"', '').split(' ')]\n",
" return ' '.join(words).lstrip().rstrip('?')\n",
"clean('What is Project Coronado?')"
]
},
{
"cell_type": "code",
"execution_count": 109,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>category</th>\n",
" <th>answer</th>\n",
" <th>predicted</th>\n",
" <th>correct</th>\n",
" <th>Score</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>The Oscars</td>\n",
" <td>Born in 1932 &amp; the son of a percussionist in t...</td>\n",
" <td>Who is John Smith?</td>\n",
" <td>Who is John Williams?</td>\n",
" <td>57.142857</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>English Literature</td>\n",
" <td>It says, \"The mind is its own place, &amp; in itse...</td>\n",
" <td>Who is William Shakespeare?</td>\n",
" <td>What is Paradise Lost?</td>\n",
" <td>31.250000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Writers' Lesser-Known Works</td>\n",
" <td>Known for more philosophical works, he wrote t...</td>\n",
" <td>Who is Machiavelli and what is his</td>\n",
" <td>Who is Niccolò Machiavelli?</td>\n",
" <td>73.684211</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Exploration</td>\n",
" <td>James Cook's account of a 1774 visit here reco...</td>\n",
" <td>What is the Giant's Causeway</td>\n",
" <td>What is Easter Island (Rapa Nui)?</td>\n",
" <td>35.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>The Bill of Rights</td>\n",
" <td>England's \"Bloody Assizes\" &amp; a 1685 life sente...</td>\n",
" <td>What is the First Amendment?</td>\n",
" <td>What is the Eighth Amendment?</td>\n",
" <td>77.419355</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Nobel Peace Prize Winners</td>\n",
" <td>At times they each lived on Vilakazi St. in So...</td>\n",
" <td>Who are Desmond Tutu and Nelson Mand</td>\n",
" <td>Who are Nelson Mandela &amp; Desmond Tutu?</td>\n",
" <td>89.230769</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Famous Names</td>\n",
" <td>In 1966, the year of his death, he shared plan...</td>\n",
" <td>Who was Walt Disney?</td>\n",
" <td>Who is Walt Disney?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Geography</td>\n",
" <td>Of the 13 nations through which the Equator pa...</td>\n",
" <td>What is Ecuador?</td>\n",
" <td>What is Colombia?</td>\n",
" <td>26.666667</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Fashion History</td>\n",
" <td>These decorative items get their name from the...</td>\n",
" <td>What are Strass brooches?</td>\n",
" <td>What are rhinestones?</td>\n",
" <td>58.823529</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Movies of the '80s</td>\n",
" <td>Based on an off-Broadway play with just 3 char...</td>\n",
" <td>What is The Drama Club?</td>\n",
" <td>What is Driving Miss Daisy?</td>\n",
" <td>28.571429</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>Novelists</td>\n",
" <td>A 2012 book review noted subjects that \"sparke...</td>\n",
" <td>Who is John Grisham?</td>\n",
" <td>Who is John Grisham?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>20th Century Eponyms</td>\n",
" <td>A 1940 headline about this included \"failure\",...</td>\n",
" <td>What is Dunkirk?</td>\n",
" <td>What is the Maginot Line?</td>\n",
" <td>21.052632</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>City History</td>\n",
" <td>Over 700 years after its traditional 1252 foun...</td>\n",
" <td>What is Venice, Italy?\\n\\nV</td>\n",
" <td>What is Stockholm?</td>\n",
" <td>16.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>Brand Names</td>\n",
" <td>The success of this brand has its roots with a...</td>\n",
" <td>What is the brand name of the hydrotherapy</td>\n",
" <td>What is Jacuzzi?</td>\n",
" <td>6.060606</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>American Authors</td>\n",
" <td>In a periodical in 1807, he called New York Ci...</td>\n",
" <td>Who is Mark Twain?</td>\n",
" <td>Who is Washington Irving?</td>\n",
" <td>44.444444</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>Symbols</td>\n",
" <td>In math, it's a rotated V; in society, it's a ...</td>\n",
" <td>What is the V for Victory symbol?</td>\n",
" <td>What is \"less than\"?</td>\n",
" <td>20.689655</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>Movie Theme Songs</td>\n",
" <td>Monty Norman, the composer of this character's...</td>\n",
" <td>What is the James Bond theme?</td>\n",
" <td>Who is James Bond?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>American Novelists</td>\n",
" <td>He served with an airman named Yohannan in Wor...</td>\n",
" <td>Who is Ernest Hemingway?</td>\n",
" <td>Who is Joseph Heller?</td>\n",
" <td>27.586207</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>Medieval Places</td>\n",
" <td>One of the participants in an 1170 event at th...</td>\n",
" <td>What is the Battle of Hastings?</td>\n",
" <td>What is Canterbury, England? (Canterbury Cathe...</td>\n",
" <td>29.508197</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>Countries of Africa</td>\n",
" <td>At one time a province of the Roman Empire, th...</td>\n",
" <td>What is Morocco?</td>\n",
" <td>What is Morocco?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>20</th>\n",
" <td>Statehood</td>\n",
" <td>Congress relented in 1890 after this prospecti...</td>\n",
" <td>What is Wyoming?\\n\\n \\n1</td>\n",
" <td>What is Wyoming?</td>\n",
" <td>82.352941</td>\n",
" </tr>\n",
" <tr>\n",
" <th>21</th>\n",
" <td>1980s Movies</td>\n",
" <td>A writer &amp; producer of this movie said he want...</td>\n",
" <td>What is Indiana Jones and the Temple of Doom</td>\n",
" <td>What is Raiders of the Lost Ark?</td>\n",
" <td>43.137255</td>\n",
" </tr>\n",
" <tr>\n",
" <th>22</th>\n",
" <td>Art Exhibitions</td>\n",
" <td>In 1898 what's been called the first blockbust...</td>\n",
" <td>What is Vincent Van Gogh?</td>\n",
" <td>Who is Rembrandt?</td>\n",
" <td>24.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>23</th>\n",
" <td>Countries of the World</td>\n",
" <td>Part of the largest contiguous land empire dur...</td>\n",
" <td>What is the Central Asian nation of Kazakhstan</td>\n",
" <td>What is Mongolia?</td>\n",
" <td>14.285714</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>Literature</td>\n",
" <td>A 2006 book was titled \"The Poem That Changed ...</td>\n",
" <td>What is The Road Not Taken by Robert F</td>\n",
" <td>What is \"Howl\"?</td>\n",
" <td>6.666667</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25</th>\n",
" <td>Invasions</td>\n",
" <td>Backed by 14,000 troops, he invaded England to...</td>\n",
" <td>Who is Louis XIV of France?\\n\\nHe</td>\n",
" <td>Who is William of Orange?</td>\n",
" <td>50.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>26</th>\n",
" <td>Landmarks</td>\n",
" <td>After its completion in the late 19th c., it w...</td>\n",
" <td>What is the Eiffel Tower?</td>\n",
" <td>What is the Eiffel Tower?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>27</th>\n",
" <td>Geographic Name's the Same</td>\n",
" <td>The busiest passenger port in the U.K., it sha...</td>\n",
" <td>What is London Heathrow airport?</td>\n",
" <td>What is Dover?</td>\n",
" <td>21.428571</td>\n",
" </tr>\n",
" <tr>\n",
" <th>28</th>\n",
" <td>Names in the Bookstore</td>\n",
" <td>This man made lists, perhaps to cope with depr...</td>\n",
" <td>Who is Charles Dickens?</td>\n",
" <td>Who is Peter Mark Roget?</td>\n",
" <td>32.258065</td>\n",
" </tr>\n",
" <tr>\n",
" <th>29</th>\n",
" <td>U.S. History</td>\n",
" <td>An 1869 presidential pardon was granted to thi...</td>\n",
" <td>Who is E.D. Cogswell</td>\n",
" <td>Who is Dr. Samuel Mudd?</td>\n",
" <td>21.428571</td>\n",
" </tr>\n",
" <tr>\n",
" <th>30</th>\n",
" <td>American Literature</td>\n",
" <td>Letters, pocket knives, C rations &amp; steel helm...</td>\n",
" <td>What is \"The Things They Carried\" by</td>\n",
" <td>What is The Things They Carried?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>31</th>\n",
" <td>Nonfiction</td>\n",
" <td>It has the line, \"The discovery of America…ope...</td>\n",
" <td>What is Karl Marx's \"The Communist</td>\n",
" <td>What is The Communist Manifesto</td>\n",
" <td>65.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>32</th>\n",
" <td>Laws in U.S. History</td>\n",
" <td>A radical Republican championed this 1875 act ...</td>\n",
" <td>What is the Civil Rights Act of 19</td>\n",
" <td>What is the Civil Rights Act?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>33</th>\n",
" <td>Names of Myth</td>\n",
" <td>Her brothers, Castor &amp; Pollux, saved her after...</td>\n",
" <td>What is Helen of Troy?</td>\n",
" <td>Who is Helen of Troy?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>34</th>\n",
" <td>African Countries</td>\n",
" <td>Once Africa's largest country in area, it drop...</td>\n",
" <td>What is Sudan?</td>\n",
" <td>What is Sudan?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>35</th>\n",
" <td>The Ancient World</td>\n",
" <td>The ancient writer Galen said books on ships a...</td>\n",
" <td>Which is Rome?</td>\n",
" <td>What is Alexandria?</td>\n",
" <td>20.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>36</th>\n",
" <td>Famous Names</td>\n",
" <td>For a special 1970s cookbook, he provided one ...</td>\n",
" <td>Who is Gordon Ramsay?</td>\n",
" <td>Who is Andy Warhol?</td>\n",
" <td>25.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>37</th>\n",
" <td>People &amp; Places</td>\n",
" <td>Thought to descend from people of Southeast As...</td>\n",
" <td>What is Guam?\\n\\nAnswer Question:</td>\n",
" <td>What is Guam?</td>\n",
" <td>30.769231</td>\n",
" </tr>\n",
" <tr>\n",
" <th>38</th>\n",
" <td>Current World Leaders</td>\n",
" <td>In office from 2022, the president of this cou...</td>\n",
" <td>Who is Ferdinand Magellan Jr.?</td>\n",
" <td>What is the Philippines?</td>\n",
" <td>18.181818</td>\n",
" </tr>\n",
" <tr>\n",
" <th>39</th>\n",
" <td>Writers &amp; The South</td>\n",
" <td>In 1939 he lived on Toulouse Street in the Fre...</td>\n",
" <td>Who is Tennessee Williams and what did he do in</td>\n",
" <td>Who is Tennessee Williams?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>40</th>\n",
" <td>National Parks</td>\n",
" <td>It's named for a river indigenous people calle...</td>\n",
" <td>What is Yellowstone National Park?</td>\n",
" <td>What is Yellowstone?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>41</th>\n",
" <td>Sports</td>\n",
" <td>In 2010 they introduced the 4-point shot, 35 f...</td>\n",
" <td>What is the 3-point line?</td>\n",
" <td>Who are the Harlem Globetrotters?</td>\n",
" <td>27.777778</td>\n",
" </tr>\n",
" <tr>\n",
" <th>42</th>\n",
" <td>The U.S. Military</td>\n",
" <td>Losses over Asia in the 1960s led to the estab...</td>\n",
" <td>What is Project X-Ray?</td>\n",
" <td>What is \"Top Gun\"?</td>\n",
" <td>10.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>43</th>\n",
" <td>Art &amp; Science</td>\n",
" <td>A craft that visited it was named for Giotto, ...</td>\n",
" <td>What is an astrolabe?</td>\n",
" <td>What is Halley's Comet?</td>\n",
" <td>30.769231</td>\n",
" </tr>\n",
" <tr>\n",
" <th>44</th>\n",
" <td>Words From World War I</td>\n",
" <td>\"Cistern\" &amp; \"reservoir\" were suggested names f...</td>\n",
" <td>What is tank?</td>\n",
" <td>What is \"tank\"?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>45</th>\n",
" <td>European History</td>\n",
" <td>Until 1806, some German nobles included among ...</td>\n",
" <td>Who is the Holy Roman Emperor Maximilian I?</td>\n",
" <td>What is Holy Roman Emperor?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>46</th>\n",
" <td>Theater History</td>\n",
" <td>In 1904, wearing a harness, actress Nina Bouci...</td>\n",
" <td>Who is Puck from A Midsummer Night</td>\n",
" <td>Who is Peter Pan?</td>\n",
" <td>17.647059</td>\n",
" </tr>\n",
" <tr>\n",
" <th>47</th>\n",
" <td>European Cities</td>\n",
" <td>Alphabetically the first German city in encycl...</td>\n",
" <td>What is Aachen?</td>\n",
" <td>What is Aachen?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>48</th>\n",
" <td>Word Origins</td>\n",
" <td>This Sanskrit word referring to a spoken word ...</td>\n",
" <td>What is Mantra?</td>\n",
" <td>What is mantra?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>49</th>\n",
" <td>Inventions</td>\n",
" <td>1917's \"Elements of Trench Warfare\" said this ...</td>\n",
" <td>What is a trench?</td>\n",
" <td>What is barbed wire?</td>\n",
" <td>23.529412</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50</th>\n",
" <td>World War II</td>\n",
" <td>Mimi Reinhard, who never learned to type using...</td>\n",
" <td>Who is Mimi Reimann?</td>\n",
" <td>What is Schindler's list?</td>\n",
" <td>28.571429</td>\n",
" </tr>\n",
" <tr>\n",
" <th>51</th>\n",
" <td>Mythology</td>\n",
" <td>Poseidon carried off the maiden Theophane &amp; tu...</td>\n",
" <td>What is a magical item?</td>\n",
" <td>What is the Golden Fleece?</td>\n",
" <td>32.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>52</th>\n",
" <td>Literature</td>\n",
" <td>Published in 2011, P.D. James' final novel, \"D...</td>\n",
" <td>What is \"Pride and Prejudice\"</td>\n",
" <td>What is Pride and Prejudice?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>53</th>\n",
" <td>U.S. State Names</td>\n",
" <td>5 U.S. states have 6-letter names; only these ...</td>\n",
" <td>The two U.S. states with six-</td>\n",
" <td>What are Oregon &amp; Nevada?</td>\n",
" <td>22.727273</td>\n",
" </tr>\n",
" <tr>\n",
" <th>54</th>\n",
" <td>Word Origins</td>\n",
" <td>Originally relating to a story of suffering, t...</td>\n",
" <td>What is agony?</td>\n",
" <td>What is passion?</td>\n",
" <td>50.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>55</th>\n",
" <td>World Cinema</td>\n",
" <td>The 2007 biopic called \"La Môme\" in France, me...</td>\n",
" <td>What is La Vie en Rose?</td>\n",
" <td>What is La Vie en Rose?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>56</th>\n",
" <td>History</td>\n",
" <td>Returning home in 1493, Columbus stopped in th...</td>\n",
" <td>What is Hispaniola?</td>\n",
" <td>What is Santa Maria?</td>\n",
" <td>47.619048</td>\n",
" </tr>\n",
" <tr>\n",
" <th>57</th>\n",
" <td>Landmarks</td>\n",
" <td>Pskov &amp; Nizhny Novgorod are 2 of the cities th...</td>\n",
" <td>What is Kremlin?</td>\n",
" <td>What is a kremlin?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>58</th>\n",
" <td>Foreign-Born Authors</td>\n",
" <td>In the 1950s the New York Times said this auth...</td>\n",
" <td>Who is Vladimir Nabokov?</td>\n",
" <td>Who is Vladimir Nabokov?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>59</th>\n",
" <td>Astronomy &amp; Geography</td>\n",
" <td>At the winter solstice, the sun is in Sagittar...</td>\n",
" <td>What is the Southern Hemisphere?</td>\n",
" <td>What is Capricorn?</td>\n",
" <td>35.714286</td>\n",
" </tr>\n",
" <tr>\n",
" <th>60</th>\n",
" <td>Television</td>\n",
" <td>Mike Post combined the sound of a slamming jai...</td>\n",
" <td>What is the theme song to \"The F.</td>\n",
" <td>What is Law &amp; Order?</td>\n",
" <td>22.222222</td>\n",
" </tr>\n",
" <tr>\n",
" <th>61</th>\n",
" <td>British Landmarks</td>\n",
" <td>Like Sir Thomas More, 3 16th century English q...</td>\n",
" <td>What is Westminster Abbey?</td>\n",
" <td>What is the Tower of London?</td>\n",
" <td>25.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>62</th>\n",
" <td>Early American History</td>\n",
" <td>In 1692 Increase Mather wrote, \"It were better...</td>\n",
" <td>What did Increase Mather write in</td>\n",
" <td>What are witches?</td>\n",
" <td>35.897436</td>\n",
" </tr>\n",
" <tr>\n",
" <th>63</th>\n",
" <td>Geography Mnemonics</td>\n",
" <td>Mimal, sometimes said to be the silhouette of ...</td>\n",
" <td>Who is Bobby Flay?\\n\\nMimal</td>\n",
" <td>What are Arkansas and Louisiana?</td>\n",
" <td>23.255814</td>\n",
" </tr>\n",
" <tr>\n",
" <th>64</th>\n",
" <td>Business Milestones</td>\n",
" <td>These were first sold in 1908, at a price equi...</td>\n",
" <td>Who are Apple Airpods?</td>\n",
" <td>What is the Ford Model T?</td>\n",
" <td>34.482759</td>\n",
" </tr>\n",
" <tr>\n",
" <th>65</th>\n",
" <td>In The Bookstore</td>\n",
" <td>The name of this author dead since 2013 now ap...</td>\n",
" <td>Jimmy Buffett and Clive Cuss</td>\n",
" <td>Who is Tom Clancy?</td>\n",
" <td>26.315789</td>\n",
" </tr>\n",
" <tr>\n",
" <th>66</th>\n",
" <td>Historic Art</td>\n",
" <td>The artwork once known in France as \"la tapiss...</td>\n",
" <td>The artwork once known in France as \"la</td>\n",
" <td>What is the Bayeux Tapestry?</td>\n",
" <td>20.408163</td>\n",
" </tr>\n",
" <tr>\n",
" <th>67</th>\n",
" <td>Pop Stars</td>\n",
" <td>In 2022 she became the first woman to have a B...</td>\n",
" <td>Who is Madonna?\\n\\nShe became the first</td>\n",
" <td>Who is Madonna?</td>\n",
" <td>43.750000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>68</th>\n",
" <td>Classic Tale Characters</td>\n",
" <td>In one 19th century translation, she \"perceive...</td>\n",
" <td>What is the Snow Queen?</td>\n",
" <td>Who is Scheherazade?</td>\n",
" <td>18.181818</td>\n",
" </tr>\n",
" <tr>\n",
" <th>69</th>\n",
" <td>USA</td>\n",
" <td>Ironically, though this company founded in the...</td>\n",
" <td>What is Walmart?</td>\n",
" <td>What is Jack Daniel's?</td>\n",
" <td>30.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>70</th>\n",
" <td>Historic People</td>\n",
" <td>After a 1789 event, he wrote, \"My first determ...</td>\n",
" <td>Who is Captain James Cook?</td>\n",
" <td>Who was William Bligh?</td>\n",
" <td>25.806452</td>\n",
" </tr>\n",
" <tr>\n",
" <th>71</th>\n",
" <td>The Movies</td>\n",
" <td>Laurence Olivier &amp; Ernest Borgnine were consid...</td>\n",
" <td>What is The Godfather?</td>\n",
" <td>What is The Godfather?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>72</th>\n",
" <td>Continental Geography</td>\n",
" <td>Until a 1903 secession, this country's contigu...</td>\n",
" <td>What is Constantinople?</td>\n",
" <td>What is Colombia?</td>\n",
" <td>27.272727</td>\n",
" </tr>\n",
" <tr>\n",
" <th>73</th>\n",
" <td>Foreign-Born Authors</td>\n",
" <td>Early in her career she translated romance nov...</td>\n",
" <td>Who is Isabel Allende?</td>\n",
" <td>Who is Isabel Allende?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>74</th>\n",
" <td>Historic Crimes</td>\n",
" <td>Saying it was stolen by Napoleon, self-styled ...</td>\n",
" <td>What happened in 1911?</td>\n",
" <td>What is the Mona Lisa?</td>\n",
" <td>24.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75</th>\n",
" <td>U.S. Bodies of Water</td>\n",
" <td>Continuing a downward trend, in July 2022 it w...</td>\n",
" <td>What is the level of Lake Mead?</td>\n",
" <td>What is Lake Mead?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>76</th>\n",
" <td>Gods &amp; Goddesses</td>\n",
" <td>Each morning she began her ride in her chariot...</td>\n",
" <td>Who is Aurora?</td>\n",
" <td>Who is Aurora (or Eos)?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>77</th>\n",
" <td>America At War</td>\n",
" <td>Until the Civil War, the Jan. 8 date of this b...</td>\n",
" <td>What is the Battle of Bull Run?</td>\n",
" <td>What is the Battle of New Orleans?</td>\n",
" <td>66.666667</td>\n",
" </tr>\n",
" <tr>\n",
" <th>78</th>\n",
" <td>Children's Books</td>\n",
" <td>Its title character is told \"By the time you a...</td>\n",
" <td>What is The Velveteen Rabbit?</td>\n",
" <td>What is The Velveteen Rabbit?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>79</th>\n",
" <td>TV Finales</td>\n",
" <td>In a reunion over 40 years in the making, Doll...</td>\n",
" <td>What is The Good Place?</td>\n",
" <td>What is Grace and Frankie?</td>\n",
" <td>37.037037</td>\n",
" </tr>\n",
" <tr>\n",
" <th>80</th>\n",
" <td>American Poems</td>\n",
" <td>In an 1847 poem this character sees her town o...</td>\n",
" <td>Who is Eliza Hanson and what happened to</td>\n",
" <td>Who is Evangeline?</td>\n",
" <td>36.842105</td>\n",
" </tr>\n",
" <tr>\n",
" <th>81</th>\n",
" <td>Famous Names</td>\n",
" <td>In 2001 he published a book called \"Banging Yo...</td>\n",
" <td>Who is Alan Watts?</td>\n",
" <td>Who is Banksy?</td>\n",
" <td>37.500000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>82</th>\n",
" <td>Children's Lit</td>\n",
" <td>The title object of this book \"never looked mo...</td>\n",
" <td>What is a spider web?</td>\n",
" <td>What is Charlotte's Web?</td>\n",
" <td>46.153846</td>\n",
" </tr>\n",
" <tr>\n",
" <th>83</th>\n",
" <td>Classic Songs</td>\n",
" <td>The shouts of excited children at a 1946 holid...</td>\n",
" <td>What is Jingle Bells?</td>\n",
" <td>What is \"Here Comes Santa Claus\"?</td>\n",
" <td>23.529412</td>\n",
" </tr>\n",
" <tr>\n",
" <th>84</th>\n",
" <td>Brand Names</td>\n",
" <td>Unable to make these candies perfectly round, ...</td>\n",
" <td>What is Hershey's Kisses</td>\n",
" <td>What are Milk Duds?</td>\n",
" <td>34.482759</td>\n",
" </tr>\n",
" <tr>\n",
" <th>85</th>\n",
" <td>Countries of the World</td>\n",
" <td>It's home to 58 UNESCO World Heritage Sites, m...</td>\n",
" <td>Which is Italy?</td>\n",
" <td>What is Italy?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>86</th>\n",
" <td>Action Movies</td>\n",
" <td>Its last line is \"If this is their idea of Chr...</td>\n",
" <td>What is Die Hard (1988)?</td>\n",
" <td>What is Die Hard?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>87</th>\n",
" <td>Presidential Facts</td>\n",
" <td>Only 3 presidents have married while in office...</td>\n",
" <td>What is the first and last name of John Ty</td>\n",
" <td>Who is Woodrow Wilson?</td>\n",
" <td>27.272727</td>\n",
" </tr>\n",
" <tr>\n",
" <th>88</th>\n",
" <td>19th Century Americans</td>\n",
" <td>Demonstrating the dignity &amp; humanity of Black ...</td>\n",
" <td>Who was Joseph Sebastian?</td>\n",
" <td>Who is Frederick Douglass?</td>\n",
" <td>29.411765</td>\n",
" </tr>\n",
" <tr>\n",
" <th>89</th>\n",
" <td>Latin Phrases</td>\n",
" <td>Originally, this 3-word phrase referred to whe...</td>\n",
" <td>Say it ain't so!</td>\n",
" <td>What is \"quid pro quo\"?</td>\n",
" <td>28.571429</td>\n",
" </tr>\n",
" <tr>\n",
" <th>90</th>\n",
" <td>1970s Movies</td>\n",
" <td>A 1975 premiere of this comedy advertised free...</td>\n",
" <td>What is The Great Gatsby?</td>\n",
" <td>What is Monty Python and the Holy Grail?</td>\n",
" <td>30.769231</td>\n",
" </tr>\n",
" <tr>\n",
" <th>91</th>\n",
" <td>Name's The Same</td>\n",
" <td>A cocktail, an island &amp; a WWII venture origina...</td>\n",
" <td>What is Operation Mincemeat?</td>\n",
" <td>What is Manhattan?</td>\n",
" <td>42.857143</td>\n",
" </tr>\n",
" <tr>\n",
" <th>92</th>\n",
" <td>U.S. Presidents</td>\n",
" <td>He was sworn in twice as President within 2 ye...</td>\n",
" <td>Who is Franklin D. Roosevelt</td>\n",
" <td>Who is Calvin Coolidge?</td>\n",
" <td>44.444444</td>\n",
" </tr>\n",
" <tr>\n",
" <th>93</th>\n",
" <td>Plays</td>\n",
" <td>A 1609 story in which an exiled king of Bulgar...</td>\n",
" <td>What is \"The Tempest\" by William Shakespeare</td>\n",
" <td>What is The Tempest?</td>\n",
" <td>100.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>94</th>\n",
" <td>Landmarks</td>\n",
" <td>In 2009, during a 20th anniversary celebration...</td>\n",
" <td>What is the Great Wall of China?</td>\n",
" <td>What is the Berlin Wall?</td>\n",
" <td>53.333333</td>\n",
" </tr>\n",
" <tr>\n",
" <th>95</th>\n",
" <td>World Capitals</td>\n",
" <td>Among its nicknames are the \"City of Classical...</td>\n",
" <td>What is Vienna?</td>\n",
" <td>What is Vienna, Austria?</td>\n",
" <td>57.142857</td>\n",
" </tr>\n",
" <tr>\n",
" <th>96</th>\n",
" <td>Language &amp; Its Meanings</td>\n",
" <td>Now meaning someone with nocturnal habits, it ...</td>\n",
" <td>What does \"Now\" mean in the poem \"</td>\n",
" <td>What is a night owl?</td>\n",
" <td>26.666667</td>\n",
" </tr>\n",
" <tr>\n",
" <th>97</th>\n",
" <td>Flags of Our Hemisphere</td>\n",
" <td>The stars on this country's flag represent sta...</td>\n",
" <td>Which is the flag of Canada?</td>\n",
" <td>What is Brazil?</td>\n",
" <td>15.384615</td>\n",
" </tr>\n",
" <tr>\n",
" <th>98</th>\n",
" <td>Names in U.S. History</td>\n",
" <td>This father was the only man among the 13 plai...</td>\n",
" <td>Who is Clarence Darrow?</td>\n",
" <td>Who is Oliver Brown?</td>\n",
" <td>29.629630</td>\n",
" </tr>\n",
" <tr>\n",
" <th>99</th>\n",
" <td>Children's Authors</td>\n",
" <td>Reversing the story of this heroine she create...</td>\n",
" <td>Who is Patricia MacLachlan?</td>\n",
" <td>Who is Sarah? (from Sarah, Plain and Tall)</td>\n",
" <td>33.333333</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" category \\\n",
"0 The Oscars \n",
"1 English Literature \n",
"2 Writers' Lesser-Known Works \n",
"3 Exploration \n",
"4 The Bill of Rights \n",
"5 Nobel Peace Prize Winners \n",
"6 Famous Names \n",
"7 Geography \n",
"8 Fashion History \n",
"9 Movies of the '80s \n",
"10 Novelists \n",
"11 20th Century Eponyms \n",
"12 City History \n",
"13 Brand Names \n",
"14 American Authors \n",
"15 Symbols \n",
"16 Movie Theme Songs \n",
"17 American Novelists \n",
"18 Medieval Places \n",
"19 Countries of Africa \n",
"20 Statehood \n",
"21 1980s Movies \n",
"22 Art Exhibitions \n",
"23 Countries of the World \n",
"24 Literature \n",
"25 Invasions \n",
"26 Landmarks \n",
"27 Geographic Name's the Same \n",
"28 Names in the Bookstore \n",
"29 U.S. History \n",
"30 American Literature \n",
"31 Nonfiction \n",
"32 Laws in U.S. History \n",
"33 Names of Myth \n",
"34 African Countries \n",
"35 The Ancient World \n",
"36 Famous Names \n",
"37 People & Places \n",
"38 Current World Leaders \n",
"39 Writers & The South \n",
"40 National Parks \n",
"41 Sports \n",
"42 The U.S. Military \n",
"43 Art & Science \n",
"44 Words From World War I \n",
"45 European History \n",
"46 Theater History \n",
"47 European Cities \n",
"48 Word Origins \n",
"49 Inventions \n",
"50 World War II \n",
"51 Mythology \n",
"52 Literature \n",
"53 U.S. State Names \n",
"54 Word Origins \n",
"55 World Cinema \n",
"56 History \n",
"57 Landmarks \n",
"58 Foreign-Born Authors \n",
"59 Astronomy & Geography \n",
"60 Television \n",
"61 British Landmarks \n",
"62 Early American History \n",
"63 Geography Mnemonics \n",
"64 Business Milestones \n",
"65 In The Bookstore \n",
"66 Historic Art \n",
"67 Pop Stars \n",
"68 Classic Tale Characters \n",
"69 USA \n",
"70 Historic People \n",
"71 The Movies \n",
"72 Continental Geography \n",
"73 Foreign-Born Authors \n",
"74 Historic Crimes \n",
"75 U.S. Bodies of Water \n",
"76 Gods & Goddesses \n",
"77 America At War \n",
"78 Children's Books \n",
"79 TV Finales \n",
"80 American Poems \n",
"81 Famous Names \n",
"82 Children's Lit \n",
"83 Classic Songs \n",
"84 Brand Names \n",
"85 Countries of the World \n",
"86 Action Movies \n",
"87 Presidential Facts \n",
"88 19th Century Americans \n",
"89 Latin Phrases \n",
"90 1970s Movies \n",
"91 Name's The Same \n",
"92 U.S. Presidents \n",
"93 Plays \n",
"94 Landmarks \n",
"95 World Capitals \n",
"96 Language & Its Meanings \n",
"97 Flags of Our Hemisphere \n",
"98 Names in U.S. History \n",
"99 Children's Authors \n",
"\n",
" answer \\\n",
"0 Born in 1932 & the son of a percussionist in t... \n",
"1 It says, \"The mind is its own place, & in itse... \n",
"2 Known for more philosophical works, he wrote t... \n",
"3 James Cook's account of a 1774 visit here reco... \n",
"4 England's \"Bloody Assizes\" & a 1685 life sente... \n",
"5 At times they each lived on Vilakazi St. in So... \n",
"6 In 1966, the year of his death, he shared plan... \n",
"7 Of the 13 nations through which the Equator pa... \n",
"8 These decorative items get their name from the... \n",
"9 Based on an off-Broadway play with just 3 char... \n",
"10 A 2012 book review noted subjects that \"sparke... \n",
"11 A 1940 headline about this included \"failure\",... \n",
"12 Over 700 years after its traditional 1252 foun... \n",
"13 The success of this brand has its roots with a... \n",
"14 In a periodical in 1807, he called New York Ci... \n",
"15 In math, it's a rotated V; in society, it's a ... \n",
"16 Monty Norman, the composer of this character's... \n",
"17 He served with an airman named Yohannan in Wor... \n",
"18 One of the participants in an 1170 event at th... \n",
"19 At one time a province of the Roman Empire, th... \n",
"20 Congress relented in 1890 after this prospecti... \n",
"21 A writer & producer of this movie said he want... \n",
"22 In 1898 what's been called the first blockbust... \n",
"23 Part of the largest contiguous land empire dur... \n",
"24 A 2006 book was titled \"The Poem That Changed ... \n",
"25 Backed by 14,000 troops, he invaded England to... \n",
"26 After its completion in the late 19th c., it w... \n",
"27 The busiest passenger port in the U.K., it sha... \n",
"28 This man made lists, perhaps to cope with depr... \n",
"29 An 1869 presidential pardon was granted to thi... \n",
"30 Letters, pocket knives, C rations & steel helm... \n",
"31 It has the line, \"The discovery of America…ope... \n",
"32 A radical Republican championed this 1875 act ... \n",
"33 Her brothers, Castor & Pollux, saved her after... \n",
"34 Once Africa's largest country in area, it drop... \n",
"35 The ancient writer Galen said books on ships a... \n",
"36 For a special 1970s cookbook, he provided one ... \n",
"37 Thought to descend from people of Southeast As... \n",
"38 In office from 2022, the president of this cou... \n",
"39 In 1939 he lived on Toulouse Street in the Fre... \n",
"40 It's named for a river indigenous people calle... \n",
"41 In 2010 they introduced the 4-point shot, 35 f... \n",
"42 Losses over Asia in the 1960s led to the estab... \n",
"43 A craft that visited it was named for Giotto, ... \n",
"44 \"Cistern\" & \"reservoir\" were suggested names f... \n",
"45 Until 1806, some German nobles included among ... \n",
"46 In 1904, wearing a harness, actress Nina Bouci... \n",
"47 Alphabetically the first German city in encycl... \n",
"48 This Sanskrit word referring to a spoken word ... \n",
"49 1917's \"Elements of Trench Warfare\" said this ... \n",
"50 Mimi Reinhard, who never learned to type using... \n",
"51 Poseidon carried off the maiden Theophane & tu... \n",
"52 Published in 2011, P.D. James' final novel, \"D... \n",
"53 5 U.S. states have 6-letter names; only these ... \n",
"54 Originally relating to a story of suffering, t... \n",
"55 The 2007 biopic called \"La Môme\" in France, me... \n",
"56 Returning home in 1493, Columbus stopped in th... \n",
"57 Pskov & Nizhny Novgorod are 2 of the cities th... \n",
"58 In the 1950s the New York Times said this auth... \n",
"59 At the winter solstice, the sun is in Sagittar... \n",
"60 Mike Post combined the sound of a slamming jai... \n",
"61 Like Sir Thomas More, 3 16th century English q... \n",
"62 In 1692 Increase Mather wrote, \"It were better... \n",
"63 Mimal, sometimes said to be the silhouette of ... \n",
"64 These were first sold in 1908, at a price equi... \n",
"65 The name of this author dead since 2013 now ap... \n",
"66 The artwork once known in France as \"la tapiss... \n",
"67 In 2022 she became the first woman to have a B... \n",
"68 In one 19th century translation, she \"perceive... \n",
"69 Ironically, though this company founded in the... \n",
"70 After a 1789 event, he wrote, \"My first determ... \n",
"71 Laurence Olivier & Ernest Borgnine were consid... \n",
"72 Until a 1903 secession, this country's contigu... \n",
"73 Early in her career she translated romance nov... \n",
"74 Saying it was stolen by Napoleon, self-styled ... \n",
"75 Continuing a downward trend, in July 2022 it w... \n",
"76 Each morning she began her ride in her chariot... \n",
"77 Until the Civil War, the Jan. 8 date of this b... \n",
"78 Its title character is told \"By the time you a... \n",
"79 In a reunion over 40 years in the making, Doll... \n",
"80 In an 1847 poem this character sees her town o... \n",
"81 In 2001 he published a book called \"Banging Yo... \n",
"82 The title object of this book \"never looked mo... \n",
"83 The shouts of excited children at a 1946 holid... \n",
"84 Unable to make these candies perfectly round, ... \n",
"85 It's home to 58 UNESCO World Heritage Sites, m... \n",
"86 Its last line is \"If this is their idea of Chr... \n",
"87 Only 3 presidents have married while in office... \n",
"88 Demonstrating the dignity & humanity of Black ... \n",
"89 Originally, this 3-word phrase referred to whe... \n",
"90 A 1975 premiere of this comedy advertised free... \n",
"91 A cocktail, an island & a WWII venture origina... \n",
"92 He was sworn in twice as President within 2 ye... \n",
"93 A 1609 story in which an exiled king of Bulgar... \n",
"94 In 2009, during a 20th anniversary celebration... \n",
"95 Among its nicknames are the \"City of Classical... \n",
"96 Now meaning someone with nocturnal habits, it ... \n",
"97 The stars on this country's flag represent sta... \n",
"98 This father was the only man among the 13 plai... \n",
"99 Reversing the story of this heroine she create... \n",
"\n",
" predicted \\\n",
"0 Who is John Smith? \n",
"1 Who is William Shakespeare? \n",
"2 Who is Machiavelli and what is his \n",
"3 What is the Giant's Causeway \n",
"4 What is the First Amendment? \n",
"5 Who are Desmond Tutu and Nelson Mand \n",
"6 Who was Walt Disney? \n",
"7 What is Ecuador? \n",
"8 What are Strass brooches? \n",
"9 What is The Drama Club? \n",
"10 Who is John Grisham? \n",
"11 What is Dunkirk? \n",
"12 What is Venice, Italy?\\n\\nV \n",
"13 What is the brand name of the hydrotherapy \n",
"14 Who is Mark Twain? \n",
"15 What is the V for Victory symbol? \n",
"16 What is the James Bond theme? \n",
"17 Who is Ernest Hemingway? \n",
"18 What is the Battle of Hastings? \n",
"19 What is Morocco? \n",
"20 What is Wyoming?\\n\\n \\n1 \n",
"21 What is Indiana Jones and the Temple of Doom \n",
"22 What is Vincent Van Gogh? \n",
"23 What is the Central Asian nation of Kazakhstan \n",
"24 What is The Road Not Taken by Robert F \n",
"25 Who is Louis XIV of France?\\n\\nHe \n",
"26 What is the Eiffel Tower? \n",
"27 What is London Heathrow airport? \n",
"28 Who is Charles Dickens? \n",
"29 Who is E.D. Cogswell \n",
"30 What is \"The Things They Carried\" by \n",
"31 What is Karl Marx's \"The Communist \n",
"32 What is the Civil Rights Act of 19 \n",
"33 What is Helen of Troy? \n",
"34 What is Sudan? \n",
"35 Which is Rome? \n",
"36 Who is Gordon Ramsay? \n",
"37 What is Guam?\\n\\nAnswer Question: \n",
"38 Who is Ferdinand Magellan Jr.? \n",
"39 Who is Tennessee Williams and what did he do in \n",
"40 What is Yellowstone National Park? \n",
"41 What is the 3-point line? \n",
"42 What is Project X-Ray? \n",
"43 What is an astrolabe? \n",
"44 What is tank? \n",
"45 Who is the Holy Roman Emperor Maximilian I? \n",
"46 Who is Puck from A Midsummer Night \n",
"47 What is Aachen? \n",
"48 What is Mantra? \n",
"49 What is a trench? \n",
"50 Who is Mimi Reimann? \n",
"51 What is a magical item? \n",
"52 What is \"Pride and Prejudice\" \n",
"53 The two U.S. states with six- \n",
"54 What is agony? \n",
"55 What is La Vie en Rose? \n",
"56 What is Hispaniola? \n",
"57 What is Kremlin? \n",
"58 Who is Vladimir Nabokov? \n",
"59 What is the Southern Hemisphere? \n",
"60 What is the theme song to \"The F. \n",
"61 What is Westminster Abbey? \n",
"62 What did Increase Mather write in \n",
"63 Who is Bobby Flay?\\n\\nMimal \n",
"64 Who are Apple Airpods? \n",
"65 Jimmy Buffett and Clive Cuss \n",
"66 The artwork once known in France as \"la \n",
"67 Who is Madonna?\\n\\nShe became the first \n",
"68 What is the Snow Queen? \n",
"69 What is Walmart? \n",
"70 Who is Captain James Cook? \n",
"71 What is The Godfather? \n",
"72 What is Constantinople? \n",
"73 Who is Isabel Allende? \n",
"74 What happened in 1911? \n",
"75 What is the level of Lake Mead? \n",
"76 Who is Aurora? \n",
"77 What is the Battle of Bull Run? \n",
"78 What is The Velveteen Rabbit? \n",
"79 What is The Good Place? \n",
"80 Who is Eliza Hanson and what happened to \n",
"81 Who is Alan Watts? \n",
"82 What is a spider web? \n",
"83 What is Jingle Bells? \n",
"84 What is Hershey's Kisses \n",
"85 Which is Italy? \n",
"86 What is Die Hard (1988)? \n",
"87 What is the first and last name of John Ty \n",
"88 Who was Joseph Sebastian? \n",
"89 Say it ain't so! \n",
"90 What is The Great Gatsby? \n",
"91 What is Operation Mincemeat? \n",
"92 Who is Franklin D. Roosevelt \n",
"93 What is \"The Tempest\" by William Shakespeare \n",
"94 What is the Great Wall of China? \n",
"95 What is Vienna? \n",
"96 What does \"Now\" mean in the poem \" \n",
"97 Which is the flag of Canada? \n",
"98 Who is Clarence Darrow? \n",
"99 Who is Patricia MacLachlan? \n",
"\n",
" correct Score \n",
"0 Who is John Williams? 57.142857 \n",
"1 What is Paradise Lost? 31.250000 \n",
"2 Who is Niccolò Machiavelli? 73.684211 \n",
"3 What is Easter Island (Rapa Nui)? 35.000000 \n",
"4 What is the Eighth Amendment? 77.419355 \n",
"5 Who are Nelson Mandela & Desmond Tutu? 89.230769 \n",
"6 Who is Walt Disney? 100.000000 \n",
"7 What is Colombia? 26.666667 \n",
"8 What are rhinestones? 58.823529 \n",
"9 What is Driving Miss Daisy? 28.571429 \n",
"10 Who is John Grisham? 100.000000 \n",
"11 What is the Maginot Line? 21.052632 \n",
"12 What is Stockholm? 16.000000 \n",
"13 What is Jacuzzi? 6.060606 \n",
"14 Who is Washington Irving? 44.444444 \n",
"15 What is \"less than\"? 20.689655 \n",
"16 Who is James Bond? 100.000000 \n",
"17 Who is Joseph Heller? 27.586207 \n",
"18 What is Canterbury, England? (Canterbury Cathe... 29.508197 \n",
"19 What is Morocco? 100.000000 \n",
"20 What is Wyoming? 82.352941 \n",
"21 What is Raiders of the Lost Ark? 43.137255 \n",
"22 Who is Rembrandt? 24.000000 \n",
"23 What is Mongolia? 14.285714 \n",
"24 What is \"Howl\"? 6.666667 \n",
"25 Who is William of Orange? 50.000000 \n",
"26 What is the Eiffel Tower? 100.000000 \n",
"27 What is Dover? 21.428571 \n",
"28 Who is Peter Mark Roget? 32.258065 \n",
"29 Who is Dr. Samuel Mudd? 21.428571 \n",
"30 What is The Things They Carried? 100.000000 \n",
"31 What is The Communist Manifesto 65.000000 \n",
"32 What is the Civil Rights Act? 100.000000 \n",
"33 Who is Helen of Troy? 100.000000 \n",
"34 What is Sudan? 100.000000 \n",
"35 What is Alexandria? 20.000000 \n",
"36 Who is Andy Warhol? 25.000000 \n",
"37 What is Guam? 30.769231 \n",
"38 What is the Philippines? 18.181818 \n",
"39 Who is Tennessee Williams? 100.000000 \n",
"40 What is Yellowstone? 100.000000 \n",
"41 Who are the Harlem Globetrotters? 27.777778 \n",
"42 What is \"Top Gun\"? 10.000000 \n",
"43 What is Halley's Comet? 30.769231 \n",
"44 What is \"tank\"? 100.000000 \n",
"45 What is Holy Roman Emperor? 100.000000 \n",
"46 Who is Peter Pan? 17.647059 \n",
"47 What is Aachen? 100.000000 \n",
"48 What is mantra? 100.000000 \n",
"49 What is barbed wire? 23.529412 \n",
"50 What is Schindler's list? 28.571429 \n",
"51 What is the Golden Fleece? 32.000000 \n",
"52 What is Pride and Prejudice? 100.000000 \n",
"53 What are Oregon & Nevada? 22.727273 \n",
"54 What is passion? 50.000000 \n",
"55 What is La Vie en Rose? 100.000000 \n",
"56 What is Santa Maria? 47.619048 \n",
"57 What is a kremlin? 100.000000 \n",
"58 Who is Vladimir Nabokov? 100.000000 \n",
"59 What is Capricorn? 35.714286 \n",
"60 What is Law & Order? 22.222222 \n",
"61 What is the Tower of London? 25.000000 \n",
"62 What are witches? 35.897436 \n",
"63 What are Arkansas and Louisiana? 23.255814 \n",
"64 What is the Ford Model T? 34.482759 \n",
"65 Who is Tom Clancy? 26.315789 \n",
"66 What is the Bayeux Tapestry? 20.408163 \n",
"67 Who is Madonna? 43.750000 \n",
"68 Who is Scheherazade? 18.181818 \n",
"69 What is Jack Daniel's? 30.000000 \n",
"70 Who was William Bligh? 25.806452 \n",
"71 What is The Godfather? 100.000000 \n",
"72 What is Colombia? 27.272727 \n",
"73 Who is Isabel Allende? 100.000000 \n",
"74 What is the Mona Lisa? 24.000000 \n",
"75 What is Lake Mead? 100.000000 \n",
"76 Who is Aurora (or Eos)? 100.000000 \n",
"77 What is the Battle of New Orleans? 66.666667 \n",
"78 What is The Velveteen Rabbit? 100.000000 \n",
"79 What is Grace and Frankie? 37.037037 \n",
"80 Who is Evangeline? 36.842105 \n",
"81 Who is Banksy? 37.500000 \n",
"82 What is Charlotte's Web? 46.153846 \n",
"83 What is \"Here Comes Santa Claus\"? 23.529412 \n",
"84 What are Milk Duds? 34.482759 \n",
"85 What is Italy? 100.000000 \n",
"86 What is Die Hard? 100.000000 \n",
"87 Who is Woodrow Wilson? 27.272727 \n",
"88 Who is Frederick Douglass? 29.411765 \n",
"89 What is \"quid pro quo\"? 28.571429 \n",
"90 What is Monty Python and the Holy Grail? 30.769231 \n",
"91 What is Manhattan? 42.857143 \n",
"92 Who is Calvin Coolidge? 44.444444 \n",
"93 What is The Tempest? 100.000000 \n",
"94 What is the Berlin Wall? 53.333333 \n",
"95 What is Vienna, Austria? 57.142857 \n",
"96 What is a night owl? 26.666667 \n",
"97 What is Brazil? 15.384615 \n",
"98 Who is Oliver Brown? 29.629630 \n",
"99 Who is Sarah? (from Sarah, Plain and Tall) 33.333333 "
]
},
"execution_count": 109,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from rapidfuzz import fuzz\n",
"\n",
"results['Score'] = results.loc[:, ['predicted', 'correct']].apply(lambda i: fuzz.token_ratio(clean(i[0]), clean(i[1])), axis=1)\n",
"results"
]
},
{
"cell_type": "code",
"execution_count": 105,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"27"
]
},
"execution_count": 105,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"(results['Score'] >= 95).sum()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.10"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment