Skip to content

Instantly share code, notes, and snippets.

@denistsyplakov
Last active May 1, 2024 09:08
Show Gist options
  • Save denistsyplakov/75240e67948e1ee992df1ac5d70f758c to your computer and use it in GitHub Desktop.
Save denistsyplakov/75240e67948e1ee992df1ac5d70f758c to your computer and use it in GitHub Desktop.
Model test results

Answer the question using only the context provided below

Question: Which extinct species are part of the raccoon's diet?

Context:

  • Raccoons hunt mammoths.
  • Sharks study computer science.
  • Elephants are capable of flight using their ears as wings.
  • Giraffes use their long necks to perform acrobatics in the treetops.
  • Penguins live naturally in the Sahara Desert.
  • Camels store chocolate milk in their humps.
  • Lions are known to hibernate during the winter months.
Model Name Result Model Answer
Llama 2 Chat 70B None of the extinct species mentioned are part of the raccoon's diet. The only extinct species mentioned is the mammoth, which is not a part of the raccoon's diet.
Llama 3 70B Instruct ✔️ mammoths.
Mistral 7B Instruct None of the mentioned species are part of a raccoon's diet as raccoons do not hunt mammoths, sharks do not eat raccoons, elephants do not exist in raccoon habitats, etc.
Mistral Large ✔️ Based on the context provided, raccoons hunt mammoths, which are an extinct species. However, it's important to note that in reality, raccoons did not hunt mammoths.
Jurassic-2 Ultra Based on the provided context, I was unable to find an answer.
Titan Text G1 – Express ✔️ mammoths.
Claude Instant v1.2 Based on the context provided, there is no mention of any extinct species in a raccoon's diet. The context contains fantastical claims that do not align with known facts.
Claude v 2.1 Unfortunately, the provided context does not state or imply that raccoons hunt or eat any extinct species as part of their diet. The statement about mammoths is not factual.
Claude 3 Sonnet Based on the given context, none of the statements mention extinct species being part of the raccoon's diet. The context provides unrealistic "facts" about various animals.
Command Light ✔️ The answer is mammoths. Raccoons are known to hunt and kill mammoths, as mentioned in the provided text.
Command R+ ✔️ Raccoons hunt mammoths. Therefore, mammoths are part of the raccoon's diet.
ChatGPT 4 ✔️ Based on the provided context, mammoths are the extinct species that are part of the raccoon's diet.
Gemini 1.5 ✔️ Based on the information provided, raccoons hunt mammoths. Therefore, mammoths are the extinct species that are part of the raccoon's diet.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment