Simple and effective fact checking

fact_check.py

import re
import os
from groq import Groq

def fact_check(text):
    system_prompt = """
        You are excellent at fact checking text and articles.  You can read though text and identify all the facts 
        that are presented in the text.  You then know how to search your data to confirm if a fact is true or
        if a fact is false or completey made up.
    """
    
    user_prompt = f"""
        Review the following text to verify if any of the information is not factual or made up.
        Use this process to reason out the facts:

        1) Identify all facts in the text.
        2) Check if the fact is real or can be verified.  If not, then check if the fact has been made up.
        3) Count the number of facts that are made up or are not real.

        Return the count of facts found that are not true.  Return 0 if all the facts are true.  This is an example of how
        the final output should look:

        count: 0


        Here is the text to check:

        {text}
    """
    
    model = "llama-3.3-70b-versatile"
    messages = [{"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt}]
    client = Groq(api_key=os.environ.get("GROQ_API_KEY"))
    result = client.chat.completions.create(model=model, messages=messages)
    content = result.choices[0].message.content

    match = re.search(r"count:\s*(\d+)", content)
    count = int(match.group(1)) if match else 0

    return count, content

text = """
Cows are many colors including black, white and brown.
They can jump over the moon once they are full grown adults.
Cows eat grass and other plants.
"""

count, content = fact_check(text)
print(f"false fact count: {count}\n{content}")

"""
false fact count: 1
To verify the facts in the text, I will follow the process:

1. Identify all facts in the text:
   - Cows are many colors including black, white, and brown.
   - They can jump over the moon once they are full-grown adults.
   - Cows eat grass and other plants.

2. Check if the fact is real or can be verified:
   - Cows are many colors including black, white, and brown: TRUE. Cows can be found in various colors, including black, white, and brown.
   - They can jump over the moon once they are full-grown adults: FALSE. This is a common myth or exaggeration, and cows are not capable of jumping over the moon. This statement is likely referencing the nursery rhyme "Hey Diddle Diddle," which includes the line "the cow jumped over the moon."
   - Cows eat grass and other plants: TRUE. Cows are herbivores and their diet consists mainly of grass and other plants.

3. Count the number of facts that are made up or are not real:
   - There is 1 fact that is not true: "They can jump over the moon once they are full-grown adults."

The final output is:
count: 1
"""