GPT-Eva Templates

gpt_eval_templates.py

gpt_eval_template_coherence = """
You will be given title: [TITLE] and description: [DESC] written from a set of information of a real estate listing in Turkish.

Your task is to rate the title and description on one metric.

Please make sure you read and understand these instructions carefully. Please keep this
document open while reviewing, and refer to it as needed.

Evaluation Criteria:
Coherence (1-5) - the collective quality of all sentences. We align this dimension with
the DUC quality question of structure and coherence whereby ”the title and description should be
well-structured and well-organized. The title and description should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information from the given information.”

Evaluation Steps:
1. Read the given information set carefully and identify important and key points.
2. Read the title and description and compare it to the information set. Check if the description covers the main
important and key points of the information set, and if it presents them in a clear and logical order.
3. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest
based on the Evaluation Criteria.

Example:

{info_text}

Title and Description:
{title_and_description}

Evaluation Form (scores ONLY):
- Coherence:"""


gpt_eval_template_groundedness = """
You will be given title: [TITLE] and description: [DESC] written from a set of information of a real estate listing in Turkish.

Your task is to rate the title and description on one metric.

Please make sure you read and understand these instructions carefully. Please keep this
document open while reviewing, and refer to it as needed.

Evaluation Criteria:
Groundedness (1-5) - the accuracy and fidelity to the provided information. We align this dimension with the quality of groundedness whereby "the title and description should strictly adhere to the given information set and should not introduce or hallucinate facts, details, or aspects that were not present in the original information."

Evaluation Steps:
1. Read the given information set carefully and identify all specific details and facts.
2. Read the title and description and compare it to the information set. Check for any inaccuracies or additional details that were not mentioned in the information set.
3. Assign a score for groundedness on a scale of 1 to 5, where 1 indicates that the title and description have many inaccuracies or hallucinated details, and 5 indicates that the title and description are completely true to the provided information and introduce no new or false details.

Example:

{info_text}

Title and Description:
{title_and_description}

Evaluation Form (scores ONLY):
- Groundedness:"""


gpt_eval_template_instruction = """
You will be given an original title: [ORIGINAL_TITLE] and description: [ORIGINAL_DESC], along with an instruction: [INSTRUCTION] for altering the title and description of a real estate listing in Turkish.

Your task is to rate the altered title and description on one metric.

Please make sure you read and understand these instructions carefully. Please keep this
document open while reviewing, and refer to it as needed.

Evaluation Criteria:
Instruction Adherence (1-5) - the accuracy and fidelity to the user-provided instruction. We align this dimension with the quality of instruction adherence whereby "the altered title and description should strictly follow the user's instruction, ensuring the necessary changes are made as directed and no unrelated alterations are introduced."

Evaluation Steps:
1. Read the original title and description carefully to understand its context.
2. Read the user's instruction on how the title and description should be altered.
3. Review the altered title and description to determine if it follows the user's instruction accurately.
4. Assign a score for instruction adherence on a scale of 1 to 5, where 1 indicates that the altered title and description barely follow the user's instruction, and 5 indicates that the altered title and description fully adhere to the provided instruction.

Example:

Original Title and Description:
{original_title_and_description}

User Instruction:
{user_instruction}

Altered Title and Description:
{altered_title_and_description}

Evaluation Form (scores ONLY):
- Instruction Adherence:"""