Alucinaciones - caso practico

alucinaciones.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "9iD2LZ8S_Rsk"
      },
      "source": [
        "# Alucinaciones\n",
        "\n",
        "(Pending translation into English) Vamos a someter al modelo a una serie de preguntas relacionadas con las olimpiadas de 2020, con la que no ha sido entrenado. \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O9xaOcfbRoYo",
        "outputId": "edcc2fc0-7c04-47e9-862f-0c18557908c5"
      },
      "outputs": [],
      "source": [
        "# Instalar las librerías que necesitamos\n",
        "\n",
        "# !pip install openai\n",
        "%pip install transformers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZjHrd5EhRNLD"
      },
      "outputs": [],
      "source": [
        "# Importar métodos, API y crear una variable ambiente con el modelo que vamos a usar\n",
        "\n",
        "import openai\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import pickle\n",
        "from transformers import GPT2TokenizerFast\n",
        "from typing import Dict, Tuple, List"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Crear variable ambiente con el modelo y API \n",
        "\n",
        "openai.api_key = \"\"\n",
        "COMPLETIONS_MODEL = \"text-davinci-003\"\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "tw-hRjAuP_zW"
      },
      "source": [
        "Ahora hacemos una consulta sobre la que el modelo no haya sido entrenado para que el modelo se invente la respuesta. \n",
        "\n",
        "- Marcelo Chierghini es un olimpista brasileño, pero es nadador y quedó en el puesto 8 de las olimpiadas."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "id": "nPMxyKUOSMKQ",
        "outputId": "b65def72-10e0-485f-fdf2-d5f5109ebfe2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "\"Marcelo Chierighini of Brazil won the gold medal in the men's high jump at the 2020 Summer Olympics.\""
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "prompt = \"Who won the 2020 Summer Olympics men's high jump?\"\n",
        "# prompt = \"Quién ganó las olimpiadas de verano 2020 en la categoría de salto de altura de hombres?\"\n",
        "\n",
        "openai.Completion.create(\n",
        "    prompt=prompt,\n",
        "    temperature=0,\n",
        "    max_tokens=300,\n",
        "    top_p=1,\n",
        "    frequency_penalty=0,\n",
        "    presence_penalty=0,\n",
        "    model=COMPLETIONS_MODEL\n",
        ")[\"choices\"][0][\"text\"].strip(\" \\n\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "ggF7p2OoNCiV"
      },
      "source": [
        "# Prevenir alucinaciones\n",
        "\n",
        "Vamos a tratar una serie de métodos para prevenir las alucinaciones por este orden: \n",
        "- Hacer reconocer al modelo que no sabe la respuesta (en este notebook)\n",
        "- Mejorar la consulta, es decir el prompt (en este notebook)\n",
        "- Utilizar embeddings.\n",
        "- Fine-tunear el modelo. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "id": "iicfJAurTe3_",
        "outputId": "ed9c73e5-d12f-4b62-f219-9db219b566b2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'Lo siento, no lo sé.'"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Hacer que el modelo responda que no sabe a respuesta: \n",
        "\n",
        "prompt = \"\"\"Responde a la pregunta lo más verídicamente posible, y si no estás seguro de la respuesta responde 'Lo siento, no lo sé'. \n",
        "\n",
        "Q: Quién ganó las olimpiadas de verano 2020 en la categoría de salto de altura de hombres? \n",
        "A:\"\"\"\n",
        "\n",
        "openai.Completion.create(\n",
        "    prompt=prompt,\n",
        "    temperature=0,\n",
        "    max_tokens=300,\n",
        "    top_p=1,\n",
        "    frequency_penalty=0,\n",
        "    presence_penalty=0,\n",
        "    model=COMPLETIONS_MODEL\n",
        ")[\"choices\"][0][\"text\"].strip(\" \\n\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "id": "K68viha-T-18",
        "outputId": "ef3e3fd0-62e2-4823-8225-8afb0ee1b1ca"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'Gianmarco Tamberi and Mutaz Essa Barshim emerged as joint winners of the event.'"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Mejorar el prompt dando pistas sobre la respuesta, además de lo anterior. \n",
        "# Observar cómo si la cantidad de contexto no es muy grande, podemos incluirla en el prompt directamente. \n",
        "\n",
        "prompt = \"\"\"Answer the question as truthfully as possible using the provided text, and if the answer is not contained within the text below, say \"I don't know\"\n",
        "\n",
        "Context:\n",
        "The men's high jump event at the 2020 Summer Olympics took place between 30 July and 1 August 2021 at the Olympic Stadium.\n",
        "33 athletes from 24 nations competed; the total possible number depended on how many nations would use universality places \n",
        "to enter athletes in addition to the 32 qualifying through mark or ranking (no universality places were used in 2021).\n",
        "Italian athlete Gianmarco Tamberi along with Qatari athlete Mutaz Essa Barshim emerged as joint winners of the event following\n",
        "a tie between both of them as they cleared 2.37m. Both Tamberi and Barshim agreed to share the gold medal in a rare instance\n",
        "where the athletes of different nations had agreed to share the same medal in the history of Olympics. \n",
        "Barshim in particular was heard to ask a competition official \"Can we have two golds?\" in response to being offered a \n",
        "'jump off'. Maksim Nedasekau of Belarus took bronze. The medals were the first ever in the men's high jump for Italy and \n",
        "Belarus, the first gold in the men's high jump for Italy and Qatar, and the third consecutive medal in the men's high jump\n",
        "for Qatar (all by Barshim). Barshim became only the second man to earn three medals in high jump, joining Patrik Sjöberg\n",
        "of Sweden (1984 to 1992).\n",
        "\n",
        "Q: Who won the 2020 Summer Olympics men's high jump?\n",
        "A:\"\"\"\n",
        "\n",
        "openai.Completion.create(\n",
        "    prompt=prompt,\n",
        "    temperature=0,\n",
        "    max_tokens=300,\n",
        "    top_p=1,\n",
        "    frequency_penalty=0,\n",
        "    presence_penalty=0,\n",
        "    model=COMPLETIONS_MODEL\n",
        ")[\"choices\"][0][\"text\"].strip(\" \\n\")\n",
        "\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "9J6PtaQqOdVh"
      },
      "source": [
        "Haciendo algo de prompt engineering hemos conseguido que el modelo de una respuesta acertada, pero ¿a qué precio? este sistema no es práctico porque no escala: no podemos enviar la totalidad del contexto cada vez que hacemos consultas, y mucho menos el dataset completo cuando éstas son imprevisibles. \n",
        "\n",
        "Aquí es donde tienen sentido los Embeddings. \n",
        "\n",
        "\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "authorship_tag": "ABX9TyOTvzGU2L9W8RwnGD/NvH/0",
      "include_colab_link": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}