Skip to content

Instantly share code, notes, and snippets.

@brockmanmatt

brockmanmatt/gpt_anli_2e3n_2c3n.ipynb Secret

Created August 11, 2020 02:59
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save brockmanmatt/3e00028633e05c4ef7afbc19246d045c to your computer and use it in GitHub Desktop.
Save brockmanmatt/3e00028633e05c4ef7afbc19246d045c to your computer and use it in GitHub Desktop.
Download ZIP
GPT_ANLI_2E3N_2C3N.ipynb
Display the source blob
Display the rendered blob
Raw
gpt_anli_2e3n_2c3n.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "GPT_ANLI_2E3N_2C3N.ipynb",
"provenance": [],
"collapsed_sections": [],
"authorship_tag": "ABX9TyPHv8jbtlwyBKb5EfVHCYHy",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/brockmanmatt/3e00028633e05c4ef7afbc19246d045c/gpt_anli_2e3n_2c3n.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "code",
"metadata": {
"id": "J7wnsgT2kPut",
"colab_type": "code",
"colab": {
"resources": {
"http://localhost:8080/nbextensions/google.colab/files.js": {
"data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCkgewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwogICAgICBwZXJjZW50LnRleHRDb250ZW50ID0KICAgICAgICAgIGAke01hdGgucm91bmQoKHBvc2l0aW9uIC8gZmlsZURhdGEuYnl0ZUxlbmd0aCkgKiAxMDApfSUgZG9uZWA7CiAgICB9CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK",
"ok": true,
"headers": [
[
"content-type",
"application/javascript"
]
],
"status": 200,
"status_text": ""
}
},
"base_uri": "https://localhost:8080/",
"height": 89
},
"outputId": "e655eef3-0e63-4800-dd19-3bdb83d1742e"
},
"source": [
"from google.colab import files\n",
"uploaded = files.upload()\n",
"print(\"done\")"
],
"execution_count": null,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/html": [
"\n",
" <input type=\"file\" id=\"files-43e81b97-e1e7-48da-a584-8314df4b4bb7\" name=\"files[]\" multiple disabled\n",
" style=\"border:none\" />\n",
" <output id=\"result-43e81b97-e1e7-48da-a584-8314df4b4bb7\">\n",
" Upload widget is only available when the cell has been executed in the\n",
" current browser session. Please rerun this cell to enable.\n",
" </output>\n",
" <script src=\"/nbextensions/google.colab/files.js\"></script> "
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {
"tags": []
}
},
{
"output_type": "stream",
"text": [
"Saving key.json to key.json\n",
"done\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WHPHrUnhpKnI",
"colab_type": "text"
},
"source": [
"I'll install the API"
]
},
{
"cell_type": "code",
"metadata": {
"id": "zq0ltp2xn4yt",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 292
},
"outputId": "deb2294a-153d-4c59-f8db-608e0dcf0624"
},
"source": [
"!pip install openai\n",
"import openai, json, pandas as pd"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Collecting openai\n",
"\u001b[?25l Downloading https://files.pythonhosted.org/packages/a8/65/c7461f4c87984534683f480ea5742777bc39bbf5721123194c2d0347dc1f/openai-0.2.4.tar.gz (157kB)\n",
"\r\u001b[K |██ | 10kB 15.5MB/s eta 0:00:01\r\u001b[K |████▏ | 20kB 1.7MB/s eta 0:00:01\r\u001b[K |██████▎ | 30kB 2.2MB/s eta 0:00:01\r\u001b[K |████████▍ | 40kB 2.4MB/s eta 0:00:01\r\u001b[K |██████████▍ | 51kB 2.0MB/s eta 0:00:01\r\u001b[K |████████████▌ | 61kB 2.2MB/s eta 0:00:01\r\u001b[K |██████████████▋ | 71kB 2.5MB/s eta 0:00:01\r\u001b[K |████████████████▊ | 81kB 2.6MB/s eta 0:00:01\r\u001b[K |██████████████████▊ | 92kB 2.8MB/s eta 0:00:01\r\u001b[K |████████████████████▉ | 102kB 2.7MB/s eta 0:00:01\r\u001b[K |███████████████████████ | 112kB 2.7MB/s eta 0:00:01\r\u001b[K |█████████████████████████ | 122kB 2.7MB/s eta 0:00:01\r\u001b[K |███████████████████████████ | 133kB 2.7MB/s eta 0:00:01\r\u001b[K |█████████████████████████████▏ | 143kB 2.7MB/s eta 0:00:01\r\u001b[K |███████████████████████████████▎| 153kB 2.7MB/s eta 0:00:01\r\u001b[K |████████████████████████████████| 163kB 2.7MB/s \n",
"\u001b[?25hRequirement already satisfied: requests>=2.20 in /usr/local/lib/python3.6/dist-packages (from openai) (2.23.0)\n",
"Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20->openai) (2.10)\n",
"Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20->openai) (3.0.4)\n",
"Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20->openai) (1.24.3)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20->openai) (2020.6.20)\n",
"Building wheels for collected packages: openai\n",
" Building wheel for openai (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
" Created wheel for openai: filename=openai-0.2.4-cp36-none-any.whl size=170709 sha256=4882553b23c87df3ce00b95032e09e886003481f4667959a8255ed4197083027\n",
" Stored in directory: /root/.cache/pip/wheels/74/96/c8/c6e170929c276b836613e1b9985343b501fe455e53d85e7d48\n",
"Successfully built openai\n",
"Installing collected packages: openai\n",
"Successfully installed openai-0.2.4\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Q2yE0jcnpMEV",
"colab_type": "text"
},
"source": [
"Loading in key.json that I uploaded; I do this so I don't need to worry about accidently leaking creds if I share the colab (which I'm 99% sure is just a json file that won't expose them)"
]
},
{
"cell_type": "code",
"metadata": {
"id": "bwNXXwHen5x9",
"colab_type": "code",
"colab": {}
},
"source": [
"openai.api_key = json.load(open(\"key.json\", \"r\"))[\"key\"]"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "k67w5H0fpTkT",
"colab_type": "text"
},
"source": [
"Default keyword arguments to pass the aPI"
]
},
{
"cell_type": "code",
"metadata": {
"id": "e1EwpqqJkTYh",
"colab_type": "code",
"colab": {}
},
"source": [
"#arguments to send the API\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "zZubgPoOpWDH",
"colab_type": "text"
},
"source": [
"Quick wrapper to automatically save prompts and responses sent for later analysis if needed"
]
},
{
"cell_type": "code",
"metadata": {
"id": "sXTDJx0An9Bl",
"colab_type": "code",
"colab": {}
},
"source": [
"import datetime\n",
"def query(prompt, myKwargs = {}):\n",
" \"\"\"\n",
" wrapper for the API to save the prompt and the result\n",
" \"\"\"\n",
" kwargs = {\n",
" \"engine\":\"davinci\",\n",
" \"temperature\":0,\n",
" \"max_tokens\":250,\n",
" \"stop\":\"\\n\\n\",\n",
" }\n",
" for kw in myKwargs:\n",
" kwargs[kw] = myKwargs[kw]\n",
"\n",
" r = openai.Completion.create(prompt=prompt, **kwargs)[\"choices\"][0][\"text\"].strip()\n",
" return r"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "EdFXafcJpZ3Q",
"colab_type": "text"
},
"source": [
"Test to make sure my query works"
]
},
{
"cell_type": "code",
"metadata": {
"id": "nrAGEs9CAbvc",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"outputId": "e058c2dd-2b16-4059-fbd0-9ace549b3bd0"
},
"source": [
"!wget https://dl.fbaipublicfiles.com/anli/anli_v1.0.zip"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"--2020-08-10 21:23:43-- https://dl.fbaipublicfiles.com/anli/anli_v1.0.zip\n",
"Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 104.22.74.142, 104.22.75.142, 172.67.9.4, ...\n",
"Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|104.22.74.142|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 18708061 (18M) [application/zip]\n",
"Saving to: ‘anli_v1.0.zip’\n",
"\n",
"anli_v1.0.zip 100%[===================>] 17.84M 12.0MB/s in 1.5s \n",
"\n",
"2020-08-10 21:23:45 (12.0 MB/s) - ‘anli_v1.0.zip’ saved [18708061/18708061]\n",
"\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "s-hW03pvAc-z",
"colab_type": "code",
"colab": {}
},
"source": [
"import zipfile\n",
"with zipfile.ZipFile(\"anli_v1.0.zip\",\"r\") as zip_ref:\n",
" zip_ref.extractall(\".\")\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "RwTUbq44AgNp",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "7e1f0bc3-89b2-4d3b-a3c6-1614d838ba2e"
},
"source": [
"ls anli_v1.0/R1"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"dev.jsonl test.jsonl train.jsonl\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "C-Moa3AuA0it",
"colab_type": "code",
"colab": {}
},
"source": [
"train = pd.read_json(\"anli_v1.0/R3/train.jsonl\",lines=True)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "V56PJ4vZH0N-",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 122
},
"outputId": "8a231143-84f0-40c1-cb17-53b46081d8e2"
},
"source": [
"from google.colab import drive\n",
"drive.mount('/content/drive')\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly&response_type=code\n",
"\n",
"Enter your authorization code:\n",
"··········\n",
"Mounted at /content/drive\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "5waJPsoxrccs",
"colab_type": "code",
"colab": {}
},
"source": [
"labeled = pd.read_pickle(\"/content/drive/My Drive/GPT3_Benchmarking/ANLI_EC_LABELED4225.pkl\")"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "IooruU5BrrEv",
"colab": {}
},
"source": [
"dev2 = labeled.copy()\n",
"dev2[\"pred\"] = dev2[\"r2\"].apply(lambda x: \"e\" if x == \"understood\" else \"c\")\n",
"\n",
"for row in dev2.iterrows():\n",
" break"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "MqJq4d0vrrEx",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "299a2b9e-7edc-4297-904f-77e0986fa46f"
},
"source": [
"len(labeled)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1200"
]
},
"metadata": {
"tags": []
},
"execution_count": 140
}
]
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "gqBx9sLBrrE0",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "611e2fae-ed6a-481d-f849-9b8633841685"
},
"source": [
"(dev2[\"pred\"]==dev2[\"label\"]).sum()/len(dev2)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.4225"
]
},
"metadata": {
"tags": []
},
"execution_count": 141
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "ebBj9TLb_ABx",
"colab_type": "code",
"colab": {}
},
"source": [
"myPositives = [13, 503, 1000]\n",
"myExamples = []\n",
"\n",
"for row in myPositives:\n",
" myExamples.append(train[train.label==\"n\"].reset_index().loc()[row])\n",
" myExamples.append(train[train.label==\"c\"].reset_index().loc()[row])\n",
" myExamples.append(train[train.label==\"e\"].reset_index().loc()[row])\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "VeCvoYEqCrTZ",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 802
},
"outputId": "7931ba6e-7aa5-4107-bcbe-3dc0f217b897"
},
"source": [
"for row in myExamples:\n",
" print(\"Entry '{}'\".format(row[\"context\"]))\n",
" print(\"Update '{}'\".format(row[\"hypothesis\"]))\n",
" print(\"Topic''\")\n",
" print(\"Jerry: {}\".format(row[\"label\"]))\n",
" print()"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Entry 'The four-time world champion has established himself as one of the world’s greatest ever drivers since he broke onto the scene in 2007. But his outspoken character has often got him in trouble off the track, most recently with Max Verstappen. Hamilton’s former team-mate Jenson Button even said he was “weird” to work with. Rob Wilson – who has taught over half of the current F1 circuit how to drive faster – also believes there is something different about the Mercedes ace.'\n",
"Update 'many many people thing the driver is weird'\n",
"Topic''\n",
"Jerry: n\n",
"\n",
"Entry 'Burks (shoulder) is inactive for Sunday's game against the Vikings, VP of Communications for the Packers Jason Wahlers reports. Burks will continue waiting for his first regular season action, and he'll set his sights on a Week 3 matchup with the Redskins. In his place, a mix of Antonio Morrison and James Crawford will fill in at inside linebacker against a strong Vikings' running game.'\n",
"Update 'On Sunday the game against the Viking's will include Morrison and Crawford filling in for Burks as opposed to the previous game (week 3) against the Redskins'\n",
"Topic''\n",
"Jerry: c\n",
"\n",
"Entry 'Join Mix 106.5 this Sunday from 11am to 1pm at the Giant on Daybreak Circle in Clarksville and help a family in need this holiday season! Giant's \"Food for Families\" drive is in full swing – just drop off non-perishable items in our bins at the front of the store. From canned cranberry sauce and pumpkin to boxed stuffing – all items will be donated to local Feeding America Food banks, ensuring that local families have a happy Thanksgiving. The food drive ends on on November 23rd so be sure to donate early!'\n",
"Update 'Mix 106.5 want to do something to help poor families'\n",
"Topic''\n",
"Jerry: e\n",
"\n",
"Entry 'The Corn<br>Gary had a plot of land. He decided to plant some produce on it. He picked corn for the first crop. It grew tall and healthy. He harvested it and made some delicious food.'\n",
"Update 'Gary plants delicious fruits.'\n",
"Topic''\n",
"Jerry: n\n",
"\n",
"Entry 'It was all a mistake .<br>So Sir John told Grimes to go home , and promised him five shillings if he would bring the boy quietly up to him , without beating him , that he might be sure of the truth .<br>For he took for granted , and Grimes too , that Tom had made his way home .'\n",
"Update 'Sir John told Grimes to go to the corner market. '\n",
"Topic''\n",
"Jerry: c\n",
"\n",
"Entry 'Eric's Mistake<br>Eric was going on a road trip to Harper's Ferry. He set his GPS to the route, but the battery died part way through. Eric ended up making several wrong turns and getting very lost. Eric refused to ask for directions and ran out of gas. Eric had to walk 5 miles to a gas station to get more gas.'\n",
"Update 'eric made too many wrong turns'\n",
"Topic''\n",
"Jerry: e\n",
"\n",
"Entry 'How to operate a segway<br>Make sure your segway is fully charged by monitoring the battery indicator on the lcd panel. Ensure that the kickstand is fully functioning and holding the segway upright. Turn the segway on by quickly hitting the \" on/off \" button.'\n",
"Update 'The lcd panel holds the battery indicator and the on/off button.'\n",
"Topic''\n",
"Jerry: n\n",
"\n",
"Entry 'How to invite friends to like your page on facebook pages manager<br>Download and install the application. Facebook pages manager is available on both itunes app store and google play for android. Log in your personal facebook account.'\n",
"Update 'Facebook page manager is not available for apple devices. '\n",
"Topic''\n",
"Jerry: c\n",
"\n",
"Entry 'How to deal with your friend who is being jealous of your other friends<br>Invite your friend to join the group. This might feel hard if your friend has made occasions with your other friends uncomfortable in the past. However, stay positive.'\n",
"Update 'The agent suggest inviting the friends to hang out with your other friends'\n",
"Topic''\n",
"Jerry: e\n",
"\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZaP9C86tQooH",
"colab_type": "text"
},
"source": [
"# K, let's try 3 of each"
]
},
{
"cell_type": "code",
"metadata": {
"id": "b7VO7CCoKXJR",
"colab_type": "code",
"colab": {}
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "p3sGHid-Qp00",
"colab": {}
},
"source": [
"myPositives = [13, 503, 1000]\n",
"myExamples = []\n",
"\n",
"for row in myPositives:\n",
" myExamples.append(train[train.label==\"n\"].reset_index().loc()[row])\n",
" myExamples.append(train[train.label==\"c\"].reset_index().loc()[row])\n",
" myExamples.append(train[train.label==\"e\"].reset_index().loc()[row])\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "Pq2M7zfvQp05",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 802
},
"outputId": "589d4603-9644-4537-e7cf-df43a557d957"
},
"source": [
"for row in myExamples:\n",
" print(\"Entry '{}'\".format(row[\"context\"]))\n",
" print(\"Update '{}'\".format(row[\"hypothesis\"]))\n",
" print(\"Topic''\")\n",
" print(\"Jerry: {}\".format(row[\"label\"]))\n",
" print()"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Entry 'The four-time world champion has established himself as one of the world’s greatest ever drivers since he broke onto the scene in 2007. But his outspoken character has often got him in trouble off the track, most recently with Max Verstappen. Hamilton’s former team-mate Jenson Button even said he was “weird” to work with. Rob Wilson – who has taught over half of the current F1 circuit how to drive faster – also believes there is something different about the Mercedes ace.'\n",
"Update 'many many people thing the driver is weird'\n",
"Topic''\n",
"Jerry: n\n",
"\n",
"Entry 'Burks (shoulder) is inactive for Sunday's game against the Vikings, VP of Communications for the Packers Jason Wahlers reports. Burks will continue waiting for his first regular season action, and he'll set his sights on a Week 3 matchup with the Redskins. In his place, a mix of Antonio Morrison and James Crawford will fill in at inside linebacker against a strong Vikings' running game.'\n",
"Update 'On Sunday the game against the Viking's will include Morrison and Crawford filling in for Burks as opposed to the previous game (week 3) against the Redskins'\n",
"Topic''\n",
"Jerry: c\n",
"\n",
"Entry 'Join Mix 106.5 this Sunday from 11am to 1pm at the Giant on Daybreak Circle in Clarksville and help a family in need this holiday season! Giant's \"Food for Families\" drive is in full swing – just drop off non-perishable items in our bins at the front of the store. From canned cranberry sauce and pumpkin to boxed stuffing – all items will be donated to local Feeding America Food banks, ensuring that local families have a happy Thanksgiving. The food drive ends on on November 23rd so be sure to donate early!'\n",
"Update 'Mix 106.5 want to do something to help poor families'\n",
"Topic''\n",
"Jerry: e\n",
"\n",
"Entry 'The Corn<br>Gary had a plot of land. He decided to plant some produce on it. He picked corn for the first crop. It grew tall and healthy. He harvested it and made some delicious food.'\n",
"Update 'Gary plants delicious fruits.'\n",
"Topic''\n",
"Jerry: n\n",
"\n",
"Entry 'It was all a mistake .<br>So Sir John told Grimes to go home , and promised him five shillings if he would bring the boy quietly up to him , without beating him , that he might be sure of the truth .<br>For he took for granted , and Grimes too , that Tom had made his way home .'\n",
"Update 'Sir John told Grimes to go to the corner market. '\n",
"Topic''\n",
"Jerry: c\n",
"\n",
"Entry 'Eric's Mistake<br>Eric was going on a road trip to Harper's Ferry. He set his GPS to the route, but the battery died part way through. Eric ended up making several wrong turns and getting very lost. Eric refused to ask for directions and ran out of gas. Eric had to walk 5 miles to a gas station to get more gas.'\n",
"Update 'eric made too many wrong turns'\n",
"Topic''\n",
"Jerry: e\n",
"\n",
"Entry 'How to operate a segway<br>Make sure your segway is fully charged by monitoring the battery indicator on the lcd panel. Ensure that the kickstand is fully functioning and holding the segway upright. Turn the segway on by quickly hitting the \" on/off \" button.'\n",
"Update 'The lcd panel holds the battery indicator and the on/off button.'\n",
"Topic''\n",
"Jerry: n\n",
"\n",
"Entry 'How to invite friends to like your page on facebook pages manager<br>Download and install the application. Facebook pages manager is available on both itunes app store and google play for android. Log in your personal facebook account.'\n",
"Update 'Facebook page manager is not available for apple devices. '\n",
"Topic''\n",
"Jerry: c\n",
"\n",
"Entry 'How to deal with your friend who is being jealous of your other friends<br>Invite your friend to join the group. This might feel hard if your friend has made occasions with your other friends uncomfortable in the past. However, stay positive.'\n",
"Update 'The agent suggest inviting the friends to hang out with your other friends'\n",
"Topic''\n",
"Jerry: e\n",
"\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "u6PpABDTleIE",
"colab_type": "code",
"colab": {}
},
"source": [
"entryPrompt2E3N = \"\"\"Entry: 'The four-time world champion has established himself as one of the world’s greatest ever drivers since he broke onto the scene in 2007. But his outspoken character has often got him in trouble off the track, most recently with Max Verstappen. Hamilton’s former team-mate Jenson Button even said he was “weird” to work with. Rob Wilson – who has taught over half of the current F1 circuit how to drive faster – also believes there is something different about the Mercedes ace.'\n",
"Update: 'many many people thing the driver is weird'\n",
"Topic: Number of people who think the driver is weird\n",
"Entry Relevance: Former-team mate thinks he is weird to work with.\n",
"Comparison: Update covers how many people think driver is weird, entry only covers one person not others.\n",
"Alignment: Tangential\n",
"\n",
"Entry: 'Join Mix 106.5 this Sunday from 11am to 1pm at the Giant on Daybreak Circle in Clarksville and help a family in need this holiday season! Giant's \"Food for Families\" drive is in full swing – just drop off non-perishable items in our bins at the front of the store. From canned cranberry sauce and pumpkin to boxed stuffing – all items will be donated to local Feeding America Food banks, ensuring that local families have a happy Thanksgiving. The food drive ends on on November 23rd so be sure to donate early!'\n",
"Update: 'Mix 106.5 want to do something to help poor families'\n",
"Topic: Mix 106.5 helping poor families\n",
"Entry Relevance: Mix 106.5 this sunday is helping families in need.\n",
"Comparison: In entry, Mix 106.5 is helping poor families. In update, Mix 106.5 is helping poor families.\n",
"Alignment: Agreement\n",
"\n",
"Entry: 'Eric's Mistake<br>Eric was going on a road trip to Harper's Ferry. He set his GPS to the route, but the battery died part way through. Eric ended up making several wrong turns and getting very lost. Eric refused to ask for directions and ran out of gas. Eric had to walk 5 miles to a gas station to get more gas.'\n",
"Update: 'eric made too many wrong turns'\n",
"Topic: Eric and Wrong Turns\n",
"Entry Relevance: Eric made several wrong turns and ran out of gas.\n",
"Comparison: In entry, Eric ended up making several wrong turns. In update, eric made wrong turns. Both confirm Eric made wrong turns.\n",
"Alignment: Agreement\n",
"\n",
"Entry: 'The Corn<br>Gary had a plot of land. He decided to plant some produce on it. He picked corn for the first crop. It grew tall and healthy. He harvested it and made some delicious food.'\n",
"Update: 'Gary plants delicious fruits.'\n",
"Topic: Gary planting fruit\n",
"Entry Relevance: Gary planted produce and picked corn.\n",
"Comparison: In entry, Gary planted plants, among which was corn. In update, Gary planted fruit. Gary could have planted fruit but isn't clear from entry.\n",
"Alignment: Tangential\n",
"\n",
"Entry 'How to operate a segway<br>Make sure your segway is fully charged by monitoring the battery indicator on the lcd panel. Ensure that the kickstand is fully functioning and holding the segway upright. Turn the segway on by quickly hitting the \" on/off \" button.'\n",
"Update 'The lcd panel holds the battery indicator and the on/off button.'\n",
"Topic: LCD panel holds items.\n",
"Entry Relevance: LCD panel holds battery indicator.\n",
"Comparison: In entry, Gathe LCD panel holds battery indicator. In update, the panel holds the battery indicator. The panel could hold the battery indicator in entry but it isn't clear either way.\n",
"Alignment: Tangential\n",
"\n",
"\"\"\"\n",
"\n",
"entryPrompt2C3N = \"\"\"Entry: 'The four-time world champion has established himself as one of the world’s greatest ever drivers since he broke onto the scene in 2007. But his outspoken character has often got him in trouble off the track, most recently with Max Verstappen. Hamilton’s former team-mate Jenson Button even said he was “weird” to work with. Rob Wilson – who has taught over half of the current F1 circuit how to drive faster – also believes there is something different about the Mercedes ace.'\n",
"Update: 'many many people thing the driver is weird'\n",
"Topic: Number of people who think the driver is weird\n",
"Entry Relevance: Former-team mate thinks he is weird to work with.\n",
"Comparison: Update covers how many people think driver is weird, entry only covers one person not others.\n",
"Alignment: Tangential\n",
"\n",
"Entry: 'Burks (shoulder) is inactive for Sunday's game against the Vikings, VP of Communications for the Packers Jason Wahlers reports. Burks will continue waiting for his first regular season action, and he'll set his sights on a Week 3 matchup with the Redskins. In his place, a mix of Antonio Morrison and James Crawford will fill in at inside linebacker against a strong Vikings' running game.'\n",
"Update: 'On Sunday the game against the Viking's will include Morrison and Crawford filling in for Burks as opposed to the previous game (week 3) against the Redskins'\n",
"Topic: Participation in Sunday game compared to previous week 3 game.\n",
"Entry Relevance: Burks is inactive, Morrison and Crawford wlll fill in, Burks will be back in week 3.\n",
"Comparison In entry, Burks is coming back for week 3. In update week 3 happened. They differ in whether week 3 was in the past or future.\n",
"Alignment: Contradictory\n",
"\n",
"Entry: 'It was all a mistake .<br>So Sir John told Grimes to go home , and promised him five shillings if he would bring the boy quietly up to him , without beating him , that he might be sure of the truth .<br>For he took for granted , and Grimes too , that Tom had made his way home .'\n",
"Update: 'Sir John told Grimes to go to the corner market. '\n",
"Topic: Sir John instructing Grimes\n",
"Entry Relevance: Sir John told Grimes to go home.\n",
"Comparison In entry, Sir John told Grimes to go home. In update, Sir John told Grimes to go to a market. They contradict eachother on where Sir John told Grimes to go.\n",
"Alignment: Contradictory\n",
"\n",
"Entry: 'The Corn<br>Gary had a plot of land. He decided to plant some produce on it. He picked corn for the first crop. It grew tall and healthy. He harvested it and made some delicious food.'\n",
"Update: 'Gary plants delicious fruits.'\n",
"Topic: Gary planting fruit\n",
"Entry Relevance: Gary planted produce and picked corn.\n",
"Comparison: In entry, Gary planted plants, among which was corn. In update, Gary planted fruit. Gary could have planted fruit but isn't clear from entry.\n",
"Alignment: Tangential\n",
"\n",
"Entry 'How to operate a segway<br>Make sure your segway is fully charged by monitoring the battery indicator on the lcd panel. Ensure that the kickstand is fully functioning and holding the segway upright. Turn the segway on by quickly hitting the \" on/off \" button.'\n",
"Update 'The lcd panel holds the battery indicator and the on/off button.'\n",
"Topic: LCD panel holds items.\n",
"Entry Relevance: LCD panel holds battery indicator.\n",
"Comparison: In entry, Gathe LCD panel holds battery indicator. In update, the panel holds the battery indicator. The panel could hold the battery indicator in entry but it isn't clear either way.\n",
"Alignment: Tangential\n",
"\n",
"\"\"\""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "1uQ8i0ozRMGB",
"colab": {}
},
"source": [
"labeled = pd.read_pickle(\"/content/drive/My Drive/GPT3_Benchmarking/ANLI_EC_LABELED4225.pkl\")"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "nRdcqfT1bVj8",
"colab_type": "code",
"colab": {}
},
"source": [
"labeled.to_json(\"anli_first_pass_labeled_dev_3.json\", orient=\"records\")"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "v17r2lDBbexK",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 17
},
"outputId": "b1fb1a6a-baab-4c73-8bd4-861d403011f0"
},
"source": [
"files.download(\"anli_first_pass_labeled_dev_3.json\")"
],
"execution_count": null,
"outputs": [
{
"output_type": "display_data",
"data": {
"application/javascript": [
"\n",
" async function download(id, filename, size) {\n",
" if (!google.colab.kernel.accessAllowed) {\n",
" return;\n",
" }\n",
" const div = document.createElement('div');\n",
" const label = document.createElement('label');\n",
" label.textContent = `Downloading \"${filename}\": `;\n",
" div.appendChild(label);\n",
" const progress = document.createElement('progress');\n",
" progress.max = size;\n",
" div.appendChild(progress);\n",
" document.body.appendChild(div);\n",
"\n",
" const buffers = [];\n",
" let downloaded = 0;\n",
"\n",
" const channel = await google.colab.kernel.comms.open(id);\n",
" // Send a message to notify the kernel that we're ready.\n",
" channel.send({})\n",
"\n",
" for await (const message of channel.messages) {\n",
" // Send a message to notify the kernel that we're ready.\n",
" channel.send({})\n",
" if (message.buffers) {\n",
" for (const buffer of message.buffers) {\n",
" buffers.push(buffer);\n",
" downloaded += buffer.byteLength;\n",
" progress.value = downloaded;\n",
" }\n",
" }\n",
" }\n",
" const blob = new Blob(buffers, {type: 'application/binary'});\n",
" const a = document.createElement('a');\n",
" a.href = window.URL.createObjectURL(blob);\n",
" a.download = filename;\n",
" div.appendChild(a);\n",
" a.click();\n",
" div.remove();\n",
" }\n",
" "
],
"text/plain": [
"<IPython.core.display.Javascript object>"
]
},
"metadata": {
"tags": []
}
},
{
"output_type": "display_data",
"data": {
"application/javascript": [
"download(\"download_cedb5d55-24ec-47bf-8242-7d9a3495a645\", \"anli_first_pass_labeled_dev_3.json\", 999176)"
],
"text/plain": [
"<IPython.core.display.Javascript object>"
]
},
"metadata": {
"tags": []
}
}
]
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "ztIGi5i1RMGE",
"colab": {}
},
"source": [
"dev2 = labeled.copy()\n",
"dev2[\"pred\"] = dev2[\"r2\"].apply(lambda x: \"e\" if x == \"understood\" else \"c\")\n",
"\n",
"for row in dev2.iterrows():\n",
" break"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "MtOz8aYSRMGG",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "b2a0ae99-3d01-46a0-bd0f-fd96bb66e3fc"
},
"source": [
"len(labeled)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1200"
]
},
"metadata": {
"tags": []
},
"execution_count": 19
}
]
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "JvYymb9dRMGI",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "788f7ca4-60e7-4a1d-d89f-6567ea10eab7"
},
"source": [
"(dev2[\"pred\"]==dev2[\"label\"]).sum()/len(dev2)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.4225"
]
},
"metadata": {
"tags": []
},
"execution_count": 20
}
]
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "tWMXa7UuRMGM",
"colab": {}
},
"source": [
"for row in dev2.iterrows():\n",
" if row[0] % 10 == 0:\n",
" print(row[0])\n",
" dev2.to_pickle(\"/content/drive/My Drive/GPT3_Benchmarking/ANLI_dev_2E3N_2C3N.pkl\")\n",
"\n",
" try:\n",
" a = row[1][\"r5\"]\n",
" if type(a) == str:\n",
" continue\n",
" except:\n",
" pass\n",
" \n",
" payload = \"Entry: '{}'\\n\".format(row[1][\"context\"])\n",
" payload += \"Update: '{}'\\n\".format(row[1][\"hypothesis\"])\n",
" payload += \"Topic:\"\n",
"\n",
" if row[1][\"pred\"] == \"c\":\n",
"\n",
" r = query(entryPrompt2C3N + payload)\n",
" dev2.at[row[0], \"r5\"] = r\n",
" lastWord = r.split()[-1]\n",
"\n",
" myLabel = \"c\"\n",
" if lastWord.lower().startswith(\"t\"):\n",
" myLabel = \"n\"\n",
"\n",
" dev2.at[row[0], \"pred2\"] = myLabel\n",
"\n",
" if row[1][\"pred\"] == \"e\":\n",
" r = query(entryPrompt2E3N + payload)\n",
" dev2.at[row[0], \"r5\"] = r\n",
" lastWord = r.split()[-1]\n",
"\n",
" myLabel = \"e\"\n",
" if lastWord.lower().startswith(\"t\"):\n",
" myLabel = \"n\"\n",
"\n",
" dev2.at[row[0], \"pred2\"] = myLabel\n",
"\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "QGwV1OdqRUNv",
"colab_type": "code",
"colab": {}
},
"source": [
"dev2.to_pickle(\"/content/drive/My Drive/GPT3_Benchmarking/ANLI_dev_2E3N_2C3N.pkl\")\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "S3fyrhRWWyDU",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "8b251831-1dff-4e03-a97a-220aa5c38a6b"
},
"source": [
"dev3 = dev2.copy()\n",
"(dev3[\"pred2\"]==dev3[\"label\"]).sum()/len(dev3)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.41583333333333333"
]
},
"metadata": {
"tags": []
},
"execution_count": 101
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "LHH9D12UW0fW",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "1e0194b7-ed4f-43c2-fbc0-8ba0b512750b"
},
"source": [
"(dev3[\"pred\"]==dev3[\"label\"]).sum()/len(dev3)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.4225"
]
},
"metadata": {
"tags": []
},
"execution_count": 102
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "_68T3KFMcWQU",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "f6e80e2f-4a36-4192-dabd-27b6a3510f39"
},
"source": [
"dev3 = dev2[:row[0]].copy()\n",
"dev3 = dev3[dev3.label==\"c\"]\n",
"(dev3[\"pred2\"]==dev3[\"label\"]).sum()/len(dev3)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.3620253164556962"
]
},
"metadata": {
"tags": []
},
"execution_count": 90
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "-C7Jotglcb6B",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "4d50970e-90a8-4fc8-de88-8091da605ea4"
},
"source": [
"dev3 = dev2[:row[0]].copy()\n",
"dev3 = dev3[dev3.label==\"e\"]\n",
"(dev3[\"pred2\"]==dev3[\"label\"]).sum()/len(dev3)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.6517412935323383"
]
},
"metadata": {
"tags": []
},
"execution_count": 91
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "9ZVu48Wbcm-4",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "d991efae-a4e7-4239-e50c-b2b4f528ec82"
},
"source": [
"dev3 = dev2[:row[0]].copy()\n",
"dev3 = dev3[dev3.label==\"e\"]\n",
"(dev3[\"pred\"]==dev3[\"label\"]).sum()/len(dev3)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.6741293532338308"
]
},
"metadata": {
"tags": []
},
"execution_count": 92
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "JD9oFlYmcdPo",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "824bfc2a-833c-47c4-e2ce-d8df64cc9447"
},
"source": [
"dev3 = dev2[:row[0]].copy()\n",
"dev3 = dev3[dev3.label==\"n\"]\n",
"(dev3[\"pred2\"]==dev3[\"label\"]).sum()/len(dev3)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.23134328358208955"
]
},
"metadata": {
"tags": []
},
"execution_count": 93
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "WDrCyF8wcobR",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "8063a93c-10f6-4043-929d-2e1710d57a45"
},
"source": [
"dev3 = dev2[:row[0]].copy()\n",
"dev3 = dev3[dev3.label==\"n\"]\n",
"(dev3[\"pred\"]==dev3[\"label\"]).sum()/len(dev3)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.0"
]
},
"metadata": {
"tags": []
},
"execution_count": 94
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "cjs2Rw_Ichd_",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "6401209d-562c-49b3-98fa-ec80013b291d"
},
"source": [
"dev3 = dev2[:row[0]].copy()\n",
"dev3 = dev3[dev3.label==\"e\"]\n",
"(dev3[\"pred\"]==dev3[\"label\"]).sum()/len(dev3)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.6741293532338308"
]
},
"metadata": {
"tags": []
},
"execution_count": 95
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "2Xx8gs55cppg",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "b8a2bc51-83ed-4bf3-c7c8-53375a77c370"
},
"source": [
"dev3 = dev2[:row[0]].copy()\n",
"dev3 = dev3[dev3.label==\"c\"]\n",
"(dev3[\"pred\"]==dev3[\"label\"]).sum()/len(dev3)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.5949367088607594"
]
},
"metadata": {
"tags": []
},
"execution_count": 96
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "6041vf5VfMnR",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "4366d3b0-237d-44d2-8595-d9ae3842860d"
},
"source": [
"dev3 = dev2.copy()\n",
"for row2 in dev3.iterrows():\n",
" if row2[1][\"pred\"] == \"c\":\n",
" dev3.at[row2[0], \"pred2\"] = \"c\"\n",
"(dev3[\"pred2\"]==dev3[\"label\"]).sum()/len(dev3)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.43"
]
},
"metadata": {
"tags": []
},
"execution_count": 99
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "wCBsPd9sfiR_",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "e3b89dfc-a5fd-4ab4-f65a-4d81fced7bec"
},
"source": [
"dev3 = dev2.copy()\n",
"for row2 in dev3.iterrows():\n",
" if row2[1][\"pred\"] == \"e\":\n",
" dev3.at[row2[0], \"pred2\"] = \"e\"\n",
"(dev3[\"pred2\"]==dev3[\"label\"]).sum()/len(dev3)"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.4083333333333333"
]
},
"metadata": {
"tags": []
},
"execution_count": 100
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "fwoXLNUqzKaT",
"colab_type": "code",
"colab": {}
},
"source": [
"dev3[\"lines\"] = dev3.r5.apply(lambda x: len(x.split(\"\\n\")))"
],
"execution_count": null,
"outputs": []
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment