Created
April 18, 2024 07:37
-
-
Save skilfoy/460fffe1e4a5990e3bccaee238b3e0df to your computer and use it in GitHub Desktop.
Analyzing Car Reviews with LLMs.ipynb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "cells": [ | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "view-in-github", | |
| "colab_type": "text" | |
| }, | |
| "source": [ | |
| "<a href=\"https://colab.research.google.com/gist/skilfoy/460fffe1e4a5990e3bccaee238b3e0df/analyzing-car-reviews-with-llms.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>" | |
| ] | |
| }, | |
| { | |
| "source": [ | |
| "# Analyzing Car Reviews with LLMs\n", | |
| "\n", | |
| "## Introduction: Harnessing the Power of AI in Automotive Customer Engagement\n", | |
| "\n", | |
| "In the rapidly evolving landscape of the automotive industry, technological advancements are not just reshaping product development but are also transforming how companies interact with their customers. Artificial Intelligence (AI), especially through the use of Large Language Models (LLMs), presents a unique opportunity to enhance customer service and operational efficiency. As businesses strive to meet the increasing expectations of personalized service and rapid response times, AI emerges as a crucial tool in redefining customer engagement strategies.\n", | |
| "\n", | |
| "At \"Car-ing is sharing,\" a forward-thinking auto dealership, the integration of AI into customer interactions is more than just a technological upgrade; it's a strategic move to deepen customer relationships and streamline internal processes. The deployment of AI-driven solutions, such as chatbots and automated analysis tools, offers a dual advantage: enhancing the customer experience through quick and accurate responses, and alleviating the workload on human agents by handling routine inquiries and data processing tasks.\n", | |
| "\n", | |
| "This notebook outlines a prototype that leverages various pre-trained LLMs from the Hugging Face transformers library to perform a series of tasks crucial for customer interaction and backend analytics. These tasks include sentiment analysis of customer reviews, translation of text for non-English speaking customers, answering specific queries with precision, and summarizing large volumes of text for quick insights.\n", | |
| "\n", | |
| "By incorporating these AI capabilities, \"Car-ing is sharing\" aims not only to meet the current needs of its customers and staff but also to adapt to future demands, ensuring the company remains at the forefront of the automotive industry's technological frontier.\n", | |
| "\n" | |
| ], | |
| "metadata": { | |
| "id": "3668f4f4-35e8-4579-834f-0f43488d15de" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "3668f4f4-35e8-4579-834f-0f43488d15de" | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "from google.colab import drive\n", | |
| "drive.mount('/content/drive')" | |
| ], | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "T2-eTdkvFTwx", | |
| "outputId": "f47c5aa7-e7c7-4fc4-aba4-0bebf7f98650" | |
| }, | |
| "id": "T2-eTdkvFTwx", | |
| "execution_count": 1, | |
| "outputs": [ | |
| { | |
| "output_type": "stream", | |
| "name": "stdout", | |
| "text": [ | |
| "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n" | |
| ] | |
| } | |
| ] | |
| }, | |
| { | |
| "source": [ | |
| "\n", | |
| "\n", | |
| "#### Fictitious Pretense:\n", | |
| "\n", | |
| "**Car-ing is sharing**, an auto dealership company for car sales and rental, is taking their services to the next level thanks to **Large Language Models (LLMs)**.\n", | |
| "\n", | |
| "As their newly recruited AI and NLP developer, you've been asked to prototype a chatbot app with multiple functionalities that not only assist customers but also provide support to human agents in the company.\n", | |
| "\n", | |
| "The solution should receive textual prompts and use a variety of pre-trained Hugging Face LLMs to respond to a series of tasks, e.g. classifying the sentiment in a car’s text review, answering a customer question, summarizing or translating text, etc.\n" | |
| ], | |
| "metadata": { | |
| "id": "9aabafca-8129-4943-b865-d5e897637253" | |
| }, | |
| "id": "9aabafca-8129-4943-b865-d5e897637253", | |
| "cell_type": "markdown" | |
| }, | |
| { | |
| "source": [ | |
| "!pip install transformers > /dev/null 2>&1\n", | |
| "!pip install evaluate > /dev/null 2>&1\n", | |
| "\n", | |
| "from transformers import AutoModelForSequenceClassification, AutoTokenizer, AutoModelForSeq2SeqLM, pipeline\n", | |
| "import evaluate\n", | |
| "from evaluate import load\n", | |
| "import pandas as pd\n", | |
| "import torch\n", | |
| "\n", | |
| "from transformers import logging as hf_logging\n", | |
| "hf_logging.set_verbosity(hf_logging.WARNING)" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 10747, | |
| "lastExecutedAt": 1713424049895, | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "!pip install transformers\n!pip install evaluate\n\nfrom transformers import AutoModelForSequenceClassification, AutoTokenizer, AutoModelForSeq2SeqLM, pipeline\nimport evaluate\nfrom evaluate import load\nimport pandas as pd\nimport torch\n\nfrom transformers import logging as hf_logging\nhf_logging.set_verbosity(hf_logging.WARNING)", | |
| "outputsMetadata": { | |
| "0": { | |
| "height": 101, | |
| "type": "stream" | |
| } | |
| }, | |
| "collapsed": true, | |
| "jupyter": { | |
| "outputs_hidden": true, | |
| "source_hidden": false | |
| }, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "id": "5325a4c0-ceb3-4b66-acd2-5eadcefe3a63" | |
| }, | |
| "id": "5325a4c0-ceb3-4b66-acd2-5eadcefe3a63", | |
| "cell_type": "code", | |
| "execution_count": 2, | |
| "outputs": [] | |
| }, | |
| { | |
| "source": [ | |
| "To address the tasks proposed by the CTO at \"Car-ing is sharing\", we will employ various pre-trained Large Language Models (LLMs) from the Hugging Face `transformers` library. This solution will provide a robust prototype for a chatbot app capable of sentiment analysis, translation, question answering, and summarization. Each task is designed to support the customer service operations of a car sales and rental company." | |
| ], | |
| "metadata": { | |
| "id": "5b12b181-0dee-444c-909e-4f46a43f371b" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "5b12b181-0dee-444c-909e-4f46a43f371b" | |
| }, | |
| { | |
| "source": [ | |
| "# Sentiment Analysis of Car Reviews\n", | |
| "\n", | |
| "We'll classify the sentiment of car reviews using a pre-trained BERT model specifically fine-tuned for sentiment analysis." | |
| ], | |
| "metadata": { | |
| "id": "89039478-62fd-4bbd-babb-bd37ca28a762" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "89039478-62fd-4bbd-babb-bd37ca28a762" | |
| }, | |
| { | |
| "source": [ | |
| "### Load and prepare data" | |
| ], | |
| "metadata": { | |
| "id": "b20f5801-3f8e-4968-9ffc-239f55d58660" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "b20f5801-3f8e-4968-9ffc-239f55d58660" | |
| }, | |
| { | |
| "source": [ | |
| "df = pd.read_csv(\"/content/drive/MyDrive/Colab Notebooks/DataCamp Projects/workspace - Analyzing Car Reviews with LLMs/data/car_reviews.csv\", delimiter=';')\n", | |
| "reviews = df['Review'].tolist()\n", | |
| "true_labels = df['Class'].tolist()" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 50, | |
| "lastExecutedAt": 1713424049946, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "df = pd.read_csv(\"data/car_reviews.csv\", delimiter=';')\nreviews = df['Review'].tolist()\ntrue_labels = df['Class'].tolist()", | |
| "collapsed": true, | |
| "jupyter": { | |
| "outputs_hidden": true, | |
| "source_hidden": false | |
| }, | |
| "id": "8d42bc1e-4db2-4836-bb8a-5557fce12182" | |
| }, | |
| "cell_type": "code", | |
| "id": "8d42bc1e-4db2-4836-bb8a-5557fce12182", | |
| "outputs": [], | |
| "execution_count": 3 | |
| }, | |
| { | |
| "source": [ | |
| "### Load a sentiment analysis LLM into the pipeline" | |
| ], | |
| "metadata": { | |
| "id": "56b9a83b-3791-400e-9c00-e2a32ab149e1" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "56b9a83b-3791-400e-9c00-e2a32ab149e1" | |
| }, | |
| { | |
| "source": [ | |
| "classifier = pipeline('sentiment-analysis', model='distilbert-base-uncased-finetuned-sst-2-english')" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 752, | |
| "lastExecutedAt": 1713424050699, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "classifier = pipeline('sentiment-analysis', model='distilbert-base-uncased-finetuned-sst-2-english')", | |
| "outputsMetadata": { | |
| "0": { | |
| "height": 80, | |
| "type": "stream" | |
| } | |
| }, | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "25c9e442-1dbf-4484-be70-d44c405ba7bb", | |
| "outputId": "bf1d9474-2cb8-4ceb-e6ed-bdbb21f9caf7" | |
| }, | |
| "cell_type": "code", | |
| "id": "25c9e442-1dbf-4484-be70-d44c405ba7bb", | |
| "outputs": [ | |
| { | |
| "output_type": "stream", | |
| "name": "stderr", | |
| "text": [ | |
| "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: \n", | |
| "The secret `HF_TOKEN` does not exist in your Colab secrets.\n", | |
| "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n", | |
| "You will be able to reuse this secret in all of your notebooks.\n", | |
| "Please note that authentication is recommended but still optional to access public models or datasets.\n", | |
| " warnings.warn(\n" | |
| ] | |
| } | |
| ], | |
| "execution_count": 4 | |
| }, | |
| { | |
| "source": [ | |
| "### Perform inference on the car reviews and display prediction results" | |
| ], | |
| "metadata": { | |
| "id": "fbce2a5f-563e-4701-bbe1-ce8834859338" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "fbce2a5f-563e-4701-bbe1-ce8834859338" | |
| }, | |
| { | |
| "source": [ | |
| "predicted_labels = classifier(reviews)\n", | |
| "for review, prediction, label in zip(reviews, predicted_labels, true_labels):\n", | |
| " print(f\"Review: {review}\\nActual Sentiment: {label}\\nPredicted Sentiment: {prediction['label']} (Confidence: {prediction['score']:.4f})\\n\")" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 3256, | |
| "lastExecutedAt": 1713424053957, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "predicted_labels = classifier(reviews)\nfor review, prediction, label in zip(reviews, predicted_labels, true_labels):\n print(f\"Review: {review}\\nActual Sentiment: {label}\\nPredicted Sentiment: {prediction['label']} (Confidence: {prediction['score']:.4f})\\n\")", | |
| "outputsMetadata": { | |
| "0": { | |
| "height": 616, | |
| "type": "stream" | |
| } | |
| }, | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "bee9e720-96ee-4862-ba9d-edd2805957b6", | |
| "outputId": "0d5a4663-978b-49a2-9485-dfca8cbcc31a" | |
| }, | |
| "cell_type": "code", | |
| "id": "bee9e720-96ee-4862-ba9d-edd2805957b6", | |
| "outputs": [ | |
| { | |
| "output_type": "stream", | |
| "name": "stdout", | |
| "text": [ | |
| "Review: I am very satisfied with my 2014 Nissan NV SL. I use this van for my business deliveries and personal use. Camping, road trips, etc. We dont have any children so I store most of the seats in my warehouse. I wanted the passenger van for the rear air conditioning. We drove our van from Florida to California for a Cross Country trip in 2014. We averaged about 18 mpg. We drove thru a lot of rain and It was a very comfortable and stable vehicle. The V8 Nissan Titan engine is a 500k mile engine. It has been tested many times by delivery and trucking companies. This is why Nissan gives you a 5 year or 100k mile bumper to bumper warranty. Many people are scared about driving this van because of its size. But with front and rear sonar sensors, large mirrors and the back up camera. It is easy to drive. The front and rear sensors also monitor the front and rear sides of the bumpers making it easier to park close to objects. Our Nissan NV is a Tow Monster. It pulls our 5000 pound travel trailer like its not even there. I have plenty of power to pass a vehicle if needed. The 5.6 liter engine produces 317 hp. I have owned Chevy and Ford vans and there were not very comfortable and had little cockpit room. The Nissan NV is the only vehicle made that has the engine forward like a pick up truck giving the driver plenty of room and comfort in the cockpit area. I dont have any negatives to say about my NV. This is a wide vehicle. The only modification I would like to see from Nissan is for them to add amber side mirror marker lights.BTW. I now own a 2016 Nissan NVP SL. Love it.\n", | |
| "Actual Sentiment: POSITIVE\n", | |
| "Predicted Sentiment: POSITIVE (Confidence: 0.9294)\n", | |
| "\n", | |
| "Review: The car is fine. It's a bit loud and not very powerful. On one hand, compared to its peers, the interior is well-built. The transmission failed a few years ago, and the dealer replaced it under warranty with no issues. Now, about 60k miles later, the transmission is failing again. It sounds like a truck, and the issues are well-documented. The dealer tells me it is normal, refusing to do anything to resolve the issue. After owning the car for 4 years, there are many other vehicles I would purchase over this one. Initially, I really liked what the brand is about: ride quality, reliability, etc. But I will not purchase another one. Despite these concerns, I must say, the level of comfort in the car has always been satisfactory, but not worth the rest of issues found.\n", | |
| "Actual Sentiment: NEGATIVE\n", | |
| "Predicted Sentiment: POSITIVE (Confidence: 0.8654)\n", | |
| "\n", | |
| "Review: My first foreign car. Love it, I would buy another.\n", | |
| "Actual Sentiment: POSITIVE\n", | |
| "Predicted Sentiment: POSITIVE (Confidence: 0.9995)\n", | |
| "\n", | |
| "Review: I've come across numerous reviews praising the Rogue, and I genuinely feel like I might be missing something. It's only been a week since I got the car, and I am genuinely disappointed. I truly wish I could return it. My main concern revolves around what I see as a significant design flaw (which I believe also exists in the Murano, though that wasn't much better and considerably pricier). The rear windshield is just too small. The headrests in the back seat obstruct the sides of the rearview window. This \"Crossover\" feels more like a cheaply made compact car. My other vehicle is a Sonata, and it provides a significantly quieter and smoother ride. I did not anticipate this car to ride so roughly; my 2006 Pathfinder had a smoother ride! I would rate this car a 5 all around.\n", | |
| "Actual Sentiment: NEGATIVE\n", | |
| "Predicted Sentiment: NEGATIVE (Confidence: 0.9935)\n", | |
| "\n", | |
| "Review: I've been dreaming of owning an SUV for quite a while, but I've been driving cars that were already paid for during an extended period. I ultimately made the decision to transition to a brand-new car, which, of course, involved taking on new payments. However, given that I don't drive extensively, I was inclined to avoid a substantial financial commitment. The Nissan Rogue provides me with the desired SUV experience without burdening me with an exorbitant payment; the financial arrangement is quite reasonable. Handling and styling are great; I have hauled 12 bags of mulch in the back with the seats down and could have held more. I am VERY satisfied overall. I find myself needing to exercise extra caution when making lane changes, particularly owing to the blind spots resulting from the small side windows situated towards the rear of the vehicle. To address this concern, I am actively engaged in making adjustments to my mirrors and consciously reducing the frequency of lane changes. The engine delivers strong performance, and the ride is really smooth.\n", | |
| "Actual Sentiment: POSITIVE\n", | |
| "Predicted Sentiment: POSITIVE (Confidence: 0.9987)\n", | |
| "\n" | |
| ] | |
| } | |
| ], | |
| "execution_count": 5 | |
| }, | |
| { | |
| "source": [ | |
| "### Convert categorical sentiment labels into binary integer labels" | |
| ], | |
| "metadata": { | |
| "id": "8adb662e-e24b-4399-a072-f62eebf69db7" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "8adb662e-e24b-4399-a072-f62eebf69db7" | |
| }, | |
| { | |
| "source": [ | |
| "references = [1 if label == \"POSITIVE\" else 0 for label in true_labels]\n", | |
| "predictions = [1 if prediction == 1 else 0 for prediction in predicted_labels]" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 76, | |
| "lastExecutedAt": 1713424054033, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "references = [1 if label == \"POSITIVE\" else 0 for label in true_labels]\npredictions = [1 if prediction == 1 else 0 for prediction in predicted_labels]", | |
| "id": "1b092edf-0987-4827-8307-cc50569ced95" | |
| }, | |
| "cell_type": "code", | |
| "id": "1b092edf-0987-4827-8307-cc50569ced95", | |
| "outputs": [], | |
| "execution_count": 6 | |
| }, | |
| { | |
| "source": [ | |
| "### Compute accuracy and F1 score" | |
| ], | |
| "metadata": { | |
| "id": "c7451e0f-7098-4be5-984f-f7807e7b6e81" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "c7451e0f-7098-4be5-984f-f7807e7b6e81" | |
| }, | |
| { | |
| "source": [ | |
| "accuracy_metric = evaluate.load(\"accuracy\")\n", | |
| "f1_metric = evaluate.load(\"f1\")\n", | |
| "\n", | |
| "accuracy_result = accuracy_metric.compute(predictions=predictions, references=references)\n", | |
| "accuracy_result = accuracy_result['accuracy']\n", | |
| "f1_result = f1_metric.compute(predictions=predictions, references=references)\n", | |
| "f1_result = f1_result['f1']\n", | |
| "print(\"Accuracy:\", accuracy_result)\n", | |
| "print(\"F1:\", f1_result)" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 1281, | |
| "lastExecutedAt": 1713424055314, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "accuracy_metric = evaluate.load(\"accuracy\")\nf1_metric = evaluate.load(\"f1\")\n\naccuracy_result = accuracy_metric.compute(predictions=predictions, references=references)\naccuracy_result = accuracy_result['accuracy']\nf1_result = f1_metric.compute(predictions=predictions, references=references)\nf1_result = f1_result['f1']\nprint(\"Accuracy:\", accuracy_result)\nprint(\"F1:\", f1_result)", | |
| "outputsMetadata": { | |
| "0": { | |
| "height": 59, | |
| "type": "stream" | |
| } | |
| }, | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "1f3bf8a4-7892-45cd-92db-62daba7ff322", | |
| "outputId": "7ed57e75-ad4c-476f-e963-29044443989c" | |
| }, | |
| "cell_type": "code", | |
| "id": "1f3bf8a4-7892-45cd-92db-62daba7ff322", | |
| "outputs": [ | |
| { | |
| "output_type": "stream", | |
| "name": "stdout", | |
| "text": [ | |
| "Accuracy: 0.4\n", | |
| "F1: 0.0\n" | |
| ] | |
| } | |
| ], | |
| "execution_count": 7 | |
| }, | |
| { | |
| "source": [ | |
| "# Translation of Car Review\n", | |
| "\n", | |
| "Next, we will translate the first review into Spanish and compute the BLEU score to assess the translation quality." | |
| ], | |
| "metadata": { | |
| "id": "c573d477-5b9f-459e-92ee-38420eb61db9" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "c573d477-5b9f-459e-92ee-38420eb61db9" | |
| }, | |
| { | |
| "source": [ | |
| "### Load model and tokenizer for translation" | |
| ], | |
| "metadata": { | |
| "id": "e24dc5a8-065f-4847-988d-b2b0f5ce63ee" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "e24dc5a8-065f-4847-988d-b2b0f5ce63ee" | |
| }, | |
| { | |
| "source": [ | |
| "translator = AutoModelForSeq2SeqLM.from_pretrained(\"Helsinki-NLP/opus-mt-en-es\")\n", | |
| "tokenizer = AutoTokenizer.from_pretrained(\"Helsinki-NLP/opus-mt-en-es\")" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 2187, | |
| "lastExecutedAt": 1713424057502, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "translator = AutoModelForSeq2SeqLM.from_pretrained(\"Helsinki-NLP/opus-mt-en-es\")\ntokenizer = AutoTokenizer.from_pretrained(\"Helsinki-NLP/opus-mt-en-es\")", | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "157f8a6c-677e-436b-8a91-5f89e552b7a1", | |
| "outputId": "f3cc20c4-85d7-4f1a-a790-47dd87070cfe" | |
| }, | |
| "cell_type": "code", | |
| "id": "157f8a6c-677e-436b-8a91-5f89e552b7a1", | |
| "outputs": [ | |
| { | |
| "output_type": "stream", | |
| "name": "stderr", | |
| "text": [ | |
| "/usr/local/lib/python3.10/dist-packages/transformers/models/marian/tokenization_marian.py:197: UserWarning: Recommended: pip install sacremoses.\n", | |
| " warnings.warn(\"Recommended: pip install sacremoses.\")\n" | |
| ] | |
| } | |
| ], | |
| "execution_count": 8 | |
| }, | |
| { | |
| "source": [ | |
| "### Extract first two sentences of the first review" | |
| ], | |
| "metadata": { | |
| "id": "03f474d9-7244-4d2e-9740-f018e971583e" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "03f474d9-7244-4d2e-9740-f018e971583e" | |
| }, | |
| { | |
| "source": [ | |
| "context = reviews[0][:reviews[0].find('.', reviews[0].find('.')+1)+1]" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 46, | |
| "lastExecutedAt": 1713424057550, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "context = reviews[0][:reviews[0].find('.', reviews[0].find('.')+1)+1]", | |
| "id": "b22e6bb4-4172-4176-aeb3-488e9a8f0050" | |
| }, | |
| "cell_type": "code", | |
| "id": "b22e6bb4-4172-4176-aeb3-488e9a8f0050", | |
| "outputs": [], | |
| "execution_count": 9 | |
| }, | |
| { | |
| "source": [ | |
| "### Translate" | |
| ], | |
| "metadata": { | |
| "id": "c78d8dca-d286-4df7-b24f-2191465f0c96" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "c78d8dca-d286-4df7-b24f-2191465f0c96" | |
| }, | |
| { | |
| "source": [ | |
| "inputs = tokenizer(context, return_tensors=\"pt\", truncation=True)\n", | |
| "with torch.no_grad():\n", | |
| " translated_tokens = translator.generate(**inputs, max_length=30)\n", | |
| "translated_review = tokenizer.decode(translated_tokens[0], skip_special_tokens=True)\n", | |
| "print(\"Translated Review:\", translated_review)" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 2894, | |
| "lastExecutedAt": 1713424060445, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "inputs = tokenizer(context, return_tensors=\"pt\", truncation=True)\nwith torch.no_grad():\n translated_tokens = translator.generate(**inputs, max_length=30)\ntranslated_review = tokenizer.decode(translated_tokens[0], skip_special_tokens=True)\nprint(\"Translated Review:\", translated_review)", | |
| "outputsMetadata": { | |
| "0": { | |
| "height": 59, | |
| "type": "stream" | |
| } | |
| }, | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "07d96d34-ed8e-452f-9c12-0cfd9d3e8657", | |
| "outputId": "8d4e621d-3cae-464d-c1f3-b2e0a68f3fbd" | |
| }, | |
| "cell_type": "code", | |
| "id": "07d96d34-ed8e-452f-9c12-0cfd9d3e8657", | |
| "outputs": [ | |
| { | |
| "output_type": "stream", | |
| "name": "stdout", | |
| "text": [ | |
| "Translated Review: Estoy muy satisfecho con mi Nissan NV SL 2014. Uso esta camioneta para mis entregas de negocios y uso personal.\n" | |
| ] | |
| } | |
| ], | |
| "execution_count": 10 | |
| }, | |
| { | |
| "source": [ | |
| "### Load reference translations and compute BLEU score" | |
| ], | |
| "metadata": { | |
| "id": "a1814591-7d05-4e95-bc72-2591828ac68c" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "a1814591-7d05-4e95-bc72-2591828ac68c" | |
| }, | |
| { | |
| "source": [ | |
| "with open(\"/content/drive/MyDrive/Colab Notebooks/DataCamp Projects/workspace - Analyzing Car Reviews with LLMs/data/reference_translations.txt\", 'r') as file:\n", | |
| " lines = file.readlines()\n", | |
| "references = [line.strip() for line in lines]\n", | |
| "\n", | |
| "bleu_metric = load(\"bleu\")\n", | |
| "bleu_score = bleu_metric.compute(predictions=[translated_review], references=[references])\n", | |
| "bleu_score = bleu_score['bleu']\n", | |
| "print(\"Blue Score\", bleu_score)" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 248, | |
| "lastExecutedAt": 1713424060693, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "with open(\"data/reference_translations.txt\", 'r') as file:\n lines = file.readlines()\nreferences = [line.strip() for line in lines]\n\nbleu_metric = load(\"bleu\")\nbleu_score = bleu_metric.compute(predictions=[translated_review], references=[references])\nbleu_score = bleu_score['bleu']\nprint(\"Blue Score\", bleu_score)", | |
| "outputsMetadata": { | |
| "0": { | |
| "height": 38, | |
| "type": "stream" | |
| } | |
| }, | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "241d1d6e-0be9-4387-8f12-eca5bfe9fb9b", | |
| "outputId": "f2221772-2f1b-4886-ff68-0d5eb6782904" | |
| }, | |
| "cell_type": "code", | |
| "id": "241d1d6e-0be9-4387-8f12-eca5bfe9fb9b", | |
| "outputs": [ | |
| { | |
| "output_type": "stream", | |
| "name": "stdout", | |
| "text": [ | |
| "Blue Score 0.7794483794144497\n" | |
| ] | |
| } | |
| ], | |
| "execution_count": 11 | |
| }, | |
| { | |
| "source": [ | |
| "# Extractive Question Answering\n", | |
| "\n", | |
| "We will then use an extractive QA LLM to answer a specific question about a car review." | |
| ], | |
| "metadata": { | |
| "id": "9bf930b9-313d-413e-8c50-1134097dab48" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "9bf930b9-313d-413e-8c50-1134097dab48" | |
| }, | |
| { | |
| "source": [ | |
| "### Load QA model" | |
| ], | |
| "metadata": { | |
| "id": "16da0fa5-8f78-4c01-b7f7-4a41d765b1ef" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "16da0fa5-8f78-4c01-b7f7-4a41d765b1ef" | |
| }, | |
| { | |
| "source": [ | |
| "qa_pipeline = pipeline(\"question-answering\", model=\"deepset/minilm-uncased-squad2\", )" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 399, | |
| "lastExecutedAt": 1713424061092, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "qa_pipeline = pipeline(\"question-answering\", model=\"deepset/minilm-uncased-squad2\")", | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "62f28572-d7a8-4dc8-98b1-b966caf59a0d", | |
| "outputId": "f9b4c1ff-9f8f-49dc-8538-96e4f308e7bf" | |
| }, | |
| "cell_type": "code", | |
| "id": "62f28572-d7a8-4dc8-98b1-b966caf59a0d", | |
| "outputs": [ | |
| { | |
| "output_type": "stream", | |
| "name": "stderr", | |
| "text": [ | |
| "Some weights of the model checkpoint at deepset/minilm-uncased-squad2 were not used when initializing BertForQuestionAnswering: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']\n", | |
| "- This IS expected if you are initializing BertForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", | |
| "- This IS NOT expected if you are initializing BertForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n" | |
| ] | |
| } | |
| ], | |
| "execution_count": 12 | |
| }, | |
| { | |
| "source": [ | |
| "### Define question and context" | |
| ], | |
| "metadata": { | |
| "id": "97d66159-7c67-4b9b-af11-a536f5437cec" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "97d66159-7c67-4b9b-af11-a536f5437cec" | |
| }, | |
| { | |
| "source": [ | |
| "question = \"What did he like about the brand?\"\n", | |
| "context = reviews[1]" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 48, | |
| "lastExecutedAt": 1713424061142, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "question = \"What did he like about the brand?\"\ncontext = reviews[1]", | |
| "id": "21a5ff92-447a-4c01-b121-5d912b6e0b79" | |
| }, | |
| "cell_type": "code", | |
| "id": "21a5ff92-447a-4c01-b121-5d912b6e0b79", | |
| "outputs": [], | |
| "execution_count": 13 | |
| }, | |
| { | |
| "source": [ | |
| "### Perform question answering" | |
| ], | |
| "metadata": { | |
| "id": "e9224d86-08e9-4303-90b8-0195e0495006" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "e9224d86-08e9-4303-90b8-0195e0495006" | |
| }, | |
| { | |
| "source": [ | |
| "answer = qa_pipeline(question=question, context=context)['answer']\n", | |
| "print(\"Question:\", question)\n", | |
| "print(\"Answer:\", answer)" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 405, | |
| "lastExecutedAt": 1713424061548, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "answer = qa_pipeline(question=question, context=context)['answer']\nprint(\"Question:\", question)\nprint(\"Answer:\", answer)", | |
| "outputsMetadata": { | |
| "0": { | |
| "height": 59, | |
| "type": "stream" | |
| } | |
| }, | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "e329284d-996a-4129-b742-3df82a63c9dc", | |
| "outputId": "81138f59-6d2d-42fe-9aed-171dc427c018" | |
| }, | |
| "cell_type": "code", | |
| "id": "e329284d-996a-4129-b742-3df82a63c9dc", | |
| "outputs": [ | |
| { | |
| "output_type": "stream", | |
| "name": "stdout", | |
| "text": [ | |
| "Question: What did he like about the brand?\n", | |
| "Answer: ride quality, reliability\n" | |
| ] | |
| } | |
| ], | |
| "execution_count": 14 | |
| }, | |
| { | |
| "source": [ | |
| "# Summarize Car Review\n", | |
| "\n", | |
| "To wrap things up, we'll generate a summary of the last review in the dataset." | |
| ], | |
| "metadata": { | |
| "id": "88e44454-9e0e-4406-b22d-06cdaba2caaf" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "88e44454-9e0e-4406-b22d-06cdaba2caaf" | |
| }, | |
| { | |
| "source": [ | |
| "### Load summarization model" | |
| ], | |
| "metadata": { | |
| "id": "be07ce8b-0353-450e-b1bf-fa92c5c45e07" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "be07ce8b-0353-450e-b1bf-fa92c5c45e07" | |
| }, | |
| { | |
| "source": [ | |
| "summarizer = pipeline(\"summarization\", model=\"cnicu/t5-small-booksum\")" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 1063, | |
| "lastExecutedAt": 1713424062611, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "summarizer = pipeline(\"summarization\", model=\"cnicu/t5-small-booksum\")", | |
| "outputsMetadata": { | |
| "0": { | |
| "height": 227, | |
| "type": "stream" | |
| } | |
| }, | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 145, | |
| "referenced_widgets": [ | |
| "a667e55e5ddf4c3ebe8911ca136c5b8f", | |
| "93d9794628ee46a5910394a3562d04a9", | |
| "e20b622669d54ae0949a5a26bb49d286", | |
| "d89a16b4214b47b29b34e988ea00206c", | |
| "65bb5b761c3c47f294eb4ecca9f5c426", | |
| "46b5a04ac9d14ea98317d530018be983", | |
| "e43a54acaa0f48c585bb552772e91bc6", | |
| "721758fe73244ae596ac78734279de0c", | |
| "772b91269f384d8eac266b693e5fa470", | |
| "2b3322ee973544e1ab43c00aa1415e67", | |
| "cf4123b4912d42a4b45b1d5b5f7a8ccd", | |
| "4ce04cb308ab4313aeb0290625b700c4", | |
| "caeae222de704d63b54fe6fced38f55d", | |
| "07e59b20268441288ee00a6bb74d0cb3", | |
| "5cf05e8086e0461b9459a97608e6630b", | |
| "8533731ed66b4f2785755d9b343cea1d", | |
| "81577f5abc8b432abbc951d31041e099", | |
| "08b95601974140738057fddcfaf25cfe", | |
| "733f4d62722a45368ba7765539f5ac39", | |
| "124c6871d87d45aaa442ba7b7be86784", | |
| "69363f13bee244969f67d1838f2189cf", | |
| "ea45123e31224b709d864edee7a61c07", | |
| "5685ae9350914f8f8eeebf63b13b7acf", | |
| "f5668f041bc64352bd48e85cc24dd1c1", | |
| "d7b3ee01e999480997f7fa07d5cdc867", | |
| "59adb33c436346be8f3f393952e2e34d", | |
| "ccc50240be51474fadeaf871a7b63501", | |
| "958d0d6af567458189ea9b11b0760209", | |
| "944346d49a944ff18f08c01fcdd228ba", | |
| "0132f1adc2ba4b16b963a55f360c470b", | |
| "fc9315398e9f4ebcb67185662693c536", | |
| "67d1a1e35097471cb404a97cefe2daae", | |
| "8b86a116594f446abfed0a487c2173f7", | |
| "437fd072159d48d4aac2e3315668b427", | |
| "09988ddd3e2144859c1d074f29eedf61", | |
| "eb1295820c3c4fffb6ab7ddcf77bdd1b", | |
| "f7ab90b3fc7b406ea8d80510227ed5e2", | |
| "9f654d992e2040ada17e4b7f4be4cfe4", | |
| "311a116d06994b9e8dc30ad257118ef0", | |
| "5c6f4c8a40964d63b2a9e21a2e4b42f2", | |
| "afbd67a9baca4fa9ac7746fc2e1380b8", | |
| "6a7d979c4ea846bcb03f20861b5e9e51", | |
| "c819d3dc14ab4295b9fb459a420aa15d", | |
| "915ab10be4c4422b9a568df036d43ed1" | |
| ] | |
| }, | |
| "id": "5bdfc426-9eef-48c3-8f4e-b951d4e38bec", | |
| "outputId": "1a3e1973-8f5b-44d8-a732-f81d119b80ad" | |
| }, | |
| "cell_type": "code", | |
| "id": "5bdfc426-9eef-48c3-8f4e-b951d4e38bec", | |
| "outputs": [ | |
| { | |
| "output_type": "display_data", | |
| "data": { | |
| "text/plain": [ | |
| "tokenizer_config.json: 0%| | 0.00/1.92k [00:00<?, ?B/s]" | |
| ], | |
| "application/vnd.jupyter.widget-view+json": { | |
| "version_major": 2, | |
| "version_minor": 0, | |
| "model_id": "a667e55e5ddf4c3ebe8911ca136c5b8f" | |
| } | |
| }, | |
| "metadata": {} | |
| }, | |
| { | |
| "output_type": "display_data", | |
| "data": { | |
| "text/plain": [ | |
| "spiece.model: 0%| | 0.00/792k [00:00<?, ?B/s]" | |
| ], | |
| "application/vnd.jupyter.widget-view+json": { | |
| "version_major": 2, | |
| "version_minor": 0, | |
| "model_id": "4ce04cb308ab4313aeb0290625b700c4" | |
| } | |
| }, | |
| "metadata": {} | |
| }, | |
| { | |
| "output_type": "display_data", | |
| "data": { | |
| "text/plain": [ | |
| "tokenizer.json: 0%| | 0.00/2.42M [00:00<?, ?B/s]" | |
| ], | |
| "application/vnd.jupyter.widget-view+json": { | |
| "version_major": 2, | |
| "version_minor": 0, | |
| "model_id": "5685ae9350914f8f8eeebf63b13b7acf" | |
| } | |
| }, | |
| "metadata": {} | |
| }, | |
| { | |
| "output_type": "display_data", | |
| "data": { | |
| "text/plain": [ | |
| "special_tokens_map.json: 0%| | 0.00/1.79k [00:00<?, ?B/s]" | |
| ], | |
| "application/vnd.jupyter.widget-view+json": { | |
| "version_major": 2, | |
| "version_minor": 0, | |
| "model_id": "437fd072159d48d4aac2e3315668b427" | |
| } | |
| }, | |
| "metadata": {} | |
| } | |
| ], | |
| "execution_count": 15 | |
| }, | |
| { | |
| "source": [ | |
| "### Summarize the last review" | |
| ], | |
| "metadata": { | |
| "id": "4270fe9c-0312-444f-ba95-e70b567ed758" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "4270fe9c-0312-444f-ba95-e70b567ed758" | |
| }, | |
| { | |
| "source": [ | |
| "summarized_text = summarizer(reviews[-1], max_length=55, min_length=50, truncation=True)[0]['summary_text']\n", | |
| "print(summarized_text)" | |
| ], | |
| "metadata": { | |
| "executionCancelledAt": null, | |
| "executionTime": 5624, | |
| "lastExecutedAt": 1713424068236, | |
| "lastExecutedByKernel": "bd45531b-d2c7-47f2-ab1f-cd54fb0db4d5", | |
| "lastScheduledRunId": null, | |
| "lastSuccessfullyExecutedCode": "summarized_text = summarizer(reviews[-1], max_length=55, min_length=50, truncation=True)[0]['summary_text']\nprint(summarized_text)", | |
| "outputsMetadata": { | |
| "0": { | |
| "height": 80, | |
| "type": "stream" | |
| } | |
| }, | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "848e5fd5-b29b-46a8-86f6-eb4c956d289a", | |
| "outputId": "83858031-cb8f-4f87-9197-4c50e98176d8" | |
| }, | |
| "cell_type": "code", | |
| "id": "848e5fd5-b29b-46a8-86f6-eb4c956d289a", | |
| "outputs": [ | |
| { | |
| "output_type": "stream", | |
| "name": "stdout", | |
| "text": [ | |
| "the Nissan Rogue provides me with the desired SUV experience without burdening me with an exorbitant payment; the financial arrangement is quite reasonable. I have hauled 12 bags of mulch in the back with the seats down and could have held more. I find\n" | |
| ] | |
| } | |
| ], | |
| "execution_count": 16 | |
| }, | |
| { | |
| "source": [ | |
| "# Conclusion and Future Directions\n", | |
| "\n", | |
| "This notebook encapsulates a comprehensive exploration of leveraging Large Language Models (LLMs) to enhance customer interaction and backend operations at \"Car-ing is sharing,\" an innovative car dealership company. Through the application of models such as sentiment analysis, translation, question answering, and summarization, we have demonstrated not only the practical utility of LLMs in processing and understanding textual data but also their potential in transforming business operations to be more efficient and customer-friendly.\n", | |
| "\n", | |
| "**Key Outcomes:**\n", | |
| "- **Sentiment Analysis**: We successfully automated the classification of customer sentiments in car reviews, providing valuable insights into customer satisfaction and product feedback.\n", | |
| "- **Language Translation**: By translating text, we catered to a non-English speaking demographic, enhancing the inclusiveness and reach of our services.\n", | |
| "- **Question Answering**: The ability to automatically extract information in response to specific inquiries can significantly speed up response times, reducing workload on human agents.\n", | |
| "- **Text Summarization**: Summarizing extensive reviews into concise reports allows for quicker assimilation of feedback and aids in swift decision-making processes.\n", | |
| "\n", | |
| "**Future Enhancements:**\n", | |
| "- **Integration with Real-Time Data Streams**: To further enhance the chatbot’s utility, integrating real-time data processing capabilities will allow it to handle live customer interactions more effectively.\n", | |
| "- **Expansion of Language Support**: Incorporating additional language models will broaden our market reach and cater to a more diverse customer base.\n", | |
| "- **Advanced Sentiment Analysis**: Implementing more nuanced sentiment analysis that can detect subtleties such as sarcasm and context-dependent sentiments could refine the accuracy of our customer sentiment assessments.\n", | |
| "- **Custom LLM Training**: Training custom models on specific datasets relevant to our automotive content can improve accuracy and relevance in outputs, tailored to the unique context of our company's data.\n", | |
| "\n", | |
| "By advancing these LLM capabilities, \"Car-ing is sharing\" can significantly enhance its customer service operations and maintain a competitive edge in the automotive industry. The implementations showcased in this notebook not only highlight the current achievements but also pave the way for future innovations that could redefine how automotive companies interact with their customers and manage internal processes." | |
| ], | |
| "metadata": { | |
| "id": "450a51f2-7e6f-45ec-97cd-3d4a062c4608" | |
| }, | |
| "cell_type": "markdown", | |
| "id": "450a51f2-7e6f-45ec-97cd-3d4a062c4608" | |
| } | |
| ], | |
| "metadata": { | |
| "colab": { | |
| "provenance": [], | |
| "include_colab_link": true | |
| }, | |
| "kernelspec": { | |
| "display_name": "Python 3 (ipykernel)", | |
| "language": "python", | |
| "name": "python3" | |
| }, | |
| "language_info": { | |
| "codemirror_mode": { | |
| "name": "ipython", | |
| "version": 3 | |
| }, | |
| "file_extension": ".py", | |
| "mimetype": "text/x-python", | |
| "name": "python", | |
| "nbconvert_exporter": "python", | |
| "pygments_lexer": "ipython3", | |
| "version": "3.8.10" | |
| }, | |
| "widgets": { | |
| "application/vnd.jupyter.widget-state+json": { | |
| "a667e55e5ddf4c3ebe8911ca136c5b8f": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "HBoxModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HBoxModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HBoxView", | |
| "box_style": "", | |
| "children": [ | |
| "IPY_MODEL_93d9794628ee46a5910394a3562d04a9", | |
| "IPY_MODEL_e20b622669d54ae0949a5a26bb49d286", | |
| "IPY_MODEL_d89a16b4214b47b29b34e988ea00206c" | |
| ], | |
| "layout": "IPY_MODEL_65bb5b761c3c47f294eb4ecca9f5c426" | |
| } | |
| }, | |
| "93d9794628ee46a5910394a3562d04a9": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "HTMLModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HTMLModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HTMLView", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_46b5a04ac9d14ea98317d530018be983", | |
| "placeholder": "", | |
| "style": "IPY_MODEL_e43a54acaa0f48c585bb552772e91bc6", | |
| "value": "tokenizer_config.json: 100%" | |
| } | |
| }, | |
| "e20b622669d54ae0949a5a26bb49d286": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "FloatProgressModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "FloatProgressModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "ProgressView", | |
| "bar_style": "success", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_721758fe73244ae596ac78734279de0c", | |
| "max": 1924, | |
| "min": 0, | |
| "orientation": "horizontal", | |
| "style": "IPY_MODEL_772b91269f384d8eac266b693e5fa470", | |
| "value": 1924 | |
| } | |
| }, | |
| "d89a16b4214b47b29b34e988ea00206c": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "HTMLModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HTMLModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HTMLView", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_2b3322ee973544e1ab43c00aa1415e67", | |
| "placeholder": "", | |
| "style": "IPY_MODEL_cf4123b4912d42a4b45b1d5b5f7a8ccd", | |
| "value": " 1.92k/1.92k [00:00<00:00, 18.3kB/s]" | |
| } | |
| }, | |
| "65bb5b761c3c47f294eb4ecca9f5c426": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "46b5a04ac9d14ea98317d530018be983": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "e43a54acaa0f48c585bb552772e91bc6": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "DescriptionStyleModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "DescriptionStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "description_width": "" | |
| } | |
| }, | |
| "721758fe73244ae596ac78734279de0c": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "772b91269f384d8eac266b693e5fa470": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "ProgressStyleModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "ProgressStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "bar_color": null, | |
| "description_width": "" | |
| } | |
| }, | |
| "2b3322ee973544e1ab43c00aa1415e67": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "cf4123b4912d42a4b45b1d5b5f7a8ccd": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "DescriptionStyleModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "DescriptionStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "description_width": "" | |
| } | |
| }, | |
| "4ce04cb308ab4313aeb0290625b700c4": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "HBoxModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HBoxModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HBoxView", | |
| "box_style": "", | |
| "children": [ | |
| "IPY_MODEL_caeae222de704d63b54fe6fced38f55d", | |
| "IPY_MODEL_07e59b20268441288ee00a6bb74d0cb3", | |
| "IPY_MODEL_5cf05e8086e0461b9459a97608e6630b" | |
| ], | |
| "layout": "IPY_MODEL_8533731ed66b4f2785755d9b343cea1d" | |
| } | |
| }, | |
| "caeae222de704d63b54fe6fced38f55d": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "HTMLModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HTMLModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HTMLView", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_81577f5abc8b432abbc951d31041e099", | |
| "placeholder": "", | |
| "style": "IPY_MODEL_08b95601974140738057fddcfaf25cfe", | |
| "value": "spiece.model: 100%" | |
| } | |
| }, | |
| "07e59b20268441288ee00a6bb74d0cb3": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "FloatProgressModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "FloatProgressModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "ProgressView", | |
| "bar_style": "success", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_733f4d62722a45368ba7765539f5ac39", | |
| "max": 791656, | |
| "min": 0, | |
| "orientation": "horizontal", | |
| "style": "IPY_MODEL_124c6871d87d45aaa442ba7b7be86784", | |
| "value": 791656 | |
| } | |
| }, | |
| "5cf05e8086e0461b9459a97608e6630b": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "HTMLModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HTMLModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HTMLView", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_69363f13bee244969f67d1838f2189cf", | |
| "placeholder": "", | |
| "style": "IPY_MODEL_ea45123e31224b709d864edee7a61c07", | |
| "value": " 792k/792k [00:00<00:00, 7.31MB/s]" | |
| } | |
| }, | |
| "8533731ed66b4f2785755d9b343cea1d": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "81577f5abc8b432abbc951d31041e099": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "08b95601974140738057fddcfaf25cfe": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "DescriptionStyleModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "DescriptionStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "description_width": "" | |
| } | |
| }, | |
| "733f4d62722a45368ba7765539f5ac39": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "124c6871d87d45aaa442ba7b7be86784": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "ProgressStyleModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "ProgressStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "bar_color": null, | |
| "description_width": "" | |
| } | |
| }, | |
| "69363f13bee244969f67d1838f2189cf": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "ea45123e31224b709d864edee7a61c07": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "DescriptionStyleModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "DescriptionStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "description_width": "" | |
| } | |
| }, | |
| "5685ae9350914f8f8eeebf63b13b7acf": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "HBoxModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HBoxModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HBoxView", | |
| "box_style": "", | |
| "children": [ | |
| "IPY_MODEL_f5668f041bc64352bd48e85cc24dd1c1", | |
| "IPY_MODEL_d7b3ee01e999480997f7fa07d5cdc867", | |
| "IPY_MODEL_59adb33c436346be8f3f393952e2e34d" | |
| ], | |
| "layout": "IPY_MODEL_ccc50240be51474fadeaf871a7b63501" | |
| } | |
| }, | |
| "f5668f041bc64352bd48e85cc24dd1c1": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "HTMLModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HTMLModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HTMLView", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_958d0d6af567458189ea9b11b0760209", | |
| "placeholder": "", | |
| "style": "IPY_MODEL_944346d49a944ff18f08c01fcdd228ba", | |
| "value": "tokenizer.json: 100%" | |
| } | |
| }, | |
| "d7b3ee01e999480997f7fa07d5cdc867": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "FloatProgressModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "FloatProgressModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "ProgressView", | |
| "bar_style": "success", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_0132f1adc2ba4b16b963a55f360c470b", | |
| "max": 2422193, | |
| "min": 0, | |
| "orientation": "horizontal", | |
| "style": "IPY_MODEL_fc9315398e9f4ebcb67185662693c536", | |
| "value": 2422193 | |
| } | |
| }, | |
| "59adb33c436346be8f3f393952e2e34d": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "HTMLModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HTMLModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HTMLView", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_67d1a1e35097471cb404a97cefe2daae", | |
| "placeholder": "", | |
| "style": "IPY_MODEL_8b86a116594f446abfed0a487c2173f7", | |
| "value": " 2.42M/2.42M [00:00<00:00, 18.0MB/s]" | |
| } | |
| }, | |
| "ccc50240be51474fadeaf871a7b63501": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "958d0d6af567458189ea9b11b0760209": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "944346d49a944ff18f08c01fcdd228ba": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "DescriptionStyleModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "DescriptionStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "description_width": "" | |
| } | |
| }, | |
| "0132f1adc2ba4b16b963a55f360c470b": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "fc9315398e9f4ebcb67185662693c536": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "ProgressStyleModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "ProgressStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "bar_color": null, | |
| "description_width": "" | |
| } | |
| }, | |
| "67d1a1e35097471cb404a97cefe2daae": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "8b86a116594f446abfed0a487c2173f7": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "DescriptionStyleModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "DescriptionStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "description_width": "" | |
| } | |
| }, | |
| "437fd072159d48d4aac2e3315668b427": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "HBoxModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HBoxModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HBoxView", | |
| "box_style": "", | |
| "children": [ | |
| "IPY_MODEL_09988ddd3e2144859c1d074f29eedf61", | |
| "IPY_MODEL_eb1295820c3c4fffb6ab7ddcf77bdd1b", | |
| "IPY_MODEL_f7ab90b3fc7b406ea8d80510227ed5e2" | |
| ], | |
| "layout": "IPY_MODEL_9f654d992e2040ada17e4b7f4be4cfe4" | |
| } | |
| }, | |
| "09988ddd3e2144859c1d074f29eedf61": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "HTMLModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HTMLModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HTMLView", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_311a116d06994b9e8dc30ad257118ef0", | |
| "placeholder": "", | |
| "style": "IPY_MODEL_5c6f4c8a40964d63b2a9e21a2e4b42f2", | |
| "value": "special_tokens_map.json: 100%" | |
| } | |
| }, | |
| "eb1295820c3c4fffb6ab7ddcf77bdd1b": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "FloatProgressModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "FloatProgressModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "ProgressView", | |
| "bar_style": "success", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_afbd67a9baca4fa9ac7746fc2e1380b8", | |
| "max": 1786, | |
| "min": 0, | |
| "orientation": "horizontal", | |
| "style": "IPY_MODEL_6a7d979c4ea846bcb03f20861b5e9e51", | |
| "value": 1786 | |
| } | |
| }, | |
| "f7ab90b3fc7b406ea8d80510227ed5e2": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "HTMLModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HTMLModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HTMLView", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_c819d3dc14ab4295b9fb459a420aa15d", | |
| "placeholder": "", | |
| "style": "IPY_MODEL_915ab10be4c4422b9a568df036d43ed1", | |
| "value": " 1.79k/1.79k [00:00<00:00, 30.8kB/s]" | |
| } | |
| }, | |
| "9f654d992e2040ada17e4b7f4be4cfe4": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "311a116d06994b9e8dc30ad257118ef0": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "5c6f4c8a40964d63b2a9e21a2e4b42f2": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "DescriptionStyleModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "DescriptionStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "description_width": "" | |
| } | |
| }, | |
| "afbd67a9baca4fa9ac7746fc2e1380b8": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "6a7d979c4ea846bcb03f20861b5e9e51": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "ProgressStyleModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "ProgressStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "bar_color": null, | |
| "description_width": "" | |
| } | |
| }, | |
| "c819d3dc14ab4295b9fb459a420aa15d": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_name": "LayoutModel", | |
| "model_module_version": "1.2.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "915ab10be4c4422b9a568df036d43ed1": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_name": "DescriptionStyleModel", | |
| "model_module_version": "1.5.0", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "DescriptionStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "description_width": "" | |
| } | |
| } | |
| } | |
| } | |
| }, | |
| "nbformat": 4, | |
| "nbformat_minor": 5 | |
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment