Skip to content

Instantly share code, notes, and snippets.

@kota7

kota7/bert-solves-eiken-problems.ipynb

Last active August 24, 2021 17:30
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save kota7/9a1219fc57f73f2247deb2bcf0112a20 to your computer and use it in GitHub Desktop.
Save kota7/9a1219fc57f73f2247deb2bcf0112a20 to your computer and use it in GitHub Desktop.
Download ZIP
BERT solves Eiken problems
Display the source blob
Display the rendered blob
Raw
bert-solves-eiken-problems.ipynb
{
"cells": [
{
"cell_type": "markdown",
"id": "2eae93bd",
"metadata": {
"papermill": {
"duration": 0.020327,
"end_time": "2021-08-01T13:32:43.140250",
"exception": false,
"start_time": "2021-08-01T13:32:43.119923",
"status": "completed"
},
"tags": []
},
"source": [
"# BERT solves Eiken problems\n",
"\n",
"Eiken (実用英語技能検定) is an English proficiency test conducted by a Japanese public-interest incorporated foundation ([Link to wikipedia](https://en.wikipedia.org/wiki/STEP_Eiken)).\n",
"One type of the questions in the test is a multiple choice problem to fill a blank in a sentence. For example:\n",
"\n",
"> My sister usually plays tennis (　　　) Saturdays. \n",
"　1. by　　2. on　　3. with　　4. at\n",
" \n",
"> Bob (　　　) five friends to his party. \n",
"　1. made　　2. visited　　3. invited　　4. spoke\n",
"\n",
"In this notebook we solve this type of questions using pre-trained BERT models.\n",
"\n",
"First, we use the masked language model, which is designed to guess a word in a blank in a sentence.\n",
"A drawback of this approach is that the model cannot guess a word not included in its vocabulary set.\n",
"\n",
"To handle unknown words, the second approach calculates perplexity scores of the sentences filled by choices.\n",
"Since a lower perplexity score indicates the sentense is more \"natural,\" we can pick the sentence with the lowest score as the answer. "
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "48ef8f87",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:32:43.200826Z",
"iopub.status.busy": "2021-08-01T13:32:43.190779Z",
"iopub.status.idle": "2021-08-01T13:32:51.700986Z",
"shell.execute_reply": "2021-08-01T13:32:51.699992Z",
"shell.execute_reply.started": "2021-08-01T13:19:54.795444Z"
},
"papermill": {
"duration": 8.541466,
"end_time": "2021-08-01T13:32:51.701165",
"exception": false,
"start_time": "2021-08-01T13:32:43.159699",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Collecting openpyxl\r\n",
" Downloading openpyxl-3.0.7-py2.py3-none-any.whl (243 kB)\r\n",
"\u001b[K |████████████████████████████████| 243 kB 2.1 MB/s \r\n",
"\u001b[?25hCollecting et-xmlfile\r\n",
" Downloading et_xmlfile-1.1.0-py3-none-any.whl (4.7 kB)\r\n",
"Installing collected packages: et-xmlfile, openpyxl\r\n",
"Successfully installed et-xmlfile-1.1.0 openpyxl-3.0.7\r\n",
"\u001b[33mWARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv\u001b[0m\r\n"
]
}
],
"source": [
"!pip install openpyxl"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "00b688f6",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:32:51.748027Z",
"iopub.status.busy": "2021-08-01T13:32:51.745318Z",
"iopub.status.idle": "2021-08-01T13:33:16.107153Z",
"shell.execute_reply": "2021-08-01T13:33:16.106656Z",
"shell.execute_reply.started": "2021-08-01T13:20:03.501385Z"
},
"id": "GPZTUze_oO-f",
"outputId": "7241d085-173e-4108-afef-e88c05727fdc",
"papermill": {
"duration": 24.385851,
"end_time": "2021-08-01T13:33:16.107288",
"exception": false,
"start_time": "2021-08-01T13:32:51.721437",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/opt/conda/lib/python3.7/site-packages/torchaudio/backend/utils.py:54: UserWarning: \"sox\" backend is being deprecated. The default backend will be changed to \"sox_io\" backend in 0.8.0 and \"sox\" backend will be removed in 0.9.0. Please migrate to \"sox_io\" backend. Please refer to https://github.com/pytorch/audio/issues/903 for the detail.\n",
" '\"sox\" backend is being deprecated. '\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a1684aa619614e8bb9c2e8e99435e825",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading: 0%| | 0.00/480 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "fd437e67b2514af0bf40eddef8e67269",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading: 0%| | 0.00/331M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "0c497fde35bc43ebb4e34942be288a20",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading: 0%| | 0.00/899k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "8b8f8e333aad4938951b0e370d1dc021",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading: 0%| | 0.00/456k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "f0c1c58c4bd545b783e5bb7e92c4e6d0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading: 0%| | 0.00/1.36M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"[{'sequence': 'HuggingFace is creating a tool that the community uses to solve NLP tasks.',\n",
" 'score': 0.17927570641040802,\n",
" 'token': 3944,\n",
" 'token_str': ' tool'},\n",
" {'sequence': 'HuggingFace is creating a framework that the community uses to solve NLP tasks.',\n",
" 'score': 0.11349428445100784,\n",
" 'token': 7208,\n",
" 'token_str': ' framework'},\n",
" {'sequence': 'HuggingFace is creating a library that the community uses to solve NLP tasks.',\n",
" 'score': 0.05243517830967903,\n",
" 'token': 5560,\n",
" 'token_str': ' library'},\n",
" {'sequence': 'HuggingFace is creating a database that the community uses to solve NLP tasks.',\n",
" 'score': 0.034935519099235535,\n",
" 'token': 8503,\n",
" 'token_str': ' database'},\n",
" {'sequence': 'HuggingFace is creating a prototype that the community uses to solve NLP tasks.',\n",
" 'score': 0.028602516278624535,\n",
" 'token': 17715,\n",
" 'token_str': ' prototype'}]"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import pipeline\n",
"# fill-mask pipeline implements a task of filling a blank word\n",
"# https://huggingface.co/transformers/main_classes/pipelines.html#transformers.FillMaskPipeline\n",
"model = pipeline(\"fill-mask\")\n",
"\n",
"# Check how it works\n",
"res = model(f\"HuggingFace is creating a {model.tokenizer.mask_token} that the community uses to solve NLP tasks.\")\n",
"res\n",
"\n",
"# notice that the matched words have a space in the beginning.\n",
"# this is to distinguish them with suffix words, which connects another word without a space"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "194e3e6a",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:33:16.161084Z",
"iopub.status.busy": "2021-08-01T13:33:16.160370Z",
"iopub.status.idle": "2021-08-01T13:33:16.163921Z",
"shell.execute_reply": "2021-08-01T13:33:16.164680Z",
"shell.execute_reply.started": "2021-08-01T13:20:32.703176Z"
},
"id": "_8mEY5K3uomv",
"outputId": "bd4b49cb-35ad-4fd4-9e5e-585b9012c41c",
"papermill": {
"duration": 0.033907,
"end_time": "2021-08-01T13:33:16.165002",
"exception": false,
"start_time": "2021-08-01T13:33:16.131095",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Help on method __call__ in module transformers.pipelines.fill_mask:\n",
"\n",
"__call__(*args, targets=None, top_k: Union[int, NoneType] = None, **kwargs) method of transformers.pipelines.fill_mask.FillMaskPipeline instance\n",
" Fill the masked token in the text(s) given as inputs.\n",
" \n",
" Args:\n",
" args (:obj:`str` or :obj:`List[str]`):\n",
" One or several texts (or one list of prompts) with masked tokens.\n",
" targets (:obj:`str` or :obj:`List[str]`, `optional`):\n",
" When passed, the model will return the scores for the passed token or tokens rather than the top k\n",
" predictions in the entire vocabulary. If the provided targets are not in the model vocab, they will be\n",
" tokenized and the first resulting token will be used (with a warning).\n",
" top_k (:obj:`int`, `optional`):\n",
" When passed, overrides the number of predictions to return.\n",
" \n",
" Return:\n",
" A list or a list of list of :obj:`dict`: Each result comes as list of dictionaries with the following keys:\n",
" \n",
" - **sequence** (:obj:`str`) -- The corresponding input with the mask token prediction.\n",
" - **score** (:obj:`float`) -- The corresponding probability.\n",
" - **token** (:obj:`int`) -- The predicted token id (to replace the masked one).\n",
" - **token** (:obj:`str`) -- The predicted token (to replace the masked one).\n",
"\n"
]
}
],
"source": [
"# Check the pipeline syntax\n",
"help(model.__call__)\n",
"\n",
"# notice that the model accepts options:\n",
"# targets: the candidate words to fill the blank\n",
"# top_k : number of candidates to show"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "a0a77e70",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:33:16.223270Z",
"iopub.status.busy": "2021-08-01T13:33:16.222190Z",
"iopub.status.idle": "2021-08-01T13:33:16.227276Z",
"shell.execute_reply": "2021-08-01T13:33:16.226644Z",
"shell.execute_reply.started": "2021-08-01T13:20:32.712378Z"
},
"id": "6QvNuEuSaBQO",
"outputId": "c449d39c-6aac-478d-9def-753e8f5e8d17",
"papermill": {
"duration": 0.037064,
"end_time": "2021-08-01T13:33:16.227424",
"exception": false,
"start_time": "2021-08-01T13:33:16.190360",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"RobertaForMaskedLM(\n",
" (roberta): RobertaModel(\n",
" (embeddings): RobertaEmbeddings(\n",
" (word_embeddings): Embedding(50265, 768, padding_idx=1)\n",
" (position_embeddings): Embedding(514, 768, padding_idx=1)\n",
" (token_type_embeddings): Embedding(1, 768)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (encoder): RobertaEncoder(\n",
" (layer): ModuleList(\n",
" (0): RobertaLayer(\n",
" (attention): RobertaAttention(\n",
" (self): RobertaSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): RobertaSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): RobertaIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): RobertaOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (1): RobertaLayer(\n",
" (attention): RobertaAttention(\n",
" (self): RobertaSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): RobertaSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): RobertaIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): RobertaOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (2): RobertaLayer(\n",
" (attention): RobertaAttention(\n",
" (self): RobertaSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): RobertaSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): RobertaIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): RobertaOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (3): RobertaLayer(\n",
" (attention): RobertaAttention(\n",
" (self): RobertaSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): RobertaSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): RobertaIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): RobertaOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (4): RobertaLayer(\n",
" (attention): RobertaAttention(\n",
" (self): RobertaSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): RobertaSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): RobertaIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): RobertaOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (5): RobertaLayer(\n",
" (attention): RobertaAttention(\n",
" (self): RobertaSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): RobertaSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): RobertaIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): RobertaOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" )\n",
" )\n",
" )\n",
" (lm_head): RobertaLMHead(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (decoder): Linear(in_features=768, out_features=50265, bias=True)\n",
" )\n",
")"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Check which pretrained model we are using\n",
"model.model"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "650b2cde",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:33:16.291768Z",
"iopub.status.busy": "2021-08-01T13:33:16.290971Z",
"iopub.status.idle": "2021-08-01T13:33:16.294065Z",
"shell.execute_reply": "2021-08-01T13:33:16.294495Z",
"shell.execute_reply.started": "2021-08-01T13:20:32.724325Z"
},
"id": "B0FoWDUAsoam",
"outputId": "cb30bf7e-3089-4e59-c3ca-76db7c5fb7e5",
"papermill": {
"duration": 0.04097,
"end_time": "2021-08-01T13:33:16.294637",
"exception": false,
"start_time": "2021-08-01T13:33:16.253667",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"[Problem(text='A: What is your {}? B: Kazumi Suzuki.', choices=['hour', 'club', 'date', 'name'], answer='name'),\n",
" Problem(text='I know Judy. She can {} French very well.', choices=['see', 'drink', 'speak', 'open'], answer='speak'),\n",
" Problem(text=\"A: Are your baseball shoes in your room, Mike? B: No, Mom. They're in my {} at school.\", choices=['window', 'shop', 'locker', 'door'], answer='locker'),\n",
" Problem(text='Mysister usually plays tennis {} Saturdays.', choices=['by', 'on', 'with', 'at'], answer='on'),\n",
" Problem(text='My mother likes {}. She has many pretty ones in the garden.', choices=['sports', 'movies', 'schools', 'flowers'], answer='flowers'),\n",
" Problem(text=\"Let's begin today's class. Open your textbooks to {} 22.\", choices=['chalk', 'ground', 'page', 'minute'], answer='page'),\n",
" Problem(text='Today is Wednesday. Tomorrow is {}.', choices=['Monday', 'Tuesday', 'Thursday', 'Friday'], answer='Thursday'),\n",
" Problem(text='I usually read magazines {} home.', choices=['of', 'on', 'with', 'at'], answer='at'),\n",
" Problem(text=\"A: It's ten o'clock, Jimmy. {} to bed. B: All right, Mom.\", choices=['Go', 'Sleep', 'Do', 'Sit'], answer='Go'),\n",
" Problem(text=\"A: Do you live {} Tokyo? B: Yes. It's a big city.\", choices=['after', 'with', 'on', 'in'], answer='in')]"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from collections import namedtuple\n",
"\n",
"# we define a problem as a named tuple below:\n",
"Problem = namedtuple(\"Problem\", \"text choices answer\")\n",
"\n",
"# Eiken grade 5\n",
"# source: https://www.eiken.or.jp/eiken/exam/grade_5/pdf/202101/2021-1-1ji-5kyu.pdf\n",
"eiken5 = [\n",
" Problem(\"A: What is your {}? B: Kazumi Suzuki.\",\n",
" [\"hour\", \"club\", \"date\", \"name\"], \"name\")\n",
" ,Problem(\"I know Judy. She can {} French very well.\",\n",
" [\"see\", \"drink\", \"speak\", \"open\"], \"speak\")\n",
" ,Problem(\"A: Are your baseball shoes in your room, Mike? B: No, Mom. They're in my {} at school.\",\n",
" [\"window\", \"shop\", \"locker\", \"door\"], \"locker\")\n",
" ,Problem(\"Mysister usually plays tennis {} Saturdays.\",\n",
" [\"by\", \"on\", \"with\", \"at\"], \"on\")\n",
" ,Problem(\"My mother likes {}. She has many pretty ones in the garden.\",\n",
" [\"sports\", \"movies\", \"schools\", \"flowers\"], \"flowers\")\n",
" ,Problem(\"Let's begin today's class. Open your textbooks to {} 22.\",\n",
" [\"chalk\", \"ground\", \"page\", \"minute\"], \"page\")\n",
" ,Problem(\"Today is Wednesday. Tomorrow is {}.\",\n",
" [\"Monday\", \"Tuesday\", \"Thursday\", \"Friday\"], \"Thursday\")\n",
" ,Problem(\"I usually read magazines {} home.\",\n",
" [\"of\", \"on\", \"with\", \"at\"], \"at\")\n",
" ,Problem(\"A: It's ten o'clock, Jimmy. {} to bed. B: All right, Mom.\",\n",
" [\"Go\", \"Sleep\", \"Do\", \"Sit\"], \"Go\")\n",
" ,Problem(\"A: Do you live {} Tokyo? B: Yes. It's a big city.\",\n",
" [\"after\", \"with\", \"on\", \"in\"], \"in\")\n",
"]\n",
"\n",
"# These are grade-5 questions (Grade 1 is the highest)\n",
"eiken5"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "ebac49c3",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:33:16.356121Z",
"iopub.status.busy": "2021-08-01T13:33:16.354910Z",
"iopub.status.idle": "2021-08-01T13:33:16.978770Z",
"shell.execute_reply": "2021-08-01T13:33:16.978234Z",
"shell.execute_reply.started": "2021-08-01T13:20:32.740352Z"
},
"id": "DfYdZ0uGopYi",
"outputId": "1096cb06-cede-4ec5-ef65-7073ba104905",
"papermill": {
"duration": 0.660246,
"end_time": "2021-08-01T13:33:16.979015",
"exception": false,
"start_time": "2021-08-01T13:33:16.318769",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" <th>grade</th>\n",
" <th>method</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A: What is your {}? B: Kazumi Suzuki.</td>\n",
" <td>name(0.245),nickname(0.101),surname(0.058),ans...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-no-target</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>I know Judy. She can {} French very well.</td>\n",
" <td>speak(0.858),read(0.053),learn(0.012),teach(0....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-no-target</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>A: Are your baseball shoes in your room, Mike?...</td>\n",
" <td>room(0.408),locker(0.132),classroom(0.107),clo...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-no-target</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Mysister usually plays tennis {} Saturdays.</td>\n",
" <td>on(0.953),every(0.021),through(0.007),until(0....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-no-target</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>My mother likes {}. She has many pretty ones i...</td>\n",
" <td>roses(0.326),flowers(0.147),strawberries(0.070...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-no-target</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Let's begin today's class. Open your textbooks...</td>\n",
" <td>page(0.285),chapter(0.171),Chapter(0.113),Page...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-no-target</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Today is Wednesday. Tomorrow is {}.</td>\n",
" <td>Friday(0.293),Thursday(0.236),Sunday(0.168),Tu...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-no-target</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>I usually read magazines {} home.</td>\n",
" <td>at(0.945),from(0.026),back(0.023),around(0.001...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-no-target</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>A: It's ten o'clock, Jimmy. {} to bed. B: All...</td>\n",
" <td>Go(0.868),Get(0.039),Back(0.039),Went(0.009),G...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-no-target</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>A: Do you live {} Tokyo? B: Yes. It's a big c...</td>\n",
" <td>in(0.874),near(0.075),outside(0.043),around(0....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-no-target</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 A: What is your {}? B: Kazumi Suzuki. \n",
"1 I know Judy. She can {} French very well. \n",
"2 A: Are your baseball shoes in your room, Mike?... \n",
"3 Mysister usually plays tennis {} Saturdays. \n",
"4 My mother likes {}. She has many pretty ones i... \n",
"5 Let's begin today's class. Open your textbooks... \n",
"6 Today is Wednesday. Tomorrow is {}. \n",
"7 I usually read magazines {} home. \n",
"8 A: It's ten o'clock, Jimmy. {} to bed. B: All... \n",
"9 A: Do you live {} Tokyo? B: Yes. It's a big c... \n",
"\n",
" scores answer_position \\\n",
"0 name(0.245),nickname(0.101),surname(0.058),ans... 1 \n",
"1 speak(0.858),read(0.053),learn(0.012),teach(0.... 1 \n",
"2 room(0.408),locker(0.132),classroom(0.107),clo... 2 \n",
"3 on(0.953),every(0.021),through(0.007),until(0.... 1 \n",
"4 roses(0.326),flowers(0.147),strawberries(0.070... 2 \n",
"5 page(0.285),chapter(0.171),Chapter(0.113),Page... 1 \n",
"6 Friday(0.293),Thursday(0.236),Sunday(0.168),Tu... 2 \n",
"7 at(0.945),from(0.026),back(0.023),around(0.001... 1 \n",
"8 Go(0.868),Get(0.039),Back(0.039),Went(0.009),G... 1 \n",
"9 in(0.874),near(0.075),outside(0.043),around(0.... 1 \n",
"\n",
" correct grade method \n",
"0 True 5 fill-mask-no-target \n",
"1 True 5 fill-mask-no-target \n",
"2 True 5 fill-mask-no-target \n",
"3 True 5 fill-mask-no-target \n",
"4 True 5 fill-mask-no-target \n",
"5 True 5 fill-mask-no-target \n",
"6 True 5 fill-mask-no-target \n",
"7 True 5 fill-mask-no-target \n",
"8 True 5 fill-mask-no-target \n",
"9 True 5 fill-mask-no-target "
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Masked Language Model without choices\n",
"# we consider the BERT model's guess \"correct\" if the correct answer is in the top 5 candidates\n",
"import pandas as pd\n",
"out = pd.DataFrame() # we will add outcomes in the dataframe\n",
"\n",
"def solve_without_choices(problems, top_k=5):\n",
" inputs = [p.text.format(model.tokenizer.mask_token) for p in problems]\n",
" res = model(inputs, top_k=top_k)\n",
" out = []\n",
" for p, r in zip(problems, res):\n",
" # suggested answers and the scores\n",
" suggested = [s[\"token_str\"].strip() for s in r]\n",
" scores = [s[\"score\"] for s in r]\n",
" suggested_scores = \",\".join(\"%s(%.3f)\" % (w,s) for w, s in zip(suggested, scores))\n",
" # location of answer\n",
" if p.answer in suggested:\n",
" position = suggested.index(p.answer) + 1\n",
" else:\n",
" position = -1\n",
" out.append((p.text, suggested_scores, position))\n",
" out = pd.DataFrame(out, columns=[\"problem\", \"scores\", \"answer_position\"])\n",
" out[\"correct\"] = (out[\"answer_position\"] > 0)\n",
" return out\n",
"\n",
"o = solve_without_choices(eiken5)\n",
"o[\"grade\"] = \"5\"\n",
"o[\"method\"] = \"fill-mask-no-target\"\n",
"out = pd.concat((out, o))\n",
"o"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "e11d5746",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:33:17.044568Z",
"iopub.status.busy": "2021-08-01T13:33:17.043038Z",
"iopub.status.idle": "2021-08-01T13:33:18.514191Z",
"shell.execute_reply": "2021-08-01T13:33:18.513313Z",
"shell.execute_reply.started": "2021-08-01T13:20:33.304098Z"
},
"id": "7Rt1bFeWpQGS",
"outputId": "17763051-ccb9-417d-a61b-583b91e3c40f",
"papermill": {
"duration": 1.508228,
"end_time": "2021-08-01T13:33:18.514350",
"exception": false,
"start_time": "2021-08-01T13:33:17.006122",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" <th>grade</th>\n",
" <th>method</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A: What is your {}? B: Kazumi Suzuki.</td>\n",
" <td>name(0.245),date(0.002),club(0.000),hour(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-with-targets</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>I know Judy. She can {} French very well.</td>\n",
" <td>speak(0.858),drink(0.000),see(0.000),open(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-with-targets</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>A: Are your baseball shoes in your room, Mike?...</td>\n",
" <td>locker(0.132),window(0.000),door(0.000),shop(0...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-with-targets</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Mysister usually plays tennis {} Saturdays.</td>\n",
" <td>on(0.953),at(0.001),with(0.000),by(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-with-targets</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>My mother likes {}. She has many pretty ones i...</td>\n",
" <td>flowers(0.147),movies(0.000),sports(0.000),sch...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-with-targets</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Let's begin today's class. Open your textbooks...</td>\n",
" <td>page(0.285),minute(0.000),chalk(0.000),ground(...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-with-targets</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Today is Wednesday. Tomorrow is {}.</td>\n",
" <td>Friday(0.293),Thursday(0.236),Tuesday(0.105),M...</td>\n",
" <td>2</td>\n",
" <td>False</td>\n",
" <td>5</td>\n",
" <td>fill-mask-with-targets</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>I usually read magazines {} home.</td>\n",
" <td>at(0.945),on(0.000),with(0.000),of(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-with-targets</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>A: It's ten o'clock, Jimmy. {} to bed. B: All...</td>\n",
" <td>Go(0.868),Sleep(0.000),Sit(0.000),Do(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-with-targets</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>A: Do you live {} Tokyo? B: Yes. It's a big c...</td>\n",
" <td>in(0.874),on(0.001),with(0.000),after(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" <td>5</td>\n",
" <td>fill-mask-with-targets</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 A: What is your {}? B: Kazumi Suzuki. \n",
"1 I know Judy. She can {} French very well. \n",
"2 A: Are your baseball shoes in your room, Mike?... \n",
"3 Mysister usually plays tennis {} Saturdays. \n",
"4 My mother likes {}. She has many pretty ones i... \n",
"5 Let's begin today's class. Open your textbooks... \n",
"6 Today is Wednesday. Tomorrow is {}. \n",
"7 I usually read magazines {} home. \n",
"8 A: It's ten o'clock, Jimmy. {} to bed. B: All... \n",
"9 A: Do you live {} Tokyo? B: Yes. It's a big c... \n",
"\n",
" scores answer_position \\\n",
"0 name(0.245),date(0.002),club(0.000),hour(0.000) 1 \n",
"1 speak(0.858),drink(0.000),see(0.000),open(0.000) 1 \n",
"2 locker(0.132),window(0.000),door(0.000),shop(0... 1 \n",
"3 on(0.953),at(0.001),with(0.000),by(0.000) 1 \n",
"4 flowers(0.147),movies(0.000),sports(0.000),sch... 1 \n",
"5 page(0.285),minute(0.000),chalk(0.000),ground(... 1 \n",
"6 Friday(0.293),Thursday(0.236),Tuesday(0.105),M... 2 \n",
"7 at(0.945),on(0.000),with(0.000),of(0.000) 1 \n",
"8 Go(0.868),Sleep(0.000),Sit(0.000),Do(0.000) 1 \n",
"9 in(0.874),on(0.001),with(0.000),after(0.000) 1 \n",
"\n",
" correct grade method \n",
"0 True 5 fill-mask-with-targets \n",
"1 True 5 fill-mask-with-targets \n",
"2 True 5 fill-mask-with-targets \n",
"3 True 5 fill-mask-with-targets \n",
"4 True 5 fill-mask-with-targets \n",
"5 True 5 fill-mask-with-targets \n",
"6 False 5 fill-mask-with-targets \n",
"7 True 5 fill-mask-with-targets \n",
"8 True 5 fill-mask-with-targets \n",
"9 True 5 fill-mask-with-targets "
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Masked Language Model with candidate words\n",
"# In this case, the model's guess is considered correct if the correct one is the first guess.\n",
"def solve_with_choices(problems):\n",
" out = []\n",
" for p in problems:\n",
" text = p.text.format(model.tokenizer.mask_token)\n",
" targets = [\" \" + c for c in p.choices] # without this, words seems to be treated as suffix\n",
" res = model(text, targets=targets)\n",
"\n",
" words = [s[\"token_str\"].strip() for s in res]\n",
" scores = [s[\"score\"] for s in res]\n",
" suggested_scores = \",\".join(\"%s(%.3f)\" % (w,s) for w, s in zip(words, scores))\n",
" # location of answer\n",
" if p.answer in words:\n",
" position = words.index(p.answer) + 1\n",
" else:\n",
" position = -1\n",
" out.append((p.text, suggested_scores, position))\n",
" out = pd.DataFrame(out, columns=[\"problem\", \"scores\", \"answer_position\"])\n",
" out[\"correct\"] = (out.answer_position == 1)\n",
" return out\n",
"o = solve_with_choices(eiken5)\n",
"\n",
"o[\"grade\"] = \"5\"\n",
"o[\"method\"] = \"fill-mask-with-targets\"\n",
"out = pd.concat((out, o))\n",
"o\n",
"\n",
"# The only failure by BERT is about the day of week:\n",
"# Today is Wednesday. Tomorrow is ( ).\n",
"#\n",
"# The model guessed Friday rather than Thursday. Close!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "53af53ac",
"metadata": {
"id": "_wRHWBvYpacS",
"papermill": {
"duration": 0.025366,
"end_time": "2021-08-01T13:33:18.567797",
"exception": false,
"start_time": "2021-08-01T13:33:18.542431",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 8,
"id": "92ae6911",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:33:18.631977Z",
"iopub.status.busy": "2021-08-01T13:33:18.630908Z",
"iopub.status.idle": "2021-08-01T13:33:20.353351Z",
"shell.execute_reply": "2021-08-01T13:33:20.353885Z",
"shell.execute_reply.started": "2021-08-01T13:20:34.472063Z"
},
"id": "e1fUC_RF07Jp",
"outputId": "859b2cd0-f587-4835-ebdb-7d3491a5f53e",
"papermill": {
"duration": 1.760249,
"end_time": "2021-08-01T13:33:20.354061",
"exception": false,
"start_time": "2021-08-01T13:33:18.593812",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>My father is a {} of a sports club. He plays t...</td>\n",
" <td>member(0.660),founder(0.068),patron(0.067),dir...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Mr. Clark told us many intesting {} about his ...</td>\n",
" <td>stories(0.447),anecdotes(0.249),tales(0.187),t...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>It's snowing a lot today, so please drive {}.</td>\n",
" <td>accordingly(0.111),responsibly(0.091),carefull...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>In spring, Jane likes to walk in her grandmoth...</td>\n",
" <td>garden(0.659),yard(0.233),gardens(0.042),backy...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Many girls in my class have {} hair.</td>\n",
" <td>curly(0.335),blonde(0.172),blond(0.074),facial...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>A: Do you live in a city? B: No. I live in a ...</td>\n",
" <td>town(0.515),city(0.139),village(0.099),suburb(...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>I {} Nancy's notebook. It was on Mary's desk</td>\n",
" <td>remember(0.180),remembered(0.115),borrowed(0.0...</td>\n",
" <td>4</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Dennis went to Japan for a year in August. He ...</td>\n",
" <td>said(0.822),waved(0.081),bid(0.039),says(0.011...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Jeff left the party at 8:00. He wanted to {} h...</td>\n",
" <td>go(0.398),get(0.369),head(0.099),come(0.050),s...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Mom's lemon cake is not as good {} her chocola...</td>\n",
" <td>as(0.998),for(0.001),than(0.000),with(0.000),a...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 My father is a {} of a sports club. He plays t... \n",
"1 Mr. Clark told us many intesting {} about his ... \n",
"2 It's snowing a lot today, so please drive {}. \n",
"3 In spring, Jane likes to walk in her grandmoth... \n",
"4 Many girls in my class have {} hair. \n",
"5 A: Do you live in a city? B: No. I live in a ... \n",
"6 I {} Nancy's notebook. It was on Mary's desk \n",
"7 Dennis went to Japan for a year in August. He ... \n",
"8 Jeff left the party at 8:00. He wanted to {} h... \n",
"9 Mom's lemon cake is not as good {} her chocola... \n",
"\n",
" scores answer_position correct \n",
"0 member(0.660),founder(0.068),patron(0.067),dir... 1 True \n",
"1 stories(0.447),anecdotes(0.249),tales(0.187),t... 1 True \n",
"2 accordingly(0.111),responsibly(0.091),carefull... -1 False \n",
"3 garden(0.659),yard(0.233),gardens(0.042),backy... 1 True \n",
"4 curly(0.335),blonde(0.172),blond(0.074),facial... -1 False \n",
"5 town(0.515),city(0.139),village(0.099),suburb(... 1 True \n",
"6 remember(0.180),remembered(0.115),borrowed(0.0... 4 True \n",
"7 said(0.822),waved(0.081),bid(0.039),says(0.011... 1 True \n",
"8 go(0.398),get(0.369),head(0.099),come(0.050),s... 2 True \n",
"9 as(0.998),for(0.001),than(0.000),with(0.000),a... 1 True "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"The specified target token ` coldly` does not exist in the model vocabulary. Replacing with `Ġcold`.\n",
"The specified target token ` busily` does not exist in the model vocabulary. Replacing with `Ġbus`.\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>My father is a {} of a sports club. He plays t...</td>\n",
" <td>member(0.660),group(0.000),festival(0.000),pic...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Mr. Clark told us many intesting {} about his ...</td>\n",
" <td>stories(0.447),books(0.000),pictures(0.000),ma...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>It's snowing a lot today, so please drive {}.</td>\n",
" <td>slowly(0.036),cold(0.001),freely(0.000),bus(0....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>In spring, Jane likes to walk in her grandmoth...</td>\n",
" <td>garden(0.659),wall(0.000),stone(0.000),sky(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Many girls in my class have {} hair.</td>\n",
" <td>short(0.023),late(0.000),busy(0.000),slow(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>A: Do you live in a city? B: No. I live in a ...</td>\n",
" <td>town(0.515),hobby(0.000),holiday(0.000),ticket...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>I {} Nancy's notebook. It was on Mary's desk</td>\n",
" <td>found(0.077),stayed(0.000),stopped(0.000),went...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Dennis went to Japan for a year in August. He ...</td>\n",
" <td>said(0.822),told(0.001),ended(0.000),hoped(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Jeff left the party at 8:00. He wanted to {} h...</td>\n",
" <td>get(0.369),send(0.000),meet(0.000),put(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Mom's lemon cake is not as good {} her chocola...</td>\n",
" <td>as(0.998),to(0.000),by(0.000),of(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 My father is a {} of a sports club. He plays t... \n",
"1 Mr. Clark told us many intesting {} about his ... \n",
"2 It's snowing a lot today, so please drive {}. \n",
"3 In spring, Jane likes to walk in her grandmoth... \n",
"4 Many girls in my class have {} hair. \n",
"5 A: Do you live in a city? B: No. I live in a ... \n",
"6 I {} Nancy's notebook. It was on Mary's desk \n",
"7 Dennis went to Japan for a year in August. He ... \n",
"8 Jeff left the party at 8:00. He wanted to {} h... \n",
"9 Mom's lemon cake is not as good {} her chocola... \n",
"\n",
" scores answer_position correct \n",
"0 member(0.660),group(0.000),festival(0.000),pic... 1 True \n",
"1 stories(0.447),books(0.000),pictures(0.000),ma... 1 True \n",
"2 slowly(0.036),cold(0.001),freely(0.000),bus(0.... 1 True \n",
"3 garden(0.659),wall(0.000),stone(0.000),sky(0.000) 1 True \n",
"4 short(0.023),late(0.000),busy(0.000),slow(0.000) 1 True \n",
"5 town(0.515),hobby(0.000),holiday(0.000),ticket... 1 True \n",
"6 found(0.077),stayed(0.000),stopped(0.000),went... 1 True \n",
"7 said(0.822),told(0.001),ended(0.000),hoped(0.000) 1 True \n",
"8 get(0.369),send(0.000),meet(0.000),put(0.000) 1 True \n",
"9 as(0.998),to(0.000),by(0.000),of(0.000) 1 True "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Apply the methods to other grades\n",
"\n",
"# Eiken grade 4\n",
"# source: https://www.eiken.or.jp/eiken/exam/grade_4/pdf/202101/2021-1-1ji-4kyu.pdf\n",
"\n",
"eiken4 = [\n",
" Problem(\"My father is a {} of a sports club. He plays tennis there every Wednesday night.\",\n",
" [\"festival\", \"picnic\", \"member\", \"group\"], \"member\")\n",
" ,Problem(\"Mr. Clark told us many intesting {} about his trip to India.\",\n",
" [\"pictures\", \"books\", \"stories\", \"magazines\"], \"stories\")\n",
" ,Problem(\"It's snowing a lot today, so please drive {}.\",\n",
" [\"slowly\", \"freely\", \"coldly\", \"busily\"], \"slowly\")\n",
" ,Problem(\"In spring, Jane likes to walk in her grandmother's {}. She enjoys looking at the beautiful flowers there.\",\n",
" [\"stone\", \"sky\", \"garden\", \"wall\"], \"garden\")\n",
" ,Problem(\"Many girls in my class have {} hair.\",\n",
" [\"late\", \"slow\", \"short\", \"busy\"], \"short\")\n",
" ,Problem(\"A: Do you live in a city? B: No. I live in a small {}\",\n",
" [\"hobby\", \"ticket\", \"town\", \"holiday\"], \"town\")\n",
" ,Problem(\"I {} Nancy's notebook. It was on Mary's desk\",\n",
" [\"stayed\", \"found\", \"stopped\", \"went\"], \"found\")\n",
" ,Problem(\"Dennis went to Japan for a year in August. He was sad when he {} goodbye to his family\",\n",
" [\"ended\", \"hoped\", \"told\", \"said\"], \"said\")\n",
" ,Problem(\"Jeff left the party at 8:00. He wanted to {} home early and go to bed.\",\n",
" [\"meet\", \"put\", \"send\", \"get\"], \"get\")\n",
" ,Problem(\"Mom's lemon cake is not as good {} her chocolate cake.\",\n",
" [\"to\", \"of\", \"as\", \"by\"], \"as\")\n",
"]\n",
"\n",
"o = solve_without_choices(eiken4)\n",
"display(o)\n",
"o[\"grade\"] = \"4\"\n",
"o[\"method\"] = \"fill-mask-no-target\"\n",
"out = pd.concat((out, o))\n",
"\n",
"o = solve_with_choices(eiken4)\n",
"display(o)\n",
"o[\"grade\"] = \"4\"\n",
"o[\"method\"] = \"fill-mask-with-targets\"\n",
"out = pd.concat((out, o))"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "9965daf2",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:33:20.425580Z",
"iopub.status.busy": "2021-08-01T13:33:20.424117Z",
"iopub.status.idle": "2021-08-01T13:33:22.398860Z",
"shell.execute_reply": "2021-08-01T13:33:22.399299Z",
"shell.execute_reply.started": "2021-08-01T13:20:36.182252Z"
},
"id": "vCFInWXT8KVH",
"outputId": "6d2bf6c0-4277-4458-aedb-ae0cbb8e59eb",
"papermill": {
"duration": 2.016298,
"end_time": "2021-08-01T13:33:22.399468",
"exception": false,
"start_time": "2021-08-01T13:33:20.383170",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A: How do you make that potato dish? B: First...</td>\n",
" <td>peel(0.218),roast(0.106),bake(0.099),boil(0.08...</td>\n",
" <td>4</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Last summer, Hiroshi's family traveled around ...</td>\n",
" <td>skiing(0.214),surfing(0.121),camping(0.098),fi...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Bob {} five friends to his party</td>\n",
" <td>invites(0.555),invited(0.367),welcomes(0.031),...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>A: John, you should go to bed soon. If you sta...</td>\n",
" <td>graduate(0.145),crash(0.076),cry(0.051),die(0....</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>A: Did you buy your father something special f...</td>\n",
" <td>one(0.087),stove(0.038),spoon(0.033),dish(0.03...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>I bought a new T-shirt for my brother, but I b...</td>\n",
" <td>big(0.277),small(0.083),expensive(0.051),bulky...</td>\n",
" <td>5</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Sarah saw some flowers by the road while she w...</td>\n",
" <td>picked(0.319),bought(0.086),collected(0.085),g...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Jenny saw her grandparents {} the first time i...</td>\n",
" <td>for(0.815),at(0.052),in(0.027),again(0.011),li...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>A: I told my mother that I would be home by 7:...</td>\n",
" <td>break(0.769),fulfill(0.066),violate(0.044),kee...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>A: Don'y say anything to Dad about the surpris...</td>\n",
" <td>anything(0.642),out(0.304),word(0.020),somethi...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 A: How do you make that potato dish? B: First... \n",
"1 Last summer, Hiroshi's family traveled around ... \n",
"2 Bob {} five friends to his party \n",
"3 A: John, you should go to bed soon. If you sta... \n",
"4 A: Did you buy your father something special f... \n",
"5 I bought a new T-shirt for my brother, but I b... \n",
"6 Sarah saw some flowers by the road while she w... \n",
"7 Jenny saw her grandparents {} the first time i... \n",
"8 A: I told my mother that I would be home by 7:... \n",
"9 A: Don'y say anything to Dad about the surpris... \n",
"\n",
" scores answer_position correct \n",
"0 peel(0.218),roast(0.106),bake(0.099),boil(0.08... 4 True \n",
"1 skiing(0.214),surfing(0.121),camping(0.098),fi... -1 False \n",
"2 invites(0.555),invited(0.367),welcomes(0.031),... 2 True \n",
"3 graduate(0.145),crash(0.076),cry(0.051),die(0.... -1 False \n",
"4 one(0.087),stove(0.038),spoon(0.033),dish(0.03... -1 False \n",
"5 big(0.277),small(0.083),expensive(0.051),bulky... 5 True \n",
"6 picked(0.319),bought(0.086),collected(0.085),g... 1 True \n",
"7 for(0.815),at(0.052),in(0.027),again(0.011),li... 1 True \n",
"8 break(0.769),fulfill(0.066),violate(0.044),kee... 1 True \n",
"9 anything(0.642),out(0.304),word(0.020),somethi... 2 True "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"The specified target token ` oversleep` does not exist in the model vocabulary. Replacing with `Ġovers`.\n",
"The specified target token ` apron` does not exist in the model vocabulary. Replacing with `Ġa`.\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A: How do you make that potato dish? B: First...</td>\n",
" <td>boil(0.087),eat(0.003),hurt(0.000),care(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Last summer, Hiroshi's family traveled around ...</td>\n",
" <td>abroad(0.019),inside(0.000),similar(0.000),oth...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Bob {} five friends to his party</td>\n",
" <td>invited(0.367),made(0.000),visited(0.000),spok...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>A: John, you should go to bed soon. If you sta...</td>\n",
" <td>graduate(0.145),return(0.001),promise(0.000),o...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>A: Did you buy your father something special f...</td>\n",
" <td>ring(0.002),field(0.000),a(0.000),contact(0.000)</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>I bought a new T-shirt for my brother, but I b...</td>\n",
" <td>tight(0.046),heavy(0.042),bright(0.002),clear(...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Sarah saw some flowers by the road while she w...</td>\n",
" <td>picked(0.319),spent(0.000),guessed(0.000),wish...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Jenny saw her grandparents {} the first time i...</td>\n",
" <td>for(0.815),from(0.003),over(0.001),out(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>A: I told my mother that I would be home by 7:...</td>\n",
" <td>break(0.769),sell(0.000),pass(0.000),lend(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>A: Don'y say anything to Dad about the surpris...</td>\n",
" <td>out(0.304),through(0.000),near(0.000),within(0...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 A: How do you make that potato dish? B: First... \n",
"1 Last summer, Hiroshi's family traveled around ... \n",
"2 Bob {} five friends to his party \n",
"3 A: John, you should go to bed soon. If you sta... \n",
"4 A: Did you buy your father something special f... \n",
"5 I bought a new T-shirt for my brother, but I b... \n",
"6 Sarah saw some flowers by the road while she w... \n",
"7 Jenny saw her grandparents {} the first time i... \n",
"8 A: I told my mother that I would be home by 7:... \n",
"9 A: Don'y say anything to Dad about the surpris... \n",
"\n",
" scores answer_position correct \n",
"0 boil(0.087),eat(0.003),hurt(0.000),care(0.000) 1 True \n",
"1 abroad(0.019),inside(0.000),similar(0.000),oth... 1 True \n",
"2 invited(0.367),made(0.000),visited(0.000),spok... 1 True \n",
"3 graduate(0.145),return(0.001),promise(0.000),o... -1 False \n",
"4 ring(0.002),field(0.000),a(0.000),contact(0.000) -1 False \n",
"5 tight(0.046),heavy(0.042),bright(0.002),clear(... 1 True \n",
"6 picked(0.319),spent(0.000),guessed(0.000),wish... 1 True \n",
"7 for(0.815),from(0.003),over(0.001),out(0.000) 1 True \n",
"8 break(0.769),sell(0.000),pass(0.000),lend(0.000) 1 True \n",
"9 out(0.304),through(0.000),near(0.000),within(0... 1 True "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Eiken grade 3\n",
"# source: https://www.eiken.or.jp/eiken/exam/grade_3/pdf/202101/2021-1-1ji-3kyu.pdf\n",
"\n",
"eiken3 = [\n",
" Problem(\"A: How do you make that potato dish? B: First, you {} the potatoes, and then cut them in half and put butter on them.\",\n",
" [\"boil\", \"care\", \"hurt\", \"eat\"], \"boil\")\n",
" ,Problem(\"Last summer, Hiroshi's family traveled around Japan. This year they went to go {}.\",\n",
" [\"abroad\", \"inside\", \"other\", \"similar\"], \"abroad\")\n",
" ,Problem(\"Bob {} five friends to his party\",\n",
" [\"made\", \"visited\", \"invited\", \"spoke\"], \"invited\")\n",
" ,Problem(\"A: John, you should go to bed soon. If you stay up too late, you'll {} and be late for school. B: OK, Mom.\",\n",
" [\"graduate\", \"promise\", \"return\", \"oversleep\"], \"oversleep\")\n",
" ,Problem(\"A: Did you buy your father something special for his birthday? B: Yes. He loves to cook, so I got him a new {}.\",\n",
" [\"apron\", \"ring\", \"contact\", \"field\"], \"apron\")\n",
" ,Problem(\"I bought a new T-shirt for my brother, but I bought the wong size. It was too {} for him.\",\n",
" [\"heavy\", \"clear\", \"tight\", \"bright\"], \"tight\")\n",
" ,Problem(\"Sarah saw some flowers by the road while she was taking a walk. She {} a few and took them home.\",\n",
" [\"spent\", \"wished\", \"picked\", \"guessed\"], \"picked\")\n",
" ,Problem(\"Jenny saw her grandparents {} the first time in years. She missed them very much.\",\n",
" [\"for\", \"from\", \"out\", \"over\"], \"for\")\n",
" ,Problem(\"A: I told my mother that I would be home by 7:00. I don't want to {} my promise, so I have to go now. B: OK.\",\n",
" [\"pass\", \"sell\", \"break\", \"lend\"], \"break\")\n",
" ,Problem(\"A: Don'y say anything to Dad about the surprise party! B: Don't worry. He won't find {} about it from me.\",\n",
" [\"within\", \"through\", \"out\", \"near\"], \"out\")\n",
"]\n",
"\n",
"o = solve_without_choices(eiken3)\n",
"display(o)\n",
"o[\"grade\"] = \"3\"\n",
"o[\"method\"] = \"fill-mask-no-target\"\n",
"out = pd.concat((out, o))\n",
"\n",
"o = solve_with_choices(eiken3)\n",
"display(o)\n",
"o[\"grade\"] = \"3\"\n",
"o[\"method\"] = \"fill-mask-with-targets\"\n",
"out = pd.concat((out, o))"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "751ace08",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:33:22.480859Z",
"iopub.status.busy": "2021-08-01T13:33:22.479770Z",
"iopub.status.idle": "2021-08-01T13:33:24.689742Z",
"shell.execute_reply": "2021-08-01T13:33:24.689196Z",
"shell.execute_reply.started": "2021-08-01T13:20:38.036834Z"
},
"id": "QQHvJ-Oi-8Nl",
"outputId": "0d91f3d0-8a2f-45ef-82d7-ebb9398783d4",
"papermill": {
"duration": 2.254988,
"end_time": "2021-08-01T13:33:24.689882",
"exception": false,
"start_time": "2021-08-01T13:33:22.434894",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Jamie visited several {} buildings when he wen...</td>\n",
" <td>ancient(0.177),Roman(0.109),historic(0.066),By...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Sally's French teacher told her to read an art...</td>\n",
" <td>translate(0.944),paste(0.033),translated(0.007...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Henry likes living in the city because there a...</td>\n",
" <td>woods(0.460),mountains(0.118),countryside(0.05...</td>\n",
" <td>3</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>A: Is it true that the things in this store on...</td>\n",
" <td>extra(0.120),yen(0.100),postage(0.098),taxes(0...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>When the bust was an hour late, one man shoute...</td>\n",
" <td>angrily(0.495),insults(0.126),loudly(0.057),cu...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Firefighters have to {} people from buildings ...</td>\n",
" <td>rescue(0.601),evacuate(0.232),remove(0.049),pu...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>John loves the singer Ann May, and he cannot w...</td>\n",
" <td>released(0.776),out(0.174),published(0.015),av...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>The news that Ms. Kelly, the art teacher, was ...</td>\n",
" <td>blew(0.197),swept(0.149),spread(0.141),came(0....</td>\n",
" <td>3</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>A: I'm really nervous about acting in the play...</td>\n",
" <td>stage(0.415),scene(0.078),screen(0.048),show(0...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Before Diane attended Professor Miller's {} at...</td>\n",
" <td>lectures(0.595),lecture(0.120),classes(0.041),...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 Jamie visited several {} buildings when he wen... \n",
"1 Sally's French teacher told her to read an art... \n",
"2 Henry likes living in the city because there a... \n",
"3 A: Is it true that the things in this store on... \n",
"4 When the bust was an hour late, one man shoute... \n",
"5 Firefighters have to {} people from buildings ... \n",
"6 John loves the singer Ann May, and he cannot w... \n",
"7 The news that Ms. Kelly, the art teacher, was ... \n",
"8 A: I'm really nervous about acting in the play... \n",
"9 Before Diane attended Professor Miller's {} at... \n",
"\n",
" scores answer_position correct \n",
"0 ancient(0.177),Roman(0.109),historic(0.066),By... 1 True \n",
"1 translate(0.944),paste(0.033),translated(0.007... 1 True \n",
"2 woods(0.460),mountains(0.118),countryside(0.05... 3 True \n",
"3 extra(0.120),yen(0.100),postage(0.098),taxes(0... -1 False \n",
"4 angrily(0.495),insults(0.126),loudly(0.057),cu... 1 True \n",
"5 rescue(0.601),evacuate(0.232),remove(0.049),pu... 1 True \n",
"6 released(0.776),out(0.174),published(0.015),av... 1 True \n",
"7 blew(0.197),swept(0.149),spread(0.141),came(0.... 3 True \n",
"8 stage(0.415),scene(0.078),screen(0.048),show(0... 1 True \n",
"9 lectures(0.595),lecture(0.120),classes(0.041),... 2 True "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Jamie visited several {} buildings when he wen...</td>\n",
" <td>ancient(0.177),exact(0.000),responsible(0.000)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Sally's French teacher told her to read an art...</td>\n",
" <td>translate(0.944),throw(0.000),guide(0.000),con...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Henry likes living in the city because there a...</td>\n",
" <td>countryside(0.057),image(0.000),experiment(0.0...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>A: Is it true that the things in this store on...</td>\n",
" <td>tax(0.011),data(0.000),total(0.000),waste(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>When the bust was an hour late, one man shoute...</td>\n",
" <td>angrily(0.495),tightly(0.000),partly(0.000),se...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Firefighters have to {} people from buildings ...</td>\n",
" <td>rescue(0.601),weigh(0.000),produce(0.000),stam...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>John loves the singer Ann May, and he cannot w...</td>\n",
" <td>released(0.776),invented(0.000),trapped(0.000)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>The news that Ms. Kelly, the art teacher, was ...</td>\n",
" <td>spread(0.141),stretched(0.000),served(0.000),s...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>A: I'm really nervous about acting in the play...</td>\n",
" <td>stage(0.415),screen(0.048),field(0.011),court(...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Before Diane attended Professor Miller's {} at...</td>\n",
" <td>lecture(0.120),comment(0.000),furniture(0.000)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 Jamie visited several {} buildings when he wen... \n",
"1 Sally's French teacher told her to read an art... \n",
"2 Henry likes living in the city because there a... \n",
"3 A: Is it true that the things in this store on... \n",
"4 When the bust was an hour late, one man shoute... \n",
"5 Firefighters have to {} people from buildings ... \n",
"6 John loves the singer Ann May, and he cannot w... \n",
"7 The news that Ms. Kelly, the art teacher, was ... \n",
"8 A: I'm really nervous about acting in the play... \n",
"9 Before Diane attended Professor Miller's {} at... \n",
"\n",
" scores answer_position correct \n",
"0 ancient(0.177),exact(0.000),responsible(0.000)... 1 True \n",
"1 translate(0.944),throw(0.000),guide(0.000),con... 1 True \n",
"2 countryside(0.057),image(0.000),experiment(0.0... 1 True \n",
"3 tax(0.011),data(0.000),total(0.000),waste(0.000) 1 True \n",
"4 angrily(0.495),tightly(0.000),partly(0.000),se... 1 True \n",
"5 rescue(0.601),weigh(0.000),produce(0.000),stam... 1 True \n",
"6 released(0.776),invented(0.000),trapped(0.000)... 1 True \n",
"7 spread(0.141),stretched(0.000),served(0.000),s... 1 True \n",
"8 stage(0.415),screen(0.048),field(0.011),court(... 1 True \n",
"9 lecture(0.120),comment(0.000),furniture(0.000)... 1 True "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Eiken grade pre2\n",
"# source: https://www.eiken.or.jp/eiken/exam/grade_p2/pdf/202101/2021-1-1ji-p2kyu.pdf\n",
"\n",
"eikenp2 = [\n",
" Problem(\"Jamie visited several {} buildings when he went to Rome. Some of them were more than 2,000 years old.\",\n",
" [\"ancient\", \"exact\", \"responsible\", \"unable\"], \"ancient\")\n",
" ,Problem(\"Sally's French teacher told her to read an article in a French newspaper and {} it into English.\",\n",
" [\"guide\", \"throw\", \"control\", \"translate\"], \"translate\")\n",
" ,Problem(\"Henry likes living in the city because there are so many things to do there. But he also loves nature, so sometimes he goes for a drive in the {}.\",\n",
" [\"decision\", \"experiment\", \"countryside\", \"image\"], \"countryside\")\n",
" ,Problem(\"A: Is it true that the things in this store only cost 100 yen? B: Yes, but you will also need to pay {}, so they actually cost a little more.\",\n",
" [\"tax\", \"data\", \"total\", \"waste\"], \"tax\")\n",
" ,Problem(\"When the bust was an hour late, one man shouted {} at the driver. He said that he had missed an important meeting.\",\n",
" [\"partly\", \"angrily\", \"secretly\", \"tightly\"], \"angrily\")\n",
" ,Problem(\"Firefighters have to {} people from buildings that are on fire. To do this, they must be strong and healthy.\",\n",
" [\"weigh\", \"produce\", \"stamp\", \"rescue\"], \"rescue\")\n",
" ,Problem(\"John loves the singer Ann May, and he cannot wait until her new CD is {} next week. He has been waiting for it since here last CD came out two years ago.\",\n",
" [\"released\", \"trapped\", \"divided\", \"invented\"], \"released\")\n",
" ,Problem(\"The news that Ms. Kelly, the art teacher, was going to get married {} through the school very quickly. By lunchtime, almost all the students knew about it.\",\n",
" [\"spread\", \"served\", \"stretched\", \"stood\"], \"spread\")\n",
" ,Problem(\"A: I'm really nervous about acting in the play next week. B: I know you're worried now, but you'll feel fine as soon a syou get on the {}.\",\n",
" [\"stage\", \"field\", \"court\", \"screen\"], \"stage\")\n",
" ,Problem(\"Before Diane attended Professor Miller's {} at the university, she was not intested in Chinese art. However, now, she wants to learn more about it.\",\n",
" [\"comment\", \"shipment\", \"lecture\", \"furniture\"], \"lecture\")\n",
"]\n",
"\n",
"o = solve_without_choices(eikenp2)\n",
"display(o)\n",
"o[\"grade\"] = \"pre2\"\n",
"o[\"method\"] = \"fill-mask-no-target\"\n",
"out = pd.concat((out, o))\n",
"\n",
"o = solve_with_choices(eikenp2)\n",
"display(o)\n",
"o[\"grade\"] = \"pre2\"\n",
"o[\"method\"] = \"fill-mask-with-targets\"\n",
"out = pd.concat((out, o))"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "fe704881",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:33:24.768441Z",
"iopub.status.busy": "2021-08-01T13:33:24.766809Z",
"iopub.status.idle": "2021-08-01T13:33:26.752962Z",
"shell.execute_reply": "2021-08-01T13:33:26.753390Z",
"shell.execute_reply.started": "2021-08-01T13:20:39.958367Z"
},
"id": "PqCrbCnLCLdM",
"outputId": "75c9ec60-8001-456d-ff55-3280528d625e",
"papermill": {
"duration": 2.03192,
"end_time": "2021-08-01T13:33:26.753571",
"exception": false,
"start_time": "2021-08-01T13:33:24.721651",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>At first, the marketing department and the sal...</td>\n",
" <td>jointly(0.257),entirely(0.139),separately(0.13...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Experts at the art gallery discovered that one...</td>\n",
" <td>Pablo(0.455),genuine(0.041),fake(0.037),signed...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>The musician Jimmy Baker had a lot of {} when ...</td>\n",
" <td>money(0.189),fame(0.113),success(0.088),proble...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Mother Teresa helped many sick people and gave...</td>\n",
" <td>orphans(0.279),humanity(0.114),children(0.082)...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>As Liam waled down the dark street, he began t...</td>\n",
" <td>feeling(0.477),impression(0.221),sense(0.122),...</td>\n",
" <td>5</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Risa buys water that comes from a mountain str...</td>\n",
" <td>nutrients(0.880),minerals(0.047),vitamins(0.02...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>The lifeguard ran into the ocean to help a you...</td>\n",
" <td>drowning(0.797),drowned(0.061),swimming(0.027)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Yesterday was a hot day at the zoo, so Heather...</td>\n",
" <td>putting(0.326),spilling(0.107),pouring(0.064),...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>In the past, sailors had to use the stars to {...</td>\n",
" <td>communicate(0.272),navigate(0.219),signal(0.07...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Daisuke's grandmother eats a lot of vegetables...</td>\n",
" <td>improve(0.144),check(0.116),boost(0.089),asses...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 At first, the marketing department and the sal... \n",
"1 Experts at the art gallery discovered that one... \n",
"2 The musician Jimmy Baker had a lot of {} when ... \n",
"3 Mother Teresa helped many sick people and gave... \n",
"4 As Liam waled down the dark street, he began t... \n",
"5 Risa buys water that comes from a mountain str... \n",
"6 The lifeguard ran into the ocean to help a you... \n",
"7 Yesterday was a hot day at the zoo, so Heather... \n",
"8 In the past, sailors had to use the stars to {... \n",
"9 Daisuke's grandmother eats a lot of vegetables... \n",
"\n",
" scores answer_position correct \n",
"0 jointly(0.257),entirely(0.139),separately(0.13... 2 True \n",
"1 Pablo(0.455),genuine(0.041),fake(0.037),signed... 2 True \n",
"2 money(0.189),fame(0.113),success(0.088),proble... -1 False \n",
"3 orphans(0.279),humanity(0.114),children(0.082)... 2 True \n",
"4 feeling(0.477),impression(0.221),sense(0.122),... 5 True \n",
"5 nutrients(0.880),minerals(0.047),vitamins(0.02... 2 True \n",
"6 drowning(0.797),drowned(0.061),swimming(0.027)... 1 True \n",
"7 putting(0.326),spilling(0.107),pouring(0.064),... 2 True \n",
"8 communicate(0.272),navigate(0.219),signal(0.07... 2 True \n",
"9 improve(0.144),check(0.116),boost(0.089),asses... -1 False "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"The specified target token ` needlessly` does not exist in the model vocabulary. Replacing with `Ġneed`.\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>At first, the marketing department and the sal...</td>\n",
" <td>entirely(0.139),scientifically(0.000),violentl...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Experts at the art gallery discovered that one...</td>\n",
" <td>genuine(0.041),severe(0.000),portable(0.000),l...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>The musician Jimmy Baker had a lot of {} when ...</td>\n",
" <td>hardship(0.002),concentration(0.000),membershi...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Mother Teresa helped many sick people and gave...</td>\n",
" <td>humanity(0.114),generation(0.000),gravity(0.00...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>As Liam waled down the dark street, he began t...</td>\n",
" <td>sensation(0.018),property(0.000),feature(0.000...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Risa buys water that comes from a mountain str...</td>\n",
" <td>minerals(0.047),operations(0.000),illustration...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>The lifeguard ran into the ocean to help a you...</td>\n",
" <td>drowning(0.797),proposing(0.000),converting(0....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Yesterday was a hot day at the zoo, so Heather...</td>\n",
" <td>spilling(0.107),maintaining(0.000),convincing(...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>In the past, sailors had to use the stars to {...</td>\n",
" <td>navigate(0.219),satisfy(0.000),respect(0.000),...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Daisuke's grandmother eats a lot of vegetables...</td>\n",
" <td>preserve(0.019),replace(0.000),betray(0.000),i...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 At first, the marketing department and the sal... \n",
"1 Experts at the art gallery discovered that one... \n",
"2 The musician Jimmy Baker had a lot of {} when ... \n",
"3 Mother Teresa helped many sick people and gave... \n",
"4 As Liam waled down the dark street, he began t... \n",
"5 Risa buys water that comes from a mountain str... \n",
"6 The lifeguard ran into the ocean to help a you... \n",
"7 Yesterday was a hot day at the zoo, so Heather... \n",
"8 In the past, sailors had to use the stars to {... \n",
"9 Daisuke's grandmother eats a lot of vegetables... \n",
"\n",
" scores answer_position correct \n",
"0 entirely(0.139),scientifically(0.000),violentl... 1 True \n",
"1 genuine(0.041),severe(0.000),portable(0.000),l... 1 True \n",
"2 hardship(0.002),concentration(0.000),membershi... 1 True \n",
"3 humanity(0.114),generation(0.000),gravity(0.00... 1 True \n",
"4 sensation(0.018),property(0.000),feature(0.000... 1 True \n",
"5 minerals(0.047),operations(0.000),illustration... 1 True \n",
"6 drowning(0.797),proposing(0.000),converting(0.... 1 True \n",
"7 spilling(0.107),maintaining(0.000),convincing(... 1 True \n",
"8 navigate(0.219),satisfy(0.000),respect(0.000),... 1 True \n",
"9 preserve(0.019),replace(0.000),betray(0.000),i... 1 True "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Eiken grade 2\n",
"# source: https://www.eiken.or.jp/eiken/exam/grade_2/pdf/202101/2021-1-1ji-2kyu.pdf\n",
"\n",
"eiken2 = [\n",
" Problem((\"At first, the marketing department and the sales department were on the project together. \"\n",
" \"But people in the sales department were too busy, so now the project is being run {} by the marketing department.\"),\n",
" [\"needlessly\", \"entirely\", \"scientifically\", \"violently\"], \"entirely\")\n",
" ,Problem(\"Experts at the art gallery discovered that one of their paitings, which they had thought was a {} Picasso, was actually just a copy.\",\n",
" [\"genuine\", \"severe\", \"logical\", \"portable\"], \"genuine\")\n",
" ,Problem(\"The musician Jimmy Baker had a lot of {} when he was a child. His family was very poor before he became a rich and famous rock star.\",\n",
" [\"permission\", \"membership\", \"concentration\", \"hardship\"], \"hardship\")\n",
" ,Problem(\"Mother Teresa helped many sick people and gave food to many hungry children in India. She was known as a person who cared about {}.\",\n",
" [\"generation\", \"gravity\", \"hesitation\", \"humanity\"], \"humanity\")\n",
" ,Problem(\"As Liam waled down the dark street, he began to feel afraid. He had the {} that someone was watching him.\",\n",
" [\"feature\", \"translation\", \"sensation\", \"property\"], \"sensation\")\n",
" ,Problem(\"Risa buys water that comes from a mountain stream. She says that drinking it is good because it has many {} that her body needs.\",\n",
" [\"campaigns\", \"operations\", \"illustrations\", \"minerals\"], \"minerals\")\n",
" ,Problem(\"The lifeguard ran into the ocean to help a young girl who looked like she was {} in the big waves\",\n",
" [\"proposing\", \"converting\", \"drowning\", \"exporting\"], \"drowning\")\n",
" ,Problem(\"Yesterday was a hot day at the zoo, so Heather bought an ice cream. It melted so quickly that she could not help {} some on her dress.\",\n",
" [\"arguing\", \"spilling\", \"convincing\", \"maintaining\"], \"spilling\")\n",
" ,Problem(\"In the past, sailors had to use the stars to {} when they were on an ocean. These days, ships have modern equipment that shows sailors which way to go.\",\n",
" [\"satisfy\", \"respect\", \"permit\", \"navigate\"], \"navigate\")\n",
" ,Problem(\"Daisuke's grandmother eats a lot of vegetables, drinks green tea, and goes for a long walk every evening to {} her health.\",\n",
" [\"interpret\", \"replace\", \"preserve\", \"betray\"], \"preserve\")\n",
"]\n",
"\n",
"o = solve_without_choices(eiken2)\n",
"display(o)\n",
"o[\"grade\"] = \"2\"\n",
"o[\"method\"] = \"fill-mask-no-target\"\n",
"out = pd.concat((out, o))\n",
"\n",
"o = solve_with_choices(eiken2)\n",
"display(o)\n",
"o[\"grade\"] = \"2\"\n",
"o[\"method\"] = \"fill-mask-with-targets\"\n",
"out = pd.concat((out, o))"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "4bfa297b",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:33:26.834296Z",
"iopub.status.busy": "2021-08-01T13:33:26.832752Z",
"iopub.status.idle": "2021-08-01T13:33:29.369033Z",
"shell.execute_reply": "2021-08-01T13:33:29.370087Z",
"shell.execute_reply.started": "2021-08-01T13:20:41.826715Z"
},
"id": "qQuKSMMdPECm",
"outputId": "e2a39d54-237f-450e-a106-0494df8d9cf3",
"papermill": {
"duration": 2.584946,
"end_time": "2021-08-01T13:33:29.370333",
"exception": false,
"start_time": "2021-08-01T13:33:26.785387",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A: Thanks for showing me the outline of your s...</td>\n",
" <td>confusing(0.174),repetitive(0.051),boring(0.04...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Lisa went to the interview even though she tho...</td>\n",
" <td>probability(0.391),likelihood(0.337),chance(0....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>It is sadly {} that, in developing counties, m...</td>\n",
" <td>ironic(0.355),sad(0.098),true(0.081),unfortuna...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>The explosion at the chemical factory {} great...</td>\n",
" <td>inflicted(0.798),caused(0.093),wrought(0.036),...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Some say the best way to overcome a {} is to e...</td>\n",
" <td>fear(0.577),stigma(0.036),dilemma(0.032),depre...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>English classes at the university were require...</td>\n",
" <td>excluded(0.375),exempted(0.224),barred(0.143),...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>E-mail and text messaging have {} the way peop...</td>\n",
" <td>changed(0.602),altered(0.146),transformed(0.10...</td>\n",
" <td>3</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Some analysts think the new treaty on CO2 emis...</td>\n",
" <td>milestone(0.416),breakthrough(0.238),landmark(...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Lying on the sunny beach with her husband on t...</td>\n",
" <td>so(0.175),incredibly(0.166),very(0.063),extrem...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Nadine spends an hour thoroughly cleaning her ...</td>\n",
" <td>clean(0.318),tidy(0.183),cleaner(0.109),cleane...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 A: Thanks for showing me the outline of your s... \n",
"1 Lisa went to the interview even though she tho... \n",
"2 It is sadly {} that, in developing counties, m... \n",
"3 The explosion at the chemical factory {} great... \n",
"4 Some say the best way to overcome a {} is to e... \n",
"5 English classes at the university were require... \n",
"6 E-mail and text messaging have {} the way peop... \n",
"7 Some analysts think the new treaty on CO2 emis... \n",
"8 Lying on the sunny beach with her husband on t... \n",
"9 Nadine spends an hour thoroughly cleaning her ... \n",
"\n",
" scores answer_position correct \n",
"0 confusing(0.174),repetitive(0.051),boring(0.04... -1 False \n",
"1 probability(0.391),likelihood(0.337),chance(0.... 1 True \n",
"2 ironic(0.355),sad(0.098),true(0.081),unfortuna... 1 True \n",
"3 inflicted(0.798),caused(0.093),wrought(0.036),... 1 True \n",
"4 fear(0.577),stigma(0.036),dilemma(0.032),depre... -1 False \n",
"5 excluded(0.375),exempted(0.224),barred(0.143),... 2 True \n",
"6 changed(0.602),altered(0.146),transformed(0.10... 3 True \n",
"7 milestone(0.416),breakthrough(0.238),landmark(... 1 True \n",
"8 so(0.175),incredibly(0.166),very(0.063),extrem... -1 False \n",
"9 clean(0.318),tidy(0.183),cleaner(0.109),cleane... -1 False "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"The specified target token ` barricade` does not exist in the model vocabulary. Replacing with `Ġbarric`.\n",
"The specified target token ` phobia` does not exist in the model vocabulary. Replacing with `Ġph`.\n",
"The specified target token ` officiated` does not exist in the model vocabulary. Replacing with `Ġoffic`.\n",
"The specified target token ` synthesized` does not exist in the model vocabulary. Replacing with `Ġsynthes`.\n",
"The specified target token ` disarmed` does not exist in the model vocabulary. Replacing with `Ġdis`.\n",
"The specified target token ` vigor` does not exist in the model vocabulary. Replacing with `Ġvig`.\n",
"The specified target token ` spotless` does not exist in the model vocabulary. Replacing with `Ġspot`.\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A: Thanks for showing me the outline of your s...</td>\n",
" <td>redundant(0.006),subjective(0.000),distinct(0....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Lisa went to the interview even though she tho...</td>\n",
" <td>probability(0.391),credibility(0.000),contenti...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>It is sadly {} that, in developing counties, m...</td>\n",
" <td>ironic(0.355),superficial(0.000),indefinite(0....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>The explosion at the chemical factory {} great...</td>\n",
" <td>inflicted(0.798),enhanced(0.000),perceived(0.0...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Some say the best way to overcome a {} is to e...</td>\n",
" <td>temptation(0.002),famine(0.000),ph(0.000),barr...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>English classes at the university were require...</td>\n",
" <td>exempted(0.224),qualified(0.000),prosecuted(0....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>E-mail and text messaging have {} the way peop...</td>\n",
" <td>transformed(0.105),dis(0.000),synthes(0.000),o...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Some analysts think the new treaty on CO2 emis...</td>\n",
" <td>milestone(0.416),backlog(0.000),confession(0.0...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Lying on the sunny beach with her husband on t...</td>\n",
" <td>profoundly(0.004),barely(0.000),harshly(0.000)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Nadine spends an hour thoroughly cleaning her ...</td>\n",
" <td>spot(0.000),rugged(0.000),impartial(0.000),min...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 A: Thanks for showing me the outline of your s... \n",
"1 Lisa went to the interview even though she tho... \n",
"2 It is sadly {} that, in developing counties, m... \n",
"3 The explosion at the chemical factory {} great... \n",
"4 Some say the best way to overcome a {} is to e... \n",
"5 English classes at the university were require... \n",
"6 E-mail and text messaging have {} the way peop... \n",
"7 Some analysts think the new treaty on CO2 emis... \n",
"8 Lying on the sunny beach with her husband on t... \n",
"9 Nadine spends an hour thoroughly cleaning her ... \n",
"\n",
" scores answer_position correct \n",
"0 redundant(0.006),subjective(0.000),distinct(0.... 1 True \n",
"1 probability(0.391),credibility(0.000),contenti... 1 True \n",
"2 ironic(0.355),superficial(0.000),indefinite(0.... 1 True \n",
"3 inflicted(0.798),enhanced(0.000),perceived(0.0... 1 True \n",
"4 temptation(0.002),famine(0.000),ph(0.000),barr... -1 False \n",
"5 exempted(0.224),qualified(0.000),prosecuted(0.... 1 True \n",
"6 transformed(0.105),dis(0.000),synthes(0.000),o... 1 True \n",
"7 milestone(0.416),backlog(0.000),confession(0.0... 1 True \n",
"8 profoundly(0.004),barely(0.000),harshly(0.000)... 1 True \n",
"9 spot(0.000),rugged(0.000),impartial(0.000),min... -1 False "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Eiken grade pre1\n",
"# source: https://www.eiken.or.jp/eiken/exam/grade_p1/pdf/202101/2021-1-1ji-p1kyu.pdf\n",
"\n",
"eikenp1 = [\n",
" Problem((\"A: Thanks for showing me the outline of your sales presentation. It's good, but it's a bit {} in some places. \"\n",
" \"B: I guess I do repeat some information too much. I'll try to take some of it out.\"),\n",
" [\"decisive\", \"subjective\", \"redundant\", \"distinct\"], \"redundant\")\n",
" ,Problem(\"Lisa went to the interview even though she thought there was a low {} of her getting the job. As she expected, she was not hired.\",\n",
" [\"restoration\", \"credibility\", \"contention\", \"probability\"], \"probability\")\n",
" ,Problem(\"It is sadly {} that, in developing counties, many of the farmers who grow nutritous crops for export do no not have enough food to feed their own families.\",\n",
" [\"indefinite\", \"ironic\", \"restless\", \"superficial\"], \"ironic\")\n",
" ,Problem(\"The explosion at the chemical factory {} great damage on the local environment. It will take years for wildlife to fully recover in the region.\",\n",
" [\"inflicted\", \"enhanced\", \"vanished\", \"perceived\"], \"inflicted\")\n",
" ,Problem(\"Some say the best way to overcome a {} is to expose oneself to what one fears. For example, people who are afraid of mice should try holding one.\",\n",
" [\"temptation\", \"barricade\", \"phobia\", \"famine\"], \"phobia\")\n",
" ,Problem(\"English classes at the university were required, but students were {} from them if they could prove they had advanced ability in the language.\",\n",
" [\"exempted\", \"prosecuted\", \"commanded\", \"qualified\"], \"exempted\")\n",
" ,Problem(\"E-mail and text messaging have {} the way people write. Many people shorten words and ignore traditional rules of grammar.\",\n",
" [\"transformed\", \"officiated\", \"synthesized\", \"disarmed\"], \"transformed\")\n",
" ,Problem((\"Some analysts think the new treaty on CO2 emissions is a {} in the fight against global warming. \"\n",
" '\"This is the most important environmental treaty ever signed,\" said one.'),\n",
" [\"milestone\", \"vigor\", \"backlog\", \"confession\"], \"milestone\")\n",
" ,Problem(\"Lying on the sunny beach with her husband on their vacation, Roberta felt {} happy. She had never been so content.\",\n",
" [\"barely\", \"profoundly\", \"improperly\", \"harshly\"], \"profoundly\")\n",
" ,Problem(\"Nadine spends an hour thoroughly cleaning her apartment evey day, so the entire place is {}.\",\n",
" [\"spotless\", \"minute\", \"rugged\", \"impartial\"], \"spotless\")\n",
"]\n",
"\n",
"o = solve_without_choices(eikenp1)\n",
"display(o)\n",
"o[\"grade\"] = \"pre1\"\n",
"o[\"method\"] = \"fill-mask-no-target\"\n",
"out = pd.concat((out, o))\n",
"\n",
"o = solve_with_choices(eikenp1)\n",
"display(o)\n",
"o[\"grade\"] = \"pre1\"\n",
"o[\"method\"] = \"fill-mask-with-targets\"\n",
"out = pd.concat((out, o))"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "4f76cb0b",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:33:29.488169Z",
"iopub.status.busy": "2021-08-01T13:33:29.487188Z",
"iopub.status.idle": "2021-08-01T13:33:31.643785Z",
"shell.execute_reply": "2021-08-01T13:33:31.644269Z",
"shell.execute_reply.started": "2021-08-01T13:20:44.084332Z"
},
"id": "XzDBtSSUUmPd",
"outputId": "110d4c11-ae11-4354-976c-99d67f3a4d3f",
"papermill": {
"duration": 2.211555,
"end_time": "2021-08-01T13:33:31.644437",
"exception": false,
"start_time": "2021-08-01T13:33:29.432882",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Cell phones have become a permanent {} in mode...</td>\n",
" <td>fixture(0.932),feature(0.011),necessity(0.011)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Colin did not have enough money to pay for the...</td>\n",
" <td>increments(0.440),excess(0.375),installments(0...</td>\n",
" <td>3</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>When she asked her boss for a raise, Melanie's...</td>\n",
" <td>calm(0.040),soft(0.034),low(0.031),quiet(0.028...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>The religious sect established a {} in a rural...</td>\n",
" <td>sanctuary(0.225),monastery(0.219),temple(0.073...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>The famous reporter was fired for {} another j...</td>\n",
" <td>criticizing(0.301),copying(0.131),publishing(0...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Now that the local steel factory has closed do...</td>\n",
" <td>bustling(0.132),abandoned(0.120),struggling(0....</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>The ambassador's failure to attend the ceremon...</td>\n",
" <td>embarrassment(0.558),insult(0.365),injustice(0...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>US border guards managed to {} the escaped pri...</td>\n",
" <td>detain(0.213),apprehend(0.205),arrest(0.067),c...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Anthony enjoyed his first day at his new job. ...</td>\n",
" <td>welcoming(0.102),upbeat(0.077),positive(0.076)...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>A: I just learned I've been {} to second violi...</td>\n",
" <td>promoted(0.438),relegated(0.294),upgraded(0.08...</td>\n",
" <td>2</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 Cell phones have become a permanent {} in mode... \n",
"1 Colin did not have enough money to pay for the... \n",
"2 When she asked her boss for a raise, Melanie's... \n",
"3 The religious sect established a {} in a rural... \n",
"4 The famous reporter was fired for {} another j... \n",
"5 Now that the local steel factory has closed do... \n",
"6 The ambassador's failure to attend the ceremon... \n",
"7 US border guards managed to {} the escaped pri... \n",
"8 Anthony enjoyed his first day at his new job. ... \n",
"9 A: I just learned I've been {} to second violi... \n",
"\n",
" scores answer_position correct \n",
"0 fixture(0.932),feature(0.011),necessity(0.011)... 1 True \n",
"1 increments(0.440),excess(0.375),installments(0... 3 True \n",
"2 calm(0.040),soft(0.034),low(0.031),quiet(0.028... -1 False \n",
"3 sanctuary(0.225),monastery(0.219),temple(0.073... -1 False \n",
"4 criticizing(0.301),copying(0.131),publishing(0... -1 False \n",
"5 bustling(0.132),abandoned(0.120),struggling(0.... -1 False \n",
"6 embarrassment(0.558),insult(0.365),injustice(0... -1 False \n",
"7 detain(0.213),apprehend(0.205),arrest(0.067),c... 2 True \n",
"8 welcoming(0.102),upbeat(0.077),positive(0.076)... -1 False \n",
"9 promoted(0.438),relegated(0.294),upgraded(0.08... 2 True "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"The specified target token ` dispositions` does not exist in the model vocabulary. Replacing with `Ġdispos`.\n",
"The specified target token ` enactments` does not exist in the model vocabulary. Replacing with `Ġenact`.\n",
"The specified target token ` speculations` does not exist in the model vocabulary. Replacing with `Ġspec`.\n",
"The specified target token ` garish` does not exist in the model vocabulary. Replacing with `Ġgar`.\n",
"The specified target token ` jovial` does not exist in the model vocabulary. Replacing with `Ġj`.\n",
"The specified target token ` pompous` does not exist in the model vocabulary. Replacing with `Ġpomp`.\n",
"The specified target token ` diffident` does not exist in the model vocabulary. Replacing with `Ġdiff`.\n",
"The specified target token ` dirge` does not exist in the model vocabulary. Replacing with `Ġdir`.\n",
"The specified target token ` prelude` does not exist in the model vocabulary. Replacing with `Ġpre`.\n",
"The specified target token ` commune` does not exist in the model vocabulary. Replacing with `Ġcommun`.\n",
"The specified target token ` alleviating` does not exist in the model vocabulary. Replacing with `Ġallev`.\n",
"The specified target token ` plagiarizing` does not exist in the model vocabulary. Replacing with `Ġplagiar`.\n",
"The specified target token ` inoculating` does not exist in the model vocabulary. Replacing with `Ġinoc`.\n",
"The specified target token ` beleaguering` does not exist in the model vocabulary. Replacing with `Ġbe`.\n",
"The specified target token ` elucidation` does not exist in the model vocabulary. Replacing with `Ġeluc`.\n",
"The specified target token ` affront` does not exist in the model vocabulary. Replacing with `Ġaff`.\n",
"The specified target token ` impasse` does not exist in the model vocabulary. Replacing with `Ġimp`.\n",
"The specified target token ` ultimatum` does not exist in the model vocabulary. Replacing with `Ġult`.\n",
"The specified target token ` pillage` does not exist in the model vocabulary. Replacing with `Ġpill`.\n",
"The specified target token ` exalt` does not exist in the model vocabulary. Replacing with `Ġex`.\n",
"The specified target token ` acclimate` does not exist in the model vocabulary. Replacing with `Ġacc`.\n",
"The specified target token ` congenial` does not exist in the model vocabulary. Replacing with `Ġcongen`.\n",
"The specified target token ` delirious` does not exist in the model vocabulary. Replacing with `Ġdel`.\n",
"The specified target token ` measly` does not exist in the model vocabulary. Replacing with `Ġmeas`.\n",
"The specified target token ` implausible` does not exist in the model vocabulary. Replacing with `Ġimpl`.\n",
"The specified target token ` jeopardized` does not exist in the model vocabulary. Replacing with `Ġjeopard`.\n",
"The specified target token ` stowed` does not exist in the model vocabulary. Replacing with `Ġst`.\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Cell phones have become a permanent {} in mode...</td>\n",
" <td>fixture(0.932),rupture(0.000),stint(0.000),cla...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Colin did not have enough money to pay for the...</td>\n",
" <td>installments(0.163),spec(0.000),dispos(0.000),...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>When she asked her boss for a raise, Melanie's...</td>\n",
" <td>j(0.000),diff(0.000),gar(0.000),pomp(0.000)</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>The religious sect established a {} in a rural...</td>\n",
" <td>repository(0.000),commun(0.000),pre(0.000),dir...</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>The famous reporter was fired for {} another j...</td>\n",
" <td>plagiar(0.000),be(0.000),allev(0.000),inoc(0.000)</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Now that the local steel factory has closed do...</td>\n",
" <td>defunct(0.001),aspiring(0.000),volatile(0.000)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>The ambassador's failure to attend the ceremon...</td>\n",
" <td>imp(0.000),aff(0.000),ult(0.000),eluc(0.000)</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>US border guards managed to {} the escaped pri...</td>\n",
" <td>apprehend(0.205),pill(0.000),acc(0.000),ex(0.000)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Anthony enjoyed his first day at his new job. ...</td>\n",
" <td>congen(0.000),del(0.000),impl(0.000),meas(0.000)</td>\n",
" <td>-1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>A: I just learned I've been {} to second violi...</td>\n",
" <td>relegated(0.294),reiterated(0.000),st(0.000),j...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 Cell phones have become a permanent {} in mode... \n",
"1 Colin did not have enough money to pay for the... \n",
"2 When she asked her boss for a raise, Melanie's... \n",
"3 The religious sect established a {} in a rural... \n",
"4 The famous reporter was fired for {} another j... \n",
"5 Now that the local steel factory has closed do... \n",
"6 The ambassador's failure to attend the ceremon... \n",
"7 US border guards managed to {} the escaped pri... \n",
"8 Anthony enjoyed his first day at his new job. ... \n",
"9 A: I just learned I've been {} to second violi... \n",
"\n",
" scores answer_position correct \n",
"0 fixture(0.932),rupture(0.000),stint(0.000),cla... 1 True \n",
"1 installments(0.163),spec(0.000),dispos(0.000),... 1 True \n",
"2 j(0.000),diff(0.000),gar(0.000),pomp(0.000) -1 False \n",
"3 repository(0.000),commun(0.000),pre(0.000),dir... -1 False \n",
"4 plagiar(0.000),be(0.000),allev(0.000),inoc(0.000) -1 False \n",
"5 defunct(0.001),aspiring(0.000),volatile(0.000)... 1 True \n",
"6 imp(0.000),aff(0.000),ult(0.000),eluc(0.000) -1 False \n",
"7 apprehend(0.205),pill(0.000),acc(0.000),ex(0.000) 1 True \n",
"8 congen(0.000),del(0.000),impl(0.000),meas(0.000) -1 False \n",
"9 relegated(0.294),reiterated(0.000),st(0.000),j... 1 True "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Eiken grade 1\n",
"# source: https://www.eiken.or.jp/eiken/exam/grade_1/pdf/202101/2021-1-1ji-1kyu.pdf\n",
"\n",
"eiken1 = [\n",
" Problem(\"Cell phones have become a permanent {} in modern society. Most perople could not imagine living without one.\",\n",
" [\"clasp\", \"stint\", \"fixture\", \"rupture\"], \"fixture\")\n",
" ,Problem(\"Colin did not have enough money to pay for the car all at onece, so he paid it off in {} of $800 a month for two years.\",\n",
" [\"dispositions\", \"installments\", \"enactments\", \"speculations\"], \"installments\")\n",
" ,Problem(\"When she asked her boss for a raise, Melanie's {} tone of voice made it obvious how nervous she was.\",\n",
" [\"garish\", \"jovial\", \"pompous\", \"diffident\"], \"diffident\")\n",
" ,Problem(\"The religious sect established a {} in a rural area where its followers could live together and share everything. No private property was allowed.\",\n",
" [\"dirge\", \"prelude\", \"repository\", \"commune\"], \"commune\")\n",
" ,Problem(\"The famous reporter was fired for {} another journalist's work. His article was almost exactly the same as that of the other journalist.\",\n",
" [\"alleviating\", \"plagiarizing\", \"inoculating\", \"beleaguering\"], \"plagiarizing\")\n",
" ,Problem(\"Now that the local steel factory has closed down, the streets of the once-busy town are lined with {} businesses. Most owners have abandoned their stores.\",\n",
" [\"rhetorical\", \"volatile\", \"defunct\", \"aspiring\"], \"defunct\")\n",
" ,Problem(\"The ambassador's failure to attend the ceremony held in honor of the king was considered an {} by his host nation and made already bad relations worse.\",\n",
" [\"elucidation\", \"affront\", \"impasse\", \"ultimatum\"], \"affront\")\n",
" ,Problem(\"US border guards managed to {} the escaped prisoner as he tried to cross into Canada. He was returned to jail immediately.\",\n",
" [\"apprehend\", \"pillage\", \"exalt\", \"acclimate\"], \"apprehend\")\n",
" ,Problem(\"Anthony enjoyed his first day at his new job. The atmosphere was {}, and his colleagues did their best to make him feel welcome.\",\n",
" [\"congenial\", \"delirious\", \"measly\", \"implausible\"], \"congenial\")\n",
" ,Problem((\"A: I just learned I've been {} to second violin in the school orchestra. I knew I should've practiced more.\"\n",
" \"B: Well, if you work hard, I'm sure you can get your previous position back.\"),\n",
" [\"relegated\", \"jeopardized\", \"reiterated\", \"stowed\"], \"relegated\")\n",
"]\n",
"\n",
"o = solve_without_choices(eiken1)\n",
"display(o)\n",
"o[\"grade\"] = \"1\"\n",
"o[\"method\"] = \"fill-mask-no-target\"\n",
"out = pd.concat((out, o))\n",
"\n",
"o = solve_with_choices(eiken1)\n",
"display(o)\n",
"o[\"grade\"] = \"1\"\n",
"o[\"method\"] = \"fill-mask-with-targets\"\n",
"out = pd.concat((out, o))"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "888a1932",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:33:31.743823Z",
"iopub.status.busy": "2021-08-01T13:33:31.743174Z",
"iopub.status.idle": "2021-08-01T13:33:31.747417Z",
"shell.execute_reply": "2021-08-01T13:33:31.746984Z",
"shell.execute_reply.started": "2021-08-01T13:20:46.390158Z"
},
"id": "eTkcZuJwb1Yc",
"papermill": {
"duration": 0.055705,
"end_time": "2021-08-01T13:33:31.747575",
"exception": false,
"start_time": "2021-08-01T13:33:31.691870",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"# As the grade gets higher, we are more likely to encounter \"unknown\" words (words not included in the vocabulary)\n",
"# and failed to make a guess for them.\n",
"#\n",
"# To overcome this, we will next employs a language model that calculates the sentence perplexity scores."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "97bc7f9f",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:33:31.846486Z",
"iopub.status.busy": "2021-08-01T13:33:31.845504Z",
"iopub.status.idle": "2021-08-01T13:35:56.830789Z",
"shell.execute_reply": "2021-08-01T13:35:56.830255Z",
"shell.execute_reply.started": "2021-08-01T13:20:46.397650Z"
},
"id": "iX3OpE_YYEg0",
"papermill": {
"duration": 145.038274,
"end_time": "2021-08-01T13:35:56.830932",
"exception": false,
"start_time": "2021-08-01T13:33:31.792658",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "09c09671ab5848b0b74a73b8271f1de0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading: 0%| | 0.00/764 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "78788e3ddd0143ae8d7ca09c9a392187",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading: 0%| | 0.00/3.25G [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2f54cef3042d48aca2f70ba2238b468f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading: 0%| | 0.00/1.04M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a542df8d3d1b4a2db71900ce7e956abc",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading: 0%| | 0.00/456k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "97a759dd64c249ea9bbe89d76e36b64e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading: 0%| | 0.00/1.36M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Reference: https://huggingface.co/transformers/perplexity.html\n",
"# https://discuss.huggingface.co/t/gpt-2-perplexity-score-normalized-on-sentence-lenght/5205\n",
"import torch\n",
"from transformers import GPT2LMHeadModel, GPT2TokenizerFast\n",
"\n",
"device = \"cuda\"\n",
"model_id = \"gpt2-large\"\n",
"\n",
"model2 = GPT2LMHeadModel.from_pretrained(model_id).to(device)\n",
"tokenizer = GPT2TokenizerFast.from_pretrained(model_id)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "24d09bce",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:35:56.930787Z",
"iopub.status.busy": "2021-08-01T13:35:56.929915Z",
"iopub.status.idle": "2021-08-01T13:35:56.932405Z",
"shell.execute_reply": "2021-08-01T13:35:56.932009Z",
"shell.execute_reply.started": "2021-08-01T13:23:14.044456Z"
},
"id": "BMDS42_-tRif",
"papermill": {
"duration": 0.055181,
"end_time": "2021-08-01T13:35:56.932530",
"exception": false,
"start_time": "2021-08-01T13:35:56.877349",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"def solve_with_choices2(problems):\n",
" out = []\n",
" for p in problems:\n",
" texts = [p.text.format(c) for c in p.choices]\n",
" res = []\n",
" for t in texts:\n",
" tmp = tokenizer(t, return_tensors='pt')\n",
" input_ids = tmp.input_ids.to(device)\n",
"\n",
" with torch.no_grad():\n",
" res.append(model2(input_ids, labels=input_ids)[0].item())\n",
"\n",
" res = list(zip(p.choices, res))\n",
" res.sort(key=lambda a: a[1])\n",
" scores = \",\".join(\"%s(%.3f)\" % a for a in res)\n",
" answer_position = [s[0] for s in res].index(p.answer) + 1\n",
"\n",
" out.append((p.text, scores, answer_position))\n",
" out = pd.DataFrame(out, columns=[\"problem\", \"scores\", \"answer_position\"])\n",
" out[\"correct\"] = (out.answer_position==1)\n",
" return out"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "b1c5dfb9",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:35:57.039204Z",
"iopub.status.busy": "2021-08-01T13:35:57.038530Z",
"iopub.status.idle": "2021-08-01T13:36:08.496760Z",
"shell.execute_reply": "2021-08-01T13:36:08.497152Z",
"shell.execute_reply.started": "2021-08-01T13:23:14.056259Z"
},
"id": "tBBYMUQ4uEAM",
"outputId": "512629aa-caa1-47fc-f50d-1c289c12102f",
"papermill": {
"duration": 11.518103,
"end_time": "2021-08-01T13:36:08.497315",
"exception": false,
"start_time": "2021-08-01T13:35:56.979212",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A: What is your {}? B: Kazumi Suzuki.</td>\n",
" <td>name(4.421),date(5.132),club(5.409),hour(5.515)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>I know Judy. She can {} French very well.</td>\n",
" <td>speak(3.975),see(4.961),drink(5.339),open(5.466)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>A: Are your baseball shoes in your room, Mike?...</td>\n",
" <td>locker(3.825),shop(4.090),door(4.124),window(4...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Mysister usually plays tennis {} Saturdays.</td>\n",
" <td>on(6.950),at(7.842),by(8.166),with(8.233)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>My mother likes {}. She has many pretty ones i...</td>\n",
" <td>flowers(3.275),movies(4.204),sports(4.250),sch...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Let's begin today's class. Open your textbooks...</td>\n",
" <td>page(3.568),minute(4.388),chalk(4.695),ground(...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Today is Wednesday. Tomorrow is {}.</td>\n",
" <td>Thursday(2.731),Friday(2.820),Tuesday(3.075),M...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>I usually read magazines {} home.</td>\n",
" <td>at(4.812),on(6.230),of(6.704),with(6.817)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>A: It's ten o'clock, Jimmy. {} to bed. B: All...</td>\n",
" <td>Go(3.637),Sleep(4.108),Do(4.159),Sit(4.297)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>A: Do you live {} Tokyo? B: Yes. It's a big c...</td>\n",
" <td>in(3.254),on(3.652),with(3.861),after(3.912)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 A: What is your {}? B: Kazumi Suzuki. \n",
"1 I know Judy. She can {} French very well. \n",
"2 A: Are your baseball shoes in your room, Mike?... \n",
"3 Mysister usually plays tennis {} Saturdays. \n",
"4 My mother likes {}. She has many pretty ones i... \n",
"5 Let's begin today's class. Open your textbooks... \n",
"6 Today is Wednesday. Tomorrow is {}. \n",
"7 I usually read magazines {} home. \n",
"8 A: It's ten o'clock, Jimmy. {} to bed. B: All... \n",
"9 A: Do you live {} Tokyo? B: Yes. It's a big c... \n",
"\n",
" scores answer_position correct \n",
"0 name(4.421),date(5.132),club(5.409),hour(5.515) 1 True \n",
"1 speak(3.975),see(4.961),drink(5.339),open(5.466) 1 True \n",
"2 locker(3.825),shop(4.090),door(4.124),window(4... 1 True \n",
"3 on(6.950),at(7.842),by(8.166),with(8.233) 1 True \n",
"4 flowers(3.275),movies(4.204),sports(4.250),sch... 1 True \n",
"5 page(3.568),minute(4.388),chalk(4.695),ground(... 1 True \n",
"6 Thursday(2.731),Friday(2.820),Tuesday(3.075),M... 1 True \n",
"7 at(4.812),on(6.230),of(6.704),with(6.817) 1 True \n",
"8 Go(3.637),Sleep(4.108),Do(4.159),Sit(4.297) 1 True \n",
"9 in(3.254),on(3.652),with(3.861),after(3.912) 1 True "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>My father is a {} of a sports club. He plays t...</td>\n",
" <td>member(3.035),group(4.094),festival(4.196),pic...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Mr. Clark told us many intesting {} about his ...</td>\n",
" <td>stories(4.840),pictures(5.429),magazines(5.593...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>It's snowing a lot today, so please drive {}.</td>\n",
" <td>slowly(3.082),busily(3.700),coldly(3.714),free...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>In spring, Jane likes to walk in her grandmoth...</td>\n",
" <td>garden(3.513),stone(4.185),wall(4.230),sky(4.275)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Many girls in my class have {} hair.</td>\n",
" <td>short(3.469),late(4.467),slow(4.653),busy(5.382)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>A: Do you live in a city? B: No. I live in a ...</td>\n",
" <td>town(2.939),hobby(3.440),holiday(3.561),ticket...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>I {} Nancy's notebook. It was on Mary's desk</td>\n",
" <td>found(4.157),stopped(4.780),went(4.977),stayed...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Dennis went to Japan for a year in August. He ...</td>\n",
" <td>said(3.226),told(3.726),ended(3.983),hoped(3.999)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Jeff left the party at 8:00. He wanted to {} h...</td>\n",
" <td>get(2.797),meet(3.353),send(3.444),put(3.580)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Mom's lemon cake is not as good {} her chocola...</td>\n",
" <td>as(3.458),of(4.667),by(4.727),to(4.805)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 My father is a {} of a sports club. He plays t... \n",
"1 Mr. Clark told us many intesting {} about his ... \n",
"2 It's snowing a lot today, so please drive {}. \n",
"3 In spring, Jane likes to walk in her grandmoth... \n",
"4 Many girls in my class have {} hair. \n",
"5 A: Do you live in a city? B: No. I live in a ... \n",
"6 I {} Nancy's notebook. It was on Mary's desk \n",
"7 Dennis went to Japan for a year in August. He ... \n",
"8 Jeff left the party at 8:00. He wanted to {} h... \n",
"9 Mom's lemon cake is not as good {} her chocola... \n",
"\n",
" scores answer_position correct \n",
"0 member(3.035),group(4.094),festival(4.196),pic... 1 True \n",
"1 stories(4.840),pictures(5.429),magazines(5.593... 1 True \n",
"2 slowly(3.082),busily(3.700),coldly(3.714),free... 1 True \n",
"3 garden(3.513),stone(4.185),wall(4.230),sky(4.275) 1 True \n",
"4 short(3.469),late(4.467),slow(4.653),busy(5.382) 1 True \n",
"5 town(2.939),hobby(3.440),holiday(3.561),ticket... 1 True \n",
"6 found(4.157),stopped(4.780),went(4.977),stayed... 1 True \n",
"7 said(3.226),told(3.726),ended(3.983),hoped(3.999) 1 True \n",
"8 get(2.797),meet(3.353),send(3.444),put(3.580) 1 True \n",
"9 as(3.458),of(4.667),by(4.727),to(4.805) 1 True "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A: How do you make that potato dish? B: First...</td>\n",
" <td>boil(3.264),eat(3.363),hurt(3.521),care(3.718)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Last summer, Hiroshi's family traveled around ...</td>\n",
" <td>abroad(4.146),inside(4.269),other(4.620),simil...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Bob {} five friends to his party</td>\n",
" <td>invited(5.150),made(6.476),spoke(6.545),visite...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>A: John, you should go to bed soon. If you sta...</td>\n",
" <td>oversleep(2.627),return(2.889),graduate(3.012)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>A: Did you buy your father something special f...</td>\n",
" <td>apron(3.104),ring(3.225),field(3.478),contact(...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>I bought a new T-shirt for my brother, but I b...</td>\n",
" <td>tight(3.286),heavy(3.405),bright(3.512),clear(...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Sarah saw some flowers by the road while she w...</td>\n",
" <td>picked(2.792),guessed(3.328),spent(3.331),wish...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Jenny saw her grandparents {} the first time i...</td>\n",
" <td>for(2.761),over(3.653),out(3.665),from(3.704)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>A: I told my mother that I would be home by 7:...</td>\n",
" <td>break(2.799),pass(3.114),sell(3.139),lend(3.226)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>A: Don'y say anything to Dad about the surpris...</td>\n",
" <td>out(3.642),through(4.512),near(4.575),within(4...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 A: How do you make that potato dish? B: First... \n",
"1 Last summer, Hiroshi's family traveled around ... \n",
"2 Bob {} five friends to his party \n",
"3 A: John, you should go to bed soon. If you sta... \n",
"4 A: Did you buy your father something special f... \n",
"5 I bought a new T-shirt for my brother, but I b... \n",
"6 Sarah saw some flowers by the road while she w... \n",
"7 Jenny saw her grandparents {} the first time i... \n",
"8 A: I told my mother that I would be home by 7:... \n",
"9 A: Don'y say anything to Dad about the surpris... \n",
"\n",
" scores answer_position correct \n",
"0 boil(3.264),eat(3.363),hurt(3.521),care(3.718) 1 True \n",
"1 abroad(4.146),inside(4.269),other(4.620),simil... 1 True \n",
"2 invited(5.150),made(6.476),spoke(6.545),visite... 1 True \n",
"3 oversleep(2.627),return(2.889),graduate(3.012)... 1 True \n",
"4 apron(3.104),ring(3.225),field(3.478),contact(... 1 True \n",
"5 tight(3.286),heavy(3.405),bright(3.512),clear(... 1 True \n",
"6 picked(2.792),guessed(3.328),spent(3.331),wish... 1 True \n",
"7 for(2.761),over(3.653),out(3.665),from(3.704) 1 True \n",
"8 break(2.799),pass(3.114),sell(3.139),lend(3.226) 1 True \n",
"9 out(3.642),through(4.512),near(4.575),within(4... 1 True "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Jamie visited several {} buildings when he wen...</td>\n",
" <td>ancient(3.093),exact(3.583),responsible(3.771)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Sally's French teacher told her to read an art...</td>\n",
" <td>translate(3.196),guide(3.870),throw(4.024),con...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Henry likes living in the city because there a...</td>\n",
" <td>countryside(2.423),experiment(2.904),image(2.9...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>A: Is it true that the things in this store on...</td>\n",
" <td>tax(3.296),data(3.545),total(3.574),waste(3.582)</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>When the bust was an hour late, one man shoute...</td>\n",
" <td>angrily(3.245),secretly(3.659),tightly(3.678),...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Firefighters have to {} people from buildings ...</td>\n",
" <td>rescue(2.607),produce(3.223),weigh(3.473),stam...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>John loves the singer Ann May, and he cannot w...</td>\n",
" <td>released(3.660),invented(4.074),divided(4.186)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>The news that Ms. Kelly, the art teacher, was ...</td>\n",
" <td>spread(3.004),stretched(3.238),served(3.492),s...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>A: I'm really nervous about acting in the play...</td>\n",
" <td>stage(3.899),screen(4.051),field(4.055),court(...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Before Diane attended Professor Miller's {} at...</td>\n",
" <td>lecture(3.845),comment(4.133),furniture(4.239)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 Jamie visited several {} buildings when he wen... \n",
"1 Sally's French teacher told her to read an art... \n",
"2 Henry likes living in the city because there a... \n",
"3 A: Is it true that the things in this store on... \n",
"4 When the bust was an hour late, one man shoute... \n",
"5 Firefighters have to {} people from buildings ... \n",
"6 John loves the singer Ann May, and he cannot w... \n",
"7 The news that Ms. Kelly, the art teacher, was ... \n",
"8 A: I'm really nervous about acting in the play... \n",
"9 Before Diane attended Professor Miller's {} at... \n",
"\n",
" scores answer_position correct \n",
"0 ancient(3.093),exact(3.583),responsible(3.771)... 1 True \n",
"1 translate(3.196),guide(3.870),throw(4.024),con... 1 True \n",
"2 countryside(2.423),experiment(2.904),image(2.9... 1 True \n",
"3 tax(3.296),data(3.545),total(3.574),waste(3.582) 1 True \n",
"4 angrily(3.245),secretly(3.659),tightly(3.678),... 1 True \n",
"5 rescue(2.607),produce(3.223),weigh(3.473),stam... 1 True \n",
"6 released(3.660),invented(4.074),divided(4.186)... 1 True \n",
"7 spread(3.004),stretched(3.238),served(3.492),s... 1 True \n",
"8 stage(3.899),screen(4.051),field(4.055),court(... 1 True \n",
"9 lecture(3.845),comment(4.133),furniture(4.239)... 1 True "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>At first, the marketing department and the sal...</td>\n",
" <td>entirely(2.498),needlessly(2.740),violently(2....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Experts at the art gallery discovered that one...</td>\n",
" <td>genuine(3.079),portable(3.520),severe(3.677),l...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>The musician Jimmy Baker had a lot of {} when ...</td>\n",
" <td>hardship(3.326),concentration(3.565),membershi...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Mother Teresa helped many sick people and gave...</td>\n",
" <td>humanity(2.718),generation(3.180),gravity(3.18...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>As Liam waled down the dark street, he began t...</td>\n",
" <td>sensation(3.421),feature(4.111),translation(4....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Risa buys water that comes from a mountain str...</td>\n",
" <td>minerals(3.101),operations(3.727),illustration...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>The lifeguard ran into the ocean to help a you...</td>\n",
" <td>drowning(3.241),proposing(4.074),converting(4....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Yesterday was a hot day at the zoo, so Heather...</td>\n",
" <td>spilling(3.160),maintaining(3.649),arguing(3.8...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>In the past, sailors had to use the stars to {...</td>\n",
" <td>navigate(3.120),respect(3.493),satisfy(3.554),...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Daisuke's grandmother eats a lot of vegetables...</td>\n",
" <td>preserve(3.041),replace(3.377),betray(3.537),i...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 At first, the marketing department and the sal... \n",
"1 Experts at the art gallery discovered that one... \n",
"2 The musician Jimmy Baker had a lot of {} when ... \n",
"3 Mother Teresa helped many sick people and gave... \n",
"4 As Liam waled down the dark street, he began t... \n",
"5 Risa buys water that comes from a mountain str... \n",
"6 The lifeguard ran into the ocean to help a you... \n",
"7 Yesterday was a hot day at the zoo, so Heather... \n",
"8 In the past, sailors had to use the stars to {... \n",
"9 Daisuke's grandmother eats a lot of vegetables... \n",
"\n",
" scores answer_position correct \n",
"0 entirely(2.498),needlessly(2.740),violently(2.... 1 True \n",
"1 genuine(3.079),portable(3.520),severe(3.677),l... 1 True \n",
"2 hardship(3.326),concentration(3.565),membershi... 1 True \n",
"3 humanity(2.718),generation(3.180),gravity(3.18... 1 True \n",
"4 sensation(3.421),feature(4.111),translation(4.... 1 True \n",
"5 minerals(3.101),operations(3.727),illustration... 1 True \n",
"6 drowning(3.241),proposing(4.074),converting(4.... 1 True \n",
"7 spilling(3.160),maintaining(3.649),arguing(3.8... 1 True \n",
"8 navigate(3.120),respect(3.493),satisfy(3.554),... 1 True \n",
"9 preserve(3.041),replace(3.377),betray(3.537),i... 1 True "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>A: Thanks for showing me the outline of your s...</td>\n",
" <td>redundant(3.329),subjective(3.454),distinct(3....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Lisa went to the interview even though she tho...</td>\n",
" <td>probability(2.479),credibility(3.207),contenti...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>It is sadly {} that, in developing counties, m...</td>\n",
" <td>ironic(3.586),superficial(4.055),indefinite(4....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>The explosion at the chemical factory {} great...</td>\n",
" <td>inflicted(3.238),perceived(3.704),enhanced(3.8...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Some say the best way to overcome a {} is to e...</td>\n",
" <td>phobia(2.733),temptation(3.107),barricade(3.29...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>English classes at the university were require...</td>\n",
" <td>exempted(3.016),qualified(3.583),prosecuted(3....</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>E-mail and text messaging have {} the way peop...</td>\n",
" <td>transformed(4.060),synthesized(4.406),disarmed...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Some analysts think the new treaty on CO2 emis...</td>\n",
" <td>milestone(2.537),vigor(2.962),backlog(3.035),c...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Lying on the sunny beach with her husband on t...</td>\n",
" <td>profoundly(3.576),barely(3.864),harshly(4.001)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Nadine spends an hour thoroughly cleaning her ...</td>\n",
" <td>spotless(3.993),rugged(4.641),impartial(4.788)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 A: Thanks for showing me the outline of your s... \n",
"1 Lisa went to the interview even though she tho... \n",
"2 It is sadly {} that, in developing counties, m... \n",
"3 The explosion at the chemical factory {} great... \n",
"4 Some say the best way to overcome a {} is to e... \n",
"5 English classes at the university were require... \n",
"6 E-mail and text messaging have {} the way peop... \n",
"7 Some analysts think the new treaty on CO2 emis... \n",
"8 Lying on the sunny beach with her husband on t... \n",
"9 Nadine spends an hour thoroughly cleaning her ... \n",
"\n",
" scores answer_position correct \n",
"0 redundant(3.329),subjective(3.454),distinct(3.... 1 True \n",
"1 probability(2.479),credibility(3.207),contenti... 1 True \n",
"2 ironic(3.586),superficial(4.055),indefinite(4.... 1 True \n",
"3 inflicted(3.238),perceived(3.704),enhanced(3.8... 1 True \n",
"4 phobia(2.733),temptation(3.107),barricade(3.29... 1 True \n",
"5 exempted(3.016),qualified(3.583),prosecuted(3.... 1 True \n",
"6 transformed(4.060),synthesized(4.406),disarmed... 1 True \n",
"7 milestone(2.537),vigor(2.962),backlog(3.035),c... 1 True \n",
"8 profoundly(3.576),barely(3.864),harshly(4.001)... 1 True \n",
"9 spotless(3.993),rugged(4.641),impartial(4.788)... 1 True "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>problem</th>\n",
" <th>scores</th>\n",
" <th>answer_position</th>\n",
" <th>correct</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Cell phones have become a permanent {} in mode...</td>\n",
" <td>fixture(3.340),rupture(4.229),clasp(4.242),sti...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Colin did not have enough money to pay for the...</td>\n",
" <td>installments(3.012),dispositions(3.537),specul...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>When she asked her boss for a raise, Melanie's...</td>\n",
" <td>jovial(3.002),diffident(3.208),pompous(3.282),...</td>\n",
" <td>2</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>The religious sect established a {} in a rural...</td>\n",
" <td>commune(3.466),repository(3.890),prelude(4.160...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>The famous reporter was fired for {} another j...</td>\n",
" <td>plagiarizing(2.819),beleaguering(3.335),inocul...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Now that the local steel factory has closed do...</td>\n",
" <td>defunct(3.112),volatile(3.456),aspiring(3.502)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>The ambassador's failure to attend the ceremon...</td>\n",
" <td>affront(3.527),ultimatum(3.723),impasse(3.901)...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>US border guards managed to {} the escaped pri...</td>\n",
" <td>apprehend(3.144),acclimate(3.557),pillage(3.58...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Anthony enjoyed his first day at his new job. ...</td>\n",
" <td>congenial(2.469),delirious(2.613),measly(2.855...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>A: I just learned I've been {} to second violi...</td>\n",
" <td>relegated(3.020),jeopardized(3.433),reiterated...</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" problem \\\n",
"0 Cell phones have become a permanent {} in mode... \n",
"1 Colin did not have enough money to pay for the... \n",
"2 When she asked her boss for a raise, Melanie's... \n",
"3 The religious sect established a {} in a rural... \n",
"4 The famous reporter was fired for {} another j... \n",
"5 Now that the local steel factory has closed do... \n",
"6 The ambassador's failure to attend the ceremon... \n",
"7 US border guards managed to {} the escaped pri... \n",
"8 Anthony enjoyed his first day at his new job. ... \n",
"9 A: I just learned I've been {} to second violi... \n",
"\n",
" scores answer_position correct \n",
"0 fixture(3.340),rupture(4.229),clasp(4.242),sti... 1 True \n",
"1 installments(3.012),dispositions(3.537),specul... 1 True \n",
"2 jovial(3.002),diffident(3.208),pompous(3.282),... 2 False \n",
"3 commune(3.466),repository(3.890),prelude(4.160... 1 True \n",
"4 plagiarizing(2.819),beleaguering(3.335),inocul... 1 True \n",
"5 defunct(3.112),volatile(3.456),aspiring(3.502)... 1 True \n",
"6 affront(3.527),ultimatum(3.723),impasse(3.901)... 1 True \n",
"7 apprehend(3.144),acclimate(3.557),pillage(3.58... 1 True \n",
"8 congenial(2.469),delirious(2.613),measly(2.855... 1 True \n",
"9 relegated(3.020),jeopardized(3.433),reiterated... 1 True "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"o = solve_with_choices2(eiken5)\n",
"display(o)\n",
"o[\"grade\"] = \"5\"\n",
"o[\"method\"] = \"perplexity\"\n",
"out = pd.concat((out, o))\n",
"\n",
"o = solve_with_choices2(eiken4)\n",
"display(o)\n",
"o[\"grade\"] = \"4\"\n",
"o[\"method\"] = \"perplexity\"\n",
"out = pd.concat((out, o))\n",
"\n",
"o = solve_with_choices2(eiken3)\n",
"display(o)\n",
"o[\"grade\"] = \"3\"\n",
"o[\"method\"] = \"perplexity\"\n",
"out = pd.concat((out, o))\n",
"\n",
"o = solve_with_choices2(eikenp2)\n",
"display(o)\n",
"o[\"grade\"] = \"pre2\"\n",
"o[\"method\"] = \"perplexity\"\n",
"out = pd.concat((out, o))\n",
"\n",
"o = solve_with_choices2(eiken2)\n",
"display(o)\n",
"o[\"grade\"] = \"2\"\n",
"o[\"method\"] = \"perplexity\"\n",
"out = pd.concat((out, o))\n",
"\n",
"o = solve_with_choices2(eikenp1)\n",
"display(o)\n",
"o[\"grade\"] = \"pre1\"\n",
"o[\"method\"] = \"perplexity\"\n",
"out = pd.concat((out, o))\n",
"\n",
"o = solve_with_choices2(eiken1)\n",
"display(o)\n",
"o[\"grade\"] = \"1\"\n",
"o[\"method\"] = \"perplexity\"\n",
"out = pd.concat((out, o))"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "9e6d078b",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:36:08.605134Z",
"iopub.status.busy": "2021-08-01T13:36:08.604485Z",
"iopub.status.idle": "2021-08-01T13:36:08.608030Z",
"shell.execute_reply": "2021-08-01T13:36:08.608427Z",
"shell.execute_reply.started": "2021-08-01T13:23:24.282304Z"
},
"papermill": {
"duration": 0.057629,
"end_time": "2021-08-01T13:36:08.608638",
"exception": false,
"start_time": "2021-08-01T13:36:08.551009",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"# The model got a full-mark except just one:\n",
"#\n",
"# When she asked her boss for a raise, Melanie's {} tone of voice made it obvious how nervous she was.\",\n",
"# \"garish\", \"jovial\", \"pompous\", \"diffident\"\n",
"#\n",
"# The correct answer is \"diffident\", meaning lack of self-confidence. This matches the fact that she was nervous.\n",
"# The BERT's guess is \"jovial\", meaning friendly. This many naturally connect to \"tone of voice\", but misses the following context."
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "de2c607e",
"metadata": {
"execution": {
"iopub.execute_input": "2021-08-01T13:36:08.731466Z",
"iopub.status.busy": "2021-08-01T13:36:08.730760Z",
"iopub.status.idle": "2021-08-01T13:36:09.267053Z",
"shell.execute_reply": "2021-08-01T13:36:09.266086Z"
},
"id": "G_dW3OCVXspa",
"papermill": {
"duration": 0.602792,
"end_time": "2021-08-01T13:36:09.267203",
"exception": false,
"start_time": "2021-08-01T13:36:08.664411",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>grade</th>\n",
" <th>problem_number</th>\n",
" <th>problem</th>\n",
" <th>scores(fill-mask-no-target)</th>\n",
" <th>correct(fill-mask-no-target)</th>\n",
" <th>scores(fill-mask-with-targets)</th>\n",
" <th>correct(fill-mask-with-targets)</th>\n",
" <th>scores(perplexity)</th>\n",
" <th>correct(perplexity)</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>5</td>\n",
" <td>1</td>\n",
" <td>A: What is your {}? B: Kazumi Suzuki.</td>\n",
" <td>name(0.245),nickname(0.101),surname(0.058),ans...</td>\n",
" <td>True</td>\n",
" <td>name(0.245),date(0.002),club(0.000),hour(0.000)</td>\n",
" <td>True</td>\n",
" <td>name(4.421),date(5.132),club(5.409),hour(5.515)</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>5</td>\n",
" <td>2</td>\n",
" <td>I know Judy. She can {} French very well.</td>\n",
" <td>speak(0.858),read(0.053),learn(0.012),teach(0....</td>\n",
" <td>True</td>\n",
" <td>speak(0.858),drink(0.000),see(0.000),open(0.000)</td>\n",
" <td>True</td>\n",
" <td>speak(3.975),see(4.961),drink(5.339),open(5.466)</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>5</td>\n",
" <td>3</td>\n",
" <td>A: Are your baseball shoes in your room, Mike?...</td>\n",
" <td>room(0.408),locker(0.132),classroom(0.107),clo...</td>\n",
" <td>True</td>\n",
" <td>locker(0.132),window(0.000),door(0.000),shop(0...</td>\n",
" <td>True</td>\n",
" <td>locker(3.825),shop(4.090),door(4.124),window(4...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>5</td>\n",
" <td>4</td>\n",
" <td>Mysister usually plays tennis {} Saturdays.</td>\n",
" <td>on(0.953),every(0.021),through(0.007),until(0....</td>\n",
" <td>True</td>\n",
" <td>on(0.953),at(0.001),with(0.000),by(0.000)</td>\n",
" <td>True</td>\n",
" <td>on(6.950),at(7.842),by(8.166),with(8.233)</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>5</td>\n",
" <td>5</td>\n",
" <td>My mother likes {}. She has many pretty ones i...</td>\n",
" <td>roses(0.326),flowers(0.147),strawberries(0.070...</td>\n",
" <td>True</td>\n",
" <td>flowers(0.147),movies(0.000),sports(0.000),sch...</td>\n",
" <td>True</td>\n",
" <td>flowers(3.275),movies(4.204),sports(4.250),sch...</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" grade problem_number problem \\\n",
"0 5 1 A: What is your {}? B: Kazumi Suzuki. \n",
"1 5 2 I know Judy. She can {} French very well. \n",
"2 5 3 A: Are your baseball shoes in your room, Mike?... \n",
"3 5 4 Mysister usually plays tennis {} Saturdays. \n",
"4 5 5 My mother likes {}. She has many pretty ones i... \n",
"\n",
" scores(fill-mask-no-target) \\\n",
"0 name(0.245),nickname(0.101),surname(0.058),ans... \n",
"1 speak(0.858),read(0.053),learn(0.012),teach(0.... \n",
"2 room(0.408),locker(0.132),classroom(0.107),clo... \n",
"3 on(0.953),every(0.021),through(0.007),until(0.... \n",
"4 roses(0.326),flowers(0.147),strawberries(0.070... \n",
"\n",
" correct(fill-mask-no-target) \\\n",
"0 True \n",
"1 True \n",
"2 True \n",
"3 True \n",
"4 True \n",
"\n",
" scores(fill-mask-with-targets) \\\n",
"0 name(0.245),date(0.002),club(0.000),hour(0.000) \n",
"1 speak(0.858),drink(0.000),see(0.000),open(0.000) \n",
"2 locker(0.132),window(0.000),door(0.000),shop(0... \n",
"3 on(0.953),at(0.001),with(0.000),by(0.000) \n",
"4 flowers(0.147),movies(0.000),sports(0.000),sch... \n",
"\n",
" correct(fill-mask-with-targets) \\\n",
"0 True \n",
"1 True \n",
"2 True \n",
"3 True \n",
"4 True \n",
"\n",
" scores(perplexity) correct(perplexity) \n",
"0 name(4.421),date(5.132),club(5.409),hour(5.515) True \n",
"1 speak(3.975),see(4.961),drink(5.339),open(5.466) True \n",
"2 locker(3.825),shop(4.090),door(4.124),window(4... True \n",
"3 on(6.950),at(7.842),by(8.166),with(8.233) True \n",
"4 flowers(3.275),movies(4.204),sports(4.250),sch... True "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>method</th>\n",
" <th>fill-mask-no-target</th>\n",
" <th>fill-mask-with-targets</th>\n",
" <th>perplexity</th>\n",
" </tr>\n",
" <tr>\n",
" <th>grade</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4</td>\n",
" <td>5</td>\n",
" <td>9</td>\n",
" </tr>\n",
" <tr>\n",
" <th>pre1</th>\n",
" <td>6</td>\n",
" <td>8</td>\n",
" <td>10</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>8</td>\n",
" <td>10</td>\n",
" <td>10</td>\n",
" </tr>\n",
" <tr>\n",
" <th>pre2</th>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" <td>10</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>7</td>\n",
" <td>8</td>\n",
" <td>10</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>8</td>\n",
" <td>10</td>\n",
" <td>10</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>10</td>\n",
" <td>9</td>\n",
" <td>10</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"method fill-mask-no-target fill-mask-with-targets perplexity\n",
"grade \n",
"1 4 5 9\n",
"pre1 6 8 10\n",
"2 8 10 10\n",
"pre2 9 10 10\n",
"3 7 8 10\n",
"4 8 10 10\n",
"5 10 9 10"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# compile output into human-friendly excel file\n",
"out2 = None\n",
"for method in out.method.unique():\n",
" tmp = out[out.method==method].copy().reset_index(drop=True)\n",
" tmp[\"problem_number\"] = list(range(1,11)) * int(len(tmp)/10)\n",
" tmp = tmp[[\"grade\", \"problem_number\", \"problem\", \"scores\", \"correct\"]]\n",
" tmp = tmp.rename(columns={\"scores\": \"scores({})\".format(method),\n",
" \"correct\": \"correct({})\".format(method)})\n",
" if out2 is None:\n",
" out2 = tmp\n",
" else:\n",
" out2 = pd.merge(out2, tmp)\n",
"display(out2.head())\n",
"\n",
"out3 = out.pivot_table(index=\"grade\", columns=\"method\", values=\"correct\", aggfunc=sum)\n",
"out3[\"grade2\"] = out3.index.tolist()\n",
"out3.grade2 = out3.grade2.replace(\"pre2\", 2.5).replace(\"pre1\", 1.5).astype(float)\n",
"out3 = out3.sort_values(\"grade2\").drop(columns=\"grade2\")\n",
"display(out3)\n",
"\n",
"\n",
"with pd.ExcelWriter(\"eiken-bert.xlsx\") as f:\n",
" out.to_excel(f, \"full-result\", index=False)\n",
" out2.to_excel(f, \"result-wideformat\", index=False)\n",
" out3.to_excel(f, \"score-summary\")\n",
"\n",
"out.to_csv(\"eiken-bert_full-result.csv\", index=False)\n",
"out2.to_csv(\"eiken-bert_result-wideformat.csv\", index=False)\n",
"out3.to_csv(\"eiken-bert_score-summary.csv\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "154ec872",
"metadata": {
"id": "pCfCHx5BBA03",
"papermill": {
"duration": 0.052601,
"end_time": "2021-08-01T13:36:09.375252",
"exception": false,
"start_time": "2021-08-01T13:36:09.322651",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "7d39d669",
"metadata": {
"papermill": {
"duration": 0.054502,
"end_time": "2021-08-01T13:36:09.484225",
"exception": false,
"start_time": "2021-08-01T13:36:09.429723",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "5dc47487",
"metadata": {
"papermill": {
"duration": 0.053026,
"end_time": "2021-08-01T13:36:09.591770",
"exception": false,
"start_time": "2021-08-01T13:36:09.538744",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.10"
},
"papermill": {
"default_parameters": {},
"duration": 216.783386,
"end_time": "2021-08-01T13:36:12.703427",
"environment_variables": {},
"exception": null,
"input_path": "__notebook__.ipynb",
"output_path": "__notebook__.ipynb",
"parameters": {},
"start_time": "2021-08-01T13:32:35.920041",
"version": "2.3.3"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"state": {
"081acc28e47e4e628d603e672f821b04": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_92ea15ac8bf044e686dfc3efbd16b710",
"placeholder": "​",
"style": "IPY_MODEL_b67b141942fd4f40a3a66a082d97d2a5",
"value": "Downloading: 100%"
}
},
"089742308a7747febdb37472fc754309": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_134b23de74d8475d8adc96f3973be5c9",
"placeholder": "​",
"style": "IPY_MODEL_43da5ebde5e24cdaa85f1567a5add4fc",
"value": "Downloading: 100%"
}
},
"09c09671ab5848b0b74a73b8271f1de0": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_1ee66763e6f34dc6bc7b218940a40704",
"IPY_MODEL_9838719854fe49a58e55dd4ae775a6ca",
"IPY_MODEL_43bc224371734e6888fc0f4a42790aba"
],
"layout": "IPY_MODEL_9a975f1832a040c091cd5cefb120ecf8"
}
},
"0c497fde35bc43ebb4e34942be288a20": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_49da21017f5845aaa6ac0bedca46233c",
"IPY_MODEL_19a3ce7c788049faaa447a590c0e6ca4",
"IPY_MODEL_b238e6516bfd4352af2d6f55ee1ef2fb"
],
"layout": "IPY_MODEL_252d4891920c4d5bb81cfaf6bb7fcc95"
}
},
"0fbc886d28fc42cdbdc631af0a73fed2": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"10969068a7234a4bbe475c6685d4e072": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_f51167ac5b5f49ab9aa96a82797ce2db",
"placeholder": "​",
"style": "IPY_MODEL_1375c6adc5cc45b1959e8e97aabea024",
"value": " 3.25G/3.25G [02:02&lt;00:00, 29.8MB/s]"
}
},
"1110a19fe71b4481a5d649ae4f157882": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"12976942782a486caba1eaa8ca80bdb6": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_1110a19fe71b4481a5d649ae4f157882",
"placeholder": "​",
"style": "IPY_MODEL_34789149fb454e0ca845714df33c115e",
"value": "Downloading: 100%"
}
},
"134b23de74d8475d8adc96f3973be5c9": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"1375c6adc5cc45b1959e8e97aabea024": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"15d56d28cfc547b3aba6958cda6cdae7": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"17fb254f61734273a6253125bbae65e4": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_3873e7e2399c4b7c99131df2ceaed489",
"max": 1355863.0,
"min": 0.0,
"orientation": "horizontal",
"style": "IPY_MODEL_62f97a460b55401e98017676f1c23fd1",
"value": 1355863.0
}
},
"189d5721c8be4f2e8951f104d925a523": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"19a3ce7c788049faaa447a590c0e6ca4": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_4ab830596da24db89f8af6cd80f4e4d9",
"max": 898823.0,
"min": 0.0,
"orientation": "horizontal",
"style": "IPY_MODEL_2a74a18d6da144e5bf84661fcf155664",
"value": 898823.0
}
},
"1c36eaf1e2b84449af93391e09e3bd83": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"1ee66763e6f34dc6bc7b218940a40704": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_8f2484d433f448f48ffac7b58a583d64",
"placeholder": "​",
"style": "IPY_MODEL_f667cb97a255427486f9ca2f59c7e4c2",
"value": "Downloading: 100%"
}
},
"2036042d45f8430e9e70acfb744f82f7": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"20c13c500df048aaa6ef1a38f8651eb9": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"2283b9295e984765b8fcb78333e1edda": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_8bc0c34a257f480bad6bf9485751bb22",
"max": 1355256.0,
"min": 0.0,
"orientation": "horizontal",
"style": "IPY_MODEL_50ab23d4c36b4b90802f0470f4689b61",
"value": 1355256.0
}
},
"23c649449bfc4b9193db1787b69b86c0": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"252c1b82ba3941d0b65c801f283367f2": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"252d4891920c4d5bb81cfaf6bb7fcc95": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"29128a23fdf84795a74dde87cac3fb23": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"2a74a18d6da144e5bf84661fcf155664": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"2f54cef3042d48aca2f70ba2238b468f": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_081acc28e47e4e628d603e672f821b04",
"IPY_MODEL_8a272c6ac1b34b3bb4e5ba262d3d808d",
"IPY_MODEL_5ff41c647feb4ed2bfb0d3a4dc1d7afe"
],
"layout": "IPY_MODEL_9ccb460774e4402c8d701df220e91f1e"
}
},
"34789149fb454e0ca845714df33c115e": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"3730bb32c3774a29bcf2ff9087268b30": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_c5c9c50a85204704a5f8f706986fc2fc",
"placeholder": "​",
"style": "IPY_MODEL_4607ce24aaaa41558b719f857f7b2755",
"value": "Downloading: 100%"
}
},
"3873e7e2399c4b7c99131df2ceaed489": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"39c0dccd88914eb19ce6a32a723c28c9": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"3bfc354034e74a8093be115823d12c74": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"43bc224371734e6888fc0f4a42790aba": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_f1ba0fcc3c424474b190f121c525e3a4",
"placeholder": "​",
"style": "IPY_MODEL_7a51ced1d3cb45019dd3d9139c8ce35e",
"value": " 764/764 [00:00&lt;00:00, 25.5kB/s]"
}
},
"43da5ebde5e24cdaa85f1567a5add4fc": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"441c133f0cb64ad590265c87a4417d49": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_7db25635257d4ba695c5ad68f003b677",
"placeholder": "​",
"style": "IPY_MODEL_75ebf688b63448529c2a70f144f7e8d3",
"value": "Downloading: 100%"
}
},
"4607ce24aaaa41558b719f857f7b2755": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"462d4a4feabc4067acb508678c54ea9f": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"4837e9ef40384faf94150fb13a75cc63": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"49da21017f5845aaa6ac0bedca46233c": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_a399e16b8c52498eb24849770d7ad723",
"placeholder": "​",
"style": "IPY_MODEL_cb807c1123954db0a2e0c7cf6393f4f9",
"value": "Downloading: 100%"
}
},
"4ab830596da24db89f8af6cd80f4e4d9": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"4b3e855fd484428bba44f60abe52b3af": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"4da82f76a6d04cbab17f45d632c9e3d0": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"4daf304d9d56422880f593c105609b57": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"4e88bdc7861747b6bc4b2d3e7061dc07": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_4b3e855fd484428bba44f60abe52b3af",
"placeholder": "​",
"style": "IPY_MODEL_a0976c78f8274a57b80aa357d3b2e90e",
"value": " 456k/456k [00:00&lt;00:00, 6.63MB/s]"
}
},
"50ab23d4c36b4b90802f0470f4689b61": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"5412612673c847f2bf930072b4fd767d": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_8b9c7f641c7146d4a3ac07f1151b9570",
"placeholder": "​",
"style": "IPY_MODEL_826211a385a2418a9f86d9741eb1cb08",
"value": " 1.36M/1.36M [00:00&lt;00:00, 11.1MB/s]"
}
},
"5761c5932b554fe88be1203b7eb5f02c": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"5a8b112ad327484eb775d5d9273b6228": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_9e569b9649d74c3199dade5351d53cba",
"placeholder": "​",
"style": "IPY_MODEL_462d4a4feabc4067acb508678c54ea9f",
"value": " 1.36M/1.36M [00:00&lt;00:00, 14.1MB/s]"
}
},
"5ff41c647feb4ed2bfb0d3a4dc1d7afe": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_975e1df5491e435a8f7119bef0af334f",
"placeholder": "​",
"style": "IPY_MODEL_29128a23fdf84795a74dde87cac3fb23",
"value": " 1.04M/1.04M [00:00&lt;00:00, 12.2MB/s]"
}
},
"62f97a460b55401e98017676f1c23fd1": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"6d40402fb10b4397a3aaea4973eacbf2": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"6f65a72c142c413590a30d0f21fd3446": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"6fe938a32d3b4c568d2860c552e8e7c7": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"75ebf688b63448529c2a70f144f7e8d3": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"78788e3ddd0143ae8d7ca09c9a392187": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_089742308a7747febdb37472fc754309",
"IPY_MODEL_8ed7e53868194fd3ab9cfc3d1f5dd76a",
"IPY_MODEL_10969068a7234a4bbe475c6685d4e072"
],
"layout": "IPY_MODEL_4837e9ef40384faf94150fb13a75cc63"
}
},
"789396fcfdf24c10afc03be127ce08eb": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"7a51ced1d3cb45019dd3d9139c8ce35e": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"7db25635257d4ba695c5ad68f003b677": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"7e3bb0fe59454010ad7dc4b8a7c3f503": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_901686b5b5794d15b3d2074d7d6099aa",
"max": 456318.0,
"min": 0.0,
"orientation": "horizontal",
"style": "IPY_MODEL_2036042d45f8430e9e70acfb744f82f7",
"value": 456318.0
}
},
"826211a385a2418a9f86d9741eb1cb08": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"86b7ece1f6de4351bbf0268e687f8579": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"8a272c6ac1b34b3bb4e5ba262d3d808d": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_252c1b82ba3941d0b65c801f283367f2",
"max": 1042301.0,
"min": 0.0,
"orientation": "horizontal",
"style": "IPY_MODEL_6fe938a32d3b4c568d2860c552e8e7c7",
"value": 1042301.0
}
},
"8b8f8e333aad4938951b0e370d1dc021": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_441c133f0cb64ad590265c87a4417d49",
"IPY_MODEL_a56f90094cc44d9da08bce74d2853aba",
"IPY_MODEL_cc1dbc6b80a149ec84bf3c3b8fa160fa"
],
"layout": "IPY_MODEL_39c0dccd88914eb19ce6a32a723c28c9"
}
},
"8b9c7f641c7146d4a3ac07f1151b9570": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"8bc0c34a257f480bad6bf9485751bb22": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"8d799c4e57e24fa3aa39bcc0581a7731": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_15d56d28cfc547b3aba6958cda6cdae7",
"placeholder": "​",
"style": "IPY_MODEL_5761c5932b554fe88be1203b7eb5f02c",
"value": "Downloading: 100%"
}
},
"8ed7e53868194fd3ab9cfc3d1f5dd76a": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_a28d491b05de4cda878ccc5be699c864",
"max": 3247202234.0,
"min": 0.0,
"orientation": "horizontal",
"style": "IPY_MODEL_6d40402fb10b4397a3aaea4973eacbf2",
"value": 3247202234.0
}
},
"8f2484d433f448f48ffac7b58a583d64": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"8f280ec0994040e79ca02ab7096553f4": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"8fdd4082639c436b9877b8d1e9ba6862": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"901686b5b5794d15b3d2074d7d6099aa": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"912cb75184144774812686ebefb557a7": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"92ea15ac8bf044e686dfc3efbd16b710": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"975e1df5491e435a8f7119bef0af334f": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"97a759dd64c249ea9bbe89d76e36b64e": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_9a3508c643fa47b6b01f58122422e2ae",
"IPY_MODEL_2283b9295e984765b8fcb78333e1edda",
"IPY_MODEL_5412612673c847f2bf930072b4fd767d"
],
"layout": "IPY_MODEL_3bfc354034e74a8093be115823d12c74"
}
},
"9838719854fe49a58e55dd4ae775a6ca": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_e7d562bea2bc41f58442eb8892ae4ee3",
"max": 764.0,
"min": 0.0,
"orientation": "horizontal",
"style": "IPY_MODEL_ec97a3ad2a124a7bbf9a48f0a226eb58",
"value": 764.0
}
},
"9a3508c643fa47b6b01f58122422e2ae": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_4da82f76a6d04cbab17f45d632c9e3d0",
"placeholder": "​",
"style": "IPY_MODEL_c25ec2bf5e05493f80124c3412c4a5ca",
"value": "Downloading: 100%"
}
},
"9a975f1832a040c091cd5cefb120ecf8": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"9ccb460774e4402c8d701df220e91f1e": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"9e569b9649d74c3199dade5351d53cba": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"a025d5f6b17b4e9dad77111afa5a9840": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"a0976c78f8274a57b80aa357d3b2e90e": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"a1684aa619614e8bb9c2e8e99435e825": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_3730bb32c3774a29bcf2ff9087268b30",
"IPY_MODEL_c13ef93aa7b44bb9b6a456456f0dbce2",
"IPY_MODEL_e3ae2deb94244df18b322ada44a9795c"
],
"layout": "IPY_MODEL_dadf353e693c421fa5b7b62bd4a59d2e"
}
},
"a28d491b05de4cda878ccc5be699c864": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"a308f1a3198f439f91b5a60929f24623": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"a399e16b8c52498eb24849770d7ad723": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"a542df8d3d1b4a2db71900ce7e956abc": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_8d799c4e57e24fa3aa39bcc0581a7731",
"IPY_MODEL_7e3bb0fe59454010ad7dc4b8a7c3f503",
"IPY_MODEL_4e88bdc7861747b6bc4b2d3e7061dc07"
],
"layout": "IPY_MODEL_8fdd4082639c436b9877b8d1e9ba6862"
}
},
"a56f90094cc44d9da08bce74d2853aba": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_b5bb3ba64ed947a18647d84486b5ac2d",
"max": 456318.0,
"min": 0.0,
"orientation": "horizontal",
"style": "IPY_MODEL_a025d5f6b17b4e9dad77111afa5a9840",
"value": 456318.0
}
},
"b031b252f7484c3e802eb955e99184f7": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_6f65a72c142c413590a30d0f21fd3446",
"placeholder": "​",
"style": "IPY_MODEL_86b7ece1f6de4351bbf0268e687f8579",
"value": "Downloading: 100%"
}
},
"b238e6516bfd4352af2d6f55ee1ef2fb": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_1c36eaf1e2b84449af93391e09e3bd83",
"placeholder": "​",
"style": "IPY_MODEL_4daf304d9d56422880f593c105609b57",
"value": " 899k/899k [00:00&lt;00:00, 10.6MB/s]"
}
},
"b5bb3ba64ed947a18647d84486b5ac2d": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"b67b141942fd4f40a3a66a082d97d2a5": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"bd8755426ffe4211b3267473e9343671": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_189d5721c8be4f2e8951f104d925a523",
"placeholder": "​",
"style": "IPY_MODEL_8f280ec0994040e79ca02ab7096553f4",
"value": " 331M/331M [00:12&lt;00:00, 29.8MB/s]"
}
},
"c13ef93aa7b44bb9b6a456456f0dbce2": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_0fbc886d28fc42cdbdc631af0a73fed2",
"max": 480.0,
"min": 0.0,
"orientation": "horizontal",
"style": "IPY_MODEL_cffc24c97db8415ba743b40219f50121",
"value": 480.0
}
},
"c25ec2bf5e05493f80124c3412c4a5ca": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"c5c9c50a85204704a5f8f706986fc2fc": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"cb807c1123954db0a2e0c7cf6393f4f9": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"cc1dbc6b80a149ec84bf3c3b8fa160fa": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_23c649449bfc4b9193db1787b69b86c0",
"placeholder": "​",
"style": "IPY_MODEL_789396fcfdf24c10afc03be127ce08eb",
"value": " 456k/456k [00:00&lt;00:00, 5.24MB/s]"
}
},
"cd47819ef91c48f7ae87b1550a64acf2": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"cffc24c97db8415ba743b40219f50121": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"dadf353e693c421fa5b7b62bd4a59d2e": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"e3ae2deb94244df18b322ada44a9795c": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_20c13c500df048aaa6ef1a38f8651eb9",
"placeholder": "​",
"style": "IPY_MODEL_912cb75184144774812686ebefb557a7",
"value": " 480/480 [00:00&lt;00:00, 15.4kB/s]"
}
},
"e7d562bea2bc41f58442eb8892ae4ee3": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"e941f62d3416448d8d17bf3d26f47cc3": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"ec97a3ad2a124a7bbf9a48f0a226eb58": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"f0c1c58c4bd545b783e5bb7e92c4e6d0": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_b031b252f7484c3e802eb955e99184f7",
"IPY_MODEL_17fb254f61734273a6253125bbae65e4",
"IPY_MODEL_5a8b112ad327484eb775d5d9273b6228"
],
"layout": "IPY_MODEL_f64c885198c74e659fab868f0b36a670"
}
},
"f1ba0fcc3c424474b190f121c525e3a4": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"f51167ac5b5f49ab9aa96a82797ce2db": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"f64c885198c74e659fab868f0b36a670": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"f667cb97a255427486f9ca2f59c7e4c2": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"f9332145491049d0834b067e010a4045": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_e941f62d3416448d8d17bf3d26f47cc3",
"max": 331070498.0,
"min": 0.0,
"orientation": "horizontal",
"style": "IPY_MODEL_a308f1a3198f439f91b5a60929f24623",
"value": 331070498.0
}
},
"fd437e67b2514af0bf40eddef8e67269": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_12976942782a486caba1eaa8ca80bdb6",
"IPY_MODEL_f9332145491049d0834b067e010a4045",
"IPY_MODEL_bd8755426ffe4211b3267473e9343671"
],
"layout": "IPY_MODEL_cd47819ef91c48f7ae87b1550a64acf2"
}
}
},
"version_major": 2,
"version_minor": 0
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Raw
eiken-bert_full-result.csv
problem scores answer_position correct grade method
A: What is your {}? B: Kazumi Suzuki. name(0.245),nickname(0.101),surname(0.058),answer(0.050),idol(0.043) 1 True 5 fill-mask-no-target
I know Judy. She can {} French very well. speak(0.858),read(0.053),learn(0.012),teach(0.011),understand(0.010) 1 True 5 fill-mask-no-target
A: Are your baseball shoes in your room, Mike? B: No, Mom. They're in my {} at school. room(0.408),locker(0.132),classroom(0.107),closet(0.050),backpack(0.035) 2 True 5 fill-mask-no-target
Mysister usually plays tennis {} Saturdays. on(0.953),every(0.021),through(0.007),until(0.002),and(0.002) 1 True 5 fill-mask-no-target
My mother likes {}. She has many pretty ones in the garden. roses(0.326),flowers(0.147),strawberries(0.070),tomatoes(0.036),butterflies(0.027) 2 True 5 fill-mask-no-target
Let's begin today's class. Open your textbooks to {} 22. page(0.285),chapter(0.171),Chapter(0.113),Page(0.050),Section(0.027) 1 True 5 fill-mask-no-target
Today is Wednesday. Tomorrow is {}. Friday(0.293),Thursday(0.236),Sunday(0.168),Tuesday(0.105),Saturday(0.063) 2 True 5 fill-mask-no-target
I usually read magazines {} home. at(0.945),from(0.026),back(0.023),around(0.001),near(0.001) 1 True 5 fill-mask-no-target
A: It's ten o'clock, Jimmy. {} to bed. B: All right, Mom. Go(0.868),Get(0.039),Back(0.039),Went(0.009),Going(0.009) 1 True 5 fill-mask-no-target
A: Do you live {} Tokyo? B: Yes. It's a big city. in(0.874),near(0.075),outside(0.043),around(0.002),inside(0.002) 1 True 5 fill-mask-no-target
A: What is your {}? B: Kazumi Suzuki. name(0.245),date(0.002),club(0.000),hour(0.000) 1 True 5 fill-mask-with-targets
I know Judy. She can {} French very well. speak(0.858),drink(0.000),see(0.000),open(0.000) 1 True 5 fill-mask-with-targets
A: Are your baseball shoes in your room, Mike? B: No, Mom. They're in my {} at school. locker(0.132),window(0.000),door(0.000),shop(0.000) 1 True 5 fill-mask-with-targets
Mysister usually plays tennis {} Saturdays. on(0.953),at(0.001),with(0.000),by(0.000) 1 True 5 fill-mask-with-targets
My mother likes {}. She has many pretty ones in the garden. flowers(0.147),movies(0.000),sports(0.000),schools(0.000) 1 True 5 fill-mask-with-targets
Let's begin today's class. Open your textbooks to {} 22. page(0.285),minute(0.000),chalk(0.000),ground(0.000) 1 True 5 fill-mask-with-targets
Today is Wednesday. Tomorrow is {}. Friday(0.293),Thursday(0.236),Tuesday(0.105),Monday(0.062) 2 False 5 fill-mask-with-targets
I usually read magazines {} home. at(0.945),on(0.000),with(0.000),of(0.000) 1 True 5 fill-mask-with-targets
A: It's ten o'clock, Jimmy. {} to bed. B: All right, Mom. Go(0.868),Sleep(0.000),Sit(0.000),Do(0.000) 1 True 5 fill-mask-with-targets
A: Do you live {} Tokyo? B: Yes. It's a big city. in(0.874),on(0.001),with(0.000),after(0.000) 1 True 5 fill-mask-with-targets
My father is a {} of a sports club. He plays tennis there every Wednesday night. member(0.660),founder(0.068),patron(0.067),director(0.038),manager(0.031) 1 True 4 fill-mask-no-target
Mr. Clark told us many intesting {} about his trip to India. stories(0.447),anecdotes(0.249),tales(0.187),things(0.030),truths(0.015) 1 True 4 fill-mask-no-target
It's snowing a lot today, so please drive {}. accordingly(0.111),responsibly(0.091),carefully(0.073),safely(0.064),wisely(0.057) -1 False 4 fill-mask-no-target
In spring, Jane likes to walk in her grandmother's {}. She enjoys looking at the beautiful flowers there. garden(0.659),yard(0.233),gardens(0.042),backyard(0.039),fields(0.005) 1 True 4 fill-mask-no-target
Many girls in my class have {} hair. curly(0.335),blonde(0.172),blond(0.074),facial(0.054),brown(0.045) -1 False 4 fill-mask-no-target
A: Do you live in a city? B: No. I live in a small {} town(0.515),city(0.139),village(0.099),suburb(0.049),neighborhood(0.016) 1 True 4 fill-mask-no-target
I {} Nancy's notebook. It was on Mary's desk remember(0.180),remembered(0.115),borrowed(0.086),found(0.077),forgot(0.056) 4 True 4 fill-mask-no-target
Dennis went to Japan for a year in August. He was sad when he {} goodbye to his family said(0.822),waved(0.081),bid(0.039),says(0.011),shouted(0.004) 1 True 4 fill-mask-no-target
Jeff left the party at 8:00. He wanted to {} home early and go to bed. go(0.398),get(0.369),head(0.099),come(0.050),stay(0.015) 2 True 4 fill-mask-no-target
Mom's lemon cake is not as good {} her chocolate cake. as(0.998),for(0.001),than(0.000),with(0.000),at(0.000) 1 True 4 fill-mask-no-target
My father is a {} of a sports club. He plays tennis there every Wednesday night. member(0.660),group(0.000),festival(0.000),picnic(0.000) 1 True 4 fill-mask-with-targets
Mr. Clark told us many intesting {} about his trip to India. stories(0.447),books(0.000),pictures(0.000),magazines(0.000) 1 True 4 fill-mask-with-targets
It's snowing a lot today, so please drive {}. slowly(0.036),cold(0.001),freely(0.000),bus(0.000) 1 True 4 fill-mask-with-targets
In spring, Jane likes to walk in her grandmother's {}. She enjoys looking at the beautiful flowers there. garden(0.659),wall(0.000),stone(0.000),sky(0.000) 1 True 4 fill-mask-with-targets
Many girls in my class have {} hair. short(0.023),late(0.000),busy(0.000),slow(0.000) 1 True 4 fill-mask-with-targets
A: Do you live in a city? B: No. I live in a small {} town(0.515),hobby(0.000),holiday(0.000),ticket(0.000) 1 True 4 fill-mask-with-targets
I {} Nancy's notebook. It was on Mary's desk found(0.077),stayed(0.000),stopped(0.000),went(0.000) 1 True 4 fill-mask-with-targets
Dennis went to Japan for a year in August. He was sad when he {} goodbye to his family said(0.822),told(0.001),ended(0.000),hoped(0.000) 1 True 4 fill-mask-with-targets
Jeff left the party at 8:00. He wanted to {} home early and go to bed. get(0.369),send(0.000),meet(0.000),put(0.000) 1 True 4 fill-mask-with-targets
Mom's lemon cake is not as good {} her chocolate cake. as(0.998),to(0.000),by(0.000),of(0.000) 1 True 4 fill-mask-with-targets
A: How do you make that potato dish? B: First, you {} the potatoes, and then cut them in half and put butter on them. peel(0.218),roast(0.106),bake(0.099),boil(0.087),chop(0.076) 4 True 3 fill-mask-no-target
Last summer, Hiroshi's family traveled around Japan. This year they went to go {}. skiing(0.214),surfing(0.121),camping(0.098),fishing(0.068),Disneyland(0.058) -1 False 3 fill-mask-no-target
Bob {} five friends to his party invites(0.555),invited(0.367),welcomes(0.031),brings(0.021),welcomed(0.008) 2 True 3 fill-mask-no-target
A: John, you should go to bed soon. If you stay up too late, you'll {} and be late for school. B: OK, Mom. graduate(0.145),crash(0.076),cry(0.051),die(0.049),starve(0.042) -1 False 3 fill-mask-no-target
A: Did you buy your father something special for his birthday? B: Yes. He loves to cook, so I got him a new {}. one(0.087),stove(0.038),spoon(0.033),dish(0.033),oven(0.031) -1 False 3 fill-mask-no-target
I bought a new T-shirt for my brother, but I bought the wong size. It was too {} for him. big(0.277),small(0.083),expensive(0.051),bulky(0.048),tight(0.046) 5 True 3 fill-mask-no-target
Sarah saw some flowers by the road while she was taking a walk. She {} a few and took them home. picked(0.319),bought(0.086),collected(0.085),gathered(0.069),found(0.058) 1 True 3 fill-mask-no-target
Jenny saw her grandparents {} the first time in years. She missed them very much. for(0.815),at(0.052),in(0.027),again(0.011),live(0.007) 1 True 3 fill-mask-no-target
A: I told my mother that I would be home by 7:00. I don't want to {} my promise, so I have to go now. B: OK. break(0.769),fulfill(0.066),violate(0.044),keep(0.029),fulfil(0.011) 1 True 3 fill-mask-no-target
A: Don'y say anything to Dad about the surprise party! B: Don't worry. He won't find {} about it from me. anything(0.642),out(0.304),word(0.020),something(0.006),news(0.002) 2 True 3 fill-mask-no-target
A: How do you make that potato dish? B: First, you {} the potatoes, and then cut them in half and put butter on them. boil(0.087),eat(0.003),hurt(0.000),care(0.000) 1 True 3 fill-mask-with-targets
Last summer, Hiroshi's family traveled around Japan. This year they went to go {}. abroad(0.019),inside(0.000),similar(0.000),other(0.000) 1 True 3 fill-mask-with-targets
Bob {} five friends to his party invited(0.367),made(0.000),visited(0.000),spoke(0.000) 1 True 3 fill-mask-with-targets
A: John, you should go to bed soon. If you stay up too late, you'll {} and be late for school. B: OK, Mom. graduate(0.145),return(0.001),promise(0.000),overs(0.000) -1 False 3 fill-mask-with-targets
A: Did you buy your father something special for his birthday? B: Yes. He loves to cook, so I got him a new {}. ring(0.002),field(0.000),a(0.000),contact(0.000) -1 False 3 fill-mask-with-targets
I bought a new T-shirt for my brother, but I bought the wong size. It was too {} for him. tight(0.046),heavy(0.042),bright(0.002),clear(0.000) 1 True 3 fill-mask-with-targets
Sarah saw some flowers by the road while she was taking a walk. She {} a few and took them home. picked(0.319),spent(0.000),guessed(0.000),wished(0.000) 1 True 3 fill-mask-with-targets
Jenny saw her grandparents {} the first time in years. She missed them very much. for(0.815),from(0.003),over(0.001),out(0.000) 1 True 3 fill-mask-with-targets
A: I told my mother that I would be home by 7:00. I don't want to {} my promise, so I have to go now. B: OK. break(0.769),sell(0.000),pass(0.000),lend(0.000) 1 True 3 fill-mask-with-targets
A: Don'y say anything to Dad about the surprise party! B: Don't worry. He won't find {} about it from me. out(0.304),through(0.000),near(0.000),within(0.000) 1 True 3 fill-mask-with-targets
Jamie visited several {} buildings when he went to Rome. Some of them were more than 2,000 years old. ancient(0.177),Roman(0.109),historic(0.066),Byzantine(0.047),medieval(0.039) 1 True pre2 fill-mask-no-target
Sally's French teacher told her to read an article in a French newspaper and {} it into English. translate(0.944),paste(0.033),translated(0.007),convert(0.004),read(0.002) 1 True pre2 fill-mask-no-target
Henry likes living in the city because there are so many things to do there. But he also loves nature, so sometimes he goes for a drive in the {}. woods(0.460),mountains(0.118),countryside(0.057),park(0.053),forest(0.028) 3 True pre2 fill-mask-no-target
A: Is it true that the things in this store only cost 100 yen? B: Yes, but you will also need to pay {}, so they actually cost a little more. extra(0.120),yen(0.100),postage(0.098),taxes(0.059),more(0.049) -1 False pre2 fill-mask-no-target
When the bust was an hour late, one man shouted {} at the driver. He said that he had missed an important meeting. angrily(0.495),insults(0.126),loudly(0.057),curses(0.042),abuse(0.041) 1 True pre2 fill-mask-no-target
Firefighters have to {} people from buildings that are on fire. To do this, they must be strong and healthy. rescue(0.601),evacuate(0.232),remove(0.049),pull(0.025),save(0.024) 1 True pre2 fill-mask-no-target
John loves the singer Ann May, and he cannot wait until her new CD is {} next week. He has been waiting for it since here last CD came out two years ago. released(0.776),out(0.174),published(0.015),available(0.009),due(0.004) 1 True pre2 fill-mask-no-target
The news that Ms. Kelly, the art teacher, was going to get married {} through the school very quickly. By lunchtime, almost all the students knew about it. blew(0.197),swept(0.149),spread(0.141),came(0.082),went(0.059) 3 True pre2 fill-mask-no-target
A: I'm really nervous about acting in the play next week. B: I know you're worried now, but you'll feel fine as soon a syou get on the {}. stage(0.415),scene(0.078),screen(0.048),show(0.036),floor(0.021) 1 True pre2 fill-mask-no-target
Before Diane attended Professor Miller's {} at the university, she was not intested in Chinese art. However, now, she wants to learn more about it. lectures(0.595),lecture(0.120),classes(0.041),seminar(0.039),courses(0.027) 2 True pre2 fill-mask-no-target
Jamie visited several {} buildings when he went to Rome. Some of them were more than 2,000 years old. ancient(0.177),exact(0.000),responsible(0.000),unable(0.000) 1 True pre2 fill-mask-with-targets
Sally's French teacher told her to read an article in a French newspaper and {} it into English. translate(0.944),throw(0.000),guide(0.000),control(0.000) 1 True pre2 fill-mask-with-targets
Henry likes living in the city because there are so many things to do there. But he also loves nature, so sometimes he goes for a drive in the {}. countryside(0.057),image(0.000),experiment(0.000),decision(0.000) 1 True pre2 fill-mask-with-targets
A: Is it true that the things in this store only cost 100 yen? B: Yes, but you will also need to pay {}, so they actually cost a little more. tax(0.011),data(0.000),total(0.000),waste(0.000) 1 True pre2 fill-mask-with-targets
When the bust was an hour late, one man shouted {} at the driver. He said that he had missed an important meeting. angrily(0.495),tightly(0.000),partly(0.000),secretly(0.000) 1 True pre2 fill-mask-with-targets
Firefighters have to {} people from buildings that are on fire. To do this, they must be strong and healthy. rescue(0.601),weigh(0.000),produce(0.000),stamp(0.000) 1 True pre2 fill-mask-with-targets
John loves the singer Ann May, and he cannot wait until her new CD is {} next week. He has been waiting for it since here last CD came out two years ago. released(0.776),invented(0.000),trapped(0.000),divided(0.000) 1 True pre2 fill-mask-with-targets
The news that Ms. Kelly, the art teacher, was going to get married {} through the school very quickly. By lunchtime, almost all the students knew about it. spread(0.141),stretched(0.000),served(0.000),stood(0.000) 1 True pre2 fill-mask-with-targets
A: I'm really nervous about acting in the play next week. B: I know you're worried now, but you'll feel fine as soon a syou get on the {}. stage(0.415),screen(0.048),field(0.011),court(0.010) 1 True pre2 fill-mask-with-targets
Before Diane attended Professor Miller's {} at the university, she was not intested in Chinese art. However, now, she wants to learn more about it. lecture(0.120),comment(0.000),furniture(0.000),shipment(0.000) 1 True pre2 fill-mask-with-targets
At first, the marketing department and the sales department were on the project together. But people in the sales department were too busy, so now the project is being run {} by the marketing department. jointly(0.257),entirely(0.139),separately(0.132),independently(0.132),solely(0.064) 2 True 2 fill-mask-no-target
Experts at the art gallery discovered that one of their paitings, which they had thought was a {} Picasso, was actually just a copy. Pablo(0.455),genuine(0.041),fake(0.037),signed(0.028),Roman(0.025) 2 True 2 fill-mask-no-target
The musician Jimmy Baker had a lot of {} when he was a child. His family was very poor before he became a rich and famous rock star. money(0.189),fame(0.113),success(0.088),problems(0.057),trouble(0.048) -1 False 2 fill-mask-no-target
Mother Teresa helped many sick people and gave food to many hungry children in India. She was known as a person who cared about {}. orphans(0.279),humanity(0.114),children(0.082),everyone(0.068),others(0.061) 2 True 2 fill-mask-no-target
As Liam waled down the dark street, he began to feel afraid. He had the {} that someone was watching him. feeling(0.477),impression(0.221),sense(0.122),idea(0.066),sensation(0.018) 5 True 2 fill-mask-no-target
Risa buys water that comes from a mountain stream. She says that drinking it is good because it has many {} that her body needs. nutrients(0.880),minerals(0.047),vitamins(0.025),benefits(0.018),calories(0.006) 2 True 2 fill-mask-no-target
The lifeguard ran into the ocean to help a young girl who looked like she was {} in the big waves drowning(0.797),drowned(0.061),swimming(0.027),dying(0.021),floating(0.020) 1 True 2 fill-mask-no-target
Yesterday was a hot day at the zoo, so Heather bought an ice cream. It melted so quickly that she could not help {} some on her dress. putting(0.326),spilling(0.107),pouring(0.064),slipping(0.026),put(0.022) 2 True 2 fill-mask-no-target
In the past, sailors had to use the stars to {} when they were on an ocean. These days, ships have modern equipment that shows sailors which way to go. communicate(0.272),navigate(0.219),signal(0.073),see(0.071),sail(0.038) 2 True 2 fill-mask-no-target
Daisuke's grandmother eats a lot of vegetables, drinks green tea, and goes for a long walk every evening to {} her health. improve(0.144),check(0.116),boost(0.089),assess(0.072),maintain(0.043) -1 False 2 fill-mask-no-target
At first, the marketing department and the sales department were on the project together. But people in the sales department were too busy, so now the project is being run {} by the marketing department. entirely(0.139),scientifically(0.000),violently(0.000),need(0.000) 1 True 2 fill-mask-with-targets
Experts at the art gallery discovered that one of their paitings, which they had thought was a {} Picasso, was actually just a copy. genuine(0.041),severe(0.000),portable(0.000),logical(0.000) 1 True 2 fill-mask-with-targets
The musician Jimmy Baker had a lot of {} when he was a child. His family was very poor before he became a rich and famous rock star. hardship(0.002),concentration(0.000),membership(0.000),permission(0.000) 1 True 2 fill-mask-with-targets
Mother Teresa helped many sick people and gave food to many hungry children in India. She was known as a person who cared about {}. humanity(0.114),generation(0.000),gravity(0.000),hesitation(0.000) 1 True 2 fill-mask-with-targets
As Liam waled down the dark street, he began to feel afraid. He had the {} that someone was watching him. sensation(0.018),property(0.000),feature(0.000),translation(0.000) 1 True 2 fill-mask-with-targets
Risa buys water that comes from a mountain stream. She says that drinking it is good because it has many {} that her body needs. minerals(0.047),operations(0.000),illustrations(0.000),campaigns(0.000) 1 True 2 fill-mask-with-targets
The lifeguard ran into the ocean to help a young girl who looked like she was {} in the big waves drowning(0.797),proposing(0.000),converting(0.000),exporting(0.000) 1 True 2 fill-mask-with-targets
Yesterday was a hot day at the zoo, so Heather bought an ice cream. It melted so quickly that she could not help {} some on her dress. spilling(0.107),maintaining(0.000),convincing(0.000),arguing(0.000) 1 True 2 fill-mask-with-targets
In the past, sailors had to use the stars to {} when they were on an ocean. These days, ships have modern equipment that shows sailors which way to go. navigate(0.219),satisfy(0.000),respect(0.000),permit(0.000) 1 True 2 fill-mask-with-targets
Daisuke's grandmother eats a lot of vegetables, drinks green tea, and goes for a long walk every evening to {} her health. preserve(0.019),replace(0.000),betray(0.000),interpret(0.000) 1 True 2 fill-mask-with-targets
A: Thanks for showing me the outline of your sales presentation. It's good, but it's a bit {} in some places. B: I guess I do repeat some information too much. I'll try to take some of it out. confusing(0.174),repetitive(0.051),boring(0.048),sloppy(0.047),vague(0.034) -1 False pre1 fill-mask-no-target
Lisa went to the interview even though she thought there was a low {} of her getting the job. As she expected, she was not hired. probability(0.391),likelihood(0.337),chance(0.263),possibility(0.004),chances(0.002) 1 True pre1 fill-mask-no-target
It is sadly {} that, in developing counties, many of the farmers who grow nutritous crops for export do no not have enough food to feed their own families. ironic(0.355),sad(0.098),true(0.081),unfortunate(0.066),evident(0.041) 1 True pre1 fill-mask-no-target
The explosion at the chemical factory {} great damage on the local environment. It will take years for wildlife to fully recover in the region. inflicted(0.798),caused(0.093),wrought(0.036),did(0.019),does(0.008) 1 True pre1 fill-mask-no-target
Some say the best way to overcome a {} is to expose oneself to what one fears. For example, people who are afraid of mice should try holding one. fear(0.577),stigma(0.036),dilemma(0.032),depression(0.031),disease(0.024) -1 False pre1 fill-mask-no-target
English classes at the university were required, but students were {} from them if they could prove they had advanced ability in the language. excluded(0.375),exempted(0.224),barred(0.143),exempt(0.065),banned(0.039) 2 True pre1 fill-mask-no-target
E-mail and text messaging have {} the way people write. Many people shorten words and ignore traditional rules of grammar. changed(0.602),altered(0.146),transformed(0.105),shaped(0.044),influenced(0.016) 3 True pre1 fill-mask-no-target
Some analysts think the new treaty on CO2 emissions is a {} in the fight against global warming. "This is the most important environmental treaty ever signed," said one. milestone(0.416),breakthrough(0.238),landmark(0.081),setback(0.068),victory(0.046) 1 True pre1 fill-mask-no-target
Lying on the sunny beach with her husband on their vacation, Roberta felt {} happy. She had never been so content. so(0.175),incredibly(0.166),very(0.063),extremely(0.046),immensely(0.043) -1 False pre1 fill-mask-no-target
Nadine spends an hour thoroughly cleaning her apartment evey day, so the entire place is {}. clean(0.318),tidy(0.183),cleaner(0.109),cleaned(0.074),pristine(0.045) -1 False pre1 fill-mask-no-target
A: Thanks for showing me the outline of your sales presentation. It's good, but it's a bit {} in some places. B: I guess I do repeat some information too much. I'll try to take some of it out. redundant(0.006),subjective(0.000),distinct(0.000),decisive(0.000) 1 True pre1 fill-mask-with-targets
Lisa went to the interview even though she thought there was a low {} of her getting the job. As she expected, she was not hired. probability(0.391),credibility(0.000),contention(0.000),restoration(0.000) 1 True pre1 fill-mask-with-targets
It is sadly {} that, in developing counties, many of the farmers who grow nutritous crops for export do no not have enough food to feed their own families. ironic(0.355),superficial(0.000),indefinite(0.000),restless(0.000) 1 True pre1 fill-mask-with-targets
The explosion at the chemical factory {} great damage on the local environment. It will take years for wildlife to fully recover in the region. inflicted(0.798),enhanced(0.000),perceived(0.000),vanished(0.000) 1 True pre1 fill-mask-with-targets
Some say the best way to overcome a {} is to expose oneself to what one fears. For example, people who are afraid of mice should try holding one. temptation(0.002),famine(0.000),ph(0.000),barric(0.000) -1 False pre1 fill-mask-with-targets
English classes at the university were required, but students were {} from them if they could prove they had advanced ability in the language. exempted(0.224),qualified(0.000),prosecuted(0.000),commanded(0.000) 1 True pre1 fill-mask-with-targets
E-mail and text messaging have {} the way people write. Many people shorten words and ignore traditional rules of grammar. transformed(0.105),dis(0.000),synthes(0.000),offic(0.000) 1 True pre1 fill-mask-with-targets
Some analysts think the new treaty on CO2 emissions is a {} in the fight against global warming. "This is the most important environmental treaty ever signed," said one. milestone(0.416),backlog(0.000),confession(0.000),vig(0.000) 1 True pre1 fill-mask-with-targets
Lying on the sunny beach with her husband on their vacation, Roberta felt {} happy. She had never been so content. profoundly(0.004),barely(0.000),harshly(0.000),improperly(0.000) 1 True pre1 fill-mask-with-targets
Nadine spends an hour thoroughly cleaning her apartment evey day, so the entire place is {}. spot(0.000),rugged(0.000),impartial(0.000),minute(0.000) -1 False pre1 fill-mask-with-targets
Cell phones have become a permanent {} in modern society. Most perople could not imagine living without one. fixture(0.932),feature(0.011),necessity(0.011),norm(0.006),novelty(0.003) 1 True 1 fill-mask-no-target
Colin did not have enough money to pay for the car all at onece, so he paid it off in {} of $800 a month for two years. increments(0.440),excess(0.375),installments(0.163),lieu(0.004),chunks(0.003) 3 True 1 fill-mask-no-target
When she asked her boss for a raise, Melanie's {} tone of voice made it obvious how nervous she was. calm(0.040),soft(0.034),low(0.031),quiet(0.028),trembling(0.024) -1 False 1 fill-mask-no-target
The religious sect established a {} in a rural area where its followers could live together and share everything. No private property was allowed. sanctuary(0.225),monastery(0.219),temple(0.073),compound(0.062),camp(0.044) -1 False 1 fill-mask-no-target
The famous reporter was fired for {} another journalist's work. His article was almost exactly the same as that of the other journalist. criticizing(0.301),copying(0.131),publishing(0.062),covering(0.055),exposing(0.051) -1 False 1 fill-mask-no-target
Now that the local steel factory has closed down, the streets of the once-busy town are lined with {} businesses. Most owners have abandoned their stores. bustling(0.132),abandoned(0.120),struggling(0.069),empty(0.067),small(0.062) -1 False 1 fill-mask-no-target
The ambassador's failure to attend the ceremony held in honor of the king was considered an {} by his host nation and made already bad relations worse. embarrassment(0.558),insult(0.365),injustice(0.022),outrage(0.011),offense(0.007) -1 False 1 fill-mask-no-target
US border guards managed to {} the escaped prisoner as he tried to cross into Canada. He was returned to jail immediately. detain(0.213),apprehend(0.205),arrest(0.067),capture(0.062),rescue(0.052) 2 True 1 fill-mask-no-target
Anthony enjoyed his first day at his new job. The atmosphere was {}, and his colleagues did their best to make him feel welcome. welcoming(0.102),upbeat(0.077),positive(0.076),great(0.064),warm(0.056) -1 False 1 fill-mask-no-target
A: I just learned I've been {} to second violin in the school orchestra. I knew I should've practiced more.B: Well, if you work hard, I'm sure you can get your previous position back. promoted(0.438),relegated(0.294),upgraded(0.082),reduced(0.053),elevated(0.038) 2 True 1 fill-mask-no-target
Cell phones have become a permanent {} in modern society. Most perople could not imagine living without one. fixture(0.932),rupture(0.000),stint(0.000),clasp(0.000) 1 True 1 fill-mask-with-targets
Colin did not have enough money to pay for the car all at onece, so he paid it off in {} of $800 a month for two years. installments(0.163),spec(0.000),dispos(0.000),enact(0.000) 1 True 1 fill-mask-with-targets
When she asked her boss for a raise, Melanie's {} tone of voice made it obvious how nervous she was. j(0.000),diff(0.000),gar(0.000),pomp(0.000) -1 False 1 fill-mask-with-targets
The religious sect established a {} in a rural area where its followers could live together and share everything. No private property was allowed. repository(0.000),commun(0.000),pre(0.000),dir(0.000) -1 False 1 fill-mask-with-targets
The famous reporter was fired for {} another journalist's work. His article was almost exactly the same as that of the other journalist. plagiar(0.000),be(0.000),allev(0.000),inoc(0.000) -1 False 1 fill-mask-with-targets
Now that the local steel factory has closed down, the streets of the once-busy town are lined with {} businesses. Most owners have abandoned their stores. defunct(0.001),aspiring(0.000),volatile(0.000),rhetorical(0.000) 1 True 1 fill-mask-with-targets
The ambassador's failure to attend the ceremony held in honor of the king was considered an {} by his host nation and made already bad relations worse. imp(0.000),aff(0.000),ult(0.000),eluc(0.000) -1 False 1 fill-mask-with-targets
US border guards managed to {} the escaped prisoner as he tried to cross into Canada. He was returned to jail immediately. apprehend(0.205),pill(0.000),acc(0.000),ex(0.000) 1 True 1 fill-mask-with-targets
Anthony enjoyed his first day at his new job. The atmosphere was {}, and his colleagues did their best to make him feel welcome. congen(0.000),del(0.000),impl(0.000),meas(0.000) -1 False 1 fill-mask-with-targets
A: I just learned I've been {} to second violin in the school orchestra. I knew I should've practiced more.B: Well, if you work hard, I'm sure you can get your previous position back. relegated(0.294),reiterated(0.000),st(0.000),jeopard(0.000) 1 True 1 fill-mask-with-targets
A: What is your {}? B: Kazumi Suzuki. name(4.421),date(5.132),club(5.409),hour(5.515) 1 True 5 perplexity
I know Judy. She can {} French very well. speak(3.975),see(4.961),drink(5.339),open(5.466) 1 True 5 perplexity
A: Are your baseball shoes in your room, Mike? B: No, Mom. They're in my {} at school. locker(3.825),shop(4.090),door(4.124),window(4.185) 1 True 5 perplexity
Mysister usually plays tennis {} Saturdays. on(6.950),at(7.842),by(8.166),with(8.233) 1 True 5 perplexity
My mother likes {}. She has many pretty ones in the garden. flowers(3.275),movies(4.204),sports(4.250),schools(4.268) 1 True 5 perplexity
Let's begin today's class. Open your textbooks to {} 22. page(3.568),minute(4.388),chalk(4.695),ground(4.828) 1 True 5 perplexity
Today is Wednesday. Tomorrow is {}. Thursday(2.731),Friday(2.820),Tuesday(3.075),Monday(3.076) 1 True 5 perplexity
I usually read magazines {} home. at(4.812),on(6.230),of(6.704),with(6.817) 1 True 5 perplexity
A: It's ten o'clock, Jimmy. {} to bed. B: All right, Mom. Go(3.637),Sleep(4.108),Do(4.159),Sit(4.297) 1 True 5 perplexity
A: Do you live {} Tokyo? B: Yes. It's a big city. in(3.254),on(3.652),with(3.861),after(3.912) 1 True 5 perplexity
My father is a {} of a sports club. He plays tennis there every Wednesday night. member(3.035),group(4.094),festival(4.196),picnic(4.511) 1 True 4 perplexity
Mr. Clark told us many intesting {} about his trip to India. stories(4.840),pictures(5.429),magazines(5.593),books(5.604) 1 True 4 perplexity
It's snowing a lot today, so please drive {}. slowly(3.082),busily(3.700),coldly(3.714),freely(3.724) 1 True 4 perplexity
In spring, Jane likes to walk in her grandmother's {}. She enjoys looking at the beautiful flowers there. garden(3.513),stone(4.185),wall(4.230),sky(4.275) 1 True 4 perplexity
Many girls in my class have {} hair. short(3.469),late(4.467),slow(4.653),busy(5.382) 1 True 4 perplexity
A: Do you live in a city? B: No. I live in a small {} town(2.939),hobby(3.440),holiday(3.561),ticket(3.678) 1 True 4 perplexity
I {} Nancy's notebook. It was on Mary's desk found(4.157),stopped(4.780),went(4.977),stayed(5.243) 1 True 4 perplexity
Dennis went to Japan for a year in August. He was sad when he {} goodbye to his family said(3.226),told(3.726),ended(3.983),hoped(3.999) 1 True 4 perplexity
Jeff left the party at 8:00. He wanted to {} home early and go to bed. get(2.797),meet(3.353),send(3.444),put(3.580) 1 True 4 perplexity
Mom's lemon cake is not as good {} her chocolate cake. as(3.458),of(4.667),by(4.727),to(4.805) 1 True 4 perplexity
A: How do you make that potato dish? B: First, you {} the potatoes, and then cut them in half and put butter on them. boil(3.264),eat(3.363),hurt(3.521),care(3.718) 1 True 3 perplexity
Last summer, Hiroshi's family traveled around Japan. This year they went to go {}. abroad(4.146),inside(4.269),other(4.620),similar(4.647) 1 True 3 perplexity
Bob {} five friends to his party invited(5.150),made(6.476),spoke(6.545),visited(6.836) 1 True 3 perplexity
A: John, you should go to bed soon. If you stay up too late, you'll {} and be late for school. B: OK, Mom. oversleep(2.627),return(2.889),graduate(3.012),promise(3.052) 1 True 3 perplexity
A: Did you buy your father something special for his birthday? B: Yes. He loves to cook, so I got him a new {}. apron(3.104),ring(3.225),field(3.478),contact(3.479) 1 True 3 perplexity
I bought a new T-shirt for my brother, but I bought the wong size. It was too {} for him. tight(3.286),heavy(3.405),bright(3.512),clear(3.672) 1 True 3 perplexity
Sarah saw some flowers by the road while she was taking a walk. She {} a few and took them home. picked(2.792),guessed(3.328),spent(3.331),wished(3.393) 1 True 3 perplexity
Jenny saw her grandparents {} the first time in years. She missed them very much. for(2.761),over(3.653),out(3.665),from(3.704) 1 True 3 perplexity
A: I told my mother that I would be home by 7:00. I don't want to {} my promise, so I have to go now. B: OK. break(2.799),pass(3.114),sell(3.139),lend(3.226) 1 True 3 perplexity
A: Don'y say anything to Dad about the surprise party! B: Don't worry. He won't find {} about it from me. out(3.642),through(4.512),near(4.575),within(4.787) 1 True 3 perplexity
Jamie visited several {} buildings when he went to Rome. Some of them were more than 2,000 years old. ancient(3.093),exact(3.583),responsible(3.771),unable(3.970) 1 True pre2 perplexity
Sally's French teacher told her to read an article in a French newspaper and {} it into English. translate(3.196),guide(3.870),throw(4.024),control(4.266) 1 True pre2 perplexity
Henry likes living in the city because there are so many things to do there. But he also loves nature, so sometimes he goes for a drive in the {}. countryside(2.423),experiment(2.904),image(2.969),decision(2.981) 1 True pre2 perplexity
A: Is it true that the things in this store only cost 100 yen? B: Yes, but you will also need to pay {}, so they actually cost a little more. tax(3.296),data(3.545),total(3.574),waste(3.582) 1 True pre2 perplexity
When the bust was an hour late, one man shouted {} at the driver. He said that he had missed an important meeting. angrily(3.245),secretly(3.659),tightly(3.678),partly(3.774) 1 True pre2 perplexity
Firefighters have to {} people from buildings that are on fire. To do this, they must be strong and healthy. rescue(2.607),produce(3.223),weigh(3.473),stamp(3.549) 1 True pre2 perplexity
John loves the singer Ann May, and he cannot wait until her new CD is {} next week. He has been waiting for it since here last CD came out two years ago. released(3.660),invented(4.074),divided(4.186),trapped(4.266) 1 True pre2 perplexity
The news that Ms. Kelly, the art teacher, was going to get married {} through the school very quickly. By lunchtime, almost all the students knew about it. spread(3.004),stretched(3.238),served(3.492),stood(3.504) 1 True pre2 perplexity
A: I'm really nervous about acting in the play next week. B: I know you're worried now, but you'll feel fine as soon a syou get on the {}. stage(3.899),screen(4.051),field(4.055),court(4.075) 1 True pre2 perplexity
Before Diane attended Professor Miller's {} at the university, she was not intested in Chinese art. However, now, she wants to learn more about it. lecture(3.845),comment(4.133),furniture(4.239),shipment(4.416) 1 True pre2 perplexity
At first, the marketing department and the sales department were on the project together. But people in the sales department were too busy, so now the project is being run {} by the marketing department. entirely(2.498),needlessly(2.740),violently(2.816),scientifically(2.817) 1 True 2 perplexity
Experts at the art gallery discovered that one of their paitings, which they had thought was a {} Picasso, was actually just a copy. genuine(3.079),portable(3.520),severe(3.677),logical(3.689) 1 True 2 perplexity
The musician Jimmy Baker had a lot of {} when he was a child. His family was very poor before he became a rich and famous rock star. hardship(3.326),concentration(3.565),membership(3.583),permission(3.654) 1 True 2 perplexity
Mother Teresa helped many sick people and gave food to many hungry children in India. She was known as a person who cared about {}. humanity(2.718),generation(3.180),gravity(3.184),hesitation(3.352) 1 True 2 perplexity
As Liam waled down the dark street, he began to feel afraid. He had the {} that someone was watching him. sensation(3.421),feature(4.111),translation(4.266),property(4.431) 1 True 2 perplexity
Risa buys water that comes from a mountain stream. She says that drinking it is good because it has many {} that her body needs. minerals(3.101),operations(3.727),illustrations(4.007),campaigns(4.023) 1 True 2 perplexity
The lifeguard ran into the ocean to help a young girl who looked like she was {} in the big waves drowning(3.241),proposing(4.074),converting(4.089),exporting(4.204) 1 True 2 perplexity
Yesterday was a hot day at the zoo, so Heather bought an ice cream. It melted so quickly that she could not help {} some on her dress. spilling(3.160),maintaining(3.649),arguing(3.892),convincing(3.983) 1 True 2 perplexity
In the past, sailors had to use the stars to {} when they were on an ocean. These days, ships have modern equipment that shows sailors which way to go. navigate(3.120),respect(3.493),satisfy(3.554),permit(3.666) 1 True 2 perplexity
Daisuke's grandmother eats a lot of vegetables, drinks green tea, and goes for a long walk every evening to {} her health. preserve(3.041),replace(3.377),betray(3.537),interpret(3.599) 1 True 2 perplexity
A: Thanks for showing me the outline of your sales presentation. It's good, but it's a bit {} in some places. B: I guess I do repeat some information too much. I'll try to take some of it out. redundant(3.329),subjective(3.454),distinct(3.513),decisive(3.533) 1 True pre1 perplexity
Lisa went to the interview even though she thought there was a low {} of her getting the job. As she expected, she was not hired. probability(2.479),credibility(3.207),contention(3.268),restoration(3.370) 1 True pre1 perplexity
It is sadly {} that, in developing counties, many of the farmers who grow nutritous crops for export do no not have enough food to feed their own families. ironic(3.586),superficial(4.055),indefinite(4.084),restless(4.266) 1 True pre1 perplexity
The explosion at the chemical factory {} great damage on the local environment. It will take years for wildlife to fully recover in the region. inflicted(3.238),perceived(3.704),enhanced(3.826),vanished(4.010) 1 True pre1 perplexity
Some say the best way to overcome a {} is to expose oneself to what one fears. For example, people who are afraid of mice should try holding one. phobia(2.733),temptation(3.107),barricade(3.298),famine(3.366) 1 True pre1 perplexity
English classes at the university were required, but students were {} from them if they could prove they had advanced ability in the language. exempted(3.016),qualified(3.583),prosecuted(3.803),commanded(3.943) 1 True pre1 perplexity
E-mail and text messaging have {} the way people write. Many people shorten words and ignore traditional rules of grammar. transformed(4.060),synthesized(4.406),disarmed(4.470),officiated(4.625) 1 True pre1 perplexity
Some analysts think the new treaty on CO2 emissions is a {} in the fight against global warming. "This is the most important environmental treaty ever signed," said one. milestone(2.537),vigor(2.962),backlog(3.035),confession(3.072) 1 True pre1 perplexity
Lying on the sunny beach with her husband on their vacation, Roberta felt {} happy. She had never been so content. profoundly(3.576),barely(3.864),harshly(4.001),improperly(4.038) 1 True pre1 perplexity
Nadine spends an hour thoroughly cleaning her apartment evey day, so the entire place is {}. spotless(3.993),rugged(4.641),impartial(4.788),minute(4.859) 1 True pre1 perplexity
Cell phones have become a permanent {} in modern society. Most perople could not imagine living without one. fixture(3.340),rupture(4.229),clasp(4.242),stint(4.295) 1 True 1 perplexity
Colin did not have enough money to pay for the car all at onece, so he paid it off in {} of $800 a month for two years. installments(3.012),dispositions(3.537),speculations(3.562),enactments(3.611) 1 True 1 perplexity
When she asked her boss for a raise, Melanie's {} tone of voice made it obvious how nervous she was. jovial(3.002),diffident(3.208),pompous(3.282),garish(3.393) 2 False 1 perplexity
The religious sect established a {} in a rural area where its followers could live together and share everything. No private property was allowed. commune(3.466),repository(3.890),prelude(4.160),dirge(4.547) 1 True 1 perplexity
The famous reporter was fired for {} another journalist's work. His article was almost exactly the same as that of the other journalist. plagiarizing(2.819),beleaguering(3.335),inoculating(3.466),alleviating(3.512) 1 True 1 perplexity
Now that the local steel factory has closed down, the streets of the once-busy town are lined with {} businesses. Most owners have abandoned their stores. defunct(3.112),volatile(3.456),aspiring(3.502),rhetorical(3.706) 1 True 1 perplexity
The ambassador's failure to attend the ceremony held in honor of the king was considered an {} by his host nation and made already bad relations worse. affront(3.527),ultimatum(3.723),impasse(3.901),elucidation(4.211) 1 True 1 perplexity
US border guards managed to {} the escaped prisoner as he tried to cross into Canada. He was returned to jail immediately. apprehend(3.144),acclimate(3.557),pillage(3.580),exalt(4.100) 1 True 1 perplexity
Anthony enjoyed his first day at his new job. The atmosphere was {}, and his colleagues did their best to make him feel welcome. congenial(2.469),delirious(2.613),measly(2.855),implausible(2.907) 1 True 1 perplexity
A: I just learned I've been {} to second violin in the school orchestra. I knew I should've practiced more.B: Well, if you work hard, I'm sure you can get your previous position back. relegated(3.020),jeopardized(3.433),reiterated(3.448),stowed(3.478) 1 True 1 perplexity
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment