Created
December 18, 2019 03:07
-
-
Save jsomers/1bb5e197dec221714df250e72265a301 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "cells": [ | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "# Automatically finding Codenames clues with GloVe vectors\n", | |
| "\n", | |
| "by [James Somers](http://jsomers.net)\n", | |
| "\n", | |
| "*Abstract: A simple vector-space model shows a surprising talent for cluing in the Codenames board game.*\n", | |
| "\n", | |
| "# What is Codenames?\n", | |
| "\n", | |
| "<a href=\"https://en.wikipedia.org/wiki/Codenames_(board_game)\">Codenames</a> is a Czech board game where the goal is to say a one-word clue to your teammates in order to get them to choose correctly from the words laid out on the table. The real game is played on a 5x5 board, but here is a typical situation faced by a clue-giver:\n", | |
| "\n", | |
| "\n", | |
| "\n", | |
| "The three blue words are the target words—that's what you want your teammates to guess. The black word is the bomb; if your teammates say that one, they instantly lose the game. The tan words are neutral or perhaps belong to your opponent.\n", | |
| "\n", | |
| "Your task is to come up with a single word that connects HAM, BEIJING, and IRON, while avoiding the others. (There are [rules](https://czechgames.com/files/rules/codenames-rules-en.pdf) about which kinds of clues are allowable: usually it has to be a single word; proper nouns are optionally allowed.)\n", | |
| "\n", | |
| "The game is interesting because it requires you to connect far-flung concepts precisely enough that other people can re-create your associations. It can be delightful, and frustrating, to see your friends' minds leap from idea to idea—often going places you never intended.\n", | |
| "\n", | |
| "Can you think of a clue for the board above?" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "# What are GloVe vectors?\n", | |
| "\n", | |
| "\"Word vectors\" attempt to quantify meaning by plotting words in a high-dimensional space; words that are semantically related end up close to each other in the space.\n", | |
| "\n", | |
| "One [way](https://en.wikipedia.org/wiki/Word2vec) to generate word vectors uses a neural network. You download a vast corpus of text, say all of Wikipedia. Then, you read the text into a small moving window, considering maybe ten words at a time—nine \"context\" words and one target word. Your goal is to predict the target from the context: you rejigger the weights of the network such that, based on the nine context words, it assigns a high probability to the tenth. At the heart of this neural network is a big matrix which has a column vector for each word. In the training process, you're esssentially nudging these vectors around. After visiting the entire corpus this way, the vectors turn out to encode whatever regularities there are in the way the words are used.\n", | |
| "\n", | |
| "It's a computationally intense procedure. Luckily, Stanford has published a data set of pre-trained vectors, the [Global Vectors for Word Representation](https://nlp.stanford.edu/projects/glove/), or GloVe for short. (It uses a slightly fancier method than the one described above.) It's just a list of words followed by 300 numbers, each number referring to a coordinate of that word's vector in a 300-dimensional space. The GloVe vectors we'll be using were trained on 42 billion words worth of text gotten from the [Common Crawl](https://commoncrawl.org/).\n", | |
| "\n", | |
| "# Codenames as an AI problem\n", | |
| "\n", | |
| "Codenames seems like a good Turing test: to come up with a clue, you need to not only understand the many shades of meaning each word can take on—\"PAN,\" for instance, can be a piece of kitchenware, a way of criticizing, or a prefix meaning \"all\"—you also seem to need a model of the world. You connect \"NARWHAL\" to \"NET\" because you know that narwhals might be caught in nets. You connect \"GRENADE\" to \"PALM\" because you know that grenades are held in your hand; when you think of the two words together, you might even mentally simulate a throw. All this seems difficult for a computer to do.\n", | |
| "\n", | |
| "But if we recast the problem in terms of our vector space model, where distance is a measure of semantic similarity, then finding a good Codenames clue becomes about finding a word that is close to the target words while being far away from all the others. That sounds a little simpler.\n", | |
| "\n", | |
| "# Getting the data\n", | |
| "\n", | |
| "Download the vectors:\n", | |
| "\n", | |
| "```sh\n", | |
| "$ wget http://nlp.stanford.edu/data/glove.42B.300d.zip\n", | |
| "$ unzip glove.42B.300d.zip\n", | |
| "```\n", | |
| "\n", | |
| "The words are sorted by the number of times they appear in the original corpus. Each word has a list of 300 coordinates associated with it. Here are the word vectors for `was`, `or`, and `your`:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 1, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "[('was',\n", | |
| " '-0.0422, -0.00044414, 0.052895, -0.051688, 0.013487, -0.79987, -3.6616, 0.47123, 0.014875, -0.58004, -0.050214, -0.25385, -0.22905, -0.56836, 0.013797, 0.23938, -0.28826, -0.04298, 0.2424...'),\n", | |
| " ('or',\n", | |
| " '0.23333, -0.30334, -0.44491, -0.030651, 0.15669, 0.16303, -4.3474, 0.75635, -0.20263, -0.30256, 0.95183, -0.41293, 0.065988, -0.27925, -0.33301, 0.028757, -0.48017, -0.087209, 0.33913...'),\n", | |
| " ('your',\n", | |
| " '0.31261, -0.21024, -0.29676, 0.042702, -0.010114, -0.057844, -4.7176, 0.52637, -0.080199, -0.54652, 0.1178, 0.034668, 0.56859, 0.070415, -0.013684, -0.049383, 0.20602, -0.048774, 0.13903...')]" | |
| ] | |
| }, | |
| "execution_count": 1, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "with open(\"./glove.42B.300d.txt\", 'r') as glove:\n", | |
| " lines = [next(glove) for x in xrange(100)]\n", | |
| "[(l.split(\" \")[0], \", \".join(l.split(\" \")[1:20]) + \"...\") for l in lines[30:33]]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "There are more than a million words in this file, which makes processing slow. So we'll write the top 50,000 words to a separate file:\n", | |
| "\n", | |
| "```sh\n", | |
| "$ head -n 50000 glove.42B.300d.txt > top_50000.txt\n", | |
| "```\n", | |
| "\n", | |
| "Now we're ready to do some analysis.\n", | |
| "\n", | |
| "# Finding words that are close to one another\n", | |
| "\n", | |
| "We'll import some common libraries for numerical analysis:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 2, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "import numpy as np\n", | |
| "import pandas as pd\n", | |
| "from scipy import spatial" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "Then, we'll create a map from words to their \"embeddings\", i.e., their 300-dimensional vector representations:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 3, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "embeddings = {}\n", | |
| "with open(\"./top_50000.txt\", 'r') as f:\n", | |
| " for line in f:\n", | |
| " values = line.split()\n", | |
| " word = values[0]\n", | |
| " vector = np.asarray(values[1:], \"float32\")\n", | |
| " embeddings[word] = vector" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "We can see which words are close to others by taking their [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity)—a measure of distance in high-dimensional space that computes the angle between two vectors:\n", | |
| "\n", | |
| "$$distance(x, y) = 1 - cos(x, y) = 1 - \\frac {\\pmb x \\cdot \\pmb y}{||\\pmb x|| \\cdot ||\\pmb y||}$$\n", | |
| "\n", | |
| "With a quick look at some neighboring words, we can see that the distance metric works pretty well:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 4, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "[('magic',\n", | |
| " 'magical, spells, wizard, spell, trick, magician, enchanted, mystical, wonder...'),\n", | |
| " ('sport',\n", | |
| " 'sports, sporting, racing, football, soccer, rugby, cycling, tennis, golf...'),\n", | |
| " ('scuba',\n", | |
| " 'diving, dive, snorkeling, divers, diver, snorkel, kayaking, snorkelling, windsurfing...'),\n", | |
| " ('sock',\n", | |
| " 'socks, yarn, shoe, knit, knitted, knitting, fetish, stocking, toe...')]" | |
| ] | |
| }, | |
| "execution_count": 4, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "def distance(word, reference):\n", | |
| " return spatial.distance.cosine(embeddings[word], embeddings[reference])\n", | |
| "\n", | |
| "def closest_words(reference):\n", | |
| " return sorted(embeddings.keys(), key=lambda w: distance(w, reference))\n", | |
| "\n", | |
| "[(w, \", \".join(closest_words(w)[1:10]) + \"...\") for w in [\"magic\", \"sport\", \"scuba\", \"sock\"]]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "# The model\n", | |
| "\n", | |
| "We can express the Codenames problem as taking a set of \"target\" words and a set of \"bad\" words, then trying to find candidate words that are close to the targets and far from the bad words.\n", | |
| "\n", | |
| "One way to do this is to calculate, for a given candidate clue, the sum of its distances from the bad words minus the sum of its distances from the target words. (When the target distances are smaller, it means the candidate is better.) That is, for each word $w$ in our dictionary we want to compute:\n", | |
| "\n", | |
| "$$goodness(w) = \\sum_{b \\in bad}{cos(w, v_b)} - c \\cdot \\sum_{t \\in targets}{cos(w, v_t)}$$\n", | |
| "\n", | |
| "Then we pick the words with the highest values—say, the top 350 of them. (The constant $c>0$ expresses the fact that closeness to the target words is more important than farness from the bad words.)\n", | |
| "\n", | |
| "The trouble is that a candidate that is close to one or two of the targets but far from the third can still score well—despite being a bad clue for that very reason. So, we sort our subset of 350 good candidates by the following:\n", | |
| "\n", | |
| "$$minimax(w) = \\underset{b \\in bad}{\\operatorname{arg min}}{cos(w, v_b)} - \\underset{t \\in targets}{\\operatorname{arg max}}{cos(w, v_t)}$$\n", | |
| "\n", | |
| "That is, we're looking to minimize the maximum distance from the targets, and maximize the mininum distance from the bad words. This is all pretty easy to express in code:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 5, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "def goodness(word, answers, bad):\n", | |
| " if word in answers + bad: return -999\n", | |
| " return sum([distance(word, b) for b in bad]) - 4.0 * sum([distance(word, a) for a in answers])\n", | |
| "\n", | |
| "def minimax(word, answers, bad):\n", | |
| " if word in answers + bad: return -999\n", | |
| " return min([distance(word, b) for b in bad]) - max([distance(word, a) for a in answers])\n", | |
| "\n", | |
| "def candidates(answers, bad, size=100):\n", | |
| " best = sorted(embeddings.keys(), key=lambda w: -1 * goodness(w, answers, bad))\n", | |
| " res = [(str(i + 1), \"{0:.2f}\".format(minimax(w, answers, bad)), w) for i, w in enumerate(sorted(best[:250], key=lambda w: -1 * minimax(w, answers, bad))[:size])]\n", | |
| " return [(\". \".join([c[0], c[2]]) + \" (\" + c[1] + \")\") for c in res]" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "# Experimental setup\n", | |
| "\n", | |
| "I've been playing lots of Codenames with my friends and have gathered some data along the way. In the \"experiments,\" there are 16 players who participate. Four players are assigned randomly to the same 3x3 board, like the one above, and are asked to give a clue independently to three receivers apiece. (The receivers don't see the colors on the board, obviously.)\n", | |
| "\n", | |
| "The scoring works as follows:\n", | |
| "\n", | |
| "- Just like in the real game, when you guess an incorrect square, you're penalized. Here, you stop earning points.\n", | |
| "- You get 1 point for the first correct answer, 2 points for the second, and 3 points for the third.\n", | |
| "- Thus, scores for a round can be 0, 1, 3, or 6 points.\n", | |
| "\n", | |
| "These experiments give a baseline of human performance, which can then be compared against the vector-space model.\n", | |
| "\n", | |
| "# Results\n", | |
| "\n", | |
| "## 1. The best clue emerges\n", | |
| "\n", | |
| "For instance, with the board above, we had the following clues and results:\n", | |
| "\n", | |
| "<img src=\"https://user-images.githubusercontent.com/21294/70329766-03d6db80-1834-11ea-89cb-112902280f65.png\" style=\"margin-left: 0;\">\n", | |
| "\n", | |
| "<div> </div>\n", | |
| "<div style=\"margin-left: 0; width: 300px;\">\n", | |
| "<b>Cluer 1: PIG</b>\n", | |
| "<table><col width='200'/><col width='200'/><col width='200'/><col width='200'/>\n", | |
| "<tr>\n", | |
| "<td>Receiver 4</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>HAM</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>BEAR</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>CAT</td>\n", | |
| "<td>1 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 5</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>HAM</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>BEIJING</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>IRON</td>\n", | |
| "<td>6 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 6</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>HAM</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>IRON</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>BEIJING</td>\n", | |
| "<td>6 pts</td>\n", | |
| "</tr>\n", | |
| "</table>\n", | |
| "<div> </div>\n", | |
| "<b>Cluer 2: CAIDAO <em>(Chinese for vegetable cleaver)</em></b>\n", | |
| "<table><col width='200'/><col width='200'/><col width='200'/><col width='200'/>\n", | |
| "<tr>\n", | |
| "<td>Receiver 7</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>BEIJING</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>BEAR</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>CAT</td>\n", | |
| "<td>1 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 8</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>BEIJING</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>IRON</td> <td style='text-align: center; background-color: black; color: white; padding: 3px;'>FALL</td>\n", | |
| "<td>0 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 9</td>\n", | |
| "<td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>WITCH</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>AMBULANCE</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>NOTE</td>\n", | |
| "<td>0 pts</td>\n", | |
| "</tr>\n", | |
| "</table>\n", | |
| "<div> </div>\n", | |
| "<b>Cluer 3: COMMODITIES</b>\n", | |
| "<table><col width='200'/><col width='200'/><col width='200'/><col width='200'/>\n", | |
| "<tr>\n", | |
| "<td>Receiver 10</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>IRON</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>HAM</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>BEAR</td>\n", | |
| "<td>3 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 11</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>IRON</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>NOTE</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>HAM</td>\n", | |
| "<td>1 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 12</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>IRON</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>HAM</td> <td style='text-align: center; background-color: black; color: white; padding: 3px;'>FALL</td>\n", | |
| "<td>0 pts</td>\n", | |
| "</tr>\n", | |
| "</table>\n", | |
| "<div> </div>\n", | |
| "<b>Cluer 4: WOK</b>\n", | |
| "<table><col width='200'/><col width='200'/><col width='200'/><col width='200'/>\n", | |
| "<tr>\n", | |
| "<td>Receiver 1</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>BEIJING</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>IRON</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>HAM</td>\n", | |
| "<td>6 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 2</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>IRON</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>BEIJING</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>HAM</td>\n", | |
| "<td>6 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 3</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>BEIJING</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>HAM</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>IRON</td>\n", | |
| "<td>6 pts</td>\n", | |
| "</tr>\n", | |
| "</table>\n", | |
| "</div>\n", | |
| "\n", | |
| "Clearly \"WOK\" was the best clue. \"CAIDAO\" might have been a good clue except that none of the receivers understood what it meant. \"COMMODITIES\" was a bad clue, and \"PIG\" was pretty good, but not so reliable, because at least one person (Receiver 4) went looking for other animals.\n", | |
| "\n", | |
| "Let's see what the computer comes up with. We'll print the first 100 candidates using the function above. The number in parens is the minimax score that we're sorting by:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 6, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "import pandas as pd\n", | |
| "from itertools import izip_longest\n", | |
| "\n", | |
| "def grouper(n, iterable, fillvalue=None):\n", | |
| " args = [iter(iterable)] * n\n", | |
| " return izip_longest(fillvalue=fillvalue, *args)\n", | |
| "\n", | |
| "from IPython.display import HTML\n", | |
| "\n", | |
| "def tabulate(data):\n", | |
| " data = list(grouper(10, data))\n", | |
| " return HTML(pd.DataFrame(data).to_html(index=False, header=False))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 7, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/html": [ | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <td>1. tong (0.10)</td>\n", | |
| " <td>2. wok (0.10)</td>\n", | |
| " <td>3. guan (0.07)</td>\n", | |
| " <td>4. kitchenware (0.07)</td>\n", | |
| " <td>5. nippon (0.06)</td>\n", | |
| " <td>6. torino (0.03)</td>\n", | |
| " <td>7. thanh (0.03)</td>\n", | |
| " <td>8. jian (0.03)</td>\n", | |
| " <td>9. bao (0.03)</td>\n", | |
| " <td>10. jia (0.02)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>11. omelette (0.02)</td>\n", | |
| " <td>12. sheng (0.01)</td>\n", | |
| " <td>13. ge (0.01)</td>\n", | |
| " <td>14. savoy (0.01)</td>\n", | |
| " <td>15. lyon (0.01)</td>\n", | |
| " <td>16. shu (0.01)</td>\n", | |
| " <td>17. intercontinental (0.01)</td>\n", | |
| " <td>18. buffet (0.01)</td>\n", | |
| " <td>19. stir-fry (0.01)</td>\n", | |
| " <td>20. badminton (0.00)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>21. guo (0.00)</td>\n", | |
| " <td>22. steamed (0.00)</td>\n", | |
| " <td>23. dijon (-0.01)</td>\n", | |
| " <td>24. xin (-0.01)</td>\n", | |
| " <td>25. kowloon (-0.01)</td>\n", | |
| " <td>26. hua (-0.01)</td>\n", | |
| " <td>27. cantonese (-0.01)</td>\n", | |
| " <td>28. sichuan (-0.01)</td>\n", | |
| " <td>29. peking (-0.01)</td>\n", | |
| " <td>30. tofu (-0.01)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>31. yi (-0.01)</td>\n", | |
| " <td>32. vietnamese (-0.01)</td>\n", | |
| " <td>33. cuisine (-0.01)</td>\n", | |
| " <td>34. xian (-0.02)</td>\n", | |
| " <td>35. pan (-0.02)</td>\n", | |
| " <td>36. yan (-0.02)</td>\n", | |
| " <td>37. molybdenum (-0.02)</td>\n", | |
| " <td>38. tenderloin (-0.02)</td>\n", | |
| " <td>39. tottenham (-0.02)</td>\n", | |
| " <td>40. swiss (-0.02)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>41. xiang (-0.02)</td>\n", | |
| " <td>42. hai (-0.02)</td>\n", | |
| " <td>43. shen (-0.02)</td>\n", | |
| " <td>44. turkish (-0.03)</td>\n", | |
| " <td>45. dong (-0.03)</td>\n", | |
| " <td>46. cookware (-0.03)</td>\n", | |
| " <td>47. xu (-0.03)</td>\n", | |
| " <td>48. mongolian (-0.03)</td>\n", | |
| " <td>49. hock (-0.03)</td>\n", | |
| " <td>50. noodles (-0.03)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>51. wembley (-0.03)</td>\n", | |
| " <td>52. hu (-0.04)</td>\n", | |
| " <td>53. cn (-0.04)</td>\n", | |
| " <td>54. omelet (-0.04)</td>\n", | |
| " <td>55. selenium (-0.04)</td>\n", | |
| " <td>56. yunnan (-0.04)</td>\n", | |
| " <td>57. pork (-0.04)</td>\n", | |
| " <td>58. jing (-0.04)</td>\n", | |
| " <td>59. ware (-0.04)</td>\n", | |
| " <td>60. seafood (-0.04)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>61. wei (-0.04)</td>\n", | |
| " <td>62. restaurant (-0.04)</td>\n", | |
| " <td>63. newcastle (-0.05)</td>\n", | |
| " <td>64. qingdao (-0.05)</td>\n", | |
| " <td>65. jiang (-0.05)</td>\n", | |
| " <td>66. barbecue (-0.05)</td>\n", | |
| " <td>67. gong (-0.05)</td>\n", | |
| " <td>68. ore (-0.05)</td>\n", | |
| " <td>69. hongkong (-0.05)</td>\n", | |
| " <td>70. minh (-0.05)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>71. tian (-0.05)</td>\n", | |
| " <td>72. corned (-0.05)</td>\n", | |
| " <td>73. jiangsu (-0.05)</td>\n", | |
| " <td>74. arsenal (-0.05)</td>\n", | |
| " <td>75. manganese (-0.05)</td>\n", | |
| " <td>76. glaze (-0.06)</td>\n", | |
| " <td>77. palace (-0.06)</td>\n", | |
| " <td>78. thai (-0.06)</td>\n", | |
| " <td>79. barbeque (-0.06)</td>\n", | |
| " <td>80. panini (-0.06)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>81. beef (-0.06)</td>\n", | |
| " <td>82. dishes (-0.06)</td>\n", | |
| " <td>83. zhao (-0.06)</td>\n", | |
| " <td>84. rice (-0.06)</td>\n", | |
| " <td>85. cheng (-0.06)</td>\n", | |
| " <td>86. brussel (-0.06)</td>\n", | |
| " <td>87. qatar (-0.07)</td>\n", | |
| " <td>88. pho (-0.07)</td>\n", | |
| " <td>89. shandong (-0.07)</td>\n", | |
| " <td>90. premier (-0.07)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>91. tianjin (-0.07)</td>\n", | |
| " <td>92. yong (-0.07)</td>\n", | |
| " <td>93. manchester (-0.07)</td>\n", | |
| " <td>94. ceramics (-0.08)</td>\n", | |
| " <td>95. cooking (-0.08)</td>\n", | |
| " <td>96. tai (-0.08)</td>\n", | |
| " <td>97. tang (-0.08)</td>\n", | |
| " <td>98. ming (-0.08)</td>\n", | |
| " <td>99. ping (-0.08)</td>\n", | |
| " <td>100. lu (-0.08)</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.HTML object>" | |
| ] | |
| }, | |
| "execution_count": 7, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "answers = [\"iron\", \"ham\", \"beijing\"]\n", | |
| "bad = [\"fall\", \"witch\", \"note\", \"cat\", \"bear\", \"ambulance\"]\n", | |
| "\n", | |
| "tabulate(candidates(answers, bad))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "I find these results pretty striking. The model here is simple geometry; it relies entirely on the meaning baked into the GloVe vectors. But `wok` appears! `wok` is basically a perfect clue—everyone was impressed with the friend who came up with it and upset they hadn't thought of it themselves—and here it is in the \\#2 spot, out of 50,000 candidates. `tong` (\\#1) might work well, though I don't quite understand the connection to \"Beijing,\" and `jian` (\\#8), a word I hadn't heard before, fits decently well: it is a kind of Chinese sword. `stir-fry` (\\#19) and `sichuan` (\\#28) seem to evoke Chinese cooking. Notably, all of these clues are vastly better than \"COMMODITIES,\" which is the one I came up with.\n", | |
| "\n", | |
| "Of course, there's plenty of garbage (`molybdenum` (\\#37) (??), `qatar` (\\#87) (!?)), and many of the candidates are over-indexed to one or two of the targets at the expense of others. `hock` (\\#49), for instance, doesn't have anything to do with \"Iron\" or \"Beijing,\" and `omelette` (\\#45), although connected to \"Ham\" and \"Iron,\" is unrelated to \"Beijing.\"\n", | |
| "\n", | |
| "I experimented with different scoring models—I tried taking the product of the distances, and the mean; I tried using the [logit](https://en.wikipedia.org/wiki/Logit) function to \"spread out\" the cosine similarity measure, so that the reward for closeness grew exponentially. And I played with the constant $c$. But so far, the model above gives the best overall performance across the largest number of scenarios.\n", | |
| "\n", | |
| "(It's probably worth saying that later, I tried a board with BEIJING, GREEN, and WORM as targets, and many of these same words appeared: `jian`, `tong`, `tian`, `sichuan`. Same if GREEN were changed to LAPTOP, but not when changed to DEER. So perhaps \"Beijing\" alone had conjured them up, and to some extent, the model got lucky.)\n", | |
| "\n", | |
| "## 2. A harder board—and superhuman performance?\n", | |
| "\n", | |
| "This was a doozy:\n", | |
| "\n", | |
| "<img src=\"https://user-images.githubusercontent.com/21294/70329852-3680d400-1834-11ea-84e7-3ba9df4248f6.png\" style=\"margin-left: 0;\">\n", | |
| "\n", | |
| "<div style=\"margin-left: 0; width: 300px;\">\n", | |
| "<br/><b>Cluer 1: MALTA</b>\n", | |
| "<table><col width='200'/><col width='200'/><col width='200'/><col width='200'/>\n", | |
| "<tr>\n", | |
| "<td>Receiver 1</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ATLANTIS</td> <td style='text-align: center; background-color: black; color: white; padding: 3px;'>FAIR</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>HOSPITAL</td>\n", | |
| "<td>0 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 2</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>CHURCH</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ATLANTIS</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>CAT</td>\n", | |
| "<td>6 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 3</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ATLANTIS</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>CAT</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>BUCK</td>\n", | |
| "<td>3 pts</td>\n", | |
| "</tr>\n", | |
| "</table>\n", | |
| "<br/><br/><b>Cluer 2: MYSTIC</b>\n", | |
| "<table><col width='200'/><col width='200'/><col width='200'/><col width='200'/>\n", | |
| "<tr>\n", | |
| "<td>Receiver 4</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ATLANTIS</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>AZTEC</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>CHURCH</td>\n", | |
| "<td>1 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 5</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ATLANTIS</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>EYE</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>AZTEC</td>\n", | |
| "<td>1 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 6</td>\n", | |
| "<td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>AZTEC</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>EYE</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>CHURCH</td>\n", | |
| "<td>0 pts</td>\n", | |
| "</tr>\n", | |
| "</table>\n", | |
| "<br/><br/><b>Cluer 3: ASLAN</b>\n", | |
| "<table><col width='200'/><col width='200'/><col width='200'/><col width='200'/>\n", | |
| "<tr>\n", | |
| "<td>Receiver 7</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ATLANTIS</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>CHURCH</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>AZTEC</td>\n", | |
| "<td>3 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 8</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>CAT</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>CHURCH</td> <td style='text-align: center; background-color: black; color: white; padding: 3px;'>FAIR</td>\n", | |
| "<td>0 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 9</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>CAT</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>CHURCH</td> <td style='text-align: center; background-color: black; color: white; padding: 3px;'>FAIR</td>\n", | |
| "<td>0 pts</td>\n", | |
| "</tr>\n", | |
| "</table>\n", | |
| "<br/><br/><b>Cluer 4: MYSTICAL</b>\n", | |
| "<table><col width='200'/><col width='200'/><col width='200'/><col width='200'/>\n", | |
| "<tr>\n", | |
| "<td>Receiver 10</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ATLANTIS</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>AZTEC</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>EYE</td>\n", | |
| "<td>1 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 11</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ATLANTIS</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>AZTEC</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>CHURCH</td>\n", | |
| "<td>1 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 12</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>CHURCH</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>AZTEC</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ATLANTIS</td>\n", | |
| "<td>1 pts</td>\n", | |
| "</tr>\n", | |
| "</table>\n", | |
| "</div>\n", | |
| "\n", | |
| "Only a single player managed to guess all three correctly, via the clue \"MALTA.\" Much to my surprise, that clue appeared 12<sup>th</sup> on the model's list:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 8, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/html": [ | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <td>1. ark (0.04)</td>\n", | |
| " <td>2. archangel (0.02)</td>\n", | |
| " <td>3. darwin (0.01)</td>\n", | |
| " <td>4. evolution (0.00)</td>\n", | |
| " <td>5. graveyard (0.00)</td>\n", | |
| " <td>6. destiny (0.00)</td>\n", | |
| " <td>7. islands (-0.00)</td>\n", | |
| " <td>8. bahamas (-0.01)</td>\n", | |
| " <td>9. cave (-0.02)</td>\n", | |
| " <td>10. pool (-0.02)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>11. mystery (-0.02)</td>\n", | |
| " <td>12. malta (-0.02)</td>\n", | |
| " <td>13. sanctuary (-0.02)</td>\n", | |
| " <td>14. island (-0.02)</td>\n", | |
| " <td>15. paradise (-0.02)</td>\n", | |
| " <td>16. ii (-0.03)</td>\n", | |
| " <td>17. lighthouse (-0.03)</td>\n", | |
| " <td>18. series (-0.04)</td>\n", | |
| " <td>19. caribbean (-0.04)</td>\n", | |
| " <td>20. cove (-0.04)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>21. swimming (-0.04)</td>\n", | |
| " <td>22. lost (-0.05)</td>\n", | |
| " <td>23. universe (-0.05)</td>\n", | |
| " <td>24. tower (-0.05)</td>\n", | |
| " <td>25. kingdom (-0.05)</td>\n", | |
| " <td>26. isle (-0.05)</td>\n", | |
| " <td>27. stargate (-0.05)</td>\n", | |
| " <td>28. mysteries (-0.06)</td>\n", | |
| " <td>29. destroyed (-0.06)</td>\n", | |
| " <td>30. journey (-0.06)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>31. magnificent (-0.06)</td>\n", | |
| " <td>32. ship (-0.06)</td>\n", | |
| " <td>33. advent (-0.06)</td>\n", | |
| " <td>34. discovery (-0.06)</td>\n", | |
| " <td>35. castle (-0.06)</td>\n", | |
| " <td>36. ocean (-0.06)</td>\n", | |
| " <td>37. beach (-0.06)</td>\n", | |
| " <td>38. views (-0.06)</td>\n", | |
| " <td>39. adventures (-0.06)</td>\n", | |
| " <td>40. secret (-0.07)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>41. scripture (-0.07)</td>\n", | |
| " <td>42. space (-0.07)</td>\n", | |
| " <td>43. voyage (-0.07)</td>\n", | |
| " <td>44. club (-0.07)</td>\n", | |
| " <td>45. built (-0.07)</td>\n", | |
| " <td>46. escape (-0.08)</td>\n", | |
| " <td>47. sermon (-0.08)</td>\n", | |
| " <td>48. eden (-0.08)</td>\n", | |
| " <td>49. sea (-0.08)</td>\n", | |
| " <td>50. build (-0.08)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>51. stories (-0.08)</td>\n", | |
| " <td>52. treasure (-0.08)</td>\n", | |
| " <td>53. queen (-0.08)</td>\n", | |
| " <td>54. souls (-0.08)</td>\n", | |
| " <td>55. king (-0.08)</td>\n", | |
| " <td>56. story (-0.08)</td>\n", | |
| " <td>57. vatican (-0.08)</td>\n", | |
| " <td>58. quest (-0.08)</td>\n", | |
| " <td>59. heaven (-0.08)</td>\n", | |
| " <td>60. supper (-0.08)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>61. planet (-0.09)</td>\n", | |
| " <td>62. grand (-0.09)</td>\n", | |
| " <td>63. liturgy (-0.09)</td>\n", | |
| " <td>64. apostles (-0.09)</td>\n", | |
| " <td>65. boat (-0.09)</td>\n", | |
| " <td>66. adventure (-0.09)</td>\n", | |
| " <td>67. theology (-0.09)</td>\n", | |
| " <td>68. century (-0.09)</td>\n", | |
| " <td>69. bible (-0.09)</td>\n", | |
| " <td>70. earth (-0.10)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>71. biblical (-0.10)</td>\n", | |
| " <td>72. rescue (-0.10)</td>\n", | |
| " <td>73. ascension (-0.10)</td>\n", | |
| " <td>74. lord (-0.10)</td>\n", | |
| " <td>75. neptune (-0.10)</td>\n", | |
| " <td>76. moon (-0.11)</td>\n", | |
| " <td>77. apostolic (-0.11)</td>\n", | |
| " <td>78. augustine (-0.11)</td>\n", | |
| " <td>79. christianity (-0.11)</td>\n", | |
| " <td>80. rock (-0.11)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>81. revelation (-0.12)</td>\n", | |
| " <td>82. easter (-0.12)</td>\n", | |
| " <td>83. prophecy (-0.12)</td>\n", | |
| " <td>84. towers (-0.12)</td>\n", | |
| " <td>85. apollo (-0.12)</td>\n", | |
| " <td>86. divine (-0.13)</td>\n", | |
| " <td>87. stone (-0.13)</td>\n", | |
| " <td>88. turtle (-0.13)</td>\n", | |
| " <td>89. resurrection (-0.13)</td>\n", | |
| " <td>90. greek (-0.13)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>91. ghost (-0.14)</td>\n", | |
| " <td>92. believers (-0.14)</td>\n", | |
| " <td>93. reformed (-0.14)</td>\n", | |
| " <td>94. jesus (-0.14)</td>\n", | |
| " <td>95. temple (-0.14)</td>\n", | |
| " <td>96. christ (-0.14)</td>\n", | |
| " <td>97. orthodox (-0.14)</td>\n", | |
| " <td>98. abandoned (-0.14)</td>\n", | |
| " <td>99. sacred (-0.14)</td>\n", | |
| " <td>100. god (-0.14)</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.HTML object>" | |
| ] | |
| }, | |
| "execution_count": 8, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "answers = [\"church\", \"cat\", \"atlantis\"]\n", | |
| "bad = [\"fair\", \"eye\", \"aztec\", \"buck\", \"pin\", \"hospital\"]\n", | |
| "\n", | |
| "tabulate(candidates(answers, bad))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "Perhaps more surprising is the model's top pick, `ark`. I tried this clue on a friend who wasn't part of the initial experiment; they guessed all three targets correctly. Indeed `ark` might be a strictly better clue than \"MALTA.\" (I like how it connects both to \"Church\" and to \"Cat,\" and actually also to \"Atlantis\"—boat, island...—though it has a little interference with \"Buck,\" which is also an animal that might end up on Noah's Ark.)\n", | |
| "\n", | |
| "Note also `mystery` (\\#11) and `mysteries` (#28), reminiscent of Cluer 2's \"MYSTIC\" and Cluer 4's \"MYSTICAL.\" `aslan` didn't have a chance of appearing since it didn't make the original cutoff for inclusion in the dictionary (it's about the 57,000<sup>th</sup> word).\n", | |
| "\n", | |
| "As before, much of the list seems kind of useless. Clearly the program is noisy. But it's capable of generating clues that are sometimes as good as, if not better than, what a person could come up with.\n", | |
| "\n", | |
| "For instance, I remember that early on, someone came up with a brilliant clue for **SOCK**, **LUCK**, and **ATLANTIS**, a board which had stumped everyone else. The clue was \"Lost.\" Sure enough, the model discovers that clue, at \\#24. Good program!" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## 3. Failures of imagination" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "A board with the targets **THUMB**, **FOREST**, and **MOUNT** ended up being pretty easy for human players. The best clue—chosen independently by three people—was \"GREEN,\" and six players got perfect scores from it. But the computer can't seem to see it:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 9, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/html": [ | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <td>1. ridges (0.02)</td>\n", | |
| " <td>2. ecosystems (0.02)</td>\n", | |
| " <td>3. elevation (0.01)</td>\n", | |
| " <td>4. foliage (0.01)</td>\n", | |
| " <td>5. ponderosa (0.00)</td>\n", | |
| " <td>6. alps (0.00)</td>\n", | |
| " <td>7. plantations (-0.00)</td>\n", | |
| " <td>8. sits (-0.00)</td>\n", | |
| " <td>9. forested (-0.00)</td>\n", | |
| " <td>10. fork (-0.01)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>11. dunes (-0.01)</td>\n", | |
| " <td>12. grassland (-0.01)</td>\n", | |
| " <td>13. grasslands (-0.01)</td>\n", | |
| " <td>14. grazing (-0.01)</td>\n", | |
| " <td>15. ecological (-0.01)</td>\n", | |
| " <td>16. palm (-0.01)</td>\n", | |
| " <td>17. etna (-0.01)</td>\n", | |
| " <td>18. beside (-0.02)</td>\n", | |
| " <td>19. volcano (-0.02)</td>\n", | |
| " <td>20. plateau (-0.02)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>21. vineyards (-0.02)</td>\n", | |
| " <td>22. lower (-0.02)</td>\n", | |
| " <td>23. yellowstone (-0.02)</td>\n", | |
| " <td>24. pines (-0.02)</td>\n", | |
| " <td>25. cherry (-0.02)</td>\n", | |
| " <td>26. preserve (-0.02)</td>\n", | |
| " <td>27. landscape (-0.02)</td>\n", | |
| " <td>28. hiking (-0.02)</td>\n", | |
| " <td>29. butte (-0.03)</td>\n", | |
| " <td>30. nestled (-0.03)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>31. yosemite (-0.03)</td>\n", | |
| " <td>32. volcanic (-0.03)</td>\n", | |
| " <td>33. situated (-0.03)</td>\n", | |
| " <td>34. pike (-0.03)</td>\n", | |
| " <td>35. plants (-0.04)</td>\n", | |
| " <td>36. plantation (-0.04)</td>\n", | |
| " <td>37. glacier (-0.04)</td>\n", | |
| " <td>38. campground (-0.04)</td>\n", | |
| " <td>39. landscapes (-0.04)</td>\n", | |
| " <td>40. mammoth (-0.04)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>41. palms (-0.04)</td>\n", | |
| " <td>42. peaks (-0.04)</td>\n", | |
| " <td>43. vegetation (-0.04)</td>\n", | |
| " <td>44. trees (-0.04)</td>\n", | |
| " <td>45. adjacent (-0.05)</td>\n", | |
| " <td>46. spruce (-0.05)</td>\n", | |
| " <td>47. cypress (-0.05)</td>\n", | |
| " <td>48. shoreline (-0.05)</td>\n", | |
| " <td>49. beneath (-0.05)</td>\n", | |
| " <td>50. terrain (-0.05)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>51. arbor (-0.05)</td>\n", | |
| " <td>52. walnut (-0.05)</td>\n", | |
| " <td>53. upper (-0.05)</td>\n", | |
| " <td>54. ranch (-0.05)</td>\n", | |
| " <td>55. atop (-0.05)</td>\n", | |
| " <td>56. wooded (-0.05)</td>\n", | |
| " <td>57. located (-0.05)</td>\n", | |
| " <td>58. bend (-0.06)</td>\n", | |
| " <td>59. middle (-0.06)</td>\n", | |
| " <td>60. overlooking (-0.06)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>61. acres (-0.06)</td>\n", | |
| " <td>62. wyoming (-0.06)</td>\n", | |
| " <td>63. slope (-0.06)</td>\n", | |
| " <td>64. beech (-0.06)</td>\n", | |
| " <td>65. drive (-0.06)</td>\n", | |
| " <td>66. reservoir (-0.06)</td>\n", | |
| " <td>67. falls (-0.06)</td>\n", | |
| " <td>68. shasta (-0.06)</td>\n", | |
| " <td>69. monument (-0.06)</td>\n", | |
| " <td>70. large (-0.06)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>71. oaks (-0.06)</td>\n", | |
| " <td>72. beaver (-0.06)</td>\n", | |
| " <td>73. ecosystem (-0.06)</td>\n", | |
| " <td>74. moved (-0.06)</td>\n", | |
| " <td>75. peninsula (-0.06)</td>\n", | |
| " <td>76. sight (-0.06)</td>\n", | |
| " <td>77. hike (-0.07)</td>\n", | |
| " <td>78. basin (-0.07)</td>\n", | |
| " <td>79. climbing (-0.07)</td>\n", | |
| " <td>80. lakes (-0.07)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>81. scenery (-0.07)</td>\n", | |
| " <td>82. timber (-0.07)</td>\n", | |
| " <td>83. mature (-0.07)</td>\n", | |
| " <td>84. small (-0.07)</td>\n", | |
| " <td>85. climb (-0.07)</td>\n", | |
| " <td>86. lick (-0.07)</td>\n", | |
| " <td>87. farms (-0.07)</td>\n", | |
| " <td>88. scenic (-0.07)</td>\n", | |
| " <td>89. remote (-0.07)</td>\n", | |
| " <td>90. brook (-0.07)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>91. peak (-0.07)</td>\n", | |
| " <td>92. nature (-0.07)</td>\n", | |
| " <td>93. habitat (-0.08)</td>\n", | |
| " <td>94. valleys (-0.08)</td>\n", | |
| " <td>95. wilderness (-0.08)</td>\n", | |
| " <td>96. montana (-0.08)</td>\n", | |
| " <td>97. trails (-0.08)</td>\n", | |
| " <td>98. maple (-0.08)</td>\n", | |
| " <td>99. tree (-0.08)</td>\n", | |
| " <td>100. rd (-0.08)</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.HTML object>" | |
| ] | |
| }, | |
| "execution_count": 9, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "answers = [\"thumb\", \"mount\", \"forest\"]\n", | |
| "bad = [\"pin\", \"tag\", \"hood\", \"princess\", \"police\", \"ball\"]\n", | |
| "\n", | |
| "tabulate(candidates(answers, bad))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "`ridges`, the top clue, might work (the connection to \"THUMB\" is via the ridges on your fingerprint, I think) but when I tested it on someone, they replied with \"mount, hood, forest.\"\n", | |
| "\n", | |
| "There was a similar misfire with a **BOND**, **PIRATE**, **BUGLE** board. The winning clue was \"GOLD,\" but the computer didn't come up with it. Its clues seem pretty weak—over-indexed to one or two targets—with the exception maybe of \"corps\" (\\#41) and \"cadets\" (\\#75):" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 10, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/html": [ | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <td>1. rallies (0.06)</td>\n", | |
| " <td>2. frock (0.05)</td>\n", | |
| " <td>3. j.p. (0.03)</td>\n", | |
| " <td>4. haircuts (0.02)</td>\n", | |
| " <td>5. keegan (0.02)</td>\n", | |
| " <td>6. davy (0.01)</td>\n", | |
| " <td>7. raiser (0.01)</td>\n", | |
| " <td>8. delilah (0.01)</td>\n", | |
| " <td>9. mayfair (0.01)</td>\n", | |
| " <td>10. cavalry (0.00)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>11. bohemian (0.00)</td>\n", | |
| " <td>12. rye (0.00)</td>\n", | |
| " <td>13. quartet (-0.00)</td>\n", | |
| " <td>14. militia (-0.00)</td>\n", | |
| " <td>15. rifles (-0.00)</td>\n", | |
| " <td>16. nightly (-0.01)</td>\n", | |
| " <td>17. clarinet (-0.01)</td>\n", | |
| " <td>18. tabloid (-0.01)</td>\n", | |
| " <td>19. minnie (-0.02)</td>\n", | |
| " <td>20. cheerleading (-0.02)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>21. homecoming (-0.02)</td>\n", | |
| " <td>22. bonanza (-0.02)</td>\n", | |
| " <td>23. brigade (-0.02)</td>\n", | |
| " <td>24. cay (-0.03)</td>\n", | |
| " <td>25. jubilee (-0.03)</td>\n", | |
| " <td>26. rum (-0.03)</td>\n", | |
| " <td>27. bn (-0.03)</td>\n", | |
| " <td>28. marching (-0.03)</td>\n", | |
| " <td>29. fireman (-0.03)</td>\n", | |
| " <td>30. ruff (-0.04)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>31. seaman (-0.04)</td>\n", | |
| " <td>32. polly (-0.04)</td>\n", | |
| " <td>33. damsel (-0.04)</td>\n", | |
| " <td>34. garrison (-0.04)</td>\n", | |
| " <td>35. beaufort (-0.04)</td>\n", | |
| " <td>36. goldeneye (-0.04)</td>\n", | |
| " <td>37. wildcat (-0.04)</td>\n", | |
| " <td>38. percussion (-0.04)</td>\n", | |
| " <td>39. shipwreck (-0.05)</td>\n", | |
| " <td>40. braid (-0.05)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>41. corps (-0.05)</td>\n", | |
| " <td>42. troop (-0.05)</td>\n", | |
| " <td>43. regiment (-0.05)</td>\n", | |
| " <td>44. leighton (-0.05)</td>\n", | |
| " <td>45. boogie (-0.05)</td>\n", | |
| " <td>46. vigilante (-0.06)</td>\n", | |
| " <td>47. mardi (-0.06)</td>\n", | |
| " <td>48. hank (-0.06)</td>\n", | |
| " <td>49. rag (-0.06)</td>\n", | |
| " <td>50. greenwich (-0.06)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>51. carribean (-0.06)</td>\n", | |
| " <td>52. putnam (-0.06)</td>\n", | |
| " <td>53. tunes (-0.07)</td>\n", | |
| " <td>54. knights (-0.07)</td>\n", | |
| " <td>55. maid (-0.07)</td>\n", | |
| " <td>56. patriotic (-0.07)</td>\n", | |
| " <td>57. arabian (-0.07)</td>\n", | |
| " <td>58. charleston (-0.07)</td>\n", | |
| " <td>59. templeton (-0.07)</td>\n", | |
| " <td>60. piggy (-0.07)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>61. beads (-0.07)</td>\n", | |
| " <td>62. sequin (-0.08)</td>\n", | |
| " <td>63. phillips (-0.08)</td>\n", | |
| " <td>64. cowboy (-0.08)</td>\n", | |
| " <td>65. gras (-0.08)</td>\n", | |
| " <td>66. flute (-0.08)</td>\n", | |
| " <td>67. connery (-0.08)</td>\n", | |
| " <td>68. parade (-0.08)</td>\n", | |
| " <td>69. jolly (-0.08)</td>\n", | |
| " <td>70. doo (-0.08)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>71. bail (-0.08)</td>\n", | |
| " <td>72. venetian (-0.08)</td>\n", | |
| " <td>73. tavern (-0.08)</td>\n", | |
| " <td>74. beaded (-0.08)</td>\n", | |
| " <td>75. cadets (-0.08)</td>\n", | |
| " <td>76. vfw (-0.09)</td>\n", | |
| " <td>77. squad (-0.09)</td>\n", | |
| " <td>78. skyfall (-0.09)</td>\n", | |
| " <td>79. barnyard (-0.09)</td>\n", | |
| " <td>80. moody (-0.09)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>81. drill (-0.09)</td>\n", | |
| " <td>82. nicholson (-0.10)</td>\n", | |
| " <td>83. corp (-0.10)</td>\n", | |
| " <td>84. us$ (-0.10)</td>\n", | |
| " <td>85. outfits (-0.10)</td>\n", | |
| " <td>86. beading (-0.10)</td>\n", | |
| " <td>87. chesapeake (-0.10)</td>\n", | |
| " <td>88. treasuries (-0.10)</td>\n", | |
| " <td>89. bahamas (-0.10)</td>\n", | |
| " <td>90. goldman (-0.10)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>91. bullion (-0.10)</td>\n", | |
| " <td>92. carnival (-0.10)</td>\n", | |
| " <td>93. hooker (-0.10)</td>\n", | |
| " <td>94. phillip (-0.10)</td>\n", | |
| " <td>95. flotilla (-0.10)</td>\n", | |
| " <td>96. pony (-0.10)</td>\n", | |
| " <td>97. drum (-0.11)</td>\n", | |
| " <td>98. ransom (-0.11)</td>\n", | |
| " <td>99. bells (-0.11)</td>\n", | |
| " <td>100. antique (-0.11)</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.HTML object>" | |
| ] | |
| }, | |
| "execution_count": 10, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "answers = [\"bond\", \"pirate\", \"bugle\"]\n", | |
| "bad = [\"alien\", \"poison\", \"seal\", \"dice\", \"link\", \"spell\"]\n", | |
| "\n", | |
| "tabulate(candidates(answers, bad))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "It's hard to know what's happening here. Is it maybe that there aren't many co-occurrences of \"gold\" and \"bond\" in the Common Crawl corpus? Look at the distance of those two vectors:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 11, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "0.625728577375412" | |
| ] | |
| }, | |
| "execution_count": 11, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "distance(\"gold\", \"bond\")" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "For reference, let's consider a word that's close to \"gold\":" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 12, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "0.3826848268508911" | |
| ] | |
| }, | |
| "execution_count": 12, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "distance(\"gold\", \"plated\")" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "...and one that bears really no relation (that I can see):" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 13, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "0.7329027354717255" | |
| ] | |
| }, | |
| "execution_count": 13, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "distance(\"gold\", \"mouse\")" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "So \"bond\" is almost as far away from \"gold\" as \"mouse\" is. More surprisingly, \"bugle\"—an instrument that is often gold-colored—is even farther away, suggesting that the two words don't appear around each other, or even in similar contexts:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 14, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "0.7875212728977203" | |
| ] | |
| }, | |
| "execution_count": 14, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "distance(\"gold\", \"bugle\")" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "We humans can use our imaginations to connect words—and in many cases this turns out to be far more powerful than a measure of conceptual distance based on co-occurence in a large corpus.\n", | |
| "\n", | |
| "Perhaps my favorite example comes with a board whose targets were **ROUND**, **FIGHTER**, and **PALM**. The model's best effort is `ufc` (\\#23); it seems preoccupied with MMA and boxing-related words:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 15, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/html": [ | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <td>1. brazilian (0.00)</td>\n", | |
| " <td>2. silva (-0.01)</td>\n", | |
| " <td>3. arcade (-0.02)</td>\n", | |
| " <td>4. featured (-0.02)</td>\n", | |
| " <td>5. 22nd (-0.02)</td>\n", | |
| " <td>6. wellington (-0.03)</td>\n", | |
| " <td>7. 26th (-0.03)</td>\n", | |
| " <td>8. 23rd (-0.03)</td>\n", | |
| " <td>9. pacquiao (-0.03)</td>\n", | |
| " <td>10. 24th (-0.03)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>11. boxing (-0.04)</td>\n", | |
| " <td>12. 27th (-0.04)</td>\n", | |
| " <td>13. ace (-0.04)</td>\n", | |
| " <td>14. pro (-0.05)</td>\n", | |
| " <td>15. 25th (-0.05)</td>\n", | |
| " <td>16. champ (-0.05)</td>\n", | |
| " <td>17. vegas (-0.05)</td>\n", | |
| " <td>18. brazil (-0.06)</td>\n", | |
| " <td>19. scheduled (-0.06)</td>\n", | |
| " <td>20. showdown (-0.06)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>21. 16th (-0.06)</td>\n", | |
| " <td>22. 9th (-0.06)</td>\n", | |
| " <td>23. ufc (-0.06)</td>\n", | |
| " <td>24. 17th (-0.06)</td>\n", | |
| " <td>25. 14th (-0.07)</td>\n", | |
| " <td>26. wrestling (-0.07)</td>\n", | |
| " <td>27. racing (-0.07)</td>\n", | |
| " <td>28. ultimate (-0.08)</td>\n", | |
| " <td>29. 15th (-0.08)</td>\n", | |
| " <td>30. picks (-0.08)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>31. 5th (-0.08)</td>\n", | |
| " <td>32. street (-0.08)</td>\n", | |
| " <td>33. 13th (-0.08)</td>\n", | |
| " <td>34. middleweight (-0.08)</td>\n", | |
| " <td>35. 8th (-0.08)</td>\n", | |
| " <td>36. football (-0.08)</td>\n", | |
| " <td>37. tennis (-0.08)</td>\n", | |
| " <td>38. 12th (-0.08)</td>\n", | |
| " <td>39. ass (-0.09)</td>\n", | |
| " <td>40. vs (-0.09)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>41. 7th (-0.09)</td>\n", | |
| " <td>42. 6th (-0.09)</td>\n", | |
| " <td>43. mma (-0.09)</td>\n", | |
| " <td>44. compete (-0.09)</td>\n", | |
| " <td>45. antonio (-0.10)</td>\n", | |
| " <td>46. june (-0.10)</td>\n", | |
| " <td>47. phoenix (-0.10)</td>\n", | |
| " <td>48. 4th (-0.10)</td>\n", | |
| " <td>49. champions (-0.10)</td>\n", | |
| " <td>50. 11th (-0.10)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>51. saturday (-0.11)</td>\n", | |
| " <td>52. vs. (-0.11)</td>\n", | |
| " <td>53. champion (-0.11)</td>\n", | |
| " <td>54. championship (-0.11)</td>\n", | |
| " <td>55. beating (-0.11)</td>\n", | |
| " <td>56. july (-0.11)</td>\n", | |
| " <td>57. championships (-0.11)</td>\n", | |
| " <td>58. kicks (-0.12)</td>\n", | |
| " <td>59. super (-0.12)</td>\n", | |
| " <td>60. march (-0.12)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>61. training (-0.12)</td>\n", | |
| " <td>62. cup (-0.12)</td>\n", | |
| " <td>63. diego (-0.12)</td>\n", | |
| " <td>64. winner (-0.12)</td>\n", | |
| " <td>65. heavyweight (-0.12)</td>\n", | |
| " <td>66. winners (-0.12)</td>\n", | |
| " <td>67. live (-0.12)</td>\n", | |
| " <td>68. competition (-0.12)</td>\n", | |
| " <td>69. cool (-0.12)</td>\n", | |
| " <td>70. tournament (-0.12)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>71. fights (-0.12)</td>\n", | |
| " <td>72. 10th (-0.12)</td>\n", | |
| " <td>73. popular (-0.12)</td>\n", | |
| " <td>74. air (-0.12)</td>\n", | |
| " <td>75. melbourne (-0.13)</td>\n", | |
| " <td>76. world (-0.13)</td>\n", | |
| " <td>77. friday (-0.13)</td>\n", | |
| " <td>78. 3rd (-0.13)</td>\n", | |
| " <td>79. apple (-0.14)</td>\n", | |
| " <td>80. matches (-0.14)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>81. kick (-0.14)</td>\n", | |
| " <td>82. rounds (-0.14)</td>\n", | |
| " <td>83. ready (-0.15)</td>\n", | |
| " <td>84. sunday (-0.15)</td>\n", | |
| " <td>85. games (-0.15)</td>\n", | |
| " <td>86. knockout (-0.15)</td>\n", | |
| " <td>87. huge (-0.15)</td>\n", | |
| " <td>88. 200 (-0.15)</td>\n", | |
| " <td>89. bet (-0.15)</td>\n", | |
| " <td>90. wins (-0.15)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>91. ball (-0.15)</td>\n", | |
| " <td>92. beat (-0.15)</td>\n", | |
| " <td>93. against (-0.15)</td>\n", | |
| " <td>94. player (-0.16)</td>\n", | |
| " <td>95. foot (-0.16)</td>\n", | |
| " <td>96. biggest (-0.16)</td>\n", | |
| " <td>97. tampa (-0.16)</td>\n", | |
| " <td>98. miami (-0.16)</td>\n", | |
| " <td>99. fight (-0.16)</td>\n", | |
| " <td>100. calendar (-0.16)</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.HTML object>" | |
| ] | |
| }, | |
| "execution_count": 15, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "answers = [\"fighter\", \"round\", \"palm\"]\n", | |
| "bad = [\"ivory\", \"knife\", \"point\", \"helicopter\", \"novel\", \"diamond\"]\n", | |
| "\n", | |
| "tabulate(candidates(answers, bad))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "One of the human cluers, though, came up with \"GRENADE.\" In vector terms, this word ends up being pretty far from all of the targets:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 16, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "0.7277005612850189" | |
| ] | |
| }, | |
| "execution_count": 16, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "distance(\"grenade\", \"fighter\")" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 17, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "0.7292716205120087" | |
| ] | |
| }, | |
| "execution_count": 17, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "distance(\"grenade\", \"round\")" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 18, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "0.8634674698114395" | |
| ] | |
| }, | |
| "execution_count": 18, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "distance(\"grenade\", \"palm\")" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "The last two of these are especially interesting. We humans know that a grenade *is* round (more or less)—but of course our computer model doesn't. It doesn't know anything. It only considers the raw token `grenade`, and only \"understands\" it in relation to other tokens. And apparently that token doesn't appear in contexts involving words like `round` all that often...\n", | |
| "\n", | |
| "Same, too, with `palm`. When we think of grenades, one of the things that immediately springs to mind is the fact that it's hand-held—particularly if that idea is primed by the presence of the word \"PALM.\" This speaks to the richness of our mental models: it's not just words in there. By contrast, the only chance our dumb model has of seeing this association is if lots of *texts* happened to talk about palms, or hands, or fingers, in the same breath as grenades. Apparently that doesn't happen too often either." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## 4. Compu-trash" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "It's worth showing an example where the computer falls flat on its face. Consider this board:" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "<img src=\"https://user-images.githubusercontent.com/21294/70329826-28cb4e80-1834-11ea-852b-98bb6e6265c3.png\" style=\"margin-left: 0;\">\n", | |
| "\n", | |
| "Here's what the humans came up with:\n", | |
| "\n", | |
| "<div style=\"margin-left: 0; width: 300px;\">\n", | |
| "<br/><b>Cluer 1: ALFRED</b>\n", | |
| "<table><col width='200'/><col width='200'/><col width='200'/><col width='200'/>\n", | |
| "<tr>\n", | |
| "<td>Receiver 1</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ROBIN</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SERVER</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>PLATE</td>\n", | |
| "<td>3 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 2</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SERVER</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ROBIN</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>PLATE</td>\n", | |
| "<td>3 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 3</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ROBIN</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SERVER</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>PLATE</td>\n", | |
| "<td>3 pts</td>\n", | |
| "</tr>\n", | |
| "</table>\n", | |
| "<br/><br/><b>Cluer 2: NET</b>\n", | |
| "<table><col width='200'/><col width='200'/><col width='200'/><col width='200'/>\n", | |
| "<tr>\n", | |
| "<td>Receiver 4</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SERVER</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SCREEN</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>POLE</td>\n", | |
| "<td>3 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 5</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SERVER</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>POLE</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SCREEN</td>\n", | |
| "<td>1 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 6</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SCREEN</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SERVER</td> <td style='text-align: center; background-color: rgb(185, 98, 0); color: white; padding: 3px;'>POLE</td>\n", | |
| "<td>3 pts</td>\n", | |
| "</tr>\n", | |
| "</table>\n", | |
| "<br/><br/><b>Cluer 3: BATCOMPUTER</b>\n", | |
| "<table><col width='200'/><col width='200'/><col width='200'/><col width='200'/>\n", | |
| "<tr>\n", | |
| "<td>Receiver 7</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SCREEN</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ROBIN</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SERVER</td>\n", | |
| "<td>6 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 8</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SCREEN</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SERVER</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ROBIN</td>\n", | |
| "<td>6 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 9</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ROBIN</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SCREEN</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SERVER</td>\n", | |
| "<td>6 pts</td>\n", | |
| "</tr>\n", | |
| "</table>\n", | |
| "<br/><br/><b>Cluer 4: TWITTER</b>\n", | |
| "<table><col width='200'/><col width='200'/><col width='200'/><col width='200'/>\n", | |
| "<tr>\n", | |
| "<td>Receiver 10</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SCREEN</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ROBIN</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SERVER</td>\n", | |
| "<td>6 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 11</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SCREEN</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ROBIN</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SERVER</td>\n", | |
| "<td>6 pts</td>\n", | |
| "</tr>\n", | |
| "<tr>\n", | |
| "<td>Receiver 12</td>\n", | |
| "<td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>ROBIN</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SCREEN</td> <td style='text-align: center; background-color: rgb(44, 89, 160); color: white; padding: 3px;'>SERVER</td>\n", | |
| "<td>6 pts</td>\n", | |
| "</tr>\n", | |
| "</table>\n", | |
| "</div>" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "There was much debate about whether \"BATCOMPUTER\" was even legitimate, but indeed we were allowing proper nouns and Wikipedia has [Batcomputer](https://en.wikipedia.org/wiki/Batcomputer) spelled as one word. Clearly, though, \"TWITTER\" is the best clue, associating as it does to computer stuff (\"screen,\" \"server\") _and_ to birds (\"robin\").\n", | |
| "\n", | |
| "How does the model compare?" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 19, | |
| "metadata": { | |
| "scrolled": true | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/html": [ | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <td>1. email (0.11)</td>\n", | |
| " <td>2. yahoo (0.09)</td>\n", | |
| " <td>3. chat (0.07)</td>\n", | |
| " <td>4. mysql (0.07)</td>\n", | |
| " <td>5. facebook (0.07)</td>\n", | |
| " <td>6. e-mail (0.07)</td>\n", | |
| " <td>7. msn (0.07)</td>\n", | |
| " <td>8. dell (0.06)</td>\n", | |
| " <td>9. name (0.06)</td>\n", | |
| " <td>10. manager (0.06)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>11. wizard (0.06)</td>\n", | |
| " <td>12. saved (0.05)</td>\n", | |
| " <td>13. sql (0.05)</td>\n", | |
| " <td>14. ftp (0.05)</td>\n", | |
| " <td>15. hello (0.04)</td>\n", | |
| " <td>16. frontpage (0.04)</td>\n", | |
| " <td>17. css (0.04)</td>\n", | |
| " <td>18. password (0.03)</td>\n", | |
| " <td>19. youtube (0.03)</td>\n", | |
| " <td>20. skype (0.03)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>21. mac (0.03)</td>\n", | |
| " <td>22. host (0.03)</td>\n", | |
| " <td>23. download (0.03)</td>\n", | |
| " <td>24. addresses (0.02)</td>\n", | |
| " <td>25. html (0.02)</td>\n", | |
| " <td>26. http (0.02)</td>\n", | |
| " <td>27. admin (0.02)</td>\n", | |
| " <td>28. deleted (0.02)</td>\n", | |
| " <td>29. preview (0.02)</td>\n", | |
| " <td>30. oracle (0.02)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>31. javascript (0.02)</td>\n", | |
| " <td>32. streaming (0.02)</td>\n", | |
| " <td>33. page (0.02)</td>\n", | |
| " <td>34. itunes (0.02)</td>\n", | |
| " <td>35. message (0.02)</td>\n", | |
| " <td>36. guest (0.01)</td>\n", | |
| " <td>37. offline (0.01)</td>\n", | |
| " <td>38. ms (0.01)</td>\n", | |
| " <td>39. editor (0.01)</td>\n", | |
| " <td>40. backup (0.01)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>41. vista (0.01)</td>\n", | |
| " <td>42. homepage (0.01)</td>\n", | |
| " <td>43. dns (0.01)</td>\n", | |
| " <td>44. tutorial (0.01)</td>\n", | |
| " <td>45. 2003 (0.00)</td>\n", | |
| " <td>46. desktops (0.00)</td>\n", | |
| " <td>47. bug (0.00)</td>\n", | |
| " <td>48. updated (0.00)</td>\n", | |
| " <td>49. google (0.00)</td>\n", | |
| " <td>50. 2000 (0.00)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>51. webpage (0.00)</td>\n", | |
| " <td>52. send (-0.00)</td>\n", | |
| " <td>53. login (-0.00)</td>\n", | |
| " <td>54. mail (-0.00)</td>\n", | |
| " <td>55. cgi (-0.01)</td>\n", | |
| " <td>56. query (-0.01)</td>\n", | |
| " <td>57. vm (-0.01)</td>\n", | |
| " <td>58. update (-0.01)</td>\n", | |
| " <td>59. flash (-0.01)</td>\n", | |
| " <td>60. screenshot (-0.01)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>61. networking (-0.01)</td>\n", | |
| " <td>62. hosting (-0.01)</td>\n", | |
| " <td>63. please (-0.02)</td>\n", | |
| " <td>64. version (-0.02)</td>\n", | |
| " <td>65. icons (-0.02)</td>\n", | |
| " <td>66. graphics (-0.02)</td>\n", | |
| " <td>67. dvd (-0.02)</td>\n", | |
| " <td>68. website (-0.02)</td>\n", | |
| " <td>69. video (-0.02)</td>\n", | |
| " <td>70. wait (-0.02)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>71. list (-0.02)</td>\n", | |
| " <td>72. xp (-0.02)</td>\n", | |
| " <td>73. info (-0.02)</td>\n", | |
| " <td>74. queue (-0.02)</td>\n", | |
| " <td>75. files (-0.02)</td>\n", | |
| " <td>76. edit (-0.02)</td>\n", | |
| " <td>77. unix (-0.02)</td>\n", | |
| " <td>78. talk (-0.03)</td>\n", | |
| " <td>79. iis (-0.03)</td>\n", | |
| " <td>80. hosted (-0.03)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>81. address (-0.03)</td>\n", | |
| " <td>82. hosts (-0.03)</td>\n", | |
| " <td>83. icon (-0.03)</td>\n", | |
| " <td>84. player (-0.03)</td>\n", | |
| " <td>85. shared (-0.03)</td>\n", | |
| " <td>86. check (-0.03)</td>\n", | |
| " <td>87. database (-0.03)</td>\n", | |
| " <td>88. virtual (-0.03)</td>\n", | |
| " <td>89. search (-0.04)</td>\n", | |
| " <td>90. hd (-0.04)</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <td>91. logon (-0.04)</td>\n", | |
| " <td>92. reboot (-0.04)</td>\n", | |
| " <td>93. thanks (-0.04)</td>\n", | |
| " <td>94. bios (-0.04)</td>\n", | |
| " <td>95. messages (-0.04)</td>\n", | |
| " <td>96. updates (-0.04)</td>\n", | |
| " <td>97. phone (-0.04)</td>\n", | |
| " <td>98. ubuntu (-0.04)</td>\n", | |
| " <td>99. xml (-0.04)</td>\n", | |
| " <td>100. ip (-0.04)</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>" | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.HTML object>" | |
| ] | |
| }, | |
| "execution_count": 19, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "answers = [\"robin\", \"screen\", \"server\"]\n", | |
| "bad = [\"pole\", \"plate\", \"ground\", \"pupil\", \"iron\", \"novel\"]\n", | |
| "\n", | |
| "tabulate(candidates(answers, bad))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "It's terrible! The over-indexing problem has basically spoiled the results. It's as if \"screen\" and \"server\" combined have so much mass that we get trapped in a gravity well far away from \"robin.\"\n", | |
| "\n", | |
| "You could imagine an interactive cluer's aid that allowed you to travel toward one target and away from the others. Indeed, a version of the model that arbitrarily weights \"robin\" as two or three times more important than \"screen\" and \"saver\" ends up with slightly more interesting clues like \"webmaster\" (perhaps a person named Robin?), but still didn't deliver \"twitter.\" (Changing the constant $c$ above from 4.0 to 3.5 brings \"twitter\" into the 7<sup>th</sup> position—perhaps by increasing the universe of possible clues?—though at the expense of worse overall performance with other boards.)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## Conclusion\n", | |
| "\n", | |
| "A simple vector space model using cosine similarities can dig up human-level clues at least some of the time.\n", | |
| "\n", | |
| "There's an over-indexing problem: words that happen to be very close to one or two of the targets will rank highly even when they're far away from the third. Minimizing the maximum distance from any target helps mitigate but doesn't entirely solve this problem.\n", | |
| "\n", | |
| "There are some triplets that humans can cleverly connect with words that are rarely *used* in similar contexts, but which make sense when you think about them. Since the computer doesn't think, it doesn't generate those clues.\n", | |
| "\n", | |
| "In general, the model's rankings are a little noisy—the 11<sup>th</sup> result is often no better than its 91<sup>st</sup>—but at a coarser level, it sorts its candidates remarkably well. If you're willing to do a little sifting, the top 100 or so results can include surprisingly good clues.\n", | |
| "\n", | |
| "I wasn't expecting that. I thought the vector space model was a neat way of *describing* the Codenames problem, but I had little faith that I'd be able to write an actually useful program with it. It's a little strange, almost magical, that so much meaning can be baked into a list of coordinates.\n", | |
| "\n", | |
| "## Acknowledgements\n", | |
| "\n", | |
| "https://medium.com/analytics-vidhya/basics-of-using-pre-trained-glove-vectors-in-python-d38905f356db\n", | |
| "\n", | |
| "Thanks to [Todd](https://toddwschneider.com), Rob, and Wilson for ideas that vastly improved the model, and for feedback on the post." | |
| ] | |
| } | |
| ], | |
| "metadata": { | |
| "kernelspec": { | |
| "display_name": "Python 2", | |
| "language": "python", | |
| "name": "python2" | |
| }, | |
| "language_info": { | |
| "codemirror_mode": { | |
| "name": "ipython", | |
| "version": 2 | |
| }, | |
| "file_extension": ".py", | |
| "mimetype": "text/x-python", | |
| "name": "python", | |
| "nbconvert_exporter": "python", | |
| "pygments_lexer": "ipython2", | |
| "version": "2.7.15" | |
| } | |
| }, | |
| "nbformat": 4, | |
| "nbformat_minor": 2 | |
| } |
Well, I was messing around with it I noticed in python 3 'izip_longest' in cell 6 should be 'zip_longest,' I think, for python3. Itertool may have depreciated izip_longest: https://realpython.com/python-itertools/.
Good to know, thanks!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hey! I really like this project, it's fun to see how the computer can 'understand' words.
Well, I was messing around with it I noticed in python 3 'izip_longest' in cell 6 should be 'zip_longest,' I think, for python3. Itertool may have depreciated izip_longest: https://realpython.com/python-itertools/.
Your writing is clear, thank you for posting this project!