Draft of a Draft

draft.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![](https://images.squarespace-cdn.com/content/5df7bcdf07ad51358b316d3b/1578603501512-N2LVHR2737YDF11HUQ1R/Orange+Logo+White+Text+Horizontal%403x.png?format=1500w&content-type=image%2Fpng)](https://rebelliondefense.com/)\n",
    "\n",
    "\n",
    "https://rebelliondefense.com\n",
    "\n",
    "* rafael AT [rebelliondefense.com](https://rebelliondefense.com)\n",
    "* aaron AT [rebelliondefense.com](https://rebelliondefense.com)\n",
    "* shay AT [rebelliondefense.com](https://rebelliondefense.com)\n",
    "* tom AT [rebelliondefense.com](https://rebelliondefense.com)\n",
    "* jackie AT [rebelliondefense.com](https://rebelliondefense.com)\n",

    "\n",
    "\n",
    "---\n",
    "## Table of Contents<a id=\"toc\"></a>\n",
    "* [Introduction](#intro)\n",
    "* [What is Approximate Nearest Neighbor Search?](#nns)\n",
    "* [Obtaining Vectors and Distance Functions From Datasets](#vectors)\n",
    "* [Applications of Approximate Nearest Neighbor Search](#use-cases)\n",
    "* [Considerations when selecting an Approximate Nearest Neighbor Search](#considerations)\n",
    "* [Hands-On](#hands-on)\n",
    "* [Installing NGT and HIPCML](#ngt-hipcml)\n",
    "* [Visualizing the Connections in a Dataset as a Graph](#viz)\n",
    "* [Manifold Learning with UMAP](#umap) \n",
    "* [Graph Clustering](#clustering)\n",
    "* [Conclusion](#conclusion)\n",
    "* [Contact](#contact)\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Introduction](#toc)<a id=\"intro\"></a>\n",
    "\n",
    "At Rebellion, we are constantly leveraging techniques to scalably organize large datasets to create insights to achieve operations goals for our customers. A common problem people face is to group data and discover relationships that would overwise go unnoticed. One way to this is by transforming your initially confusing sea of data into a nice clear map. Using this new representation, we can easily ask questions such as, \"Give me things similar to this?\", \"What is the most different thing from this other things?\", \"What is unique?\" \n",
    "\n",
    "To start this process, we need data. From the data, we derive a list of numerical observations. Usually, this is called a vector or matrix. An image, for example, is just a matrix, after all. Network traffic is another example, where we isolate certain observations and want to understand how our network traffic evolves. \n",
    "\n",
    "Usually, our first pass at creating a numerical representation is very complicated and redundant. It is hard to see any patterns. This is where the techniques we describe in this post can help. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# [What is Approximate Nearest Neighbor Search?](#toc)<a id=\"nns\"></a>\n",
    "\n",
    "Say we have a vector $q \\in \\mathbb{R}^n$ and a collection vectors, $V \\subset \\mathbb{R}^n$, and we would like to know which vectors in $V$ are \"neighbors\" of $q$. We can define \"neighbors\" in a few ways. We can say vectors in $V$ that are at most $\\epsilon >= 0$ distance way from $q$ are its neighbors. We can also ask for the $k$ nearest points from $V$ to $q$. These kinds of neighbors are known as $\\epsilon$-neighbors and $k$-nearest neighbors.  \n",
    "\n",
    "However, we can do more than query individual vectors for nearest neighbors. We can also summarize entire datasets using a network representation. If we consider the nodes to be the vectors and edges to be drawn if two vectors are in the same neighborhood, we can visualize, explore, and isolate relational patterns across the entire dataset. Good examples of this using graph summarization of vector data are [Yale Digital Humanities Lab Team's pix-plot](https://s3-us-west-2.amazonaws.com/lab-apps/pix-plot/index.html) and [Google's Projector](https://projector.tensorflow.org/). \n",
    "\n",
    "\n",
    "[![](https://raw.githubusercontent.com/YaleDHLab/pix-plot/master/pixplot/web/assets/images/preview.png)](https://raw.githubusercontent.com/YaleDHLab/pix-plot/master/pixplot/web/assets/images/preview.png)\n",
    "\n",
    "Unfortunately, performing an exact nearest neighbor search using simple methods has a runtime of $O(n|V|)$ where $|V|$ is how many vectors you are considering, and $n$ is the dimension of each vector. In practice, there is a tradeoff between the speed of retrieval and the accuracy of neighbors returned.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Obtaining Vectors and Distance Functions From Datasets](#toc)<a id=\"vectors\"></a>\n",
    "\n",
    "\n",
    "While some datasets are already vectors and have natural notion distance, many datasets do not. A vector representation of data is a numerical representation that helps capture some aspect of the meaning of the data. Datasets made-up of words from natural language do not have a clear vector representation and distance. In these cases, neural networks can transform words into vectors. An entire field of Machine Learning called [representation learning](https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html) can be seen as a methodical attempt of transforming raw data into vectors that have a meaningful notion of distance. For example, Facebook's [fasttext](https://fasttext.cc/) and Google's [BERT](https://github.com/google-research/bert) are examples of techniques of turning natural language into a vector representation. Another way to turn words into vectors is through so-called [context vectors](https://en.wikipedia.org/wiki/Vector_space_model). \n",
    "\n",
    "In the case of images, vectors could be created by taking second to the last layer of a pre-trained neural network, such as [Inception](https://www.tensorflow.org/api_docs/python/tf/keras/applications/InceptionV3). Techniques include using [autoencoders](https://arxiv.org/abs/1812.05069) or flattening the RGB matrices and concatenating them together to form a vector. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Applications of Approximate Nearest Neighbor Search](#toc)<a id=\"use-cases\"></a>\n",
    "\n",
    "\n",
    "There are two general uses of the nearest neighbor search. One application is to search a database of vectors to retrieve similar vectors to the query vector. This type of usage is a local estimate to determine the closest items from the perspective of a single vector. [Recommendation engines](https://instagram-engineering.com/powered-by-ai-instagrams-explore-recommender-system-7ca901d2a882) and georegistration are example applications that can be powered by approximate nearest neighbor search. \n",
    "\n",
    "Other applications need to estimate how the entire dataset is globally related. In these use-cases, a graph is created by transforming all the vectors into nodes and edges where an edge connects two vectors if the vectors are neighbors. While this graph forgets the vectors' exact values, it maintains the relationship between the vectors and is a scalable and sparse representation of the [topology](https://en.wikipedia.org/wiki/Topology). Creating this graph is often the first step in applications such as [dimensionality reduction](https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction), where one can define neighbors based on either [k-nearest neighbor graph](https://en.wikipedia.org/wiki/Nearest_neighbor_graph) or $\\epsilon$-neighborhood graph.   [UMAP](https://github.com/lmcinnes/umap), for example, creates a k-nearest neighbor graph using the [nn descent algorthim](https://www.cs.princeton.edu/cass/papers/www11.pdf).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Considerations when selecting an Approximate Nearest Neighbor Search](#toc)<a id=\"considerations\"></a>\n",
    "An excellent place to learn about the performance of different ANN libraries is using Martin Aumueller's and Erik Bernhardsson's [ann-benchmark](https://github.com/erikbern/ann-benchmarks). Here is an example performance assessment from 2019-06-10 showing the tradeoff between the speed of retrieval and accuracy 784-dimensional vectors from [Fashion MNIST](https://github.com/zalandoresearch/fashion-mnist):\n",
    "\n",
    "\n",
    "[![](https://github.com/erikbern/ann-benchmarks/raw/master/results/fashion-mnist-784-euclidean.png)](https://github.com/erikbern/ann-benchmarks/raw/master/results/fashion-mnist-784-euclidean.png)\n",
    "\n",
    "Another consideration is the deployment of the nearest neighbor library and if there is a serving front-end that is useable within your existing infrastructure.  While many backend libraries will perform an approximate nearest neighbor search, not all of them have an abstracted API that is easy for external clients to use the library.  \n",
    "\n",
    "Here is a chart outlining which ANN libraries have built RPC for serving requests: \n",
    "\n",
    "| Library  | Serving Layer  | Technique  | Hardware Acceleration | Organization Usage |\n",
    "|---|---|---|---|---|\n",
    "|  [NGT](https://github.com/yahoojapan/NGT) |  [GRPC](https://vald.vdaas.org/) | [ANNG](https://arxiv.org/abs/1810.07355)  | None   |  Yahoo |\n",
    "|  [ANNOY](https://github.com/spotify/annoy) |  None |  [Random Projections](https://en.wikipedia.org/wiki/Locality-sensitive_hashing#Random_projection) |  None |   Spotify |\n",
    "|  [Faiss](https://github.com/facebookresearch/faiss) | None   | [Many](https://github.com/facebookresearch/faiss/wiki#research-foundations-of-faiss)  |  GPU |  Facebook |\n",
    "|  [NMSLIB](https://github.com/nmslib/nmslib) | [HTTP](https://opendistro.github.io/for-elasticsearch/features/knn.html)   | [Many](https://github.com/nmslib/nmslib#related-publications)  |  None |  [Amazon](https://aws.amazon.com/about-aws/whats-new/2020/03/build-k-nearest-neighbor-similarity-search-engine-with-amazon-elasticsearch-service/) |\n",
    "|  [SPTAG](https://github.com/microsoft/SPTAG) | Custom TCP Protocol   | [Many](https://github.com/microsoft/SPTAG#references)  |  None |  Microsoft |\n",
    "|  [Scann](https://github.com/google-research/google-research/tree/master/scann) | None | [Many](https://github.com/google-research/google-research/blob/master/scann/docs/algorithms.md)  |  AVX2 |  Google |\n",
    "\n",
    "\n",
    "As we outlined above, there are two broad applications of approximate nearest neighbor search. The critical question is, \"what type of intermediate results does the approximate nearest neighbor search library expose in a performant manner?\".  A recommendation engine or image search engine may only need to make local queries to the approximate nearest neighbor library.  Dimensionality reduction, on the other hand, will process an initial graph summary of the data. While you can create the graph by making subsequent calls to the library one vector at a time, it can be more convenient and faster if the library can return the graph. NGT, for example, allows you to create multiple graphs from datasets of vectors. Let's see how we do that in the next section. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Hands-On](#toc)<a id=\"hands-on\"></a>\n",
    "Now that we have an understanding of possible applications, we go through two use cases. First, we use NGT to construct several different graphs from word embeddings and load the graph structures into Python. Next, we will cluster the graphs using [HIPMCL](https://bitbucket.org/azadcse/hipmcl/wiki/Home). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Installing NGT and HIPMCL](#toc)<a id=\"ngt-hipmcl\"></a>\n",
    "We will be using a development branch of NGT located at https://github.com/masajiro/NGT/tree/devel . Let's pull it down and install it.\n",
    "\n",
    "`\n",
    "git clone https://github.com/masajiro/NGT.git NGT_DEV\n",
    "cd NGT_DEV/\n",
    "git checkout devel\n",
    "/usr/bin/ruby -e \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)\"\n",
    "brew install cmake\n",
    "brew install gcc@9\n",
    "export CXX=/usr/local/bin/g++-9\n",
    "export CC=/usr/local/bin/gcc-9\n",
    "mkdir build\n",
    "cd build\n",
    "cmake ..\n",
    "make\n",
    "make install\n",
    "`\n",
    "\n",
    "Running the `ngt` command should now return:\n",
    "\n",
    "`Usage : ngt command [options] index [data]\n",
    "           command : info create search remove append export import prune reconstruct-graph optimize-search-parameters optimize-#-of-edges repair\n",
    "Version : 1.11.5` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Summarizing Connections in a Dataset as a Graph](#toc)<a id=\"viz\"></a>\n",
    "\n",
    "In this example, we are going to work with word vectors from Facebook and explore how the model is relating words to one another. First, we download the data and create 4 different graphs: [ANNG](https://arxiv.org/abs/1810.07355), [Refined ANNG](https://arxiv.org/abs/1810.07355), [KNNG](https://en.wikipedia.org/wiki/Nearest_neighbor_graph), [CKNNG](https://arxiv.org/abs/1606.02353). \n",
    "\n",
    "\n",
    "`curl -O https://dl.fbaipublicfiles.com/fasttext/vectors-english/wiki-news-300d-1M-subword.vec.zip`\n",
    "\n",
    "`unzip wiki-news-300d-1M-subword.vec.zip`\n",
    "\n",
    "`tail -n +2 wiki-news-300d-1M-subword.vec | cut -d \" \" -f 2- > wiki-news-300d-1M-subword.ssv`\n",
    "\n",
    "## Insert objects and build an initial index.\n",
    "`ngt create -d 100 -o f -E 15 -D c anng wiki-news-300d-1M-subword.ssv`\n",
    "\n",
    "## Optimize search parameters to enable the expected accuracy for search\n",
    "`ngt optimize-search-parameters anng`\n",
    "## Export an ANNG\n",
    "`ngt export-graph anng > anng.graph`\n",
    "## reconstruct a refined ANNG with an expected accuracy is 0.95\n",
    "`ngt refine-anng -a 0.95 anng refined-anng ranng` \n",
    "## Export a refined ANNG\n",
    "`ngt export-graph refined-anng > refined-anng.graph`\n",
    "## Export an approximate KNNG (k=15)\n",
    "`ngt export-graph -k 15 refined-anng > approximate-knng.graph`\n",
    "## Export an approximate CKNNG (k=15)\n",
    "`ngt reconstruct-cknng -k 15 -d 1 ranng cknng`\n",
    "\n",
    "`ngt export-graph cknng > approximate-cknng.graph` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting a million vertex graph is a little overwhelming (that is part of the usefulness of UMAP we which we discuss next). Let us plot a simpler graph to get an intuition for what the graph summary offers us. I created a KNN graph from the Iris dataset. Note the colors correspond to different flows. The graph creation step did not use the flower's identity, just measurements of the width and length of the petals and sepals. We can see the clear separation just by looking at the graph:\n",
    "\n",
    "\n",
    "[![](https://i.imgur.com/Bm31yet.png)](https://i.imgur.com/Bm31yet.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Manifold Learning with UMAP](#toc)<a id=\"umap\"></a>\n",
    "Now that we have our graphs, we can start to do analysis over. In this first example, we will apply UMAP to our KNN graph. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from umap.umap_ import compute_membership_strengths, smooth_knn_dist, make_epochs_per_sample, simplicial_set_embedding, find_ab_params\n",
    "import scipy\n",
    "\n",
    "   \n",
    "# Load graphs created by NGT\n",
    "def load_knn_graph(ngt_graph):\n",
    "    knn_indices = []\n",
    "    knn_dist = []\n",
    "\n",
    "    nodes_with_no_edges = set()\n",
    "    with open(ngt_graph, \"r\") as f:\n",
    "        while True:\n",
    "            line = f.readline().strip(\"\\n\").strip()\n",
    "            if not line:\n",
    "                break\n",
    "            token = line.split(\"\\t\")\n",
    "            start_node = int(token[0])\n",
    "            end_nodes = token[1:]\n",
    "\n",
    "            if end_nodes == []:\n",
    "                nodes_with_no_edges.add(start_node)\n",
    "                continue\n",
    "\n",
    "            knn_entry = []\n",
    "            knn_entry_dist = []\n",
    "            for ith in range(0, len(end_nodes), 2):\n",
    "                try:\n",
    "                    end_node = int(end_nodes[ith])\n",
    "                except Exception as e:\n",
    "                    nodes_with_no_edges.add(start_node)\n",
    "                    continue\n",
    "                knn_entry.append(end_node)\n",
    "                weight = float(end_nodes[ith+1])\n",
    "                knn_entry_dist.append(weight)\n",
    "            knn_indices.append(knn_entry)\n",
    "            knn_dist.append(knn_entry_dist)\n",
    "\n",
    "\n",
    "    return np.array(knn_indices), np.array(knn_dist)\n",
    "    \n",
    "    \n",
    "def knn_fuzzy_simplicial_set(knn_indices, knn_dists, local_connectivity, set_op_mix_ratio, apply_set_operations):\n",
    "\n",
    "    n_samples = knn_indices.shape[0]\n",
    "    n_neighbors = knn_indices.shape[1]\n",
    "\n",
    "    knn_dists = knn_dist.astype(np.float32)\n",
    "\n",
    "    sigmas, rhos = smooth_knn_dist(\n",
    "        knn_dists, float(n_neighbors), local_connectivity=float(local_connectivity),\n",
    "    )\n",
    "\n",
    "    rows, cols, vals = compute_membership_strengths(\n",
    "        knn_indices, knn_dists, sigmas, rhos\n",
    "    )\n",
    "\n",
    "    result = scipy.sparse.coo_matrix(\n",
    "        (vals, (rows, cols)), shape=(n_samples, n_samples)\n",
    "    )\n",
    "    result.eliminate_zeros()\n",
    "\n",
    "    if apply_set_operations:\n",
    "        transpose = result.transpose()\n",
    "\n",
    "        prod_matrix = result.multiply(transpose)\n",
    "\n",
    "        result = (\n",
    "            set_op_mix_ratio * (result + transpose - prod_matrix)\n",
    "            + (1.0 - set_op_mix_ratio) * prod_matrix\n",
    "        )\n",
    "\n",
    "    result.eliminate_zeros()\n",
    "\n",
    "    return result, sigmas, rhos\n",
    "\n",
    "\n",
    "def transform(graph, metric=\"l2\", n_components = 2, n_epochs = 500, spread=1.0, min_dist = 0.0, initial_alpha=1.0, negative_sample_rate=5, repulsion_strength=7):\n",
    "    graph = graph.tocoo()\n",
    "    graph.sum_duplicates()\n",
    "    n_vertices = graph.shape[1]\n",
    "\n",
    "    if n_epochs <= 0:\n",
    "        # For smaller datasets we can use more epochs\n",
    "        if graph.shape[0] <= 10000:\n",
    "            n_epochs = 500\n",
    "        else:\n",
    "            n_epochs = 200\n",
    "\n",
    "    graph.data[graph.data < (graph.data.max() / float(n_epochs))] = 0.0\n",
    "    graph.eliminate_zeros()\n",
    "\n",
    "    epochs_per_sample = make_epochs_per_sample(graph.data, n_epochs)\n",
    "\n",
    "    head = graph.row\n",
    "    tail = graph.col\n",
    "    weight = graph.data\n",
    "\n",
    "    a, b = find_ab_params(spread, min_dist)\n",
    "    \n",
    "    emebedding = simplicial_set_embedding(None, graph, n_components=2, initial_alpha=1.0, a=a, b=b, gamma=repulsion_strength, negative_sample_rate=negative_sample_rate, random_state=np.random, metric=metric, metric_kwds=None, verbose=True, parallel=True, n_epochs=n_epochs, init=\"random\")\n",
    "    \n",
    "    return emebedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Example data for Iris can be found here: https://github.com/lmcinnes/umap/files/4921774/approximate-knng.txt\n",
    "knn_indices, knn_dist = load_knn_graph(\"approximate-knng.txt\")\n",
    "\n",
    "knn_indices = knn_indices - 1\n",
    "\n",
    "local_connectivity = 1 \n",
    "apply_set_operations = True\n",
    "set_op_mix_ratio=1.0       \n",
    "\n",
    "graph, sigmas, rhos = knn_fuzzy_simplicial_set(knn_indices, knn_dist, local_connectivity, set_op_mix_ratio, apply_set_operations)\n",
    "\n",
    "emebedding = transform(graph, metric=\"l2\", n_components = 2, n_epochs = 500, spread=1.0, min_dist = 0.0, initial_alpha=1.0, negative_sample_rate=5, repulsion_strength=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what our embedding looks like. I have already plotted two embeddings, one for the word vectors and another one for the graph we created from the iris dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQgAAAD8CAYAAACLgjpEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAboElEQVR4nO2dfZAdVZnGn9cYcUStwApsHAgTs1FEiBMzFePG3UIQiBlK4h8K1KrZkhLLAhd0y2WyoKCSzazlR7QsqQKkDCvysaVEKpMQ4gSLlQJhJCEEQkxCRk3IJiCyQi2rEN79o08PPT19uk9/3u6+z6/q1r339Nc5M/c8fc7b55xHVBWEEBLFazqdAUJIfaFAEEKsUCAIIVYoEIQQKxQIQogVCgQhxErlAiEiS0Rkp4jsFpGhqq9PCHFHqhwHISLTAPwGwJkA9gF4CMAFqvp4ZZkghDhTdQtiIYDdqvqkqv4FwK0Azq04D4QQR15b8fV6Afw+8H0fgPeGdxKRiwBcBABHHnnkgpNOOqma3BHShYyPj+OZZ56RqG1VC0RUJqb0cVT1OgDXAcDAwICOjY2VnS9CupaBgQHrtqq7GPsAnBD4fjyApyrOAyHEkaoF4iEAc0Vktoi8DsD5AO6sOA+EEEcq7WKo6ssicgmAjQCmAbhRVR+rMg+EEHeqjkFAVdcDWF/1dQkh6eFISkKIFQoEIcQKBYIQYoUCQQixQoEghFihQBBCrFAgCCFWKBCEECsUCEKIFQoEIcQKBYIQYoUCQQixQoEghFihQBBCrFAgCCFWKBCEECu5BEJExkXkURHZKiJjJu1oEdkkIrvM+1GB/VcYw5ydInJ23swTQsqliBbEB1S1X1X9pXGHAIyq6lwAo+Y7RORkeGtQvgvAEgDfN0Y6hJCaUkYX41wAa8znNQCWBdJvVdU/q+peALvhGekQQmpKXoFQAHeLyK+N2Q0AHKeqBwDAvB9r0qNMc3pzXp8QUiJ5F61drKpPicixADaJyBMx+zqZ5gCTnbVmzZqVM4uEkKzkakGo6lPm/RCAO+B1GQ6KyEwAMO+HzO7Opjmqep2qDqjqwDHHHJMni4SQHGQWCBE5UkTe5H8GcBaA7fCMcJab3ZYD+Jn5fCeA80XkCBGZDWAugAezXp8QUj55uhjHAbhDRPzz/FhV7xKRhwDcLiIXAvgdgI8CgKo+JiK3A3gcwMsALlbVw7lyTwgplcwCoapPAnh3RPofAJxhOWYlgJVZr0kIqRaOpCSEWKFAEEKsUCAIIVYoEIQQKxQIwznnnNPpLBBSOygQhnXr1nU6C4TUDgoEIcQKBYIQYoUCQTpG39BIp7NAEqBAkI4xPjyY63gKTPlQIEhjySswJBkKBCHECgWCEGKFAkEywxhA+6FAkMzkiQFQXJoBBYJ0BAYYmwEFghBiJVEgRORGETkkItsDaands0RkgXHh2i0i3xWzVh0hpL64tCB+CM8JK0gW96xr4S1lP9e8wuckhNSMRIFQ1XsBPBtKTuWeZZa/f7Oq3q+qCuCmwDGkIdQlsFiXfHQDWWMQad2zes3ncHokInKRiIyJyNjTTz+dMYukaOoSWKxLPrqBooOUNvcsZ1ctgMY5ZcE7L0lLVoFI6561z3wOp5MK4Z2XpCWrQKRyzzLdkOdFZJF5evHJwDGkxswuodVRZkvmb1awlVQk4sUMY3YQuQXAaQDeAuAggKsArAVwO4BZMO5Zqvqs2f8KAJ+C5551mapuMOkD8J6I9ADYAOBzmnRxAAMDAzo2NpahaIQQFwYGBjA2NhY57CDRWUtVL7BsSuWepapjAE5Juh4hpD5wJCWpDAZJmwcFokuoQ+VkkLR5UCC6hCorZx3EiBQDBYIUDlsK7YEC0WDqfqfuZP7q/rdpChSIBlP3O7Wfv3Bl7RsaKb0C1/1v0xQoEA0jS8VKc0wZFdevrGu37J/0vazrkeJIHCjVaThQqln0DY3w7t0w4gZKsQVBCqVMcWBro3ooECQR14qZpwK7HJskPlXENroNCgSx4k/UiquYUQHIJKL2ibpG2so+PjzI7k3BMAbRQhgHIGlgDKLLaJI4BFsJ7B7UDwpEF1JVRYyq/OFrB8WsScLWLVAgupA8FTFqARmb4IwPD04KHLLr0zwYgyClENdKoUjUi1wxCItxztUisl9EtprX0sA2GueQWBhraA5ZjXMA4Nuq2m9e6wEa5zSFMsYLRMUWsjx2rEo8Fg9vLvycbRQ+lyXn7hWRPsfzTRjnANgrIr5xzjiMcQ4AiIhvnLMhS6ZJPmyVNk+MIO643hk92P/ci1Ou1TN9Gl586XCm6+XlvqHTCz9nG7tOeYKUl4jINtMF8b05aZxjoW53l6hWRBk/8MXDm6eIg0+UOHx80azC8xBF3f4fdSWrQFwLYA6AfgAHAHzTpNM4x0LWylfUsvO2LkDZ+OLger1rlp1adpYAtPNuXwaZBEJVD6rqYVV9BcD1ABaaTY01zunv7+90FiLZW9AP2TbFet5VdxVy/rjrRlXG1ef1T7z726uotGw5pERVE18A+gBsD3yfGfj8eXhxB8ALTj4C4AgAswE8CWCa2fYQgEXwWhMbACx1ufaCBQuUVM+Jl68r7Xzhz/73YPrbhkYKvT6xY+pYZP1zecx5C4D7AbxDRPaJyIUAvm4eWW4D8AEjElDVx+AZ6jwO4C4AF6uq39H8LIAb4Dl+7wEDlI3Ej13E3YnD26OCn/52Wwtjz6qlU9KyUvYiO22GA6UqJM9TgrVb9mPZfGtctzLiZmK6li8oDkXD0Zrp4WStmpCm8oSJEodOTHTyYwc+RcyleOeXiouD2FoqJBtsQZAJXO6+SfMuwmlR5y+rBbF4eDP++3/+D3tWLWVLIgVsQXSQsu9gVd4hbddafV5/pDiE4xW9M3omjWB0iWekydv+517EYdVCVrZiy8ODLYgGUMbdMOmctu1FVpzgCMus5UvKj3/ed37pLuz42tTR/WxpxLcgKBBdTJrKEVcRV5/Xj8tu25rq2i5dEldcRavbhcAGuxgkkjQVJm7fz6cUByBdSyRuYtXbVhTTPanDOeoIBaLBuIxFKPKHGzVm4YjX5v8JxTlwAfETq55c5SZycQJXRMuira0TCkSDcflRlv3D3XnNh+yTalIQHHKd9inH+PAgemf0TEkPTvyaPTQy4exF3GEMosUE78hFCUX4nHGzNV0oO/jqd0/KmN7dFhikrAF1jZbnyVd4XEMW6vg36TYYpOwwaR8ZVhnwylNBixCHtVv2T4qXMGBYLygQJeOvnBSFrXKWdVcte5m5LMcHH48WsUZFXIsoKr8Uk3goECUzPjzYsWXVwhS5PmRRFSuNKGT174yzEMzbCmo7FIgKKCMQV+RxrvMrgp/Dk7byYlvQxrZPGmyL7vQNjUwEMSkS0VAgGoZLPCNsVhN+bBju69vSg9vCfPE/H8lRCu+8q8/rn/LosYjAZxLBc3/x7HeUdp02kLiqNYmnLk8nbNOug5Xfn/tgW006jYv3rn9bWsikKGDyVPbeGT2l/0391k9WIarL/7wKEh9zisgJAG4C8NcAXgFwnap+R0SOBnAbvOXoxgF8TFX/aI5ZAeBCAIcB/JOqbjTpC+D5bPQAWA/gUk3IQFsec3aa4IzKNGMCgq2PvqERLJ5zNO7b82wpd/lOVLpgGaaJ4LBq11R+n7yPOV8G8M+q+k54a0pebAxyhgCMqupcAKPmO81zakRUNyLKnyKOcBDx5k+/r5QK5I+E7GQs4HBBY4LaFM9IFAhVPaCqD5vPzwPYAc/T4lwAa8xua+AZ4QAB8xxV3QtvDcqFIjITxjzHtBpuChxDMpC0dkG4Ioe/z//q3U7BweDxaVd/Cl9zWoTj4vjw4KRp30XPIbGRtHxeVtrUAkkVpDQOW/MB/ArAcap6APBEBMCxZrfc5jltNM4pgyzLxQer55YvnxV5vij8ChNcU8HlEWW4otnu0uG4SREVNS0zeqZH5qebcRYIEXkjgJ8AuExV/xS3a0RaKvMcbaFxTlZcK4jLfn1DI4k+G/55XCc25b3TJlXEIoQi7thgHrZedVarugdF4CQQIjIdnjjcrKo/NckHTbcB5v2QSW+seU5duHLtoxOf08xoTBrYlGb2Z3CEY/DRY3jJONeBTlEthGBXYvHw5imPZ23Hp8VlxKqfH7YcJuPiiyEAfgBgh6p+K7DpTgDLzeflAH4WSD9fRI4QkdnwgpEPmm7I8yKyyJzzk4FjSIBrlp2a6U5W5JoHtsem/hOQYGXKUrHC5btv6PRJYuO/R1kP2taNyAvFYSouLYjFAD4B4HQR2WpeSwEMAzhTRHYBONN8p3lOQUS1CIrqbqSpULY8RK1Y7TM+PIhpr5FJ330R6Z3Rg7Vb9jvPl4jqEgWFye8KlbFAjo0i/751h9O9a07wx+YPdKr6ThfVQpj/1bsngpx9QyMQ2N2Y4+IILmVJMxs2/FSGrYJkON27wQSb3H4zvCr8u3NUayH4BGR8eNBZHLI8+YhrrUQdm3ZFKmKHAtEAilwNKk3zN7xSddScDp+oMRfhGEWw2xT3KDN4rvB6Ef72uHIUPZEsSJu6Dy5QIAqmzj8g1ycOUXfgxcObEwdeRZ07PBjLli/b323Z/N6JfAeffNjW2PCPCVN0ILNb1rekQBRME5q1cdO+bdtc52/0zuiZ5LS15ctnTWoBuDxydCFujY2oyjv32CNjz5dWQOpgpFwFDFI2nLyBuOAkrrDLVZ5zlx0gDFZoweSnHWVeu42BTwYpW0yagF9c1D84icvv96epCEnxjTJnfoZvcWVW4LaJQyKqWuvXggULtI2cePk6PenKDaqqE+9lXMM1/cTL1016uZ4rab+4svn7up47LUnnLeu6TcPUscj613EBSHo1USDK+OGlOWfW688e8o674+F9ma8TTPPFZuE1d1uPCQtT2ryH81pmpW+roMQJBGMQpNB+dXjJuCLGPOQZTEWSYQyiC4kaIh0VJ5h31V2FzJh878pNk75nPWeUYCStexG8HikWtiBaRHgCVRRlVyRbIDTpDp938lddaGLe2YIokToMjIoa4RhlZltlHqK2ueBP5Io6pqxZnFFkPXeaFbqaAAUiJ3W4W0TlwWaoG9XNiBs4ZdsnvM2lS3HSFeut28LrULgO5w4ugVfkY9Yi/q91+G3khQLREOJ+4P1fuXvS97RDm213vagmf9R8iOC21ef1TxoaHeSJlUtj8+UTXJQmqSw7vrbEadyGy/Tybhk+nQbGIDJS976mS0CvijiFyzWihCY4OjLubz17aGTSflHnv3Lto7hm2anZC5GQh6YTF4OgQNQclx9m1D5xAhEcVh2Fa0VIytvi4c2TruNajrpWxrrmKy+5gpQicoKI3CMiO0TkMRG51KRfLSL7Q6tM+cesEJHdIrJTRM4OpC8QkUfNtu+apee6giwBK9cfZNQ+ccfFiUPw2nHf/WvYyjVnxfpJ1znqDdMj9wufL+56SZQdFMwbgM2yf6dxcdaaCWCmqj4sIm8C8Gt4fhYfA/CCqn4jtP/JAG4BsBDAWwH8HMDbVfWwiDwI4FIAD8Bz1vquqsYuO9etLYiiJkrlicbb4hFJ7ly2Fk1cTCMpL2muFUfa/ddu2T9l5mbbWhKFdjFE5GcAvgdvrcoogVgBAKq6ynzfCOBqePZ896jqSSb9AgCnqepnEjLfNQJhCwZGfXc5fkbPdDz34kux1zzqDdPx3P++BE1x7tkrRrB31eCkWEFURYrDdfGXOnc52kJhAmGMc+4FcAqALwD4RwB/AjAGz57vjyLyPQAPqOqPzDE/gLc47TiAYVX9oEn/OwCXq+o5Ede5CJ5FH2bNmrXgt7/9rXMeyatkaT24RPtd9k/K04TYDI1Yl6sDgI8vmuUUYOyEiLRFuAoZKBVhnHMtgDkA+gEcAPBNf9eIw2mcU1PGhwcnHk0Ck4dk9w2NTHmEGsbf7hKziGohJRn5/OiB38WeM1iOKMrs87dBHJJwakEY45x1ADbqZG8Mf3sfgHWqegq7GHaqvOOkqRhJzfwkVp/Xj8tu25o6FpCWbqiQnSDvU4xI4xzfVcvwEQDbzeeuN87J2xyPMotxvW7Swi3h/Ph5ylP5Lrtt65SFYpPykLeyz1lhH5VZNk17EpEHl6cY7wfwXwAeBfCKSf5XABfA614ovNbBZ4wIQESuAPApAC/D65JsMOkDAH4IoAdeXOJzmpCBNrUgyqKIu3F4mnYWjnrD9CmGwDbCYyRcKaMV0ZZYQlZytSBU9ZeqKqo6T1X7zWu9qn5CVU816R/2xcEcs1JV56jqO4KPMVV1TFVPMdsuSRIH4k7U0GYbAuAfrr9/4ntRFSRopJNEFnEoi7Rlnxszp6RtcC5GTrLecdOO+087dDrYdQiLx97hQdz86ffFHp+HcMDTtk+aijk+PDgx2CpNPsuYa7HLcU5JG+BQa4LFw5tx39DpuQUiy2pQ4fEQWVeUItnhXIwuIOucjSiyxAfyVGKXQWCkPLhgTBfgMg7AtQKm9QDNOwYhqrsRnPJNOgcFouW4TjAKLwDjWrl9Fy3bZK6suDp55aWbHllmgQKRg7r/uGwrQgWDiGd+6xeT0vzPSUFEf/RlcIBU1vEbnYTdmngoEDmo84/Ln3WZ9PRj0xdOczpflFluMKYxPjyYOGw6C3lEuO4C3gQoEC3Er7jBWIJrnCD83T8uaJY7Pjw44boddUwR+N2WLCJsazmR9FAgWkCeSpA0vXz2ipEp3Y3wkG6XLkbaPKYdJxE+lk9GioEC0QLClSBLpbCdI/gUPLiPRBwTfGAenAXqV9a4O3uawKgLUfkjGbB58tXl1URvziKp0g/S1UQ4q5Fv3Pa/XTXqdM4q6WupF2cYmveSSFwqcBECFefifcfD+2ohDm015nWBAkGmEOWynbSPLY00G7p7E0KscKg1aRx8PFkPXtvpDJDmYhsz4cKcFeuxZ5V92jSfQNQDlyXnXi8iD4rII8Y45ysm/WgR2SQiu8z7UYFjaJxTIEXcTcsYkRhecyINceJA6oNLF+PPAE5X1XfDW2JuiYgsAjAEYFRV5wIYNd9945zzAbwLwBIA3xcRf5zutfCWs59rXksKLEsrKHrSUxHEXT9pMljSxLAo2L2oDy5LzqmqvmC+TjcvBXAugDUmfQ08ty2Y9FtV9c+quhfAbgALzSK3b1bV+9WLjN4UOIYY2rDmYnjiVzAf4bTwtixQUMrDKUgpItNEZCuAQwA2qeqvABynZh1K836s2b0XwO8Dh+8zab3mczg96noXiciYiIw9/fTTacpDIuh0C8THtSWSNr91KV8bcRIIVT2sqv0AjofXGjglZnca5zSUqu7EvOM3h1SPOVX1OQC/gBc7OOh7Y5j3Q2a3fQBOCBx2PICnTPrxEemkJhR1Jy7bE4NUh8tTjGNEZIb53APggwCegGeQs9zsthyvmuB0vXFOt0MBaA8uLYiZAO4RkW0AHoIXg1gHYBjAmSKyC8CZ5jtU9TEAtwN4HMBdAC5WVX8xgc8CuAFe4HIPPPMcQjoCuzrJcKg1SU0dl6KvY56aAodak0KpS0UM+nPWJU9tgwJBOkqehW45GrN8KBBkEklrVBZN1EK3jA3UBwpESyjaV7OT1CEPxIMC0RKKrFRXrn20lPOS5kGBaDFZWxXXLDu14Jx4sOvQPCgQLSbr3b+siszWSPOgQBAA2Ux+SfuhQBAA0dOyCaFAkElwRCIJQoEgE1AcSBgKBJmA4kDCUCAIIVYoEA0lbHZb98Bi3fNHouF0b0K6HE737lLqdNeuU16IO3mMc64Wkf0istW8lgaOoXFODahT0LFOeSHuuFjv+cY5L4jIdAC/FBF/qbhvq+o3gjuHjHPeCuDnIvJ2s+ycb5zzAID18Ba/5bJzhNSUPMY5NmicQzLDrki9yGOcAwCXiMg2Ebkx4M2Z2ziHdC/sitSLPMY51wKYA8+v8wCAb5rdcxvn0FmLkHqQ2ThHVQ8a4XgFwPUAFprdchvn0FmLkHqQ2TjHd9UyfATAdvOZxjkthjGC7sLlKcZMAGtEZBo8QbldVdeJyH+ISD+8bsI4gM8AnnGOiPjGOS9jqnHODwH0wHt6wScYDYMxgu4iUSBUdRuA+RHpn4g5ZiWAlRHpYwDijH9JF8BZo82BIylJ5VAcmgMFghBihQJBCLFCgSCEWKFAEEKsUCAIIVYoEIQQKxQIQogVCgQhxAoFooWcffbZyTu1mDPOOKPwc27btq3wczYBCkQL2bhxY6ez0FFGR0cLP+e8efMKP2cToEAQQqxQIAghVigQhBArFAhCiBUKBCHECgWCEGLFWSDM0vdbRGSd+X60iGwSkV3m/ajAvnTWIqQFpGlBXApgR+D7EIBRVZ0LYNR8DztrLQHwfbOeJfCqs9Zc81qSK/eEkFJxNc45HsAggBsCyecCWGM+r8GrLll01iKkJbi2IFYD+BcArwTSjjNL2cO8H2vScztr0TiHkHrg4otxDoBDqvprx3PmdtaicQ4h9cDFF2MxgA+LyFIArwfwZhH5EYCDIjJTVQ+Y7sMhs39uZy1CSD1wcfdeoarHq2ofvODjZlX9ODwHreVmt+V41SWLzlqEtASXFoSNYQC3i8iFAH4H4KMAnbUIaROpBEJVfwHPvBeq+gcAkRPv6axFSDvgSEpCiBUKBCHECgWCEGKFAkEIsUKBIIRYoUAQQqxQIAghVigQhBArFAhCiBUKBCHEinhrt9QXEXkewM5O56MA3gLgmU5nogDaUg6gPWXJW44TVTVyXYU8k7WqYqeqDnQ6E3kRkTGWo160pSxlloNdDEKIFQoEIcRKEwTiuk5noCBYjvrRlrKUVo7aBykJIZ2jCS0IQkiHoEAQQqzUViBEZImx7tstIkOdzk8YEblRRA6JyPZAWuPsCEXkBBG5R0R2iMhjInJpg8vyehF5UEQeMWX5SlPLYvLQebtLVa3dC8A0AHsAvA3A6wA8AuDkTucrlMe/B/AeANsDaV8HMGQ+DwH4d/P5ZFOGIwDMNmWbZrY9COB98HxDNgD4UMXlmAngPebzmwD8xuS3iWURAG80n6cD+BWARU0si8nDFwD8GMC6Tv2+6tqCWAhgt6o+qap/AXArPEu/2qCq9wJ4NpTcODtCVT2gqg+bz8/D81/tRTPLoqr6gvk63bwUDSxLXewu6yoQNvu+ulOaHWEViEgfgPnw7ryNLItplm+FZ+S0SVWbWpZK7S5t1FUgnG36GkJuO8KyEZE3AvgJgMtU9U9xu0ak1aYsqnpYVfvhObctFJE4m4ValqUTdpc26ioQNvu+unPQNOvQJDtCEZkOTxxuVtWfmuRGlsVHVZ+D5+GyBM0ri293OQ6ve3160O4SqK4cdRWIhwDMFZHZIvI6eJZ/d3Y4Ty40zo7QXPcHAHao6rcCm5pYlmNEZIb53APggwCeQMPKonWyu6w6MpsigrsUXkR9D4ArOp2fiPzdAuAAgJfgKfWFAP4KwCiAXeb96MD+V5iy7EQgkgxgAMB2s+17MKNbKyzH++E1O7cB2GpeSxtalnkAtpiybAfwZZPeuLIE8nEaXn2KUXk5ONSaEGKlrl0MQkgNoEAQQqxQIAghVigQhBArFAhCiBUKBCHECgWCEGLl/wFhNyOOYzyNEAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "image = plt.imread('wordvector_embedding.png')\n",
    "plt.imshow(image);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVgAAADnCAYAAABbh05UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZcUlEQVR4nO3de2xb53kG8OcVw6RU0FX2om4xY9eu57iI60RahVygbUXSLHKTJVa8Jk7gbis2xC3Q/mE3UCHBRuOuSS1MS52h6y0Zgg2L1yhNbdatgylJVayAEbeVRzmymqi1c3FCB406l9kWczZNv/uDPPTh4TmHlHxuPHp+AGHxkOL5TMuPPr7nu4iqgoiIvNcWdgOIiOKKAUtE5BMGLBGRTxiwREQ+YcASEfmEAUtE5BMGLBGRTxiwREQ+YcASEfmEAUtE5JOLwm6Ak8suu0yXL18edjOIiKoOHTr0G1XtbPb5kQ3Y5cuXY2JiIuxmEBFVicjrc3k+SwRERD5hwBIR+YQBS0TkEwYsEZFPGLBEFDuZbA69w+NYMbgfmWwutHYEPopARF4D8D8ASgDOqmpP0G0govjKZHMY2jOFQrEEABjaMwUA6O9OB96WsHqwN6pqF8OViLw2MjZTDVcAKBRL2LFvOpS2sERARLFyIl+oO5YvFHHltmcCLxeEEbAK4FkROSQim0M4PxHF2JKOlO3xMyXF1tHJQEM2jID9I1X9QwAfB/BZ8wMisllEJkRkYnZ2NoSmEVGrG+hb7fiYAoGWCwIPWFXNVf58G8Bey2OPqmqPqvZ0djY93ZeIqKq/O41F7UnHx/OFIlYOPYPtmSnf2xJowIrIB0XkcOU2DaAY5PmJaGHIfvEW9K5c7Ph4SRVPHDzue7kg0IBV1VdU9ZrKbY2qPhTk+Ylo4dh93w2uIQsAA08f9jVkOYqAiGJr9303IO1w0QsAiiXFl37gX02WAUtEC9pvT/lXqWTAElFsZbI55GzGxQaFAUtEsTUyNtPwOR0p5xEHF4oBS0Sx1EzvNdkm2HHHGt/aENktY4iI5qt3eNw2XFPJBHZuWBvYwi/swRJR7NitRwAg0HAFGLBEFEN26xGkO1KBL1nIgCWi2BnoW41UMlG9n0omXNco8AtrsEQUO0ZP1RhFMNC3OpQFtxmwRBQrmWwOO/ZNI18oTyB4ZGNXKOEKMGCJKEYy2Rw+PzqJc6ZjA08fBrCwtowhIvLcjn3TNeEKlNcbaGbCgR/YgyWilpDJ5jAyNoMT+QKWdKRq6qrbM1PYffA4NOQ2WjFgiSjyrAGayxdqdottFK5hjCAAWCIgohZgF6CFYqn60d8tXD95/bLQLnIxYIko8pwC1GnGliEhggf713rfoCYxYIko0tx2HHDaQdbw3vdchN7h8cC36zawBktEkZXJ5jDw3cO2jwnO11ZTyQQKxVLdc/KFIvKFYk29NkjswRJRZO3YN43iufoCgQDYZKqt7tywtro1jNi8TqFYwv1P+bv/lh1RjdrAhrKenh6dmJgIuxlEFKLlg/ttj782fJvt8RWD+xsO1Up3pHBg8KZ5tUdEDqlqT7PPZ4mAiCJrUXuybs+sRe3lHQiMcbHGuq/pjhTel0pWp8g6aXRhzEuxDdje4fHqgOQbP9SJH788W70/399eRBSsB25fg4GnD6NYKvdLkwnBA7evQSabw9CeqZq6ay5fQDIhSLaJbVnB0OjCmJdiGbDbM1PV32q5fAFPHDxefSyXLyCTzYU2Lo6IGjPP2npfKol3CsWa2Vu9w+O2F7WKJcWi9iTaL74IJ/IFdFh6wEEvWxh4wIrIOgD/ACAB4J9Uddir17auouNkZGyGAUsUUdbeab5QrFsRy+1jfv5UEdkv3lK9b/40G/SyhYEGrIgkAHwdwJ8CeBPAzwF4ErB2HxmcBFmDIaK5GRmbqft/fP9TtStiLelIOW5oaC0BhFkSDHqY1rUAjqrqK6p6BsCTXr2w3T+Kk452/7bpJaILY9cBKqliaM9UdZiVdccCQ1g7FzgJOmDTAN4w3X8z4PMDAH57qlid3dE7PI4Vg/tDne1BROft2thlG56FYglbRycBABOvn8SZs9aFCcvPMQdx2IIO2ByApab7V3j1wjd+qNN2gLFjQ/IFbBmdRC5fgKJ2dR4iCk9/dxo7N6xFQur/R3e0J7Hmi/+OJw4eR8lhDL95EZiwBR2wPwewSkRWiMjFAO7x4kUz2Ry+dyjnOMC42eBttsRARP7q707j4buvqenJJhOC//2/s3j3TOtcZwn0IpeqnhWRzwEYQ3kUweNevK5b/TVdGQf7vUM5BihRC7FuXPju6bMNRwgZghzr6ibwYVqq+gyAZ7x8TaffVsaUuEw2h/0vvsWAJWox/d3patA6TZu1itKFrlgs9uL02+pEZVLB0J6puul2duZSwyWi4PQOjzf1vEXtSezcsDYy49xjMZNroG81to5O1tVgl3Sk5jR8a9P1y7xv3BzY7SsU5pbDRFGQyeYcx7wa0iFMImhGrFbTMk+v21UJpt7h8bp/HHPpwDo5wa9AazSbxG0VoAtZ/Yeo1bn930glE4H2WOe6mlYsSgRu7AYkG/UZu96tH8M7jN/A5uFg1nF6br/monJFlCgMTiXAdEcqUuUAO7EoEQD1U2WtK5ibt/s1jtkFlx9hZg1tY5xesz8YUbkiShSGgb7V2FKZYAAE32u9ELEJWGtv1FjBHKi9EmlmN5/ZjzC70CCPyhVRojD0d6drOkhRrLU6iU3Aus1fBuz34hnoW11Xg3UKs0w2h62jk/P6B3YLcuN1L704YTuAOswth4miolWvQcSmBuvU83Tbi8eYkpfuSEFQrunYhZlRfjBqqFtHJ7E90/y0WmtoG+P0zK/77pkSEm21A8Xak22hbjlMRBcmNj1Yu96owa0n61Q+MLOWHxTAEwePo+cDi5vqXTp9xLEuGlw6p0iI4Jxq9XlE1LpiE7BG0N3/1GHbRSDmemHJzKleumPfdNOvZ/cRx6msAZR7ygPfPczyAFELi03AAud7o5lsruaqoyGXL2D54H48srELAPD50UlYFzyzGwe7a2OX7etZ50Wbx+E2U6vdtbHLdZFwt32FiCj6YlODNevvTtuuJ2nYOjqJLTbhCtiPg+3vTqM9Wf9WLbIs3L3VZvlDt3UpjRowEcVTLAMWQPXilZ35DOr/yoara+4bu1saMtn65RKbWZeyUQmAC4ITta5YTZW1YzdV1o1xkamjPQlV1Oxm6fbx3+k8AuDV4dtcz+m0SlCiTVCyKRP0rlyM3ffd0PTfiYi8MdepsrGqwdqxzgJpxLjIZF59y/i4/9KX1zl+n1PPd74TFxIieO8lF9muf3ng2ElseuwFhixRxMW2RGDo707X1Urno9GKXHZBKnCfhbU9M4WVQ/VL46aSCTx89zV4x2Vx4QPHTrq2h4jCF/uABYAHbl/jetELgGO9tlnWIBWUlz90qrFuz0zZ7itkXsCCaxAQtbbYlwiA2uFb5mFR1kUjnIZ3Gc9t5hzNeuLgcdvj5vGyBwZvcqzP9q5c3PS5iCgcC6IHazCGRRnTYq0r8jgFZELE8+FUdjtm2h2zC/ZV77+U9VeiFrAgerBmjXqaqWSipt7q19Jo9163tK4Xe+91S+uet3PD2pZdSYhooVtwAdtIUIFmLOLynZ++AaAcrnYLu8y19EBE0RH7cbBERF7hljFERBERWMCKyA4RyYnIZOV2a1DnJiIKQ9A12F2q+vcBn5OIKBRBlwg+JyIvisjjIrLI+qCIbBaRCRGZmJ2dDbhpRETe8jxgReR5ETliua0H8E0AKwF0AXgLwMPW71XVR1W1R1V7Ojs7vW4aEVGgPC8RqOrNjZ4jIo8B+KHX5yYiipIgL3Jdbrp7J4AjQZ2biCgMQV7k+jsR6UJ5vevXAHw6wHMTEQUusIBV1b8I6lxERFHAiQZERD5hwBIR+YQBS0TkEwYsEZFPGLBERD5hwBIR+YQBS0TkEwYsEZFPGLBERD5hwBIR+YQBS0TkEwYsEZFPGLBERD5hwBIR+YQBS0TkEwYsEZFPGLBERD5hwBIR+YQBS0TkEwYsEZFPGLBERD5hwBIR+cTzgBWRu0RkWkTOiUiP6fiQiBwVkRkR6fP6vEREUXORD695BMAGAN82DojIVQDuAbAGwBIAz4vIlapa8uH8RESR4HkPVlVfUtUZy+H1AJ5U1dOq+iqAowCu9frcRERRElQNNg3gDdP9NyvHaojIZhGZEJGJ2dnZgJpGROSPeQesiDwvIkcst/UX0hhVfVRVe1S1p7Oz80JeiogodPOuwarqzXN4eg7AUtP9KyrHiIhiK6gSwT4A94jIJSKyAsAqAD8L6NxERKHwfBSBiNwJ4GsAOgHsF5FJVe0TkacA/ALAWQCf5QgCIoo7zwNWVfcC2Gtz/CEAD3l9PiKiqOJMLiIin/gx0YCIIiyTzWFkrDxUfaBvNfq760ZMkkcYsEQtxgjIE/kClnSkcGDwpjl979CeKRSK5UsgQ3umAIAh6xMGLFGL2PTYCzhw7GTNsVy+gEw213RAjozNVMMVAArFEkbGZhiwPmENlqgF2IWrwfi434wT+UJTx8gbDFiiFuAUrkB9QGayOfQOj2PF4H5ksrXzeZZ0pOq+3+4YeYMlAqKIs4ak1a6NXdWvu770LPKFYvX+ltFJjIzNVOu0BwZv4kWuADFgiSLOrQSQbDt/gSqTzdWEq8Haw+3vTjNUA8ISAVHEOdVIe1cuxq++clv1vlMQK4De4fGGPWHyHgOWKOLel0rWHetIJbH7vhtqjrldrMrlCxjaM8WQDRgDlijiRJo71uhiVaFYwo590x61iprBgCWKuPyp+rqq3bGBvtVIJRPur1UoshcbIAYsUcQ1O7SqvzuNnRvWIt2gJzuXcbN0YTiKgCjiDgzehN7h8erUWOvQKutjFB0MWKIWYLfeQCabw7a9U3j3THnqq3EhyzwV1iqVTDCEA8SAJWpBmWwOA08fRrGkNccLxRISIiip1n1P2qb3S/5iwBK1oJGxmbpwNZRUkUomqj3ZVDKBnRvWMlhDwIAlahHmZQrto7XM6KlyOmz4GLBELcC6jqsTwflAZaiGj8O0iFqAdR1XJ5uuX8ZgjRD2YIlaWHqOOxpQsNiDJWoBdrO0OOQq+jwPWBG5S0SmReSciPRUji0XkYKITFZu3/L6vERxZp6lJSj3XDkyIPr8KBEcAbABwLctx4+papfN84moCbxw1Xo8D1hVfQkAxG65HyKiBSTIGuwKEcmKyH+IyB/bPUFENovIhIhMzM7OBtg0IiLvzbsHKyLPA/h9y+Ftqvp9m6e/BWCZqv6XiHwEQEZE1qjqf5ufpKqPAngUAHp6etzGUhMteNxbK/rmHbCqevMcnnsawOnK14dE5BiAKwFMzPf8RAuZdeLB0J4pAGDIRoyozaIQF/SCIncC+BqATgB5AJMo90r/FkARwDkAD6jqD9xep6enRycmmL9EdlYM7redLptsExTPnX/kkY1dDF0PicghVe1p9vl+XOTaC2CvzUPf8/pcRAvVko4UcjZ7cJnDFSjPAGPAhocTDYhaUDPbwwDuGyGS/zhVlqgFGb3SLaOTrs9rtBEi+YsBSxQR5uUI7baGsXue0+LaBk6lDRcDligCrKMCcvkCto5OYuL1k3iwf231edszU9h98Hj1Apc1XJMJwaUXX4R3CkUs6Uix/hoy1mCJIsBuOUIFsPvg8eo225lsriZcDQmR6voEI5+4BjvuWIMlHSmcyBea3qI7k82hd3gcKwb3o3d4/ML/QgTAh2FaXuEwLVpIeofHbUcFAOVFtN9j2gLGyrxkod3ruC1pmMnmcP9Th+t6wlwG0d5ch2mxB0sUAQN9q+G0eseSjpTrYtvmOqvdqAGnkQSbHnsBW0YnbWu4HH3gDQYsUQT0d6ex6fpldSHbaM1XQe3sLbtRA04jCQ4cO+n4uhx94A0GLFFEPNi/Frs2djW95mtCBLs21q4Aah0f6xTQjWqzHH3gDY4iIIoQpzVfU5YarNNW3Mb9kbEZ5PIFFIql6oIwxmPGiAUnCRGOPvAIe7BELaDZ3QyMMbK5fKFabsjlCxjaM1XttTbaQPHhu6/x4W+wMLEHS9QCmtnNwDqW1nzpqlAsYUtlXK3bBazelYvZe/UQA5aohVhDFDi/YtbW0UnbFbbMnjh4vO6YU7mBLhxLBEQtxO7jvVFjneuVf26e6D/2YIlaiNs414G+1XW9WzevDt/maduoHnuwRC3EbZyr3dbeFC72YIlaiF0v1Txm1XoxbHtmCrt/ehzWyVrNrCVLF449WKIWYtdLdaufPti/Fq/uvA2P2ExgIP9xsReigDS73itF11wXe2HAEgUgk83Z7j4gAHaZNia0G4aVSibw0pfXBdVUchH6podEVGbusbaJ/VpZitqNCe2GYTU7KoCihwFL5ANrT9RtWxfz0CsuExgvDFgiD5nXAmiWeehVR3sSvz1V9KNpFALPRxGIyIiIvCwiL4rIXhHpqBwfEpGjIjIjIn1en5coCob2TM0pXK3LCUb0kgjNkx/DtJ4D8GFVvRrALwEMichVAO4BsAbAOgDfEBEOxKPYcauXJip1WONPu2mq7xTYe40Tz0sEqvqs6e5BAJ8AsB7Ak6p6GsCrInIUwLUAXvD6/ERBsg69cvPw3dc0HJa1pCM1px4wRZvfNdi/BjAK4HqUw9bwJoC6nzQR2QxgMwAsW7bM56YRzV8mm8OXfjBdUy91C8aECLbtnaqueJUQwb3XLa3Zkhuwn6nFWVety5dxsCLyKQCfBvAxVT0lIkMAoKo7K4+PAdihqo49WI6Dpahy2wEWKH/0N3q0p86cdb1olRDBsZ23+tFM8kHo42BFZB2ALwD4qKqeqhzeB+DfROSrAJYAWAXgZ16fmygIjcLVvN31isH9rq/lNnyLWp8fJYJ/BHAJgOekXMw/qKqfEZGnAPwCwFkAn1VVjp6mlpQQcQxG62aBjWqqCYcJCBQPflzk+gOH4w8BeMjr8xEFza3Xab2INdC32naKrOHe65Z61i6KHk40IHJgnjRg9FrTHSmkHXqlduuv9nensWPfNPI2w69Syba6i1wUL1yukMiGMdXVCFKj15rLF3DjhzrrruxL5TFj51azHXesQbKtthSQTAh2brjan8ZTZDBgiWy4bW3945dnq2uyAuVwNYoG5u2xDf3daYzcdX4r7EXtSYx8ovGYWGp9XK6QyEYzQ7Hc6quvcb+rWJrrMC32YIlsWEcDWOXyBQztmYLdGACODCADA5bIRn93Gu1J9/8ehWIJdp//OLaVDAxYIgdfmedFKO7mSgYGLJGD/u40Pnm9+5oYi9qTNSMKrMsP0sLGgCVy8WD/2uqOrABqaq6pZAIP3L6mZpdX6/KDtLBxFAHRHGSyOWwdneSusAtU6Iu9EMXNpsdewIFjJ2uOvcU1W6kJLBEQuchkc3XhCgDnUJ6MQOSGAUvkwi1EuQMsNcKAJXLhFqKNtoghYsASuXAK0TY0nu1FxIAlcjHQt7pu5axkG/DVjV0cQUANcRQBkQsjRM07x5q3hCFyw4AlaqC/O83eKs0LSwRERD5hwBIR+YQBS0TkEwYsEZFPIrvYi4jMAnh9jt92GYDf+NCc+WJ73LE97tged2G05wOq2tnskyMbsPMhIhNzWenGb2yPO7bHHdvjLmrtscMSARGRT+IWsI+G3QALtscd2+OO7XEXtfbUiVWJgIgoSuLWgyUiigwGLBGRT2IRsCIyIiIvi8iLIrJXRDoqx4dE5KiIzIhIX4DtuUtEpkXknIj0VI4tF5GCiExWbt8Ksz2V46G8P5a27RCRXOU9uTWkNqyrvAdHRWQwjDZY2vOaiExV3pNQNqYTkcdF5G0ROWI6tlhEnhORX4nIogi0J/SfnUZiEbAAngPwYVW9GsAvAQyJyFUA7gGwBsA6AN8QkYTLa3jpCIANAH5iOX5MVbsqt88E1Bbb9oT8/ljtqrwnzwR94srf+esAPg7gKgD3Vt6bsN1YeU/CGob0zyj/XJgNAviRqq6qfB12e4AQf3aaEYuAVdVnVfVs5e5BAFcAWA/gSVU9raqvAjgK4NqA2vOSqkZmwyaH9oT2/kTMtQCOquorqnoGwJMovzcLmqr+BIB1M7L1AP6l8nV/BNoTeS0VsCLyvIgcsdzWmx7fBuAvAXwSQBrAG6Zvf7NyLLD2WLwF4KOV7/uIiLwhIr8TYnt8f38sbbO2y2jbNwHsEpGHRORxv87vItD3oUm9qvobEXm/iBwWkT8JuT2G31PVtypfvz/UlpQZPzttIf3sNNRS68Gq6s1Oj4nIpwD8GYCPqaqKSA7AUtNTrgCQC6o9Ns89DeB05etDInIMwJUAPKuxzaU9KL8Xvr4/Zqr6YbfHReQxAD/06/wuAn0fmqGqucqfb4vIXpR72dZyUxh+LSKXV0L27bAbo6q/Nr4WkUh++mqpHqwTEVkH4AsA7lDVU5XD+wDcIyKXiMgKAKsA/CzENnYaNU4R+WClPa+E1R5E5P0RkcsrX96Jcq04aD8HsEpEVojIxSjXpfeF0A4AgIhcKiLvNb4GcAvCeV/s7APwV5Wvvx9mQ4Canx0gOu9RLVVt+RvK9cM3AExWbt+qHN8G4BiAGQAfD7A9d6L8UfM0gF8DGAPw5wCmK+37TwC3h9meMN8fS9v+FcAUyv95Lw+pDbeifHH0GIBtYbTB1JYPAjhcuU2H1R4A30G5rFWs/Oz8DYDfBfAjAL8CsDgC7TF+dl4M62en0Y0zuYiIfBKLEgERURQxYImIfMKAJSLyCQOWiMgnDFgiIp8wYImIfMKAJSLyCQOWiMgnDFgiIp8wYImIfMKAJSLyCQOWiMgnDFgiIp/8P1OgDcCOzTovAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "image = plt.imread('iris_umap.png')\n",
    "plt.axis('off')\n",
    "plt.imshow(image);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Graph Clustering](#toc)<a id=\"clustering\"></a>\n",
    "\n",
    "In this section, we show how to cluster the graphs created above using [HipMCL](https://bitbucket.org/azadcse/hipmcl/src). [HipMCL](https://bitbucket.org/azadcse/hipmcl/src) is a high-performance implementation of [Markov clustering algorithm](https://micans.org/mcl/) from the Department of Energy. After you install [HipMCL](https://bitbucket.org/azadcse/hipmcl/src) you can run it on one of the graphs created below using: \n",
    "\n",
    "`./bin/hipmcl -M approximate-knng.triples -I 2 -per-process-mem 20 -o knng_clusters.txt` \n",
    "\n",
    "You can also the following Python library to compute the clustering: https://github.com/GuyAllard/markov_clustering . When we cluster the graph above, we see that we recover three groups, each one referring to one of the three plants in the [dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set). \n",
    "\n",
    "\n",
    "[![](https://i.imgur.com/If8C70b.png)](https://i.imgur.com/If8C70b.png)\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Conclusion](#toc)<a id=\"conclusion\"></a>\n",
    "\n",
    "In conclusion, we have shown how to summarize data using graphs and to use these graphs to create embeddings with UMAP as well as cluster data. By isolating the graph construction from the embedding, we can create more modular systems and iterate faster on different graphs. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Contact](#toc)<a id=\"contact\"></a>\n",
    "\n",
    "\n",
    "[![](https://images.squarespace-cdn.com/content/5df7bcdf07ad51358b316d3b/1578603501512-N2LVHR2737YDF11HUQ1R/Orange+Logo+White+Text+Horizontal%403x.png?format=1500w&content-type=image%2Fpng)](https://rebelliondefense.com/)\n",
    "\n",
    "\n",
    "https://rebelliondefense.com\n",
    "\n",
    "* rafael AT rebelliondefense.com \n",
    "* aaron AT rebelliondefense.com \n",
    "* shay AT rebelliondefense.com \n",
    "* tom AT rebelliondefense.com "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}