word2vec_gnadenhutten.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word2Vec Analysis on the Gnadenhutten Massacre\n",
    "\n",
    "**Author**: [Ung, Lik Teng](https://github.com/unglikteng) <br>\n",
    "**Class**: [DH150, Winter 2019](http://asandersgarcia.humspace.ucla.edu/courses/dh150w19/) <br>\n",
    "**Instructor**: [Professor Ashley Sanders Garcia](http://asandersgarcia.humspace.ucla.edu/)\n",
    "\n",
    "Word2Vec is a popular word embedding, which is able to model words in high-dimensional space beyond frequency count. The advantage of Word2Vec is that it can capture the \"contexts\" of a word within a specific body of corpus. I trained a Word2Vec model on 9 newspaper articles on the Gnadenhutten Massacre that happened on March 8, 1782. I am interested in how different sides involved in this massacre were being discussed in public discourse. Specifically, I am interested in words that are most associated with the Moravian Indians and the American militia.\n",
    "\n",
    "**Table of Contents**\n",
    "* [1.Documents Import](#import)\n",
    "* [2.Text Preprocessing](#preprocessing)\n",
    "* [3.Word2Vec Training](#training)\n",
    "* [4.Word2Vec to Tensor](#tensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cython, os #ENSURE cython package is installed on computer/canopy\n",
    "import string, re, collections\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from string import ascii_letters, digits\n",
    "from smart_open import smart_open\n",
    "\n",
    "import gensim\n",
    "from gensim.models import phrases \n",
    "from gensim import corpora, models, similarities #calc all similarities at once, from http://radimrehurek.com/gensim/tut3.html\n",
    "from gensim.models import Word2Vec, KeyedVectors\n",
    "\n",
    "from sklearn.manifold import TSNE\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\n",
    "# import plotly.offline as py \n",
    "# py.init_notebook_mode(connected=True)\n",
    "# import plotly.graph_objs as go\n",
    "# import plotly.tools as tls\n",
    "\n",
    "import plotly.plotly as py\n",
    "import plotly.tools as plotly_tools\n",
    "import plotly.graph_objs as go\n",
    "\n",
    "plotly_tools.set_credentials_file(username='unglikteng', api_key='ho4TAl3mWMMNK6DnRSCL')\n",
    "\n",
    "from nltk import word_tokenize\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "plt.style.use('ggplot')\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "<a id=\"import\"></a>\n",
    "## 1. Documents Import\n",
    "\n",
    "The directory path and filename are hardcoded here. Import your text documents if you would like to analyze them with Word2Vec.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "primaries = []\n",
    "primaryPath = os.path.join(os.path.realpath(\"\"),\"primary\")\n",
    "for root, directories, file in os.walk('primary'):\n",
    "    for txt in file:\n",
    "        path = os.path.join(primaryPath, txt)\n",
    "        file = open(path, \"r\")\n",
    "        primaries.append(\"\".join(file.read().splitlines()))\n",
    "        file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 9 Newspaper articles\n",
    "len(primaries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Centenary of GnadenhuttenInformation about the Old Ohio Moravian Settlement and its MassacreSpecial Correspondence of the Cincinnati Gazette Newark, O., April 27, - Gnadenhutten was established by a Christian Indian named Joshua, who brought with him a party of Mohicans, and proceeded to lay out the town on 24th day of September, 1772. It was on the west side of the Tuscarawas river, four miles above Schronbrunn (a Moravian village already established), and was called “Upper Town”. This location, however, was not satisfactory to the Netawatwees, then the reigning chief of the Delaware nation, who caused it to be removed to a point about eight miles below Schonbrunn, on the east side of the river. Here Gnadenhutten (Tents of Grace) was laid out October 9, 1772, by Joshua and his party, who were from the Moravian village of Friedenstadt (City of Peace), located on the Beaver river, in Pennsylvania. This village was subsequently removed and added to the villages of Gnadenhutten and Schonbrunn. Rev. David Zeisberger preached the first sermon in Gnadenhutten October 17, 1772.The British seized the Moravian Gnadenhutten, and with their horses, cattle, etc., drove them prisoners to the “Sandusky plains,” by Captain Matthew Elliott of the British army (a while American renegade, however), who had under his command at the time 300 hostile Indians. They were made captives September 11, 1781, and the party reached Sandusky river on the first day of October following, when they went into camp. The leaders of these Moravians at the time of the removal were Revs. Zeisberger, Senseman, and Jungman, of New Schonbrunn; Revs. Heckwelder and Jung, of Salem, and Rev. William Edwards, of Gnadenhutten. This camp, subsequently known as “Captives’ Town,” was located in the heart of the then hostile Wyandot country, on the Sandusky river, about a mile above the mouth of Broken Sword creek, and ten miles from the present town of Upper Sandusky. Here the captives were allowed to build huts and go into winter quarters. Late in October, 1781, leaders only were ordered to Detroit, there to go before the British commandant, Major DePeyster, to answer to the charge against them of aiding the Americans. They soon proved themselves innocent, and were sent back to “Captives’ Town,” on the Sandusky.'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get a sense of how the article look like \n",
    "primaries[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "<a id=\"preprocessing\"></a>\n",
    "## 2. Text Preprocessing\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define our Text Preprocessor class\n",
    "## Tokenize -> Remove stopwords -> Stemming \n",
    "class Preprocessor:\n",
    "    def tokenize_word(self, sentence, to_token = None):\n",
    "        # all lower case \n",
    "        lower = sentence.strip().lower()\n",
    "        \n",
    "        # remove punctuation\n",
    "        punctuation_table = str.maketrans(string.punctuation, len(string.punctuation)*' ' )\n",
    "        noPunc = lower.translate(punctuation_table)\n",
    "        \n",
    "        # remove digit\n",
    "        nodigit = re.sub(r'\\d+', '', noPunc)\n",
    "        nodigit = re.sub(r'\\s+', ' ', nodigit).strip()\n",
    "        if to_token:\n",
    "            tokenized = word_tokenize(nodigit)\n",
    "            return tokenized\n",
    "        return nodigit \n",
    "    \n",
    "    def stem_word(self, tokens):\n",
    "        stemmer = SnowballStemmer(\"english\")\n",
    "        stemmed = []\n",
    "        for token in tokens:\n",
    "            stemmed.append(stemmer.stem(token))\n",
    "        return stemmed\n",
    "    \n",
    "    def remove_stopwords(self, tokens):\n",
    "        stopword_list = stopwords.words(\"english\")\n",
    "        filtered = [w for w in tokens if w not in stopword_list]\n",
    "        return filtered\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define preprocessor object\n",
    "preprocessor = Preprocessor()\n",
    "\n",
    "primariesToken = [preprocessor.stem_word(\n",
    "                        preprocessor.remove_stopwords(\n",
    "                            preprocessor.tokenize_word(line, to_token=True))) \n",
    "                              for line in primaries]\n",
    "\n",
    "primariesUnstemmed = [preprocessor.remove_stopwords(\n",
    "                        preprocessor.tokenize_word(line, to_token=True)) \n",
    "                              for line in primaries]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build stemming dictionary\n",
    "# This dictionary will help us trace back to the unstemmed words\n",
    "stem_dict = {}\n",
    "for i, row in enumerate(primariesUnstemmed):\n",
    "    for j, token in enumerate(row):\n",
    "        stem_dict.update({primariesToken[i][j]:token})\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize the corpus - Frequency Analysis\n",
    "\n",
    "def word_counter(list_of_doc):\n",
    "    countVec = CountVectorizer()\n",
    "    df_cv = countVec.fit_transform(list_of_doc)\n",
    "    word_freq = dict(zip(countVec.get_feature_names(), np.asarray(df_cv.sum(axis=0)).ravel()))\n",
    "    word_counter = collections.Counter(word_freq)\n",
    "    word_counter_df = pd.DataFrame(word_counter.most_common(20), columns = ['word','freq'])\n",
    "\n",
    "    a4_dims = (15, 10)\n",
    "    fig, ax = plt.subplots(figsize = a4_dims)\n",
    "    sns.barplot(x=\"word\", y=\"freq\", data=word_counter_df, palette= \"PuBuGn_d\",ax=ax)\n",
    "    return word_counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4IAAAJTCAYAAABdHKhfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XmcJGldJ/5PM4PXwoLQCjSgAy7HAsogl8g1nAoih8oXkB+3jLgisj8EERFQ1l0UEVEEbRQHlGX4AgKziByiMNzI5QzIjSDDzA40IIdcAr1/RBSTU1PVXT1dVVnVz/v9evWrMyMjI78VGfFEfOJ5MnPPwYMHAwAAwDgusuwCAAAA2F6CIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABiMIAgAADAYQRAAAGAwxy+7gE10cNkFAAAALNmejcx0LAXBnH322csuAQAAYCn27du34XkNDQUAABiMIAgAADAYQRAAAGAwgiAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABiMIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAzm+GUXsJXueNe7L7uEnPb8U5ddAgAAwPnoEQQAABiMIAgAADAYQRAAAGAwgiAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMMdvx4tU1RWTPDvJZZN8M8n+7n5KVV0qyfOSnJDko0mquz9bVXuSPCXJ7ZN8Kcl9u/sd21ErAADAsW67egS/nuRh3f1fk/xIkl+sqmskeWSSV3f3VZK8er6fJLdLcpX538lJnr5NdQIAABzztiUIdvc5Kz163f2FJO9Ncvkkd0ryrHm2ZyW583z7Tkme3d0Hu/vNSS5ZVZfbjloBAACOddsyNHRRVZ2Q5DpJ3pLkMt19TjKFxar63nm2yyf5+MLTzpqnnbNqWSdn6jFMd2fv3r1bW/yFsBNrAgAAxratQbCqLpbkhUke2t2fr6r1Zt2zxrSDqyd09/4k+1ceP3DgwKbUuZl2Yk0AAMCxZ9++fRued9u+NbSqLpopBD6nu/96nnzuypDP+f9PztPPSnLFhadfIcnZ21UrAADAsWy7vjV0T5I/T/Le7v79hYdOS3KfJE+Y/3/JwvQHV9WpSW6Y5HMrQ0gBAAA4Ots1NPTGSe6V5Myqetc87VGZAmBX1QOS/GuSu86PvSzTT0d8KNPPR9xvm+oEAAA45u05ePACH73brQ6effb5R4/e8a53X1Ip5znt+acuuwQAAGAA82cE1/q+lQvYts8IAgAAsDMIggAAAIMRBAEAAAYjCAIAAAxGEAQAABiMIAgAADAYQRAAAGAwgiAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABiMIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABiMIAgAADAYQRAAAGAwgiAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABiMIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABjM8dvxIlX1zCR3SPLJ7r7WPO15Sa42z3LJJP/W3SdW1QlJ3pvk/fNjb+7uB21HnQAAACPYliCY5JQkT03y7JUJ3X23ldtV9aQkn1uY/8PdfeI21QYAADCUbRka2t2nJ/nMWo9V1Z4kleS521ELAADA6LarR/BQbprk3O7+4MK0K1XVO5N8Psmju/t1yykNAADg2LMTguA9cv7ewHOSfF93f7qqrpvkxVV1ze7+/OonVtXJSU5Oku7O3r17t6XgI7ETawIAAMa21CBYVccn+akk112Z1t1fTfLV+fbbq+rDSa6a5G2rn9/d+5Psn+8ePHDgwJbXfKR2Yk0AAMCxZ9++fRued9k/H3HrJO/r7rNWJlTV91TVcfPtKye5SpKPLKk+AACAY862BMGqem6SNyW5WlWdVVUPmB+6ey74JTE3S3JGVf1TkhckeVB3r/lFMwAAABy5PQcPHlx2DZvl4Nlnn32+CXe8692XVMp5Tnv+qcsuAQAAGMA8NHTPRuZd9tBQAAAAtpkgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABiMIAgAADAYQRAAAGAwgiAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABiMIAgAADEYQBAAAGMzxyy5gdHe61/2XXUKS5CV/+cxllwAAAGwTPYIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABiMIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABiMIAgAADAYQRAAAGAwgiAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYzPHb8SJV9cwkd0jyye6+1jztcUkemORT82yP6u6XzY/9WpIHJPlGkod09yu2o04AAIARbEsQTHJKkqcmefaq6U/u7t9bnFBV10hy9yTXTLIvyd9V1VW7+xvbUSgAAMCxbluGhnb36Uk+s8HZ75Tk1O7+anf/S5IPJbnBlhUHAAAwmO3qEVzPg6vq3kneluRh3f3ZJJdP8uaFec6apwEAALAJlhkEn57k8UkOzv8/Kcn9k+xZY96Day2gqk5OcnKSdHf27t27NZUehZ1Y01p2S50AAMDRW1oQ7O5zV25X1TOSvHS+e1aSKy7MeoUkZ6+zjP1J9s93Dx44cGALKj06O7GmteyWOgEAgLXt27dvw/Mu7ecjqupyC3fvkuTd8+3Tkty9qr69qq6U5CpJ3rrd9QEAAByrtuvnI56b5KQke6vqrCSPTXJSVZ2YadjnR5P8fJJ093uqqpP8c5KvJ/lF3xgKAACwebYlCHb3PdaY/OeHmP+3k/z21lUEAAAwrqUNDQUAAGA5BEEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABiMIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABiMIAgAADAYQRAAAGAwgiAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABiMIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABiMIAgAADAYQRAAAGAwgiAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYzPHb8SJV9cwkd0jyye6+1jztiUl+MsnXknw4yf26+9+q6oQk703y/vnpb+7uB21HnQAAACPYliCY5JQkT03y7IVpr0rya9399ar6nSS/luRX58c+3N0nblNtAAAAQ9mWoaHdfXqSz6ya9sru/vp8981JrrAdtQAAAIxuu3oED+f+SZ63cP9KVfXOJJ9P8ujuft1yygIAADj2LD0IVtWvJ/l6kufMk85J8n3d/emqum6SF1fVNbv782s89+QkJydJd2fv3r3bVfaG7cSa1rJb6gQAAI7eUoNgVd0n05fI3Kq7DyZJd381yVfn22+vqg8nuWqSt61+fnfvT7J/vnvwwIED21L3kdiJNa1lt9QJAACsbd++fRued2k/H1FVP57py2Hu2N1fWpj+PVV13Hz7ykmukuQjy6kSAADg2LNdPx/x3CQnJdlbVWcleWymbwn99iSvqqrkvJ+JuFmS36qqryf5RpIHdfdn1lwwAAAAR2xbgmB332ONyX++zrwvTPLCra0IAABgXEsbGgoAAMByCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABiMIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABiMIAgAADAYQRAAAGAwgiAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAM5viNzFRVV97IfN39kaMrBwAAgK22oSCY5ENJDs639yzcXrmfedpxm1QXAAAAW2SjQfABSW6d5HFJPpbk+5M8Jsmru/uULakMAACALbHRIPj4JFfp7i/P9z9YVT+f5ANJTtmKwgAAANgaG/2ymIskOWHVtO+PoaAAAAC7zkZ7BJ+c5O+r6i+SfDzJFZPcd54OAADALrKhHsHufmKS+yW5TJI7Jrlskvt39+9uYW0AAABsgY32CKa7X57k5VtYCwAAANtgo78j+O2ZviX0Hkku3d2XqKrbJrlqdz91KwsEAABgc230y2KenORaSe6Z835D8D1JfmErigIAAGDrbDQI3iXJz3b3m5J8M0m6+xNJLr9VhQEAALA1NhoEv5ZVw0ir6nuSfHrTKwIAAGBLbTQIPj/Js6rqSklSVZdL8tQkp25VYQAAAGyNjQbBRyX5aJIzk1wyyQeTnJ3kN7emLAAAALbKYb81tKoukuQmSX61ux86Dwk90N0HD/NUAAAAdqDD9gh29zeTvKS7vzrf/5QQCAAAsHttdGjo6VX1I1taCQAAANtiQz8on+RjSf62ql6S5OM577cE092P2YrCAAAA2Brr9ghW1YMX7l4iyYszBcArJLniwj8AAAB2kUP1CP52pp+ISJKf7O7/vA31AAAAsMUOFQQ/XFVPSvKeJBetqvsl2bN6pu5+5lYVBwAAwOY7VBC8e5JHJLlHkosmufca8xxMIggCAADsIusGwe7+QJKfS5KqenV332rbqgIAAGDLbOjnI4RAAACAY8dGf0cQAACAY4QgCAAAMJiN/qD8UauqZya5Q5JPdve15mmXSvK8JCck+WiS6u7PVtWeJE9JcvskX0py3+5+x3bVCgAAcCzbzh7BU5L8+Kppj0zy6u6+SpJXz/eT5HZJrjL/OznJ07epRgAAgGPetgXB7j49yWdWTb5TkmfNt5+V5M4L05/d3Qe7+81JLllVl9ueSgEAAI5t2zY0dB2X6e5zkqS7z6mq752nXz7JxxfmO2ueds7ik6vq5Ew9hunu7N27d+srPkI7saa17JY6AQCAo7fsILiePWtMO7h6QnfvT7J/5fEDBw5saVEXxk6saS27pU4AAGBt+/bt2/C8y/7W0HNXhnzO/39ynn5WkisuzHeFJGdvc20AAADHpGX3CJ6W5D5JnjD//5KF6Q+uqlOT3DDJ51aGkAIAAHB0tvPnI56b5KQke6vqrCSPzRQAu6oekORfk9x1nv1lmX464kOZfj7ifttVJwAAwLFu24Jgd99jnYdutca8B5P84tZWBAAAMKZlf0YQAACAbSYIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABiMIAgAADAYQRAAAGAwgiAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABnP8sgtgd7jTzz142SUkSV7yZ09ddgkAALDr6REEAAAYjCAIAAAwGEEQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABiMIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAzm+GUXAJvpTg9+xLJLSJK85Km/u+wSAABgXXoEAQAABiMIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABjM8ct88aq6WpLnLUy6cpLHJLlkkgcm+dQ8/VHd/bJtLg8AAOCYtNQg2N3vT3JiklTVcUk+keRFSe6X5Mnd/XtLLA8AAOCYtJOGht4qyYe7+2PLLgQAAOBYttQewVXunuS5C/cfXFX3TvK2JA/r7s+ufkJVnZzk5CTp7uzdu3dbCj0SO7Gmtahzc+2WOgEAGNOOCIJV9W1J7pjk1+ZJT0/y+CQH5/+flOT+q5/X3fuT7J/vHjxw4MDWF3uEdmJNa1Hn5totdQIAcOzYt2/fhufdEUEwye2SvKO7z02Slf+TpKqekeSlyyoMAADgWLNTPiN4jywMC62qyy08dpck7972igAAAI5RS+8RrKrvSnKbJD+/MPl3q+rETENDP7rqMQAAAI7C0oNgd38pyaVXTbvXksoBAAA45u2UoaEAAABsE0EQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABiMIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABiMIAgAADAYQRAAAGAwgiAAAMBgjl92ATCiOz3sccsuIS950vJrAABgOfQIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABiMIAgAADAYQRAAAGAwgiAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABiMIAgAADOb4ZRcA7Fx3fvQTl11CXvw/Hr7sEgAAjjl6BAEAAAajRxDY9e78+D9edgl58W/84rJLAADYMD2CAAAAgxEEAQAABmNoKMA2uPMT/2LZJSRJXvzw+y27BABgB9AjCAAAMBhBEAAAYDCCIAAAwGB2xGcEq+qjSb6Q5BtJvt7d16uqSyV5XpITknw0SXX3Z5dVIwAAwLFiRwTB2S26+8DC/UcmeXV3P6GqHjnf/9XllAYwhjv/4anLLiFJ8uKH3P2Qj99l/2nbVMmhvejkOx7y8bs/61XbVMn6Tr3PbZZdAgA70E4eGnqnJM+abz8ryZ2XWAsAAMAxY6f0CB5M8sqqOpjkT7t7f5LLdPc5SdLd51TV965+UlWdnOTkeZ7s3bt3O2vekJ1Y01rUubl2Q527ocZEnZtNnZtrN9S5kRrv/Ecv3IZKDu3Fv/TTyy4BYCg7JQjeuLvPnsPeq6rqfRt50hwY9893Dx44cOBQsy/FTqxpLercXLuhzt1QY6LOzabOzbUb6twNNSa7p06AnWzfvn0bnndHDA3t7rPn/z+Z5EVJbpDk3Kq6XJLM/39yeRUCAAAcO5YeBKvqP1XVxVduJ7ltkncnOS3JfebZ7pPkJcupEAAA4Niy9CCY5DJJXl9V/5TkrUn+prtfnuQJSW5TVR9Mcpv5PgAAAEdp6Z8R7O6PJLn2GtM/neRW218RAADAsW0n9AgCAACwjQRBAACAwQiCAAAAgxEEAQAABiMIAgAADEYQBAAAGIwgCAAAMJil/44gAMBGPPSFb1x2CfmDn/7RZZcAsCn0CAIAAAxGjyAAwCb6jZe+bdkl5PF3uN4hH/+dV56xTZUc2q/e9ocO+fjTXvu+bark0P7bza++7BJg0+kRBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAbjdwQBAOAo/OWbPrLsEpIk97rRlQ/5+Ive/vFtqmR9d7nuFZddAjM9ggAAAIMRBAEAAAYjCAIAAAxGEAQAABiML4sBAAB2jL9797nLLiG3vtZlll3CltMjCAAAMBg9ggAAAEfozR/47LJLyI9c9bsv9HP1CAIAAAxGEAQAABiMIAgAADAYQRAAAGAwgiAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABiMIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABiMIAgAADAYQRAAAGAwxy/zxavqikmeneSySb6ZZH93P6WqHpfkgUk+Nc/6qO5+2XKqBAAAOLYsNQgm+XqSh3X3O6rq4kneXlWvmh97cnf/3hJrAwAAOCYtNQh29zlJzplvf6Gq3pvk8susCQAA4Fi37B7Bb6mqE5JcJ8lbktw4yYOr6t5J3pap1/CzSywPAADgmLEjgmBVXSzJC5M8tLs/X1VPT/L4JAfn/5+U5P5rPO/kJCcnSXdn796921f0Bu3Emtaizs21G+rcDTUm6txs6txcu6HO3VBjos7Nps7Ndfg6P7ItdRzO4ev8+LbUcSgbe8/P3fI6DmdDdX5g+f1UR7MPLT0IVtVFM4XA53T3XydJd5+78Pgzkrx0red29/4k++e7Bw8cOLDF1R65nVjTWtS5uXZDnbuhxkSdm02dm2s31LkbakzUudnUubnUuXl2Q43J7q1z3759G37uUn8+oqr2JPnzJO/t7t9fmH65hdnukuTd210bAADAsWrZPYI3TnKvJGdW1bvmaY9Kco+qOjHT0NCPJvn55ZQHAABw7Fn2t4a+PsmeNR7ym4EAAABbZKlDQwEAANh+giAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABiMIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABiMIAgAADAYQRAAAGAwgiAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMIIgAADAYARBAACAwQiCAAAAgxEEAQAABiMIAgAADEYQBAAAGIwgCAAAMBhBEAAAYDCCIAAAwGAEQQAAgMEIggAAAIMRBAEAAAYjCAIAAAxGEAQAABiMIAgAADAYQRAAAGAwgiAAAMBgBEEAAIDBCIIAAACDEQQBAAAGIwgCAAAM5vhlF3AoVfXjSZ6S5Lgkf9bdT1hySQAAALveju0RrKrjkvxxktsluUaSe1TVNZZbFQAAwO63Y4Ngkhsk+VB3f6S7v5bk1CR3WnJNAAAAu95ODoKXT/LxhftnzdMAAAA4CnsOHjy47BrWVFV3TfJj3f1z8/17JblBd//SwjwnJzk5Sbr7ukspFAAAYOfYs5GZdnKP4FlJrrhw/wpJzl6cobv3d/f1uvt6mf7gTf1XVW/fiuWqc2f/U+d4de6GGtWpzp3+bzfUuRtqVKc6d/o/de6KGjdkJ39r6D8muUpVXSnJJ5LcPcnPLrckAACA3W/H9gh299eTPDjJK5K8d5rU71luVQAAALvfTu4RTHe/LMnLlljC/iW+9pFQ5+ZS5+baDXXuhhoTdW42dW6u3VDnbqgxUedmU+fmUufmWWqNO/bLYgAAANgaO3ZoKAAAAFtjiCBYVW88wvlPqqqXzrfvWFWP3JrKtkdVXa+q/nCTl3lKVf3MhXzufavqqUf4nDtX1TVWLWPfhXn9naSqTqiqd29gvt+qqlvPtx9aVd+18NjLquqSh3juR6tq7xHW9ZCqem9VPWedxw+5TVXVc6vqjKr674u1H2L+NbenxX1xs1TVF9eZ/qCquvd8+5jYvnaCI21/j3DZF6p93sqaFl7jW9vTBuc/5H68E6zsO1W1r6pesOx6VlTVJavqv823d1RtG3Go48BG2k9Y3Ad20rJ2ogtzDnqY5W34fPhwx56qetSRzL8ZdvRnBDdLd//oUTz3tCSnbWI5R6Wqjp+/SGfDuvttSd62RSVtlzsneWmSf57v3zfJu7PqJ0WORVV1XHc/ZmHSQ5P8VZIvJUl3334LXva/Jbldd//LWg+ut01V1fFJ9ib50e7+/i2oa8t0958s3L1vBtm+tsq83X7jaNrfw7mw7fNW1pR8q53+k8PP+a359yS5Q3d/8yhfd0+SPUe7nMPp7rOTXKgLgVvkkpnarKftwNqOyqq2f9dZaQeWXcdG7bZ6k6nmLOwDm7DIzVzWUm1Gm3hhzrvXWMZGj4ePSvI/V+5s9bEqGSQIVtUXu/tiVXVSksclOZDkWknenuT/6+6DVfXjSf5gfuwdC8+9b5LrdfeDq+onkzw6ybcl+XSSe3b3uVX1uCTfl+TK8/9/0N3n6y2pqhOSvDzJ65P8SJJ/SvIXSX4zyfcmuWeSDyV55rycLyU5ubvPmJe/L8kJSQ7MVwz+Msl/mhf/4O5+Y1U9L8mz5i/ZSVWdkuT/JLl9krsleVOSbya5WpJLJ/n8/Hd8R5IXJLn6PP0GST6X6SR4Zdl7kvxRklsm+Zcs/EZJVV03ye8nudi8/u7b3edU1WuSvCXJLTI1LA/o7tfNT9tXVS9P8gNJXtTdj1h8r+bbP5PkDpk+SHvHJDevqkcneW6S6yV5TlV9OcmNklxjozUkuX6S+891/FmSFyf52/m9+dFMP1dyp+7+clX9QJI/TvI983vywO5+XzbX8VX1rCTXSfKBJPfOFHifmeS2SZ46b58vzbQd7EvyD1V1oLtvUVUfndfHl5N0pt/cPC7J47v7efNr/NK8/V40yV0P9TdU1Z9k2gZPq6q/SnKnJN85L/9+3f3+eV/6le6+w+rtM9O+9b1V9a4kv5Rpnb+0u1+w3ray6vXX3Bc3qqoekeQr3f2HVfXzkQ7ZAAASsklEQVTkJNfu7ltW1a2S3G+e57czbVtfzvRer+zHX0yysj4Pu30daW2HqPmEHL59eE+mffAHM7Xdj+vul8xt1B2TfFdW7U8beN0XZ/q91u9I8pTu3j/3+vxxklsn+WymA9PvZmrbHtrdp80nHk9IclKSb0/yx939p/N28dgk5yQ5Mck1Vu3Tj0hyr0zt0N929yOr6oFJTs7Urn4oyb26+0tzr84tk/xHku9O8uYkv7VqnVwj57XPd51f+xtJPtfdN6uqa87r8dsyjYD56e7+4MIxYc/8t90uycEk/6O7n3eYY8Vjkvxkpn3ijUl+fp7+mvn+jTPtOxdP8sXu/r111v0Jmdqdf8i0jZ1YVd+T5OFJPtbdT5vne1ySL3T3k6rq4UlqXucv6u7HrrGcOyf52Prv+tGbX/Ol3X2tqnpLkvuvfKv3vB4eluR9WWN73aKSnpDkB+Y254NJ/utc230zrY/jMr2PT8q0LdwryVeT3L67P7NN7fy3VNX/nwseg46rqmfkgsegU3Je+3mrJL+XaX3+Y5Jf6O6vbmGdh2tLv5DpePqdSV7Q3Y+dn/fRLBy/kpy6RfU9PsmB7n7KfP+3k5ybaf84334yP36B9m6e/sVM7fuPZdp2X38harl3kl/J1I6ckelc8ZmZtqlPZTp2/uv8fn4+0zHmskkeMb+3652/vTgba2vPyHn7wKu6++FH+jcsWNyf3plpHZ5WVS9K8tnuvn9VPSDJlbr70au35+7+g6N47SQbPibePgtt7Nyrfod5EedrE6vqlkl+LdP6+kCm/T9zm/snmY5vyXSMe8Ma5zXf+vm6Nd7rbyS52bweFt/Tk7LO8bCqLpfkeUn+c6b9+ReS/ESS75zX+3u6+54L818syUsyHQsvmuTR8/H/hKxz/rrRdT3E0NBVrpOpR+UamU52b1xV35HkGZkO7jfN9Eau5fVJfqS7r5OpYVs82bp6pkbkBkkeW1UXXeP5/yXJU5L80Dz/zya5SaYN6lGZNvB3dvcPzfefvfDc62Z6c382ySeT3Ka7fzhTwFsJnafO91NV35bkVkn+b5KbJzk9yU/NNTwt04nNGzMdNB+a6WT/xPn5l820k/zywrLvkilA/mCSB2ba4DL/nX+U5Ge6+7qZGr7fXqj7+O6+wfwaj12YvvJaP5jkblV1xTXWV5Kku9+Y6ar/w7v7xO7+nUy9Uffs7hOTfP0Iavi9TAewG2ZqXB6Yace6SqYG9ppJ/i3JT8/P3Z/kl+bl/kq25grZ1ZLsn9/3z2e6EpdMB+CbdPe3DqLzBYazk9yiu2+xajk/nuTs7r52d18rUyO64sC8vTx9/jvW1d0PWnmNef6bzdv8Y7JwpWqVxe3zjkk+PL9XK8F/I9tKjmBfPJTT5+cm08H2YvNr3yTJ6zJdQHlzd197nveBq/7+F+TItq/Ncrj24deT/H13Xz/Te/PEqlq5GLTh/WmV+89/0/WSPKSqLp1p/bxmnv6FJP8jyW0ytQG/NT/vAZnC1vUznQg+sKbffE2mNvDXu/saC6+TqrpdppPyG87r/nfnh/66u68/T3vvvOwVl5xf+zrzclevk0WPSfJj83LuOE97UKYTvhPnv/GsVc/5qUzr7tqZgu8T5wN0ssaxYp7+1Lnea2U6Ab7DwvIu2d037+4nZWOuluTZ8/61Et6+1Y7PKsnzq+q2mdqpG8w1X7eqbrZ6Od29pSFwDafONWZed/u6++059Pa62R6Zuc3JFKQXXSvTdnODTPvtl+b1/aZMF92S7Wnnk3zrwumRHINWnvcdSU5JcrfuXgnXv7BVdc4O15b+endfL1ObdfOq+qGF517g+LUF/jzJfZKkqi6S6femz836+8la7V0ytXnv7u4bdveFCYHXzLS933Juf345UwB+9nxcf07OO5dKkstlWod3yBS6krXP316Wjbe139oHjjIEJuffn16R87aBy2dqDzPX/7q1tuequs5Rvv6Kwx0TD2Wxbf1apvPrG2c6niwem56S5Mnz+v3pTBdmViye1yRZ971O1n5Pk3WOh/Pf8op5HV87ybu6+5FJvjy/h/dcNf9XktxlPo+7RZInzRcyk8O0HYczYhB8a3ef1VM38bsypf2rJ/mX7v5gdx/MNOxuLVdI8oqqOjPTweaaC4/9TXd/tbsPZApql1nj+f/S3WfOr/2eJK+eX+/MuY6bZOrpS3f/fZJLV9Ul5ueetpDwL5rkGXMdz895G/XfJrllVX17pivcp2dqOF6f5Jvd/YUkr850NfT6SX54/hvenmk436sz9ZA8NcmlMp3wriz7Zkme21PX9tlJ/n6efrVMB9pXzVcxHj2vpxV/Pf//9vlvXPHq7v5cd38lU+/X0QwjPJIarpzp6ta/d/cX58dumum9eddirfMVmB/NdBL2riR/mmln32wf7+43zLf/KtN2kExXi47EmUluXVW/U1U37e7PLTy23vtwOJfI9Pe/O8mTc/5tftHi9rmew71Pycb3xUN5e6aD/8UzXdB4U6YD/00znbx8LVPv6sq8J2xC3ZvhcO3DbZM8cq7hNZmuaq9cxbyw+9NDquqfMvW2XTHTAeVrOe8iwplJXtvd/7FQR+Za7j3X8pZMIwmuMj/21l57SPGtk/xFd68Maf7MPP1aVfW6uT27Z86/jX2yu8/oqbfpuDXWyaI3JDll7mE8bp72piSPqqpfTfL9a2yjN8l57dq5SV6bqW1c+TtWHyuS5BZV9Za53luuqvdI99mPdfebFyd09zsz9ajvq6prZ7oC/6+Z1vltM12hf0emfeUq6y1nG3WSu863K9MxKTn09rqd/qG7v9Ddn8o00uX/zNPPzPa28ytukg0eg1Y972rzPB+Y7z8r03F5Kx2uLa2qekembfKaOf8J9pHuC0esuz+a5NNz8FjZN66f9feTtdq7ZOrReeFRlHLLTD2iB+a6PpOpJ+p/z4//Zc47rifJi7v7m939zznvXPEC529ze3Vh2trN9LokN63pOxr+Ocm58wWfG2XqTFhve94MhzsmHspim3jDTBc3P9XdX8v5t81bZxp19a5MHQ7/ed7ek7XPa9Z6r5O139Nk/ffoH5Pcb+55/MH5/PxQ9iT5n1V1RpK/yxTKV17ncG3HIQ0xNHSVxWEU38h562Ajv6PxR0l+v6cu8pMyDR063HLXe+1vLtz/5jz/WmOQV+r694Vp/z3TVa9rZwrzX0mS7v7KPCznxzJdWXpupisqi26cqRfw05mGyfzRXO9xcz0ry35tphP/V6xRy6I9mbqwb7TGY1n4G1evk428D9+xzjKPpobjDjPPynzfmWnd/tt8xWYrrV6va73nh9XdH5ivzt0+yf+qqld290oPznrvw+E8PtOJ1F3mIQivWWe+jdR6uPdpxVH9pk13/8c8NOl+mQ5UZ2S6gvYDmXqc/mM+mCQbWx8brftoHa59+EamoY3vX3xSVd0wG2t/zmduw26d5EY9DcV8TaZ9bnH9fKuO7v5mTZ8BTaZ18kvd/Yo1lrnetrAna7+3pyS5c3f/0zyU76SFx762at7V6+RbuvtB87r4iSTvqqoTu/t/1zR08ScyXcT7ufki22JN67nAOp17ZZ6WaTjqx+eD+GI7dUT77CHmf0Gmz7ldNucNq9uT5H91958uzjjvl0f6upumuz9RVZ+ee4PuluTn54f2ZI3tdQkOt19tVzu/Yr1tbq1j0Eaet2UO05Z+OVPPzPW7+7PzUMaj2RcurD/L9Jnuy2a6eH2rrL2fnJS127tk6r08ms8Frte2LVp8fPG93pOse/628viRtrWbZt6/vzvTiKPTM3USVKYhmV9Y6JXaChs5Z17s0DrU9rfe+3ORTNvE+QJfVa21jGT99/oC7+k6dSRJuvv0uaf6J5L8ZVU9sbufvda8s3tmGmZ83YX9cuXvPVzbcUgj9giu5X1JrlTT5wSS5B7rzHeJTONvk3k4wiY7PdObvbKTH+juz69TxznzVZJ75fzh5tRMjfZNM4W4lXHDF5mvfH5fpquiyTT8bs1lz7dvs7Ds05PcvaqOm68GrQxLfH+S76mqG811X3TuOr+wzq2q/zoP87jLwvQvJLn4OvePpIavZBov/l3zMKW7ZLridQHzuv+Xmj57lKraM1+h32zft1J7pm3vcENTVq+LJNO35WUa9vRXmYbA/vAm1La4zd/3KJe1kfdpo/vi4Zye6STl9Ezv74MyDb3YaMi8sNvXVnpFps967pnrONrhN5fI1Nv0paq6eqZhPUdSyy/Mw8RSVVfdwLC/Vya5f83feFtVl5qnXzzJOfOyVg+H2bCq+oHufktPX65xIMkVq+rKST7S05Dq0zINMVp0eqahtMfV9FmRmyV56yFeZuXAe2BuT7fqS0lOzTTM7WcyhcJkWuf3n183VXX5qvreLXr9I7XyUYlLdPeZ87TN3l4PZc02cSO2sZ1fcXo2eAxa5X2ZejBXLu7eK9MF2622Zlua6bNN/57kc1V1mUy9WMvwokwh5fqZtrn19pOjae8O59XTS01DTee27Y2Z9uFkatc2MuR09flbsvG29kLvAxtY1psyDZNf2QZ+Jedtsxd2e94MH818nlNVP5zkSuvM95YkJ1XVpef1eNeFx16Z5MErd6rqcBeE1nqvj1hVfX+mES/PyDTEeeV87T9q7Y+WXWKe/z+q6hY5ulF05yMIZroSk+nLCv6mql6f9T9o/7hMw0del+lEY7M9Lsn15q7fJ2T9sPm0JPepqjcnuWrOf8XhlZlOZv6uu7/W3f+YqUG6WaYu+zdmuuJ0naz9/j9tft0bZxo6sbLsF2XqSTwz0+fGXpskczf7zyT5nXnIxbsyf37wQnpkpmF7f5/zAmkyNZAPr6p3ziHhlCR/MnfnH3cENXxtfu5bMzUOf5bpCzHWc88kD5iX+55Mn6XcbO/N9H6ekelq29MPM//+JH9bVf+wavoPJnnrvE5+PdNnu47W72bqXXxD1u9N3ZCNbCtHsC8ezusyDe960zzk7ys5soPTKblw29dWenymYeFn1DRU9/FHubyXZ+rlOmNe1pEMLfyzTMOE3jHX8qc5TC9kd788Uxh727xeVz6r+huZ9sVXZTrZvbCeWFVnzvWcnunLBe6W5N3z61095//cdTK1a2fM8/59pg/5/99D/A3/lukzrGdm+hKHfzyKetc1D4W9eJJP9PylRN39ykxDzd5U07DUF2TzTvyO1gsynfT2wrTN3l7X1d2fTvKG+XWeeCEWsR3tfJKku9+RIzsGrTzvK5lCwvPn9/+bmb7kYqut2ZZ29z9lGn75nkw9cW9YfxFbZz6u/MN0s79xiP3kaNq7w9XwnkyfP33tvA39fpKHZBr6d0am0P7Lh1jEivOdv83TNtTWLu4DVXVh9oFDLet1mb5r4UOZhtteap625vY8D2/fDi9Mcqm5ff+FTF8CcwFzG/q4TIH273L+L6F7SObz7qr650wXOta1znt9YZyUaeTKOzN9pu8p8/T9mdrM1T/d9Zy5zrdlaq827cus9hw8eFSjsNgFqupi3f3F+Ur86Zm+jfSIv40RAGCnmEcPvSPTt2F/cNn1wG6jR3AM++crJu9I8kIhEADYzWr6ApMPZfoSESEQLgQ9ggAAAIPRIwgAADAYQRAAAGAwgiAAAMBgBEEAWIKqelxV/dWy6wBgTIIgAADAYARBANhCVbVn/r0zANgxjl92AQCwk1TV/ZL8VHf/5Hz/Q0ne0d013/94kp9M8l1JnpLkqkk+kOSXu/uN8zyvSfKGJCcl+eEkP1hV30hyynz/zUnev21/FACs4golAJzfa5PctKouUlWXS3LRJDdOkqq6cpKLJfnXJH+T5A+TXDrJ7yf5m6q69MJy7pXk5CQXT/KxJP87yduT7E3y+CT32Za/BgDWIAgCwILu/kiSLyQ5McnNk7wiySeq6urz/dcl+YkkH+zuv+zur3f3c5O8L1NP4YpTuvs93f31JJdLcv0kv9HdX+3u05P8n+37qwDg/AwNBYALem2mYZ3/Zb79b5lC4I3m+/sy9fIt+liSyy/c//jC7X1JPtvd/75q/ituatUAsEF6BAHgglaC4E3n26/NFARvPt8+O8n3r3rO9yX5xML9gwu3z0ny3VX1n1bNDwBLIQgCwAW9Nsktknxnd5+VaTjoj2f6POA7k7wsyVWr6mer6viquluSayR56VoL6+6PJXlbkt+sqm+rqpvk/MNIAWBbCYIAsEp3fyDJFzMFwHT355N8JMkbuvsb3f3pJHdI8rAkn07yiCR36O4Dh1jszya5YZLPJHlskmdv3V8AAIe25+DBg4efCwAAgGOGHkEAAIDBCIIAAACDEQQBAAAGIwgCAAAMRhAEAAAYjCAIAAAwGEEQAABgMIIg8P/arwMBAAAAAEH+1oNcFgEAMCOCAAAAMwG1k5qTbEUMbgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "wc_primaries = word_counter([\" \".join(tokens) for tokens in primariesToken])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "<a id=\"training\"></a>\n",
    "## 3. Word2Vec Training \n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Text Preprocessing -> Phrase Detection with Gensim -> Word2Vec Training \n",
    "## Phrase Detection\n",
    "\n",
    "bigram_transformer = phrases.Phrases(primariesToken) \n",
    "bigram= phrases.Phraser(bigram_transformer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Word2Vec Hyperparameters**:\n",
    "* skip-gram method is used instead of CBOW (Continuous Bag of Words) since skip-gram generally performs better on small dataset\n",
    "* Dimension of word vectors: 500\n",
    "* min_count: since the corpus is pretty small, set min_count to 2 is reasonable\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_primaries = Word2Vec(bigram[primariesToken], workers=4, sg=1,size=500,window=5, min_count = 2, sample=1e-3)\n",
    "\n",
    "model_primaries.init_sims(replace=True) #Precompute L2-normalized vectors. If replace is set to TRUE, forget the original vectors and only keep the normalized ones. Saves lots of memory, but can't continue to train the model.\n",
    "model_primaries.save(\"model_primaries\") #save your model for later use! change the name to something to remember the hyperparameters you trained it with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the model\n",
    "model_p = Word2Vec.load(\"model_primaries\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1318"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# There are 1318 words in the vocabulary\n",
    "len(model_p.wv.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['old', 'ohio', 'moravian', 'settlement', 'correspond', 'cincinnati', 'newark', 'april', 'gnadenhutten', 'establish', 'christian_indian', 'name', 'joshua', 'brought', 'parti', 'proceed', 'lay', 'town', 'th', 'day', 'septemb', 'west', 'side', 'tuscarawa_river', 'four', 'mile', 'villag', 'alreadi', 'call', '“', 'upper', '”', 'locat', 'howev', 'reign', 'chief', 'delawar', 'nation', 'caus', 'remov', 'point', 'eight', 'schonbrunn', 'east', 'river', 'tent', 'grace', 'laid', 'octob', 'peac', 'pennsylvania', 'subsequ', 'rev', 'david_zeisberg', 'preach', 'first', 'british', 'seiz', 'hors', 'cattl', 'etc', 'drove', 'prison', 'sanduski', 'plain', 'captain', 'matthew', 'elliott', 'armi', 'american', 'command', 'time', 'hostil', 'indian', 'made', 'captiv', 'reach', 'follow', 'went', 'camp', 'leader', 'zeisberg', 'senseman', 'new', 'heckweld', 'jung', 'salem', 'william', 'known', '’', 'heart', 'wyandot', 'countri', 'broken', 'creek', 'ten', 'present', 'allow', 'build', 'hut', 'go', 'winter', 'quarter', 'late', 'order', 'detroit', 'major', 'answer', 'charg', 'aid', 'soon', 'prove', 'innoc', 'sent', 'back', 'fear', 'massacr', 'interest', 'event', 'advoc', 'week', 'two', 'happen', 'tuscarawa', 'counti', 'took', 'visit', 'site', 'ancient', 'trust', 'brief', 'account', 'place', 'terribl', 'may', 'reader', 'conclud', 'make', 'subject', 'communic', 'deserv', 'among', 'respect', 'valuabl', 'church', 'great', 'britain', 'origin', 'brethren', 'law', 'christ', 'know', 'unit', 'one', 'peculiar', 'unusu', 'belief', 'say', 'exhibit', 'submit', 'import', 'concern', 'member', 'lot', 'consist', 'number', 'small', 'cylind', 'inch', 'long', 'half', 'construct', 'end', 'pull', 'apart', 'disclos', 'word', 'yes', 'case', 'alik', 'far', 'appear', 'contain', 'use', 'princip', 'matter', 'instanc', 'young', 'man', 'mind', 'would', 'like', 'certain', 'woman', 'wife', 'minist', 'state', 'take', 'littl', 'put', 'thorough', 'consid', 'provid', 'approv', 'match', 'reason', 'although', 'need', 'marri', 'yet', 'also', 'much', 'decid', 'whether', 'accept', 'missionari', 'field', 'labor', 'character', 'hold', 'exampl', 'other', 'everi', 'bodi', 'christian', 'whatev', 'persuad', 'engag', 'mission', 'care', 'quarrel', 'carri', 'address', 'men', 'given', 'bethlehem', 'alway', 'still', 'center', 'unit_state', 'home', 'offic', 'societi', 'sever', 'year', 'revolutionari', 'war', 'wilder', 'convert', 'tribe', 'met', 'good', 'success', 'larg', 'savag', 'built', 'inhabit', 'three', 'within', 'goshen', 'washington', 'stand', 'beauti', 'situat', 'bank', 'south', 'philadelphia', 'columbus', 'railroad', 'hundr', 'faith', 'neat', 'meet', 'hous', 'pass', 'street', 'struck', 'quiet', 'throughout', 'modern', 'mani', 'gabl', 'ret', 'tradit', 'past', 'peopl', 'rush', 'hurri', 'world', 'around', 'simpl', 'tast', 'deepli', 'religi', 'earth', 'desir', 'daili', 'prayer', 'strife', 'sober', 'wish', 'never', 'learn', 'along', 'life', 'keep', 'way', 'attend', 'servic', 'sunday', 'inform', 'school', 'upon', 'congreg', 'born', 'seventi', 'year_ago', 'kind', 'accompani', 'spot', 'edg', 'eye', 'sacr', 'found', 'modest', 'foot', 'part', 'embrac', 'adjoin', 'graveyard', 'enclos', 'fenc', 'stood', 'occur', 'cruel', 'honor', 'grow', 'forest', 'tree', 'grown', 'sinc', 'seen', 'appl', 'plant', 'garden', 'ground', 'cellar', 'visibl', 'taken', 'char', 'corn', 'wood', 'pick', 'stone', 'piec', 'burn', 'red', 'hard', 'bore', 'mark', 'heat', 'lie', 'heap', 'purpos', 'erect', 'monument', 'last', 'rest', 'perhap', 'incid', 'histori', 'cruelti', 'equal', 'butcheri', 'pale', 'bloodi', 'slaughter', 'murder', 'king', 'excus', 'issu', 'mistak', 'true', 'addit', 'civil', 'nineti', 'act', 'without', 'even', 'grassi', 'could', 'scarc', 'horror', 'listen', 'recit', 'dark', 'deed', 'commit', 'sat', 'said', 'live', 'attach', 'wa', 'troubl', 'whose', 'territori', 'alli', 'look', 'leagu', 'white', 'hand', 'station', 'fort', 'pitt', 'pretend', 'harass', 'fire', 'summer', 'band', 'came', 'threat', 'promis', 'safeti', 'leav', 'crop', 'drag', 'hear', 'governor', 'noth', 'discharg', 'suffer', 'untold', 'privat', 'cold', 'permit', 'return', 'women_children', 'gather', 'advis', 'near', 'starv', 'els', 'arm', 'hunt', 'depred', 'frontier', 'rob', 'famili', 'mingo', 'cloth', 'stolen', 'pursu', 'compani', 'immedi', 'rais', 'colonel', 'williamson', 'set', 'arriv', 'night', 'march', 'next', 'morn', 'discov', 'advanc', 'cross', 'see', 'accost', 'come', 'protect', 'give', 'began', 'differ', 'spirit', 'bound', 'shut', 'nineti_six', 'consult', 'held', 'done', 'soldier', 'form', 'line', 'col_williamson', 'question', 'favor', 'save', 'step', 'forward', 'death', 'eighteen', 'whole', 'resolv', 'blood', 'men_women', 'children', 'meantim', 'suspect', 'dread', 'result', 'prepar', 'fate', 'pray', 'sing', 'hymn', 'aw', 'execut', 'commenc', 'doom', 'victim', 'brain', 'cooper', 'mallet', 'continu', 'cours', 'left', 'work', 'accomplish', 'repeat', 'treacheri', 'tragedi', 'right', 'escap', 'unobserv', 'warn', 'stun', 'blow', 'scalp', 'recov', 'conscious', 'departur', 'tell', 'stori', 'anoth', 'boy', 'hide', 'confin', 'flame', 'third', 'beg', 'conceal', 'horrid', 'enact', 'god', 'disast', 'die', 'kill', 'lightn', 'fiendish', 'torn', 'least', 'us', 'crawford', 'autumn', 'reveng', 'inhuman', 'chin', 'seem', 'silent', 'think', 'join', 'martyr', 'love', 'well', 'bone', 'sad', 'gray', 'hair', 'forc', 'august', 'breez', 'chant', 'depart', 'dreami', 'hill', 'safe', 'breath', 'break', 'sleep', 'shriek', 'aros', 'pain', 'chicago', 'find', 'month', 'scene', 'horribl', 'record', 'detail', 'previous', 'compos', 'english', 'mask', 'friendship', 'endur', 'hardship', 'persecut', 'blame', 'thank', 'prais', 'divis', 'fifti', 'forsaken', 'greater', 'portion', 'fall', 'face', 'actor', 'notori', 'david', 'destroy', 'suppos', 'trade', 'wild', 'actual', 'wrong', 'busi', 'usual', 'captur', 'militari', 'pittsburgh', 'vote', 'eighti', 'twenti', 'determin', 'news', 'almost', 'implicit', 'confid', 'devout', 'serv', 'offer', 'strength', 'encourag', 'infant', 'closer', 'mother', 'breast', 'brave', 'women', 'chosen', 'led', 'enter', 'butcher', 'pretti', 'poor', 'merci', 'dead', 'captor', 'manner', 'shot', 'tomahawk', 'various', 'attempt', 'rise', 'despatch', 'slaughter_hous', 'partial', 'consum', 'remain', 'buri', 'friend', 'person', 'perpetr', 'light', 'evid', 'fail', 'secur', 'kept', 'fortun', 'togeth', 'forti', 'thirti', 'former', 'six', 'acr', 'purchas', 'organ', 'object', 'perpetu', 'memori', 'exact', 'precious', 'nine', 'dollar', 'view', 'solicit', 'public', 'general', 'receiv', 'acknowledg', 'suggest', 'moravian_missionari', 'strong', 'beyond', 'doubt', 'grave', 'marbl', 'slab', 'commemor', 'tribun', 'histor', 'sun', 'testimoni', 'afterward', 'june', 'appropri', 'inscript', 'hope', 'midway', 'increas', 'popul', 'store', 'effort', 'fix', 'capit', 'today', 'celebr', 'white_settler', 'circumst', 'lead', 'render', 'induc', 'covert', 'land', 'clear', 'thrive', 'enjoy', 'comfort', 'outrag', 'treatment', 'endeavor', 'valley', 'refus', 'natur', 'thus', 'avoid', 'border', 'bring', 'start', 'human', 'expedit', 'plunder', 'pillag', 'influenc', 'knew', 'atroc', 'wallac', 'eastern', 'fled', 'toward', 'mrs', 'settler', 'pursuit', 'harbor', 'dress', 'earli', 'col', 'recent', 'barbar', 'finish', 'treat', 'harm', 'sooner', 'surrend', 'accus', 'told', 'propos', 'thirsti', 'share', 'decis', 'therefor', 'ask', 'moment', 'devot', 'tie', 'behind', 'scatter', 'room', 'progress', 'aliv', 'gun', 'knife', 'brutal', 'turn', 'pen', 'regist', 'regret', 'feel', 'fact', 'centenni', 'translat', 'mean', 'seven', 'fell', 'cheer', 'mingl', 'shout', 'signific', 'necessari', 'rare', 'pieti', 'heard', 'heathen', 'sold', 'trader', 'gentl', 'evil', 'effect', 'forcibl', 'becam', 'spread', 'flourish', 'convers', 'conspiraci', 'famous', 'ottawa', 'plot', 'though', 'harsh', 'ignor', 'canaanit', 'exist', '…', 'nativ', 'fit', 'fair', 'high', 'settl', 'fertil', 'soil', 'pleasant', 'prevail', 'pipe', 'coloni', 'broke', 'teacher', 'seat', 'headquart', 'cruelli', 'accord', 'crime', 'move', 'head', 'surround', 'smile', 'final', 'conduct', 'molest', 'complet', 'deceiv', 'prevent', 'manifest', 'council', 'mode', 'excit', 'idea', 'futur', 'campaign', 'bleed', 'written', 'demonstr', 'must', 'morrow', 'affair', 'believ', 'show', 'realiz', 'fortitud', 'larger', 'watch', 'outsid', 'produc', 'babe', 'parent', 'etern', 'quick', 'cut', 'similar', 'dwell', 'stay', 'short', 'frequent', 'retir', 'canada', 'pious', 'sixti', 'reflect', 'imposs', 'except', 'st', 'proper', 'observ', 'approach', 'especi', 'presid', 'arthur', 'hay', 'invit', 'deliv', 'new_fairfield', 'particip', 'wyom', 'probabl', 'struggl', '–', 'gospel', 'mahon', 'lehighton', 'elev', 'posit', 'novemb', 'attack', 'burnt', 'inscrib', 'lord', 'renew', 'mohagan', 'maintain', 'north', 'lehigh', 'possess', 'separ', 'begin', 'road', 'path', 'mountain', 'warrior', 'plantat', 'martin', 'resid', 'succeed', 'polici', 'chang', 'might', 'alon', 'mohegan', 'languag', 'bishop', 'foundat', 'shawne', 'hatchet', 'french', 'resolut', 'chapel', 'weissport', 'cultur', 'defeat', 'open', 'neighbor', 'caution', 'enemi', 'possibl', 'compli', 'nov', 'georg', 'custard', 'expect', 'joseph', 'sturg', 'got', 'door', 'ran', 'partch', 'window', 'child', 'stair', 'best', 'jump', 'hid', 'stump', 'saw', 'abus', 'perish', 'twelv', 'stabl', 'five', 'parich', 'blanket', 'meant', 'abl', 'deliver', 'report', 'delay', 'contrari', 'bed', 'brother', 'notic', 'militia', 'ventur', 'troop', 'stockad', 'properti', 'strategi', 'quit', 'later', 'januari', 'benjamin', 'susanna', 'lost', 'surpris', 'moravian_convert', 'search', 'canadian', 'borderland', 'john', 'p', 'bow', 'journal', 'entri', 'narrow', 'lt', 'muskingum', 'develop', 'ohio_valley', 'close', 'communiti', 'eighteenth', 'centuri', 'aftermath', 'surviv', 'most', 'munse', 'refug', 'rumor', 'offici', 'earlier', 'complic', 'passag', 'decad', 'often', 'resist', 'presenc', 'lake', 'region', 'disappear', 'deal', 'particular', 'symbol', 'rang', 'impact', 'fairfield', 'moraviantown', 'upper_canada', 'northern', 'thame_river', 'stabil', 'boundari', 'problem', 'incurs', 'immigr', 'intern', 'migrat', 'simpli', 'govern', 'resid_fairfield', 'difficult', 'demand', 'relat', 'british_offici', 'inde', 'southern', 'illustr', 'avail', 'option', 'nineteenth', 'articl', 'despit', 'pressur', 'examin', 'specif', 'individu', 'second', 'polit', 'elimin', 'tragic', 'backcountri', 'defin', 'appalachian', 'conflict', 'warfar', 'local', 'neutral', 'neither', 'violenc', 'readili', 'label', 'limit', 'mid', 'zone', 'grew', 'immin', 'conclus', 'agreement', 'affect', 'raid', 'constant', 'includ', 'impend', 'sometim', 'michigan_histor', 'review', 'headman', 'power', 'jaw', 'danger', 'battl', 'risk', 'play', 'spring', 'tri', 'reloc', 'want', 'across', 'sens', 'wit', 'movement', 'potenti', 'extent', 'convinc', 'juli', 'clinton', 'northwest', 'refuge', 'slowli', 'longer', 'thing', 'lennachgo', 'destruct', 'negat', 'ripen', 'ojibw', 'away', 'welcom', 'son', 'agre', 'father', 'instead', 'journey', 'josiah', 'harmar', 'headmen', 'alcohol', 'stop', 'huron', 'better', 'drop', 'shift', 'auglaiz', 'becom', 'milit', 'confederaci', 'gen', 'messag', 'wampum', 'singular', 'repres', 'advic', 'easi', 'declin', 'area', 'spiritu', 'assist', 'support', 'thought', 'remark', 'agent', 'mckee', 'treati', 'sign', 'unfortun', 'volatil', 'jay', 'term', 'paper', 'econom', 'thame', 'cultiv', 'suffici', 'travel', 'knowledg', 'lieuten', 'simco', 'request', 'bushel', 'mere', 'cano', 'vital', 'season', 'fur', 'heighten', 'pari', 'contend', 'perform', 'western', 'fort_malden', 'negoti', 'somewhat', 'uneasi', 'america', 'worri', 'threaten', 'tension', 'capt', 'mississippi', 'compar', 'anxieti', 'surfac', 'niagara', 'nevertheless', 'begun', 'period', 'regul', 'strict', 'rule', 'requir', 'sin', 'social', 'standard', 'reveal', 'dramat', 'initi', 'primari', 'practic', 'deterior', 'group', 'less', 'diari', 'read', 'michael', 'diarist', 'complain', 'difficulti', 'relationship', 'gift', 'expans', 'wrote', 'count', 'enough', 'connect', 'respons', 'chesapeak', 'alleg', 'desert', 'drew', 'elliot', 'prophet', 'indiana', 'tecumseh', 'amherstburg', 'indic', 'revit', 'fellow', 'onim', 'either', 'gave', 'belong', 'stream', 'malden', 'denk', 'deep', 'hospit', 'chao', 'henri', 'schnall', 'food', 'dispers', 'daughter', 'meanwhil', 'cass', 'kinship', 'charl', 'killbuck', 'gelemend', 'perfect', 'jstor', 'www', 'org', 'scienc', 'art', 'heckeweld', 'guidanc', 'secret', 'verifi', 'exhort', 'resign', 'applic', 'liberti', 'acquaint', 'untim', 'impress', 'justic', 'mr_heckeweld', 'charact', 'intercours', 'marietta', 'amidst', 'full', 'publish', 'putnam', 'wabash', 'fever', 'common', 'barg', 'alarm', 'chimney', 'pocket', 'assur', 'releas', 'intellig', 'messeng', 'shebosh', 'wound', 'latter', 'display', 'glorious', 'youth', 'abel'])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_p.wv.vocab.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model_p.wv.most_similar(positive = [\"white\", \"american\"],\n",
    "#                         negative = [\"british\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\overrightarrow{Dimension_g} = \\overrightarrow{white} + \\overrightarrow{american} - \\overrightarrow{british}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "whiteAmerican_british = [('moravian', 0.9998588562011719),\n",
    "                         ('indian', 0.9998587369918823),\n",
    "                         ('fairfield', 0.9998579621315002),\n",
    "                         ('mani', 0.9998571872711182),\n",
    "                         ('live', 0.9998558163642883),\n",
    "                         ('murder', 0.9998539686203003),\n",
    "                         ('day', 0.9998528957366943),\n",
    "                         ('missionari', 0.9998528957366943),\n",
    "                         ('massacr', 0.9998518824577332),\n",
    "                         ('ohio', 0.999851405620575)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "mode": "markers+text",
         "name": "Similar to White American",
         "text": [
          "moravian",
          "indians",
          "fairfield",
          "many",
          "lives",
          "murder",
          "day",
          "missionary",
          "massacre",
          "ohio"
         ],
         "textposition": "bottom center",
         "type": "scatter",
         "uid": "d99dc882-fc0a-43d7-a522-e0bc37f373fb",
         "x": [
          -13.265983581542969,
          -3.1503617763519287,
          13.101271629333496,
          5.719395637512207,
          -17.57438850402832,
          -5.6888861656188965,
          27.676631927490234,
          33.34973907470703,
          -47.03227615356445,
          -37.47283172607422
         ],
         "y": [
          -45.59456253051758,
          -60.21220397949219,
          -47.7032470703125,
          -93.46844482421875,
          -76.33037567138672,
          -22.85382080078125,
          -70.43335723876953,
          -27.5128231048584,
          -71.16353607177734,
          -33.79647445678711
         ]
        }
       ],
       "layout": {
        "paper_bgcolor": "rgb(243,243,243)",
        "plot_bgcolor": "rgb(243,243,243)",
        "title": {
         "text": "Most similar Words to White American"
        },
        "xaxis": {
         "title": {
          "text": "Dimension1"
         }
        },
        "yaxis": {
         "title": {
          "text": "Dimension2"
         }
        }
       }
      },
      "text/html": [
       "<div id=\"21ebc5bc-2926-4fee-9131-033b18843374\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"21ebc5bc-2926-4fee-9131-033b18843374\", [{\"mode\": \"markers+text\", \"name\": \"Similar to White American\", \"text\": [\"moravian\", \"indians\", \"fairfield\", \"many\", \"lives\", \"murder\", \"day\", \"missionary\", \"massacre\", \"ohio\"], \"textposition\": \"bottom center\", \"x\": [-13.265983581542969, -3.1503617763519287, 13.101271629333496, 5.719395637512207, -17.57438850402832, -5.6888861656188965, 27.676631927490234, 33.34973907470703, -47.03227615356445, -37.47283172607422], \"y\": [-45.59456253051758, -60.21220397949219, -47.7032470703125, -93.46844482421875, -76.33037567138672, -22.85382080078125, -70.43335723876953, -27.5128231048584, -71.16353607177734, -33.79647445678711], \"type\": \"scatter\", \"uid\": \"d99dc882-fc0a-43d7-a522-e0bc37f373fb\"}], {\"paper_bgcolor\": \"rgb(243,243,243)\", \"plot_bgcolor\": \"rgb(243,243,243)\", \"title\": {\"text\": \"Most similar Words to White American\"}, \"xaxis\": {\"title\": {\"text\": \"Dimension1\"}}, \"yaxis\": {\"title\": {\"text\": \"Dimension2\"}}}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"21ebc5bc-2926-4fee-9131-033b18843374\"));});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"21ebc5bc-2926-4fee-9131-033b18843374\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"21ebc5bc-2926-4fee-9131-033b18843374\", [{\"mode\": \"markers+text\", \"name\": \"Similar to White American\", \"text\": [\"moravian\", \"indians\", \"fairfield\", \"many\", \"lives\", \"murder\", \"day\", \"missionary\", \"massacre\", \"ohio\"], \"textposition\": \"bottom center\", \"x\": [-13.265983581542969, -3.1503617763519287, 13.101271629333496, 5.719395637512207, -17.57438850402832, -5.6888861656188965, 27.676631927490234, 33.34973907470703, -47.03227615356445, -37.47283172607422], \"y\": [-45.59456253051758, -60.21220397949219, -47.7032470703125, -93.46844482421875, -76.33037567138672, -22.85382080078125, -70.43335723876953, -27.5128231048584, -71.16353607177734, -33.79647445678711], \"type\": \"scatter\", \"uid\": \"d99dc882-fc0a-43d7-a522-e0bc37f373fb\"}], {\"paper_bgcolor\": \"rgb(243,243,243)\", \"plot_bgcolor\": \"rgb(243,243,243)\", \"title\": {\"text\": \"Most similar Words to White American\"}, \"xaxis\": {\"title\": {\"text\": \"Dimension1\"}}, \"yaxis\": {\"title\": {\"text\": \"Dimension2\"}}}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"21ebc5bc-2926-4fee-9131-033b18843374\"));});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Visualize words most similar to White American\n",
    "my_word_list=[]\n",
    "my_word_vectors=[]\n",
    "# label=[]\n",
    "\n",
    "for i in whiteAmerican_british: \n",
    "    if my_word_list not in my_word_list:\n",
    "        my_word_list.append(i[0])\n",
    "        my_word_vectors.append(model_p.wv[i[0]])\n",
    "        \n",
    "tsne_model = TSNE(perplexity=5, n_components=2, init='pca', n_iter=3000, random_state=23) #you may need to tune these, epsecially the perplexity. #Use PCA to reduce dimensionality to 2-D, an \"X\" and a \"Y \n",
    "new_values = tsne_model.fit_transform(my_word_vectors)\n",
    "\n",
    "x = []\n",
    "y = []\n",
    "for value in new_values:\n",
    "    x.append(value[0])\n",
    "    y.append(value[1])\n",
    "\n",
    "\n",
    "trace1 = go.Scatter(\n",
    "    x = x,\n",
    "    y = y,\n",
    "    mode = 'markers+text',\n",
    "    name = \"Similar to White American\",\n",
    "    text = [stem_dict[word] if stem_dict.get(word) else word for word in my_word_list],\n",
    "    textposition='bottom center'\n",
    ")\n",
    "\n",
    "\n",
    "data = [trace1]\n",
    "\n",
    "layout = go.Layout(dict(title = \"Most similar Words to White American\",\n",
    "                    yaxis   = dict(title = \"Dimension2\"),\n",
    "                    xaxis   = dict(title = \"Dimension1\"),\n",
    "                    plot_bgcolor  = \"rgb(243,243,243)\",\n",
    "                    paper_bgcolor = \"rgb(243,243,243)\",\n",
    "                   )\n",
    "              )\n",
    "\n",
    "fig = go.Figure(data=data,layout=layout)\n",
    "py.iplot(fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Some of the significant words that come up from  this specific dimension include**;\n",
    "* Massacre\n",
    "* Moravian Indians\n",
    "* War \n",
    "* Missionary \n",
    "* Ohio\n",
    "\n",
    "These words construct what we know about the Gnadenhutten Massacre - \"The Moravian Indians, who were not allies of Britain, were massacred by American militia in Gnadenhutte, Ohio. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "<a id=\"tensor\"></a>\n",
    "## 4. Word2Vec to Tensor\n",
    "\n",
    "I am using [Google Embedding Projector](https://projector.tensorflow.org/) to visualize my Word2Vec model. Since it is built on tensorflow, we need to convert our Word2Vec output to the tensor format. The function below does just that.\n",
    "\n",
    "The final visualization of this Word2Vec model can be found [here](https://projector.tensorflow.org/?config=https://gist.githubusercontent.com/unglikteng/3d31526c9e090ff8123c4e7a1b07d2bb/raw/74fa1165c92c98a890d7837b97f33599146ebb0f/projector_config.json).\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def word2vec2tensor(model, tensor_filename):\n",
    "    outfiletsv = tensor_filename + '_tensor.tsv'\n",
    "    outfiletsvmeta = tensor_filename + '_metadata.tsv'\n",
    "                \n",
    "    with smart_open(outfiletsv, 'wb') as file_vector, smart_open(outfiletsvmeta, 'wb') as file_metadata:\n",
    "        for word in model.wv.index2word:\n",
    "            word = stem_dict[word] if stem_dict.get(word) else word\n",
    "            file_metadata.write(gensim.utils.to_utf8(word) + gensim.utils.to_utf8('\\n'))\n",
    "            vector_row = '\\t'.join(str(x) for x in model.wv.__getitem__(word))\n",
    "            file_vector.write(gensim.utils.to_utf8(vector_row) + gensim.utils.to_utf8('\\n'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# word2vec2tensor(model_p, \"model_primaries\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}