Skip to content

Instantly share code, notes, and snippets.

@akashpalrecha

akashpalrecha/Kaggle-toxic-comments-classification.ipynb

Created May 27, 2019 18:33
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save akashpalrecha/1f636e3a82f6e7f9802656d31c96f477 to your computer and use it in GitHub Desktop.
Save akashpalrecha/1f636e3a82f6e7f9802656d31c96f477 to your computer and use it in GitHub Desktop.
Download ZIP
Kaggle Toxic Comments Classification Challenge using Fastai V1 and ULMFIT
Display the source blob
Display the rendered blob
Raw
Kaggle-toxic-comments-classification.ipynb
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"%reload_ext autoreload\n",
"%autoreload 2\n",
"%matplotlib inline\n",
"\n",
"from fastai.text import *\n",
"from fastai import *\n",
"from sklearn.metrics import roc_auc_score"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# WARNING :\n",
"***THE NOTEBOOK MAY CONTAIN OBSCENE AND INAPPROPRIATE CONTENT BECAUSE OF THE NATURE OF THE DATASET BEING USED. READER DISCRETION IS ADVISED.***"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Language Model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Preparing Data"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# PATH = Path('data/Data_Processed/Data_Classifier').absolute()\n",
"PATH = Path('data/').absolute()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/models'),\n",
" PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/classifier_train_data'),\n",
" PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/Data_Processed'),\n",
" PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/kaggle_submission_1.csv'),\n",
" PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/test_labels.csv'),\n",
" PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/sample_submission.csv'),\n",
" PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/train.csv'),\n",
" PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/test.csv')]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"PATH.ls()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For training the language model, we use both the `train` and `test` data. This is because the label for each word is the next word in the sequence and the test data can be used in this case. We ignore the classification labels and only use the column containing the text for the language model."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>comment_text</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Explanation\\nWhy the edits made under my usern...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>D'aww! He matches this background colour I'm s...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Hey man, I'm really not trying to edit war. It...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>\"\\nMore\\nI can't make any real suggestions on ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>You, sir, are my hero. Any chance you remember...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" comment_text\n",
"0 Explanation\\nWhy the edits made under my usern...\n",
"1 D'aww! He matches this background colour I'm s...\n",
"2 Hey man, I'm really not trying to edit war. It...\n",
"3 \"\\nMore\\nI can't make any real suggestions on ...\n",
"4 You, sir, are my hero. Any chance you remember..."
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train = pd.DataFrame(pd.read_csv(PATH/'train.csv')['comment_text'])\n",
"test = pd.DataFrame(pd.read_csv(PATH/'test.csv')['comment_text'])\n",
"df_lm = train.append(test)\n",
"\n",
"df_lm.head()"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [],
"source": [
"bs = 48"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [],
"source": [
"data_lm = (TextList.from_df(train, path=PATH, cols='comment_text')\n",
" .split_by_rand_pct(0.1)\n",
" .label_for_lm()\n",
" .databunch(bs=bs))"
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>idx</th>\n",
" <th>text</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ) xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>\\n face all sorts of physical and legal threats , both on- and off - wiki , as well as real - life stalking over prolonged periods by multiple people \\n \\n xxmaj all with little or no support from the xxup wmf and large swathes of the community who are more interested in the never - ending tendentious discussion of \" \" meta \" \" issues -</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>utc ) \\n i think she xxunk the show and / or what callers are saying since xxmaj rush has a hearing problem , as you must well know . \\n xxmaj plus she probably searches the web for current news in certain categories . 05:24 , 7 xxmaj march 2010 xxbos \" \\n i 'd maintain it was only a personal attack , the user i</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>\\n person : xxup xxunk xxmaj global xxup ip - xxmaj addressing \\n address : xxmaj deutsche xxmaj telekom xxup ag \\n address : xxup xxunk xxmaj xxunk \\n address : xxmaj germany \\n phone : 49 180 xxunk \\n fax - no : 49 180 xxunk \\n e - mail : xxunk xxbos i liked it too and it inspired me to</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>xxmaj german state ; xxmaj germany as an empire ( the xxmaj first xxmaj reich ) . . . and xxmaj third xxmaj reich as a dictatorship under the xxmaj nazi regime . . . ) \" xxbos xxmaj which one of us is having trouble with xxmaj wiki is not a xxmaj battlefield ? xxbos xxmaj thanks for clarifying . xxbos xxmaj this talk page is full of xxunk</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"data_lm.show_batch()"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [],
"source": [
"data_lm.save('tmp_lm_model')"
]
},
{
"cell_type": "code",
"execution_count": 185,
"metadata": {},
"outputs": [],
"source": [
"data_lm = load_data(PATH, 'tmp_lm_model', bs=bs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Preparing the Language model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we load a pretrained model trained on the Wikitext 103 dataset."
]
},
{
"cell_type": "code",
"execution_count": 186,
"metadata": {},
"outputs": [],
"source": [
"learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def find_lr(model=learn):\n",
" \"\"\"\n",
" Convenience function to perform learning rate range test and plot results.\n",
" \"\"\"\n",
" model.lr_find()\n",
" model.recorder.plot(skip_end=15)"
]
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VPW9//HXZ7KSEBIgC5Cwg6yyBgRFKy4odYNqXbpZbWtpXW5vf7Xtvd621tbeervetrZu1XprXSoWK+Be9xWJEHaUnRAgYQsJ2ZPv748ZJGJWMmfOZPJ+Ph7zYObMOXM+Xybwzvmec75fc84hIiISbgG/CxARkdikgBEREU8oYERExBMKGBER8YQCRkREPKGAERERTyhgRETEEwoYERHxhAJGREQ8Ee93AeGUmZnphgwZ4ncZIiJdRkFBwT7nXJYXnx1TATNkyBCWL1/udxkiIl2GmW336rPVRSYiIp5QwIiIiCcUMCIi4gkFjIiIeEIBIyIinlDAiIiIJxQwIiLiCQVMNxbp6bLrGxqpa2iM6D47q7S8hofe2c7OA5V+lyLS5cTUjZbSfq9sLOG6/ytg7IBenD06m9mjsxk3oBdmRm19I/sqajhwpJYhman0TOr4j0nB9oPc/epmlm07QE1dI7UNjTQ0BgOtb2oi2b2S6dcriay0JFIS40lKCJAcH0ePxDhSk+LplRxPz6R4eqcmMjEvg7iAhfuvoFW19Y08+NY2fvevDymvqSc+YFw2NY/rZ49gYJ+UVrd1zmEW2XpFopFF+rdYL+Xn5zvdyd+2A0dqmfOb10hNiiMjJZFVRYdwLvgff4NzHKqs+2jdxLgAM4b35dwx2Zw9JocBGT1a/FznHK9sLOVPr25m2dYDZKQkMHd8P9KSE0iMC5AUH6DBOUrKa9hbVs3e8mpKy2uoqm2gur6R2vrmj25OG9GX3105mb49kzrc1vvf2Mrz6/aQ1zuFwX1SGNQ3hZ5J8RQfqqLoUBXFh6o5UlPPoD4pDM/uyfDMVI7UNvDfz6xnS+kRzhyVxTfPHMHTq3fz8LIdNDY65k3OpX96Mocq6zhUVUdZVR1llbWUVQVfH66qY2R2GpdPG8j8ybn0SU3scN0ikWJmBc65fE8+WwHTvTjnWPBQAS9vKOWfN5zGmP692FdRwysbS3lny356JMSRlRY8skjvkcD72w/y4vq9bNsf7CKaNDCDS6fmcfGEAaSnJACwp6yaf6woYmFBEVtKjzAgPZmvnD6MK6cNJLUDRz+NjY7q+gYqquspr6mnvLqelTsO8rNnNpCZmsgfvzCVSQMz2v15TxUWc9MjKxiWmcqR2nr2Hq752PuJcQH6ZySTkhjP9v1HqKxt+Oi9oZmp/ODCMZw1OuejZXvKqrnr1c08vGwH9Q2NpPdIOPZISSQj9Dw1KZ63N++jsKiMxLgA547N4RtnDmd8bnq7axeJFAVMOylg2vb48p3cvHAV/zF3NF//1PB2beOcY3PpEV5Yt5cnV+xi495yEuMCnDM2myM1Dbz+YSmNDqYP6cOV0wdy0cQBJMSF7/Te6qIyFjxUQGl5DbdePI6rpg9sswtqbXEZl/7pLU7OTedvX51BYnyAqtoGdh6s5EhNPbkZPcjsmUQg1PXmnGPv4Rq2lFZQVlXHWWOySYqPa/az6xoaiTP7aNuWbNhzmMfe28miFbuoq2/ksa/PVMhI1OmyAWNm24ByoAGob6kRZjYNeAe4wjm3MLSsAVgdWmWHc+7itvangGndzgOVzP3f1xk3oBcPf23GCZ3XcM6xtvgwCwuKeKqwmOT4AJdOzePSKXkMyUz1oOqgg0dq+bfHVvLaB6WcPjKTm84eybQhfZpd98CRWi76/Rs0NDoW3ziLrLSOd62F097D1Xzmj29RU9/AwgWnevr3JNJRXT1g8p1z+1pZJw54AagG7m8SMBXOuZ4d2Z8CpmUNjY4r7n6bjXvKeeZbp5PXu/UT1e0R6ZPZDY2OB97cyl2vbmZfRS0zh/XlprNHMmNYn4/qqG9o5It/XkbBjoM8/vWZTOxAl5qXNpdWcNmf3iItOYGF35hJdlqy3yWJALEfMN8C6oBpwBIFTPjtq6jhewtX8a8NJfz68ol8Zkqe3yV1SlVtAw8v28Hdr26mpLyGlMQ4MnokkJGSiAPW7z7Mrz47kUunRlc7V+w4yOfufZehmak89vUZpCUn+F2SSJcOmK3AQcABdzvn7jnu/VzgYeAs4M98PGDqgZVAPfBz59yTbe1PAfNJL23Yy3cXruJwdT3/OXc0V586JGYuoa2ua+DJFbvYVFLBwco6DlXWcqiqjnPH5rCgneeXIu2VjSV89cHl9EiMIz5g1DU4ahsa6Z+ezO3zTmbWyEy/S5RupisHzADnXLGZZRPsBrvROfdak/cfB37lnHvHzP7CxwPm6LbDgJeAs51zm5vZx3XAdQCDBg2aun27Z3PndCnVdQ38dOk6HnpnB6P7pfG/V05mVL80v8sS4OWNJTy3Zg8JcYHgI954Yd1etpQe4QszBvEfc8d06Oo7kc7osgHzsR2Z3QpUOOd+2WTZVuDor9OZQCVw3fFHK8eHT0t0BHPM7UvXce/rW/nqrKHcfP6oFq+IkuhQXdfAL5/byJ/f3MrA3in88rMTmT60+YsYRMLJy4DxbKgYM0s1s7Sjz4E5wJqm6zjnhjrnhjjnhgALgW865540s95mlhTaNhM4DVjnVa2xpry6jkeW7WTepAH814VjFS5dQHJCHP914Vgeu24mAFfd+w4vbyjxuSqRzvFyLLIc4A0zKwSWAUudc8+a2QIzW9DGtmOA5aFtXyZ4DkYB006PvbeTipp6vjJrmN+lSAdNH9qHpTfNYnS/NK5/+H1WF5X5XZLICdONljGmodHxqV+8zID0Hvx9wUy/y5ETVHK4mvl/fIua+kYWffPUNsc/EzlRXbKLTPzx/No9FB2s4tpZQ/0uRTohu1cyf7lmGrX1DXz5gWUcqqz1uySRDlPAxJg/v7GVQX1SOHdsTtsrS1QbmZPGPV/KZ+eBKq75y3v89e1tPLd2Dyt3HqK0vKbN7UX8pmshgbKqOtKS4tscWyraFe48xPLtB/nhhWMjPry9eGPGsL786vKJfHfhKn6wY+3H3rvl02P42hk6zybRq9sHzMEjtcz745tcOKE/N5832u9yOuXPb2wlLSmey6cN9LsUCaOLJg7g0yf358CRWvYerqakvJrH3tvJ7U+vxwy+erpCRqJTtw+YjJQETh3elztf3szA3ilcOX2Q3yWdkOJDVSxdvZtrTh1yQhOESXSLC9hH0yhAOqePzOKmR1bw06XrCZjpnJtEpW5/DsbMuO2S8ZxxUha3PLmG1z4o9bukDmtsdPzyuY0457j61CF+lyMRkBAX4HdXTea8cTnctmQdD761ze+SRD5BlymHlFfX8dm73qboYBWPL5jJmP69wlydN5xz/OCfa3jonR3cdNYIvj1nlN8lSQTV1jdy/cPv88K6vWSlJdEzKf6jR2pSHMkJcfRICE5Fff74fpw6XGOdycfFxFAxkdDZ+2B2l1Ux7843CZjx5PWnkdMrMkOqO+d4qrCYvN4pTBmU0e7BKJ1z/PCfa/nrO9tZ8KnhfO/8UTEzkKW0X219I/e9sYWdB6qoqKmnorqO8up6quoaqKproLq2gbKqOqrqGvjPT4/hK7OG6udEPqKAaadw3Gi5triMy+96m/whfXjw2ulhqqx1L28s4ZoH3gNgdL80rpo+iHmTc0nv0fJw7s45bn1qLQ++vZ2vnzGM788drf80pEWVtfX8v78X8syaPVw2NY/b54/XEEIC6EbLiBo3IJ1vzh7Bqx+UsnLnobB9bktB3tDouOOZDQzqk8LP5p9MQlyAHz21llN+9iK/ffEDGho/uV1NfQP/8Y/VPPj2dr52+lCFi7QpJTGeOz83hX87eyQLC4r43L3v6l4a8ZwCphlfmjmY9B4J/OGlDzv9WR/sLeemR1Zw8q3P8/qHn7yAYNGKXWzYU87N543ic6cMYvGNs1h8wyzOHpPDb1/8kC8/sIz9Fcf+I9ixv5LL/vQ2j763k+tnD+c/Pz1G4SLtEggY/37uSdz5uSmsLS7j8/e9Q2Vtvd9lSQxTwDQjLTmBa04bwovrS1hbfGKDDa4tLuMbDxUw5zev8eL6vfRKjueGh1ewY3/lR+tU1zXw6+c3MiEvnQtO7v/R8pPz0vnDVZP5+WdO5t2tB7jw929QsP0gz63dwwW/f53t+49w75fyufk8HblIx10woT/3fWkaH5ZUcMuiNS0eXYt0lgKmBdecOpSeSfH84aVNHd52YUERF/zuDd74cB83zB7BG987i0eumwHAdX9d/tFvjQ++tY3ismq+P3f0J0YRMDOunD6If3zjVOLjjMvvfpuv/7WAoZmpLL3pdA0FI50ya2Qm3z7nJBat2MXDy3b4XY7EKAVMC9JTErj61ME8s2YPH+wtb/d2JYer+fHitUwf0oc3vn8W3zlvFH1SExncN5XfXTWZD/aWc/PjqzhUWcudL2/izFFZrV46Oj43nSU3nM4lEwfwtdOH8viCmRpZV8Li+tkjOHNUFj9+ap2mBRBPKGBa8ZVZw0hJjOvQUcyti9dSU9/IHZdN+MRVYJ86KYvvnj+apat3c9ldb1NeU8/3zm97eJr0lAR+fcUkbrlAk4dJ+AQCxm8un0RWWhLf+FuBRmyWsNOYIq3ok5rIF2YM5r7Xt/Ctc0YyLKtnq+s/v3YPT6/ew83njWJoZmqz63z9jGGs2VXGklW7uXRKXpe5oVNiU+/URO78/BQ+e9dbnPfb1+iREBe8f6a2gfi4AEP6pjAsqyfDslIZ078Xs0ZkkhCn30ulfXQfTBtKyqs5/Y6XmTq4Nz//zAQG9W2+e+pwdR3n/vpVeqcksvjGWa3+I6ysrefBt7ZzeX4efXsmhbVekRPx7JrdLFqx66M7/5MT4qipb2Trvgq2lB6hJHRJc1ZaElfkD+TK6QPJ662u2ligGy3byasZLf/v7W3cvnQ9DY2OK6YN5IazRtA/vcfH1vmvJ1fz8Ls7+Mc3T2PSwIyw1yDip/LqOt7dcoBHlu3g5Y0lOOCsUdn84MKxDGnhaF26BgVMO3k5ZfKesmrufHkTj763AzPjrFHZpCTGEQgYzsET7xdx7WlD+eFFYz3Zv0i02HWoiseW7eAvb22jodHx40vGc+mUXF0y30UpYNrJy4A5aueBSv7w0ibe3bqfBudoaHA0OMewzJ7cd3U+qRoqX7qJ4kNV/PtjK3l36wEumjiAn84b3+rwRhKdFDDtFImAEZFjGhodd726mV+/8AH9eiXzyNdmtHieUqKTxiITkagUFzCunz2ChQtmUlZVx62L17a9kXQbChgR6bTJg3rzb2eP5KUNJby0Ya/f5UiUUMCISFhcfeoQhmWlctviddTUN/hdjkQBBYyIhEVifIBbLxrHtv2V/PmNrX6XI1FAASMiYXPGSVnMGZvDH17axJ6yar/LEZ8pYEQkrP7rgrHUNzp+9vR6v0sRnylgRCSsBvVNYcEZw3iqsJh3tuz3uxzxkQJGRMLuG2eOIK93D/7zH6uprtMJ/+5KASMiYdcjMY6fzT+ZLfuO8PswTD0uXZMCRkQ8ccZJWVw6JY+7X93CuuLDfpcjPlDAiIhnfnDhGDJSEvjeE6uob2j0uxyJMAWMiHgmIyWRWy8ex+pdZdz/pu6N6W48DRgz22Zmq81spZm1OAqlmU0zswYzu6zJsqvN7MPQ42ov6xQR71xwcn/OGZPDr1/4gO37j/hdjkRQJI5gZjvnJrU0WqeZxQF3AM81WdYH+BFwCjAd+JGZ9Y5ArSISZmbGT+eNJz4Q4I5nN/hdjkRQNHSR3Qg8AZQ0WXYe8IJz7oBz7iDwAnC+H8WJSOf1S0/my6cO4Zk1e/hwb7nf5UiEeB0wDnjezArM7Lrj3zSzXGA+cNdxb+UCO5u8LgotE5Eu6tpZQ+mREMcfXt7kdykSIV4HzGnOuSnAXOB6MzvjuPd/C3zPOXf8nVjNzb3a7MxoZnadmS03s+WlpaWdr1hEPNEnNZEvzhjM4sJitpRW+F2ORICnAeOcKw79WQIsIng+pal84FEz2wZcBvzRzOYRPGIZ2GS9PKC4hX3c45zLd87lZ2VlhbkFIhJOXz19GInxAe58ebPfpUgEeBYwZpZqZmlHnwNzgDVN13HODXXODXHODQEWAt90zj1J8IT/HDPrHTq5P4cmFwGISNeUlZbEVdMH8eTKXezYX+l3OeIxL49gcoA3zKwQWAYsdc49a2YLzGxBaxs65w4APwHeCz1uCy0TkS5uwaeGExcw/vSqzsXEunivPtg5twWY2Mzy40/oH13+5eNe3w/c70lxIuKbnF7JXJE/kEff28ENZ40kN6OH3yWJR6LhMmUR6WYWnDkcgHtf2+JzJeIlBYyIRFxuRg/OHZvDklXFNDQ2e4GoxAAFjIj44tMn92dfRS3vbtWkZLFKASMivjhrdDbJCQGeXr3b71LEIwoYEfFFSmI8Z43O5tk1e9RNFqMUMCLimwtOHqBushimgBER38wenaVushimgBER36QkxnP26Bx1k8UoBYyI+EpXk8UuBYyI+Gr26Cx6JMSpmywGKWBExFe6mix2KWBExHcXTFA3WSxSwIiI72aPyqZHQhxLV6mbLJYoYETEdz0S4zhnbA5LVu2muu74CW6lq1LAiEhUuGraQMqq6nhmjY5iYoUCRkSiwoxhfRnSN4VHlu30uxQJEwWMiESFQMC4Ytoglm09wKaSCr/LkTBQwIhI1Lhsah7xAePRZTv8LkXCQAEjIlEjKy2JOeNyeOL9ImrqdbK/q1PAiEhUuXLaIA5W1vHc2r1+lyKdpIARkagya0Qmeb178Mi76ibr6hQwIhJVAgHjqumDeHvLfrbuO+J3OdIJChgRiTqfnZpHXMB49D0dxXRlChgRiTrZvZI5a3Q2TxQUUd/Q6Hc5coIUMCISleZPzg0NgHnA71LkBClgRCQqzR6VTUpiHEtWFftdipwgBYyIRKUeiXGcOzaHZ9bsoU7dZF2SAkZEotaFEwZwqLKONzbt87sUOQEKGBGJWmeclElacjyLC9VN1hUpYEQkaiXFx3HeuH68sHav5onpghQwIhLVLpo4gPKael79oNTvUqSDFDAiEtVOHd6X3ikJLNF0yl2OpwFjZtvMbLWZrTSz5c28f4mZrTr6vpnNavJeQ2j5SjN7yss6RSR6JcQFmHtyf15ct5fK2nq/y5EOiMQRzGzn3CTnXH4z7/0LmOicmwRcC9zX5L2q0HaTnHMXR6BOEYlSF07oT1VdAy9tKPG7FOkAX7vInHMVzjkXepkKuNbWF5Hu6ZShfclKS2JJobrJupJ2BYyZDTezpNDzM83sJjPLaMemDnjezArM7LoWPnu+mW0AlhI8ijkqOdRt9o6ZzWtPnSISm+ICxgUn9+eljSWUVdb5XY60U3uPYJ4AGsxsBPBnYCjwcDu2O805NwWYC1xvZmccv4JzbpFzbjQwD/hJk7cGhbrVPgf81syGN7cDM7suFETLS0t1lYlIrLpsah619Y0sWlHkdynSTu0NmEbnXD0wH/itc+7fgf5tbeScKw79WQIsAqa3su5rwHAzyzxu2y3AK8DkFra7xzmX75zLz8rKamdzRKSrGZ+bzsm56Tz63k6O9axLNGtvwNSZ2VXA1cCS0LKE1jYws1QzSzv6HJgDrDlunRFmZqHnU4BEYL+Z9W7SJZcJnAasa2etIhKjrpw+kA17yiksKvO7FGmH9gbMNcBM4Hbn3FYzGwo81MY2OcAbZlYILAOWOueeNbMFZrYgtM6lwBozWwncCVwROuk/Blge2vZl4OfOOQWMSDd38cQB9EiI49FlmoisK7COHmqaWW9goHNulTclnbj8/Hy3fPknbrcRkRhy8+OFLF29m2W3nEPPpHi/y+nyzKyghdtIOq29V5G9Yma9zKwPUAg8YGa/9qIgEZHWXDl9EJW1DRoAswtobxdZunPuMPAZ4AHn3FTgHO/KEhFp3pRBGZyU01PdZF1AewMm3sz6A5dz7CS/iEjEmRlXThtEYVEZ64oP+12OtKK9AXMb8Byw2Tn3npkNAz70riwRkZbNn5xLYlyAx97TUUw0a1fAOOced85NcM59I/R6i3PuUm9LExFpXu/URM4f349FK3Zpnpgo1t6T/HlmtsjMSsxsr5k9YWZ5XhcnItKSy/MHcri6nn+t1wCY0aq9XWQPAE8BA4BcYHFomYiIL2YO70t2WhJPrtzldynSgvYGTJZz7gHnXH3o8RdA47KIiG/iAsZFEwfwysYSDlXW+l2ONKO9AbPPzL5gZnGhxxeA/V4WJiLSlnmTcqlrcDy9eo/fpUgz2hsw1xK8RHkPsBu4jODwMSIivhmf24thWanqJotS7b2KbIdz7mLnXJZzLts5N4/gTZciIr4xM+ZNymXZ1gPsOlTldzlynM7MaPntsFUhInKCLpk0AICnVmromGjTmYCxsFUhInKCBvdNZfKgDP6pbrKo05mA0Yw/IhIV5k3KZcOecjbs0dAx0aTVgDGzcjM73MyjnOA9MSIivrtgQn/iAsaTK9RNFk1aDRjnXJpzrlczjzTnnCZiEJGokNkzidNHZrK4sJjGRnWuRIvOdJGJiESNeZNy2XWoivd3HPS7FAlRwIhITDh7TDaJcQGeXaObLqOFAkZEYkJacgKzRmby7No9dHQqePGGAkZEYsb54/pRdLCKtZqILCooYEQkZpw9JpuAwXNr1U0WDRQwIhIz+vZMYvrQPgqYKKGAEZGYcv64fnywt4LNpRV+l9LtKWBEJKbMGdcPUDdZNFDAiEhMGZDRg4kDM3hOlyv7TgEjIjHn/HH9KCwq0xD+PlPAiEjMOW9cDgDPq5vMVwoYEYk5w7J6MionTedhfKaAEZGYdN74fizbeoD9FTV+l9JtKWBEJCadNy6HRgcvrt/rdyndlgJGRGLS2P69yM3owYvrS/wupdtSwIhITDIzzhmTzesfllJd1+B3Od2SAkZEYtY5Y3OormvkzU37/C6lW/I0YMxsm5mtNrOVZra8mfcvMbNVR983s1lN3rvazD4MPa72sk4RiU2nDO1LWlK8zsP4JBLTHs92zrX068O/gKecc87MJgB/B0abWR/gR0A+4IACM3vKOaep6kSk3RLjA3xqVBYvri/h9kZHIGB+l9St+NpF5pyrcMdmBkolGCYA5wEvOOcOhELlBeB8P2oUka7t3LE5lJbXUFh0yO9Suh2vA8YBz5tZgZld19wKZjbfzDYAS4FrQ4tzgZ1NVisKLRMR6ZAzT8omLmDqJvOB1wFzmnNuCjAXuN7Mzjh+BefcIufcaGAe8JPQ4uaOY5udA9XMrgudv1leWloarrpFJEakpyQwfUgfXlyny5UjzdOAcc4Vh/4sARYB01tZ9zVguJllEjxiGdjk7TyguIXt7nHO5Tvn8rOyssJWu4jEjnPG5rBxbzk79lf6XUq34lnAmFmqmaUdfQ7MAdYct84IM7PQ8ylAIrAfeA6YY2a9zax3aNvnvKpVRGLbOWOyAd3VH2leXkWWAywK5Uc88LBz7lkzWwDgnLsLuBT4kpnVAVXAFaGT/gfM7CfAe6HPus05d8DDWkUkhg3um8pJOT15cf1erp011O9yug3PAsY5twWY2Mzyu5o8vwO4o4Xt7wfu96o+EelezhmTw92vbaGsso70lAS/y+kWdCe/iHQL547NoaHR8fJGneyPFAWMiHQLE/MyGJCezFOFzV4vJB5QwIhItxAIGBdPyuXVD0o1R0yEKGBEpNv4zJRcGhodi3UUExEKGBHpNk7KSWNs/14sWqmAiQQFjIh0K/Mn51K48xBbSiv8LiXmKWBEpFu5eNIAAgZPrtjldykxTwEjIt1KTq9kThuRyaKVuzg2mLt4QQEjIt3OvEm57DxQRcF2TTHlJQWMiHQ7543vR3JCgEXqJvOUAkZEup2eSfGcN64fS1btpra+0e9yYpYCRkS6pXmTcymrqtPQMR5SwIhIt3T6iEwyeyay6H11k3lFASMi3VJ8XICLJg7gpQ0llFXW+V1OTFLAiEi39ZnJedQ2NLJkdde9s/+VjSU8+NY2Ghuj75JrBYyIdFvjc3sxIrtnl+4m+/vyndz7+hYCAfO7lE9QwIhIt2VmzJ+cy/LtB9mxv9LvcjrMOUfB9oNMHdzb71KapYARkW5t3uRczOiS98TsOlTF3sM1ChgRkWiUm9GDGUP7smhFUZcbOuboSARTBilgRESi0vwpuWzbX8mKnYf8LqVD3t9+kJTEOEb3S/O7lGYpYESk25s7vh9J8YEud7K/YMdBJg3MID4uOv8rj86qREQiKC05gTnj+rF4VXGXGTrmSE0963eXR+35F1DAiIgA8JnJuRyq7DpDxxQWHaKh0TFFASMiEt1OHxkcOmZhQZHfpbTL+0dP8A9UwIiIRLX4uACXTR3Iv9bvZXdZld/ltKlg+0FGZvckPSXB71JapIAREQn5/CmDcMAjy3b6XUqrGhsd7+84RP6Q6D16AQWMiMhHBvZJ4VMnZfHosh3UNUTvyf4t+yooq6qL2vtfjlLAiIg08YVTBlNSXsOL6/b6XUqLlm8Lnn+J5ivIQAEjIvIxs0dnk5vRg4fe3e53KS0q2H6Q3ikJDM1M9buUVilgRESaiAsYV00fyJub9rOltMLvcppVsCM4wKVZ9I2g3JQCRkTkOJdPG0h8wPjbuzv8LuUTDhypZUvpkai+/+UoBYyIyHGy05I5b3w/FhYUUV3X4Hc5H7NiR+j8S5Sf4AePA8bMtpnZajNbaWbLm3n/82a2KvR4y8wmtndbEREvfeGUwZRV1bG4MLpmuyzYfpD4gDEhL8PvUtoUH4F9zHbO7Wvhva3Ap5xzB81sLnAPcEo7txUR8cyMYX0Ykd2TR5bt4LP5A/0u5yNvbt7PuNx0eiTG+V1Km3ztInPOveWcOxh6+Q6Q52c9IiJHHZ3t8v0dhyg6GB2zXe48UEnhzkPMHd/P71LaxeuAccDzZlZgZte1se5XgGdOcFsRkbC7aMIAAJau2u1zJUGLVwW76y44ub/PlbSP1wFzmnNuCjAXuN7MzmhuJTNsJgfyAAAMyUlEQVSbTTBgvncC215nZsvNbHlpaWmYyxeR7mxQ3xQm5qWzJEoCZknhbiYNzGBgnxS/S2kXTwPGOVcc+rMEWARMP34dM5sA3Adc4pzb35FtQ+/f45zLd87lZ2Vlhb8RItKtXThhAKt3lbFt3xFf69hSWsG63Ye5cELXOHoBDwPGzFLNLO3oc2AOsOa4dQYB/wC+6Jz7oCPbiohEwgWh/9CXrPL3arIlq3ZjFgy8rsLLI5gc4A0zKwSWAUudc8+a2QIzWxBa54dAX+CPx12O3Oy2HtYqItKsARk9mDq4t+/dZIsLi5k2uA/90pN9raMjPLtM2Tm3BZjYzPK7mjz/KvDV9m4rIuKHCyf058eL17GppJwR2WkR3//GPeV8WFLBbZeMi/i+O0N38ouItOHTJ/fHDBYX+nMUs2RVMQGDueO7zvkXUMCIiLQpp1cy04f0YcmqYpxzEd23c44lq3Yzc3hfstKSIrrvzlLAiIi0w0UTB7C59Agb9pRHdL9riw+zdd+RLnVy/ygFjIhIO8wd34+4gEX8arLFq4qJDxjnj+sad+83pYAREWmHvj2TOHV4X54o2EV5dV1E9llb38g/VxRz+shMeqcmRmSf4aSAERFpp2+dM5LSihpuWbQmIudilq4uZs/har506hDP9+UFBYyISDtNHdyHb509kqcKi1lYUOTpvpxz3PPaVkZm9+TMk7rmKCUKGBGRDvjm7BHMGNaHHz21ls0eTqn85qb9rN99mK+dPizqp0ZuiQJGRKQD4gLGb6+YTFJ8gJseWUFNvTczXt77+hYyeyZxyeSud/XYUQoYEZEO6peezC8um8ja4sP8/JkNYf/8jXvKefWDUq45bQhJ8dE/sVhLFDAiIifgnLE5fPnUITzw5jZeWLc3rJ997+tb6JEQx+dPGRTWz400BYyIyAn6/tzRjM/txXceLwzbrJclh6v558pdXJ6fR0ZK17s0uSkFjIjICUpOiOPOz02hsdFx4yMrqGto7PRn/uWtbdQ3Oq6dNTQMFfpLASMi0gmD+6by80snsGLHIX7x3MZOfdZLG/bywJvbOH9cPwb3TQ1Thf5RwIiIdNIFE/rzxRmDuee1Lbx4gudj/vrOdr764HJGZPfkx11sWP6WKGBERMLglgvGMG5AL77995U8Vdj+UZcbGx0/e3o9P3hyDWeNzuaxr88gO63rTCrWGgWMiEgYJCfEcdcXpjK4byo3PbKCz9/3LptKWr8Rc1NJBQseKuCe17bwpZmDufuL+aQkejYPZMRZpOc28FJ+fr5bvnx52yuKiHikodHx8Lvb+Z/nNlJd18A1pw1l1ohMBvdNITejBwEzXvmghAfe3MbrH+4jMS7Ad88fxVdmDfXljn0zK3DO5Xvy2QoYEZHw21dRw38/vYEn3j82Zll8wOiZHM+hyjpyeiXxxRmDuXL6IDJ7+jeRmAKmnRQwIhJtSsqr2Vp6hO37K9l+4Ah7ymo4c1QW54/vR0Kc/2cpvAyY2OnsExGJQtlpyWSnJXPKsL5+lxJx/seniIjEJAWMiIh4QgEjIiKeUMCIiIgnFDAiIuIJBYyIiHhCASMiIp5QwIiIiCdi6k5+MysFth+3OB0oa2NZ09dtPc8E9nWizObqae86HW3L8a+PPo+ltjR93pn2dKYtLb2nn7Njy/TdtK/Wttbx4rsZ5ZxLa7vsE+Cci+kHcE9by5q+bus5sDzc9bR3nY62pZU2xExbwtWezrRFP2et/5zpu4nd76atR3foIlvcjmWLO/g83PW0d52OtuX414tbWOdERUNb2ltHWzrTlpbe089ZeOi7aX25n99Nq2KqiywSzGy582hguEiLpbZAbLUnltoCsdWeWGoLeNue7nAEE273+F1AGMVSWyC22hNLbYHYak8stQU8bI+OYERExBM6ghEREU9064Axs/vNrMTM1pzAtlPNbLWZbTKz31mTuU7N7EYz22hma83sf8JbdYv1hL0tZnarme0ys5Whx6fDX3mLNXny3YTe/46ZOTPLDF/FrdbjxXfzEzNbFfpenjezAeGvvNl6vGjLL8xsQ6g9i8wsI/yVt1iTF+35bOjffqOZeX6upjNtaOHzrjazD0OPq5ssb/XfVbO8ujytKzyAM4ApwJoT2HYZMBMw4Blgbmj5bOBFICn0OrsLt+VW4Dux8t2E3hsIPEfwfqnMrtoWoFeTdW4C7urCbZkDxIee3wHc0ZV/zoAxwCjgFSA/WtsQqm/Iccv6AFtCf/YOPe/dWntbe3TrIxjn3GvAgabLzGy4mT1rZgVm9rqZjT5+OzPrT/Af+Nsu+Df/f8C80NvfAH7unKsJ7aPE21YEedQW33jYnt8A3wUidvLRi7Y45w43WTWVCLXHo7Y875yrD636DpDnbSuO8ag9651zGyNRf2h/J9SGFpwHvOCcO+CcOwi8AJx/ov9PdOuAacE9wI3OuanAd4A/NrNOLlDU5HVRaBnAScDpZvaumb1qZtM8rbZ1nW0LwA2hrov7zay3d6W2S6faY2YXA7ucc4VeF9oOnf5uzOx2M9sJfB74oYe1tiUcP2dHXUvwt2M/hbM9fmlPG5qTC+xs8vpou06ovfHt3Gm3YGY9gVOBx5t0LyY1t2ozy47+BhlP8NByBjAN+LuZDQulfsSEqS1/An4Sev0T4FcE/wOIuM62x8xSgFsIdsf4KkzfDc65W4BbzOw/gBuAH4W51DaFqy2hz7oFqAf+Fs4aOyKc7fFLa20ws2uAfwstGwE8bWa1wFbn3HxabtcJtVcB83EB4JBzblLThWYWBxSEXj5F8D/epofxeUBx6HkR8I9QoCwzs0aCYxeVell4MzrdFufc3ibb3Qss8bLgNnS2PcOBoUBh6B9dHvC+mU13zu3xuPbjhePnrKmHgaX4EDCEqS2hk8kXAmdH+pex44T7u/FDs20AcM49ADwAYGavAF92zm1rskoRcGaT13kEz9UUcSLt9foEVLQ/gCE0OTkGvAV8NvTcgIktbPcewaOUoye8Ph1avgC4LfT8JIKHm9ZF29K/yTr/Djzalb+b49bZRoRO8nv03Yxsss6NwMIu3JbzgXVAViR/vrz+OSNCJ/lPtA20fJJ/K8FemN6h533a095m6/LjC42WB/AIsBuoI5jQXyH4W+6zQGHoh/6HLWybD6wBNgN/4NhNq4nAQ6H33gfO6sJt+SuwGlhF8Le2/pFoi1ftOW6dbUTuKjIvvpsnQstXERxXKrcLt2UTwV/EVoYeEbkizsP2zA99Vg2wF3guGttAMwETWn5t6DvZBFzTVntbe+hOfhER8YSuIhMREU8oYERExBMKGBER8YQCRkREPKGAERERTyhgJKaZWUWE93efmY0N02c1WHC05DVmtritUYbNLMPMvhmOfYuEgy5TlphmZhXOuZ5h/Lx4d2xgRk81rd3MHgQ+cM7d3sr6Q4AlzrnxkahPpC06gpFux8yyzOwJM3sv9DgttHy6mb1lZitCf44KLf+ymT1uZouB583sTDN7xcwWWnAek78dnRsjtDw/9LwiNCBloZm9Y2Y5oeXDQ6/fM7Pb2nmU9TbHBu3saWb/MrP3LTg/xyWhdX4ODA8d9fwitO7Nof2sMrMfh/GvUaRNChjpjv4X+I1zbhpwKXBfaPkG4Azn3GSCoxP/rMk2M4GrnXNnhV5PBr4FjAWGAac1s59U4B3n3ETgNeBrTfb/v6H9tzmeU2gcrLMJjqYAUA3Md85NITj/0K9CAfd9YLNzbpJz7mYzmwOMBKYDk4CpZnZGW/sTCRcNdind0TnA2CYjzfYyszQgHXjQzEYSHCk2ock2Lzjnms65scw5VwRgZisJjgX1xnH7qeXYAKEFwLmh5zM5NpfGw8AvW6izR5PPLiA4NwcEx4L6WSgsGgke2eQ0s/2c0GNF6HVPgoHzWgv7EwkrBYx0RwFgpnOuqulCM/s98LJzbn7ofMYrTd4+ctxn1DR53kDz/5bq3LGTnC2t05oq59wkM0snGFTXA78jOP9LFjDVOVdnZtuA5Ga2N+C/nXN3d3C/ImGhLjLpjp4nOH8KAGZ2dFjzdGBX6PmXPdz/OwS75gCubGtl51wZwWmRv2NmCQTrLAmFy2xgcGjVciCtyabPAdeG5gfBzHLNLDtMbRBpkwJGYl2KmRU1eXyb4H/W+aET3+sITrEA8D/Af5vZm0CchzV9C/i2mS0D+gNlbW3gnFtBcGTcKwlOyJVvZssJHs1sCK2zH3gzdFnzL5xzzxPsgnvbzFYDC/l4AIl4Spcpi0RYaHbNKuecM7Mrgaucc5e0tZ1IV6NzMCKRNxX4Q+jKr0P4NA21iNd0BCMiIp7QORgREfGEAkZERDyhgBEREU8oYERExBMKGBER8YQCRkREPPH/AdVBU29SxSWAAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"find_lr()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It looks like the loss is still decreasing steeply at a learning rate of about `3e-2`.\n",
"\n",
"We first fit the head of the model before unfreezing it."
]
},
{
"cell_type": "code",
"execution_count": 68,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>epoch</th>\n",
" <th>train_loss</th>\n",
" <th>valid_loss</th>\n",
" <th>accuracy</th>\n",
" <th>time</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>4.330651</td>\n",
" <td>4.091124</td>\n",
" <td>0.307680</td>\n",
" <td>12:01</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"learn.fit_one_cycle(1, 3e-2, moms=(0.8, 0.7))"
]
},
{
"cell_type": "code",
"execution_count": 69,
"metadata": {},
"outputs": [],
"source": [
"learn.save('fit_head')"
]
},
{
"cell_type": "code",
"execution_count": 70,
"metadata": {
"collapsed": true
},
"outputs": [
{
"data": {
"text/plain": [
"LanguageLearner(data=TextLMDataBunch;\n",
"\n",
"Train: LabelList (143614 items)\n",
"x: LMTextList\n",
"xxbos xxmaj explanation \n",
" xxmaj why the edits made under my username xxmaj hardcore xxmaj metallica xxmaj fan were reverted ? xxmaj they were n't vandalisms , just closure on some gas after i voted at xxmaj new xxmaj york xxmaj dolls xxup fac . xxmaj and please do n't remove the template from the talk page since i 'm retired xxunk,xxbos xxmaj xxunk ! xxmaj he matches this background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ),xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about the formatting than the actual info .,xxbos \" \n",
" xxmaj more \n",
" i ca n't make any real suggestions on improvement - i wondered if the section statistics should be later on , or a subsection of \" \" types of accidents \" \" xxup -i think the references may need tidying so that they are all in the exact same format ie date format etc . i can do that later on , if no - one else does first - if you have any preferences for formatting style on references or want to do it yourself please let me know . \n",
" \n",
" xxmaj there appears to be a backlog on articles for review so i guess there may be a delay until a reviewer turns up . xxmaj it 's listed in the relevant form eg xxmaj wikipedia : xxmaj xxunk # xxmaj transport \",xxbos xxmaj you , sir , are my hero . xxmaj any chance you remember what page that 's on ?\n",
"y: LMLabelList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data/Data_Processed/Data_Classifier;\n",
"\n",
"Valid: LabelList (15957 items)\n",
"x: LMTextList\n",
"xxbos would continue to be , delayed due to my xxup irl schedule .,xxbos xxmaj get xxmaj well \n",
" \n",
" xxmaj get well soon . ),xxbos \" \n",
" \n",
" xxmaj gerry xxmaj xxunk \n",
" xxmaj hey , amigo , take the protect off \" \" xxmaj gerry xxmaj xxunk . \" \" i think you 're letting your sexual preference cloud your rational thought . xxmaj the edits you have called \" \" vandalism \" \" ( not done by me ) are nowhere near vandalism . xxmaj what or who are you trying to protect . xxmaj unlock it now and lighten up . xxmaj you 've been an admin for too long to act in this juvenile way . xxmaj stop showing favoritism and be even - handed . xxmaj now ! ! ! rossp \",xxbos xxup hd xxup dvd winning ? \n",
" \n",
" xxmaj the section previously stated that on xxmaj january 8 , 2006 , at the xxmaj consumer xxmaj electronics xxmaj show , it had been said that hd dvd was ahead of xxunk . however they did n't cite it . i looked and the article i cited said they were neck and neck . \n",
" \n",
" xxup ok , my reasoning on the cnn article was probably weak , but it still may be true . \n",
" xxmaj xxunk 's website claims to have passed hd dvd , but i doubt that that is a very reliable place to look for info . xxmaj can someone find something else or revert it back ? ( i do n't know how to do that ) \n",
" \n",
" \n",
" xxup hd - xxup dvd has the backing of the porn industry , so its practically won already . \n",
" \n",
" xxmaj blu xxmaj ray fanboy stay out \n",
" \n",
" xxmaj some people keep changing articles with no proof . xxmaj stop it . —the preceding unsigned comment was added by ( talk • contribs ) . \n",
" \n",
" xxmaj like yourself . 17 xxup gb per layer discs and triple - layer discs are not part of xxup hd - xxup dvd spec and as such are xxup possible xxup future additions that have no annouced released date , no schedule for when they may be released , and are 99 % likely to be incompatible with the optical heads in current and near future drives . xxmaj if you have some proof proving otherwise , then by all means add the 51 xxup gb disc back in and xxup refference it .,xxbos xxmaj terrorism \n",
" \n",
" xxmaj would it be possible for me to edit the terrorism article to add a section on state sponsored terrorism that is unrelated to the current dispute ?\n",
"y: LMLabelList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data/Data_Processed/Data_Classifier;\n",
"\n",
"Test: None, model=SequentialRNN(\n",
" (0): AWD_LSTM(\n",
" (encoder): Embedding(60004, 400, padding_idx=1)\n",
" (encoder_dp): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (rnns): ModuleList(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (2): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" )\n",
" (input_dp): RNNDropout()\n",
" (hidden_dps): ModuleList(\n",
" (0): RNNDropout()\n",
" (1): RNNDropout()\n",
" (2): RNNDropout()\n",
" )\n",
" )\n",
" (1): LinearDecoder(\n",
" (decoder): Linear(in_features=400, out_features=60004, bias=True)\n",
" (output_dp): RNNDropout()\n",
" )\n",
"), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x7faa93d3c1e0>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/Data_Processed/Data_Classifier'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
"learn: LanguageLearner(data=TextLMDataBunch;\n",
"\n",
"Train: LabelList (143614 items)\n",
"x: LMTextList\n",
"xxbos xxmaj explanation \n",
" xxmaj why the edits made under my username xxmaj hardcore xxmaj metallica xxmaj fan were reverted ? xxmaj they were n't vandalisms , just closure on some gas after i voted at xxmaj new xxmaj york xxmaj dolls xxup fac . xxmaj and please do n't remove the template from the talk page since i 'm retired xxunk,xxbos xxmaj xxunk ! xxmaj he matches this background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ),xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about the formatting than the actual info .,xxbos \" \n",
" xxmaj more \n",
" i ca n't make any real suggestions on improvement - i wondered if the section statistics should be later on , or a subsection of \" \" types of accidents \" \" xxup -i think the references may need tidying so that they are all in the exact same format ie date format etc . i can do that later on , if no - one else does first - if you have any preferences for formatting style on references or want to do it yourself please let me know . \n",
" \n",
" xxmaj there appears to be a backlog on articles for review so i guess there may be a delay until a reviewer turns up . xxmaj it 's listed in the relevant form eg xxmaj wikipedia : xxmaj xxunk # xxmaj transport \",xxbos xxmaj you , sir , are my hero . xxmaj any chance you remember what page that 's on ?\n",
"y: LMLabelList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data/Data_Processed/Data_Classifier;\n",
"\n",
"Valid: LabelList (15957 items)\n",
"x: LMTextList\n",
"xxbos would continue to be , delayed due to my xxup irl schedule .,xxbos xxmaj get xxmaj well \n",
" \n",
" xxmaj get well soon . ),xxbos \" \n",
" \n",
" xxmaj gerry xxmaj xxunk \n",
" xxmaj hey , amigo , take the protect off \" \" xxmaj gerry xxmaj xxunk . \" \" i think you 're letting your sexual preference cloud your rational thought . xxmaj the edits you have called \" \" vandalism \" \" ( not done by me ) are nowhere near vandalism . xxmaj what or who are you trying to protect . xxmaj unlock it now and lighten up . xxmaj you 've been an admin for too long to act in this juvenile way . xxmaj stop showing favoritism and be even - handed . xxmaj now ! ! ! rossp \",xxbos xxup hd xxup dvd winning ? \n",
" \n",
" xxmaj the section previously stated that on xxmaj january 8 , 2006 , at the xxmaj consumer xxmaj electronics xxmaj show , it had been said that hd dvd was ahead of xxunk . however they did n't cite it . i looked and the article i cited said they were neck and neck . \n",
" \n",
" xxup ok , my reasoning on the cnn article was probably weak , but it still may be true . \n",
" xxmaj xxunk 's website claims to have passed hd dvd , but i doubt that that is a very reliable place to look for info . xxmaj can someone find something else or revert it back ? ( i do n't know how to do that ) \n",
" \n",
" \n",
" xxup hd - xxup dvd has the backing of the porn industry , so its practically won already . \n",
" \n",
" xxmaj blu xxmaj ray fanboy stay out \n",
" \n",
" xxmaj some people keep changing articles with no proof . xxmaj stop it . —the preceding unsigned comment was added by ( talk • contribs ) . \n",
" \n",
" xxmaj like yourself . 17 xxup gb per layer discs and triple - layer discs are not part of xxup hd - xxup dvd spec and as such are xxup possible xxup future additions that have no annouced released date , no schedule for when they may be released , and are 99 % likely to be incompatible with the optical heads in current and near future drives . xxmaj if you have some proof proving otherwise , then by all means add the 51 xxup gb disc back in and xxup refference it .,xxbos xxmaj terrorism \n",
" \n",
" xxmaj would it be possible for me to edit the terrorism article to add a section on state sponsored terrorism that is unrelated to the current dispute ?\n",
"y: LMLabelList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data/Data_Processed/Data_Classifier;\n",
"\n",
"Test: None, model=SequentialRNN(\n",
" (0): AWD_LSTM(\n",
" (encoder): Embedding(60004, 400, padding_idx=1)\n",
" (encoder_dp): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (rnns): ModuleList(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (2): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" )\n",
" (input_dp): RNNDropout()\n",
" (hidden_dps): ModuleList(\n",
" (0): RNNDropout()\n",
" (1): RNNDropout()\n",
" (2): RNNDropout()\n",
" )\n",
" )\n",
" (1): LinearDecoder(\n",
" (decoder): Linear(in_features=400, out_features=60004, bias=True)\n",
" (output_dp): RNNDropout()\n",
" )\n",
"), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x7faa93d3c1e0>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/Data_Processed/Data_Classifier'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[...], layer_groups=[Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): Embedding(60004, 400, padding_idx=1)\n",
" (1): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (2): LinearDecoder(\n",
" (decoder): Linear(in_features=400, out_features=60004, bias=True)\n",
" (output_dp): RNNDropout()\n",
" )\n",
")], add_time=True, silent=None)\n",
"alpha: 2.0\n",
"beta: 1.0], layer_groups=[Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): Embedding(60004, 400, padding_idx=1)\n",
" (1): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (2): LinearDecoder(\n",
" (decoder): Linear(in_features=400, out_features=60004, bias=True)\n",
" (output_dp): RNNDropout()\n",
" )\n",
")], add_time=True, silent=None)"
]
},
"execution_count": 70,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn.load('fit_head')"
]
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>text</th>\n",
" <th>target</th>\n",
" <th>pred</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>xxbos would continue to be , delayed due to my xxup irl schedule . xxbos xxmaj get xxmaj well \\n</td>\n",
" <td>\\n xxmaj get well soon . ) xxbos \" \\n \\n xxmaj gerry xxmaj xxunk \\n xxmaj</td>\n",
" <td>xxmaj hi the done , xxmaj xxmaj xxmaj \\n \\n xxmaj the xxmaj adams \\n \\n xxmaj hi</td>\n",
" </tr>\n",
" <tr>\n",
" <td>though , so please do nt delete cited factual information and i wo nt have to ... \" xxbos )</td>\n",
" <td>\\n \\n xxmaj for now i will leave your edit until i find proper sources to back myself .</td>\n",
" <td>\\n \\n xxmaj the example , 've be a message to i can that references . be up .</td>\n",
" </tr>\n",
" <tr>\n",
" <td>xxup rs now . if any editors can specify some content they believe should remain or possibly be merged to</td>\n",
" <td>the fear mongering article , make your suggestions here . cheers xxbos xxup redirect xxmaj talk : xxmaj history of</td>\n",
" <td>the article of , , you sure own on . xxmaj . xxmaj redirect xxmaj talk : xxmaj xxunk of</td>\n",
" </tr>\n",
" <tr>\n",
" <td>this great business . xxmaj it will be xxmaj wikipedia : wikiproject xxmaj music of wrestling . xxmaj join if</td>\n",
" <td>you are remotely interested - every little helps ! xxmaj thanks . xxbos xxmaj of course i am sorry ,</td>\n",
" <td>you want interested related in xxmaj time welcome to xxmaj please . xxmaj xxmaj the course , am not .</td>\n",
" </tr>\n",
" <tr>\n",
" <td>you created , xxmaj xxunk xxmaj villa xxmaj xxunk xxmaj soccer xxmaj club , has been tagged for deletion ,</td>\n",
" <td>as it meets one or more of the criteria for speedy deletion ; specifically , it serves only to attack</td>\n",
" <td>and you is the of more of the criteria for speedy deletion . it , the is as as prevent</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"learn.show_results()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**We will now unfreeze the whole model to fine tune al the weights for this particular dataset.**"
]
},
{
"cell_type": "code",
"execution_count": 105,
"metadata": {},
"outputs": [],
"source": [
"learn.unfreeze()"
]
},
{
"cell_type": "code",
"execution_count": 107,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>epoch</th>\n",
" <th>train_loss</th>\n",
" <th>valid_loss</th>\n",
" <th>accuracy</th>\n",
" <th>time</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>3.805411</td>\n",
" <td>3.787183</td>\n",
" <td>0.341519</td>\n",
" <td>13:29</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>3.590235</td>\n",
" <td>3.664578</td>\n",
" <td>0.357810</td>\n",
" <td>13:30</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>3.558987</td>\n",
" <td>3.597754</td>\n",
" <td>0.366857</td>\n",
" <td>13:27</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>3.434307</td>\n",
" <td>3.562573</td>\n",
" <td>0.371835</td>\n",
" <td>13:27</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>3.357494</td>\n",
" <td>3.556864</td>\n",
" <td>0.372336</td>\n",
" <td>13:27</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"learn.fit_one_cycle(5, 1e-3, moms=(0.8, 0.7))"
]
},
{
"cell_type": "code",
"execution_count": 108,
"metadata": {},
"outputs": [],
"source": [
"learn.save('fine_tuned')"
]
},
{
"cell_type": "code",
"execution_count": 109,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>epoch</th>\n",
" <th>train_loss</th>\n",
" <th>valid_loss</th>\n",
" <th>accuracy</th>\n",
" <th>time</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>3.424021</td>\n",
" <td>3.561982</td>\n",
" <td>0.371912</td>\n",
" <td>13:28</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>3.436336</td>\n",
" <td>3.554538</td>\n",
" <td>0.372866</td>\n",
" <td>13:31</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>3.403637</td>\n",
" <td>3.539753</td>\n",
" <td>0.374899</td>\n",
" <td>13:28</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>3.341969</td>\n",
" <td>3.531037</td>\n",
" <td>0.376424</td>\n",
" <td>13:30</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>3.260996</td>\n",
" <td>3.532155</td>\n",
" <td>0.376360</td>\n",
" <td>13:27</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"learn.fit_one_cycle(5, 5e-4, moms=(0.8, 0.7))"
]
},
{
"cell_type": "code",
"execution_count": 110,
"metadata": {},
"outputs": [],
"source": [
"learn.save('fine_tuned')"
]
},
{
"cell_type": "code",
"execution_count": 188,
"metadata": {
"collapsed": true
},
"outputs": [
{
"data": {
"text/plain": [
"LanguageLearner(data=TextLMDataBunch;\n",
"\n",
"Train: LabelList (143614 items)\n",
"x: LMTextList\n",
"xxbos xxmaj explanation \n",
" xxmaj why the edits made under my username xxmaj hardcore xxmaj metallica xxmaj fan were reverted ? xxmaj they were n't vandalisms , just closure on some gas after i voted at xxmaj new xxmaj york xxmaj dolls xxup fac . xxmaj and please do n't remove the template from the talk page since i 'm retired xxunk,xxbos xxmaj xxunk ! xxmaj he matches this background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ),xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about the formatting than the actual info .,xxbos \" \n",
" xxmaj more \n",
" i ca n't make any real suggestions on improvement - i wondered if the section statistics should be later on , or a subsection of \" \" types of accidents \" \" xxup -i think the references may need tidying so that they are all in the exact same format ie date format etc . i can do that later on , if no - one else does first - if you have any preferences for formatting style on references or want to do it yourself please let me know . \n",
" \n",
" xxmaj there appears to be a backlog on articles for review so i guess there may be a delay until a reviewer turns up . xxmaj it 's listed in the relevant form eg xxmaj wikipedia : xxmaj xxunk # xxmaj transport \",xxbos xxmaj you , sir , are my hero . xxmaj any chance you remember what page that 's on ?\n",
"y: LMLabelList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data/Data_Processed/Data_Classifier;\n",
"\n",
"Valid: LabelList (15957 items)\n",
"x: LMTextList\n",
"xxbos would continue to be , delayed due to my xxup irl schedule .,xxbos xxmaj get xxmaj well \n",
" \n",
" xxmaj get well soon . ),xxbos \" \n",
" \n",
" xxmaj gerry xxmaj xxunk \n",
" xxmaj hey , amigo , take the protect off \" \" xxmaj gerry xxmaj xxunk . \" \" i think you 're letting your sexual preference cloud your rational thought . xxmaj the edits you have called \" \" vandalism \" \" ( not done by me ) are nowhere near vandalism . xxmaj what or who are you trying to protect . xxmaj unlock it now and lighten up . xxmaj you 've been an admin for too long to act in this juvenile way . xxmaj stop showing favoritism and be even - handed . xxmaj now ! ! ! rossp \",xxbos xxup hd xxup dvd winning ? \n",
" \n",
" xxmaj the section previously stated that on xxmaj january 8 , 2006 , at the xxmaj consumer xxmaj electronics xxmaj show , it had been said that hd dvd was ahead of xxunk . however they did n't cite it . i looked and the article i cited said they were neck and neck . \n",
" \n",
" xxup ok , my reasoning on the cnn article was probably weak , but it still may be true . \n",
" xxmaj xxunk 's website claims to have passed hd dvd , but i doubt that that is a very reliable place to look for info . xxmaj can someone find something else or revert it back ? ( i do n't know how to do that ) \n",
" \n",
" \n",
" xxup hd - xxup dvd has the backing of the porn industry , so its practically won already . \n",
" \n",
" xxmaj blu xxmaj ray fanboy stay out \n",
" \n",
" xxmaj some people keep changing articles with no proof . xxmaj stop it . —the preceding unsigned comment was added by ( talk • contribs ) . \n",
" \n",
" xxmaj like yourself . 17 xxup gb per layer discs and triple - layer discs are not part of xxup hd - xxup dvd spec and as such are xxup possible xxup future additions that have no annouced released date , no schedule for when they may be released , and are 99 % likely to be incompatible with the optical heads in current and near future drives . xxmaj if you have some proof proving otherwise , then by all means add the 51 xxup gb disc back in and xxup refference it .,xxbos xxmaj terrorism \n",
" \n",
" xxmaj would it be possible for me to edit the terrorism article to add a section on state sponsored terrorism that is unrelated to the current dispute ?\n",
"y: LMLabelList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data/Data_Processed/Data_Classifier;\n",
"\n",
"Test: None, model=SequentialRNN(\n",
" (0): AWD_LSTM(\n",
" (encoder): Embedding(60004, 400, padding_idx=1)\n",
" (encoder_dp): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (rnns): ModuleList(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (2): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" )\n",
" (input_dp): RNNDropout()\n",
" (hidden_dps): ModuleList(\n",
" (0): RNNDropout()\n",
" (1): RNNDropout()\n",
" (2): RNNDropout()\n",
" )\n",
" )\n",
" (1): LinearDecoder(\n",
" (decoder): Linear(in_features=400, out_features=60004, bias=True)\n",
" (output_dp): RNNDropout()\n",
" )\n",
"), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x7faa93d3c1e0>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/Data_Processed/Data_Classifier'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
"learn: LanguageLearner(data=TextLMDataBunch;\n",
"\n",
"Train: LabelList (143614 items)\n",
"x: LMTextList\n",
"xxbos xxmaj explanation \n",
" xxmaj why the edits made under my username xxmaj hardcore xxmaj metallica xxmaj fan were reverted ? xxmaj they were n't vandalisms , just closure on some gas after i voted at xxmaj new xxmaj york xxmaj dolls xxup fac . xxmaj and please do n't remove the template from the talk page since i 'm retired xxunk,xxbos xxmaj xxunk ! xxmaj he matches this background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ),xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about the formatting than the actual info .,xxbos \" \n",
" xxmaj more \n",
" i ca n't make any real suggestions on improvement - i wondered if the section statistics should be later on , or a subsection of \" \" types of accidents \" \" xxup -i think the references may need tidying so that they are all in the exact same format ie date format etc . i can do that later on , if no - one else does first - if you have any preferences for formatting style on references or want to do it yourself please let me know . \n",
" \n",
" xxmaj there appears to be a backlog on articles for review so i guess there may be a delay until a reviewer turns up . xxmaj it 's listed in the relevant form eg xxmaj wikipedia : xxmaj xxunk # xxmaj transport \",xxbos xxmaj you , sir , are my hero . xxmaj any chance you remember what page that 's on ?\n",
"y: LMLabelList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data/Data_Processed/Data_Classifier;\n",
"\n",
"Valid: LabelList (15957 items)\n",
"x: LMTextList\n",
"xxbos would continue to be , delayed due to my xxup irl schedule .,xxbos xxmaj get xxmaj well \n",
" \n",
" xxmaj get well soon . ),xxbos \" \n",
" \n",
" xxmaj gerry xxmaj xxunk \n",
" xxmaj hey , amigo , take the protect off \" \" xxmaj gerry xxmaj xxunk . \" \" i think you 're letting your sexual preference cloud your rational thought . xxmaj the edits you have called \" \" vandalism \" \" ( not done by me ) are nowhere near vandalism . xxmaj what or who are you trying to protect . xxmaj unlock it now and lighten up . xxmaj you 've been an admin for too long to act in this juvenile way . xxmaj stop showing favoritism and be even - handed . xxmaj now ! ! ! rossp \",xxbos xxup hd xxup dvd winning ? \n",
" \n",
" xxmaj the section previously stated that on xxmaj january 8 , 2006 , at the xxmaj consumer xxmaj electronics xxmaj show , it had been said that hd dvd was ahead of xxunk . however they did n't cite it . i looked and the article i cited said they were neck and neck . \n",
" \n",
" xxup ok , my reasoning on the cnn article was probably weak , but it still may be true . \n",
" xxmaj xxunk 's website claims to have passed hd dvd , but i doubt that that is a very reliable place to look for info . xxmaj can someone find something else or revert it back ? ( i do n't know how to do that ) \n",
" \n",
" \n",
" xxup hd - xxup dvd has the backing of the porn industry , so its practically won already . \n",
" \n",
" xxmaj blu xxmaj ray fanboy stay out \n",
" \n",
" xxmaj some people keep changing articles with no proof . xxmaj stop it . —the preceding unsigned comment was added by ( talk • contribs ) . \n",
" \n",
" xxmaj like yourself . 17 xxup gb per layer discs and triple - layer discs are not part of xxup hd - xxup dvd spec and as such are xxup possible xxup future additions that have no annouced released date , no schedule for when they may be released , and are 99 % likely to be incompatible with the optical heads in current and near future drives . xxmaj if you have some proof proving otherwise , then by all means add the 51 xxup gb disc back in and xxup refference it .,xxbos xxmaj terrorism \n",
" \n",
" xxmaj would it be possible for me to edit the terrorism article to add a section on state sponsored terrorism that is unrelated to the current dispute ?\n",
"y: LMLabelList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data/Data_Processed/Data_Classifier;\n",
"\n",
"Test: None, model=SequentialRNN(\n",
" (0): AWD_LSTM(\n",
" (encoder): Embedding(60004, 400, padding_idx=1)\n",
" (encoder_dp): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (rnns): ModuleList(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (2): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" )\n",
" (input_dp): RNNDropout()\n",
" (hidden_dps): ModuleList(\n",
" (0): RNNDropout()\n",
" (1): RNNDropout()\n",
" (2): RNNDropout()\n",
" )\n",
" )\n",
" (1): LinearDecoder(\n",
" (decoder): Linear(in_features=400, out_features=60004, bias=True)\n",
" (output_dp): RNNDropout()\n",
" )\n",
"), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x7faa93d3c1e0>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/Data_Processed/Data_Classifier'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[...], layer_groups=[Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): Embedding(60004, 400, padding_idx=1)\n",
" (1): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (2): LinearDecoder(\n",
" (decoder): Linear(in_features=400, out_features=60004, bias=True)\n",
" (output_dp): RNNDropout()\n",
" )\n",
")], add_time=True, silent=None)\n",
"alpha: 2.0\n",
"beta: 1.0], layer_groups=[Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): Embedding(60004, 400, padding_idx=1)\n",
" (1): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (2): LinearDecoder(\n",
" (decoder): Linear(in_features=400, out_features=60004, bias=True)\n",
" (output_dp): RNNDropout()\n",
" )\n",
")], add_time=True, silent=None)"
]
},
"execution_count": 188,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn.load('fine_tuned')"
]
},
{
"cell_type": "code",
"execution_count": 158,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"I like to see you return and look for you . The Red Pen of\n",
"I like to know what the hell have you done , but i did n't much of that\n",
"I like to be careful of the proper style of the British and Irish Isles\n",
"I like to incorporate some of the information into the article . In this case , the\n",
"I like to see how i can get a chance to respond . xxbos While i appreciate\n"
]
}
],
"source": [
"print('\\n'.join(learn.predict(\"I like to\", n_words=15, temperature=0.75) for i in range(5)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we can see, the model has somewhat of an understanding of grammar and of how an opinion is expressed.<br>\n",
"It also has an understanding of how possessive pronouns work.<br>\n",
"Also, the model seems to have learnt that the comments generally have a speaker that is expressing an opinion and that it is frequently addressed to some other people. This seems enough information to go forward with training the classifier."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The encoder of this model will be used to train a classifier on the dataset."
]
},
{
"cell_type": "code",
"execution_count": 189,
"metadata": {},
"outputs": [],
"source": [
"learn.save_encoder('fine_tuned_enc')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Training the Classifier"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## Preparing the classification dataset"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"hidden": true
},
"outputs": [],
"source": [
"PATH = Path('data/').absolute()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"[PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/models'),\n",
" PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/classifier_train_data'),\n",
" PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/Data_Processed'),\n",
" PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/test_labels.csv'),\n",
" PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/sample_submission.csv'),\n",
" PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/train.csv'),\n",
" PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data/test.csv')]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"PATH.ls()"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {
"hidden": true
},
"outputs": [],
"source": [
"df = pd.read_csv(PATH/'train.csv').drop('id', axis=1)"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>comment_text</th>\n",
" <th>toxic</th>\n",
" <th>severe_toxic</th>\n",
" <th>obscene</th>\n",
" <th>threat</th>\n",
" <th>insult</th>\n",
" <th>identity_hate</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Explanation\\nWhy the edits made under my usern...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>D'aww! He matches this background colour I'm s...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Hey man, I'm really not trying to edit war. It...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>\"\\nMore\\nI can't make any real suggestions on ...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>You, sir, are my hero. Any chance you remember...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" comment_text toxic severe_toxic \\\n",
"0 Explanation\\nWhy the edits made under my usern... 0 0 \n",
"1 D'aww! He matches this background colour I'm s... 0 0 \n",
"2 Hey man, I'm really not trying to edit war. It... 0 0 \n",
"3 \"\\nMore\\nI can't make any real suggestions on ... 0 0 \n",
"4 You, sir, are my hero. Any chance you remember... 0 0 \n",
"\n",
" obscene threat insult identity_hate \n",
"0 0 0 0 0 \n",
"1 0 0 0 0 \n",
"2 0 0 0 0 \n",
"3 0 0 0 0 \n",
"4 0 0 0 0 "
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"hidden": true
},
"outputs": [],
"source": [
"bs = 48"
]
},
{
"cell_type": "code",
"execution_count": 179,
"metadata": {
"hidden": true
},
"outputs": [],
"source": [
"data_clas = (TextList.from_csv(path=PATH, csv_name='train.csv', cols='comment_text', vocab=data_lm.vocab)\n",
" .split_by_rand_pct(0.1)\n",
" .label_from_df(cols=['toxic', 'severe_toxic', 'obscene', 'threat', 'insult', 'identity_hate'])\n",
" .databunch(bs=bs))\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 180,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>text</th>\n",
" <th>target</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>xxbos xxmaj take that ! \\n \\n xxup in xxup the xxup ass xxup in xxup the xxup ass xxup in xxup the xxup ass xxup in xxup the xxup ass xxup in xxup the xxup ass xxup in xxup the xxup ass xxup in xxup the xxup ass xxup in xxup the xxup ass xxup in xxup the xxup ass xxup in xxup the xxup ass xxup in</td>\n",
" <td>toxic;severe_toxic;obscene</td>\n",
" </tr>\n",
" <tr>\n",
" <td>xxbos \" \\n \\n xxmaj fourth xxmaj examination , 17th xxmaj december , 1455 . [ xxmaj additional statements . ] \\n xxmaj the sum of a thousand pounds , or crowns , was given by the xxmaj king of xxmaj england for the surrender of the xxmaj maid ; and an annuity of 300 pounds to the soldier of the xxmaj duke of xxmaj burgundy who had</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <td>xxbos \" \\n \\n xxmaj sitush you are a menace to xxmaj wikipedia as this conversation from the page of another editor shows - xxmaj are you a 15 year old kid trying his hand in editing academic articles ? ? ? \\n \\n xxmaj first xxmaj xxunk xxmaj shastri writes to xxmaj jonathan . xxmaj this is because xxmaj sitush has just vandalized his article about xxmaj</td>\n",
" <td></td>\n",
" </tr>\n",
" <tr>\n",
" <td>xxbos xxmaj after all the times you have thwarted me ... xxmaj after all the times my plans for world domination were foiled by your xxunk interference ... xxmaj after all the countless times you escaped at the very last moment , finally , i , xxmaj xxunk xxmaj the xxmaj mighty , have defeated you , xxmaj crum375 , xxmaj space xxmaj commander xxmaj from xxmaj swalwell ! \\n</td>\n",
" <td>toxic</td>\n",
" </tr>\n",
" <tr>\n",
" <td>xxbos \" \\n \\n ( r to my favorite editor , started before xxmaj spotfixer commented : ) \\n xxmaj not months , years . xxmaj he 'd been xxunk that shit on xxmaj shapiro since 2006 . xxmaj anyway , thanks for the kind words - coming from one in your time and place it 's especially touching that you 'd be aggrieved over my two week</td>\n",
" <td></td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"data_clas.show_batch()"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"We save the data object because processing the dataset each time requires significant time."
]
},
{
"cell_type": "code",
"execution_count": 181,
"metadata": {
"hidden": true
},
"outputs": [],
"source": [
"data_clas.save('classifier_train_data')"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"hidden": true
},
"outputs": [],
"source": [
"data_clas = load_data(PATH, 'classifier_train_data', bs=bs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Preparing Model"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"# Reclaiming up some memory\n",
"learn = None\n",
"gc.collect()\n",
"\n",
"learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)\n",
"learn.load_encoder('../Data_Processed/Data_Classifier/models/fine_tuned_enc')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def roc_score(model=learn):\n",
" \"\"\"\n",
" Calculates the ROC score for all classes separately and returns their mean.\n",
" \"\"\"\n",
" y_preds, y_true = learn.get_preds()\n",
" return roc_auc_score(y_true.numpy(), y_preds.numpy(), average='macro')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This metric will classify as correct any prediction above `thresh = 0.4`.<br>\n",
"We will then simply calculate the accuracy taking into account each label as in the normal accuracy measure"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"acc = partial(accuracy_thresh, thresh=0.4)\n",
"learn.metrics.append(acc)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"def find_lr(model=learn):\n",
" \"\"\"\n",
" Convenience function to perform learning rate range test and plot results.\n",
" \"\"\"\n",
" model.lr_find()\n",
" model.recorder.plot(skip_end=15)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Start Training"
]
},
{
"cell_type": "code",
"execution_count": 201,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8XHW9//HXJ3uapE3bJKVtui+U0tKWlkJZhELFylUQUIQrCi6gXgUVvXq9/n6gVVGvenFfQFlcUBEuPwFFBaQUCm0JNC3d6L4kpc3W7Hvy+f0xkzA3Jk3a5GQmk/fz8ZhHz5z5zjmfb6edz5zv95zPMXdHREQEICHaAYiISOxQUhARkU5KCiIi0klJQUREOikpiIhIJyUFERHppKQgIiKdlBRERKSTkoKIiHRKinYAJyonJ8enTp0a7TBERIaUV155pczdc3trF1hSMLM0YA2QGt7Pw+5+R5c2U4B7gVygArje3YuOt92pU6dSUFAQTNAiInHKzA70pV2Qw0dNwMXuvgBYCKw0s3O6tPkO8Ct3PwNYBXwjwHhERKQXgSUFD6kNP00OP7pW35sLPBNefha4Iqh4RESkd4FONJtZopkVAiXAU+6+vkuTTcDV4eUrgSwzGxtkTCIi0rNAk4K7t7n7QiAfWGpm87o0+RxwoZltBC4EioHWrtsxs5vNrMDMCkpLS4MMWURkWBuUU1LdvRJYDazssv6wu1/l7ouAL4XXVXXz/rvdfYm7L8nN7XXyXERETlJgScHMcs0sO7ycDqwAdnRpk2NmHTF8kdCZSCIiEiVBHimMB541s83Ay4TmFJ4ws1Vmdnm4zUXA62a2ExgHfD3AeEREpBeBXafg7puBRd2svz1i+WHg4aBiEBGJB40tbfzgmV287fRTWDApO9B9qcyFiEiMK6tt4ier97DjSHXg+1JSEBGJcRV1zQCMzUgNfF9KCiIiMa48nBTGZKYEvi8lBRGRGFde23GkoKQgIjLsVdQ1ATA2U8NHIiLDXnltMylJCWSkJAa+LyUFEZEYV17XTE5GCmYW+L6UFEREYlxFXfOgTDKDkoKISMwrr21izCCcjgpKCiIiMa9j+GgwKCmIiMS48tpmxigpiIhIQ3MbDS1tmlMQEREoD1+jkKM5BRER6biaWcNHIiLyZjE8DR+JiEj5IFZIBSUFEZGYVl4bmlPQRLOIiFBR10zqINU9AiUFEZGYVlbbzNhBqnsEASYFM0szsw1mtsnMtprZV7ppM9nMnjWzjWa22cwuCyoeEZGhqKKuadCGjiDYI4Um4GJ3XwAsBFaa2Tld2vwf4CF3XwRcC/wkwHhERIacirrmQZtkhgCTgofUhp8mhx/etRkwMrw8CjgcVDwiIkNRx/DRYAl0TsHMEs2sECgBnnL39V2afBm43syKgL8AtwQZj4jIUFNR1zxo1yhAwEnB3dvcfSGQDyw1s3ldmlwH3O/u+cBlwK/N7J9iMrObzazAzApKS0uDDFlEJGbUN7eG6h7Fw/BRJHevBFYDK7u89GHgoXCbl4A0IKeb99/t7kvcfUlubm7A0YqIxIaOEhdxMXxkZrlmlh1eTgdWADu6NDsIXBJucxqhpKBDARERBr/EBUBSgNseDzxgZomEks9D7v6Ema0CCtz9MeCzwD1m9hlCk843unvXyWgRkWGpo0LqYBXDgwCTgrtvBhZ1s/72iOVtwHlBxSAiMpS9OXwUZ3MKIiJy4sqjMHykpCAiEqM66h6NGKS6R6CkICISs8oHue4RKCmIiMSs8romxmYO3nwCKCmIiMSsirrmQT3zCJQURERiVnnt4Ja4ACUFEZGYVV7XNKhXM4OSgohITKpvbqWxpX1Q6x6BkoKISEzqvHBNw0ciItJ54ZqGj0REpCIKdY9ASUFEJCaVhYePcnSdgoiIdJTN1pGCiIhQXttEWvLg1j0CJQURkZhUXtfM2IzUQa17BEoKIiIxKRolLkBJQUQkJkWjxAUoKYiIxCQdKYiICADuTlnt4Nc9ggDv0WxmacAaIDW8n4fd/Y4ube4CloefjgDy3D07qJhERIaC+uY2mlrbB/1eChBgUgCagIvdvdbMkoEXzOxJd1/X0cDdP9OxbGa3AIsCjEdEZEiI1jUKEODwkYfUhp8mhx9+nLdcB/wuqHhERIaKstpQiYuceJtoNrNEMysESoCn3H19D+2mANOAfwQZj4jIUPDmkcLgDx8FmhTcvc3dFwL5wFIzm9dD02sJzTm0dfeimd1sZgVmVlBaWhpUuCIiMaGzbHY8DR9FcvdKYDWwsocm13KcoSN3v9vdl7j7ktzc3AAiFBGJHZ1ls+Np+MjMcs0sO7ycDqwAdnTT7lRgNPBSULGIiAwlFXUddY+CPBeoe0EeKYwHnjWzzcDLhOYUnjCzVWZ2eUS764Dfu/vxJqFFRIaN8tpQ3aNoCCwNuftmujnF1N1v7/L8y0HFICIyFJXXRafEBeiKZhGRmFN0rJ5xI9Oism8lBRGRGFLb1MresjrmTxwVlf0rKYiIxJDXiqpwh/n5SgoiIsPea8WVACzIj04ZOCUFEZEYsqmoivzR6VGpewRKCiIiMWVzUSVnRGnoCJQURERixrG6Zg5VNHBGlIaOQElBRCRmbC6uAtCRgoiIwOZDoUnmeVE6HRWUFEREYsamoiqm52YwMi05ajEoKYiIxIjXiiujdipqByUFEZEYcLS6kaPVTVG7krmDkoKISAzYFJ5PWDBJSUFEZNh7rbiKxARj7nglBRGRYW9TURWz8jJJT0mMahxKCiIiUebubC6K/iQzKCmIiERd0bEGKutbolYZNZKSgohIlG0qim5l1EhKCiIiUba5qIqUxAROPSUr2qEoKYiIRNumQ5WcNj6LlKTofyUHFoGZpZnZBjPbZGZbzewrPbS7xsy2hds8GFQ8IiKxqL3d2VJcFdXKqJGSAtx2E3Cxu9eaWTLwgpk96e7rOhqY2Szgi8B57n7MzPICjEdEJObsLaulrrktqpVRIwWWFNzdgdrw0+Tww7s0uwn4sbsfC7+nJKh4RERi0aZDHeWyY+NIIdABLDNLNLNCoAR4yt3Xd2kyG5htZmvNbJ2ZrexhOzebWYGZFZSWlgYZsojIoHqtuIr05ERm5mVGOxQg4KTg7m3uvhDIB5aa2bwuTZKAWcBFwHXAL8zsn9Klu9/t7kvcfUlubm6QIYuIDKqth6uYO2EkiQkW7VCAQTr7yN0rgdVA1yOBIuBP7t7i7vuA1wklCRGRuNfe7mw9XM28CSOjHUqnIM8+yu341W9m6cAKYEeXZv8PWB5uk0NoOGlvUDGJiMSSfeV11De3cXqUy2VHCvLso/HAA2aWSCj5POTuT5jZKqDA3R8D/gZcambbgDbg3929PMCYRERixpbwPZnnTRgGScHdNwOLull/e8SyA7eFHyIiw8rWw9WkJCUwa1xsTDKDrmgWEYma14qqOO2ULJITY+erOHYiEREZRtydLYerYmo+AZQURESi4lBFAzWNrTE1nwBKCiIiUbHlcHiSeWLsnI4KSgoiIlGxpbiKpARj9rjol8uO1KekYGYzzCw1vHyRmd3a3ZXHIiLSN1sOVzN7XBZpydG9J3NXfT1SeARoM7OZwC+BaYDKXIuInAR3Z2txVcwNHUHfk0K7u7cCVwLfc/fPELo4TURETtAbVY2U1zUzL8bOPIK+J4UWM7sOuAF4IrwuOZiQRETiW8eVzKfH2JlH0Pek8EFgGfB1d99nZtOA3wQXlohI/NpyuJoEg9PGx9YkM/SxzIW7bwNuBTCz0UCWu38zyMBEROLV1uIqZuRmMiIlyPJzJ6evZx+tNrORZjYG2ATcZ2b/HWxoIiLxacvhKubH4HwC9H34aJS7VwNXAfe5+2JCpbBFROQElNQ0crS6KebKW3Toa1JIMrPxwDW8OdEsIiInaGtxNUBM3VgnUl+TwipC9z7Y4+4vm9l0YFdwYYmIxKeOM4/mxmhS6OtE8x+BP0Y83wtcHVRQIiLxasvhKqblZJCVFptn9fd1ojnfzB41sxIzO2pmj5hZftDBiYjEmy3F1TF50VqHvg4f3Qc8BkwAJgKPh9eJiEgfHatrpriyIWbnE6DvSSHX3e9z99bw434gN8C4RETizqaiSgDm5w/9I4UyM7vezBLDj+uB8uO9wczSzGyDmW0ys61m9pVu2txoZqVmVhh+fORkOiEiMhRsPFiJGZyRH7tFpvt6Od2HgB8BdwEOvEio9MXxNAEXu3utmSUDL5jZk+6+rku7P7j7J08kaBGRoajwUCWz87LITI29K5k79OlIwd0Puvvl7p7r7nnu/i5CF7Id7z3u7rXhp8nhh/cvXBGRocndKTxUyaLJsXuUAP2789ptvTUIDzUVAiXAU+6+vptmV5vZZjN72Mwm9SMeEZGYta+sjqqGFhZOit+kYL01cPc2d18I5ANLzWxelyaPA1Pd/QzgaeCBbndkdrOZFZhZQWlpaT9CFhGJjsJDoUnmhXF8pNDnoSB3rwRWAyu7rC9396bw03uAxT28/253X+LuS3JzddKTiAw9hYcqyUhJZFZe7JXLjnTc2Q4zq6H7L38D0nt5by7Q4u6VZpZOqIDet7q0Ge/ub4SfXg5s72vgIiJDSeGhSs7IzyYxoddBlqg6blJw9/6ktPHAA2aWSOiI5CF3f8LMVgEF7v4YcKuZXQ60AhXAjf3Yn4hITGpsaWPb4Wpuesv0aIfSq8DOi3L3zcCibtbfHrH8ReCLQcUgIhILth6uorXdY36SGfo3pyAiIn2w8WBoknmRkoKIiGw8VMnE7HTyRqZFO5ReKSmIiASs8GDlkBg6AiUFEZFAldQ0UlzZoKQgIiKhowQg5stbdFBSEBEJUOGhSpISLKZvrBNJSUFEJECFhyqZMz6LtOTEaIfSJ0oKIiIBaWt3NhdVsWjS6GiH0mdKCiIiAdldUkttU+uQmWQGJQURkcAUHjoGxH5l1EhKCiIiASk8VMnItCSmjc2Idih9pqQgIhKQVw4cY8GkbBJivDJqJCUFEZEAlNY0sfNoLctmjI12KCdESUFEJAAv7ikD4LwZOVGO5MQoKYiIBGDt7jJGpiUNmYvWOigpiIgMMHdn7e5yzp2RE/N3WutKSUFEZIAdKK+nuLKB82YOrfkEUFIQERlwL+wOzyfMHFrzCaCkICIy4F7cU8aEUWlMyxk61yd0CCwpmFmamW0ws01mttXMvnKctu82MzezJUHFIyIyGNranRf3lHPuzBzMhtZ8AkBSgNtuAi5291ozSwZeMLMn3X1dZCMzywJuBdYHGIuIyKDYdriayvoWzh+CQ0cQ4JGCh9SGnyaHH95N068C/wU0BhULQEVdMw+9fCjIXYiIsDZ8fcK5Q+yitQ6BzimYWaKZFQIlwFPuvr7L64uASe7+RJBxANz/4n4+/8jmzgtKRESCsHZ3GbPHZZI3Mi3aoZyUQJOCu7e5+0IgH1hqZvM6XjOzBOAu4LO9bcfMbjazAjMrKC0tPalYPn7hDCaPGcGXHt1CY0vbSW1DROR4GlvaeHl/xZA866jDoJx95O6VwGpgZcTqLGAesNrM9gPnAI91N9ns7ne7+xJ3X5Kbm3tSMaSnJPL1K+exr6yOnzy7+6S2ISJyPK8ePEZjS/uQK20RKcizj3LNLDu8nA6sAHZ0vO7uVe6e4+5T3X0qsA643N0Lgorpglm5vGvhBH763B52Ha0JajciMkyt3V1GYoJx9vQx0Q7lpAV5pDAeeNbMNgMvE5pTeMLMVpnZ5QHu97j+zzvmkpGaxBf/5zXa27ub9xYROTlrd5ezcFI2WWnJ0Q7lpAV59tFmd1/k7me4+zx3XxVef7u7P9ZN+4uCPErokJOZyn9edhoFB47xu5cPBr07ERkmqhpa2FxUyXlD9KyjDsPyiub3LM7nnOlj+OaTOyipDvRMWBEZJtbvLafdh2Zpi0jDMimYGXdeOZ+m1nZue2gTDc06G0lE+ufxzW+QlZrEosmjox1KvwzLpAAwPTeTr10xj7V7ynj/L9dT1dAS7ZBEZIgqOlbPX157g+vOnkxK0tD+Wh3a0ffTNWdN4ofXLWJTUSXv/flLGkoSkZNy7wv7MeDGc6dGO5R+G9ZJAeAdZ0zg3hvP4mBFPe/+2UscKK+LdkgiMoRU1bfw+5cP8s4FE5iQnR7tcPpt2CcFCF2/8OBN51Dd2MLVP32J14/oGgYR6ZsHNxykvrmNmy6YHu1QBoSSQtjCSdk8/LFlJBhc/8v1OmIQkV41tbZx39p9XDArh7kTRkY7nAGhpBBhZl4Wv/3I2bS2tfO+X6znSJXmGESkZ48VHqakpilujhJASeGfzBqXxf0fXMqxumbe/8v1VNQ1RzskEYlB7s49z+9lzilZXDBraF+bEElJoRsLJmXzixvO4kBFPTfet4GaxuOfrtrc2k5FXTPuKpshMlw8t7OUnUdruemC6UPyDms9CfLOa0Pashlj+en7zuSjv36Fa36+jpl5mbS0ttPS1k5zWzu1Ta0cq2umvK6ZmsZWAC6dO47vXrNgSNc9EZG+uXvNXk4ZmcY7F0yIdigDSknhOC45bRzfu3Yh//33nWwpriI50UhOTCA5MYGM1ETyR2czNiOFsRkp1Le0cfeavbzrx2u5+wNLmJGbGe3wRSQgz+4o4cU95fzH2+cM+YvVurKhNuSxZMkSLygIvG7eSXlpTzmfePBVWlrbueu9C1kxd1y0QxKRAbavrI7Lf/QCk0aP4JGPn0t6SmK0Q+oTM3vF3f/pfjVdxVeKi7JlM8by+C3nMyVnBB/5VQHff3qX5hlE4khtUys3/aqApATj5+9fPGQSwolQUhhgE7PTefhj53LVmRO56+md3PmX7UoMInGgvd35zB8K2VdWx4//9UwmjRkR7ZACoTmFAKQlJ/Ld9ywgKzWJe57fR3pyIrddemq0wxKRfvjBP3bx1Laj3P6OuZw7xMtjH4+SQkDMjDveeTqNLe384B+7SU1O5BPLZ0Y7LBE5CX/feoTvPb2Lq86cyAfPmxrtcAKlpBCghATjzqvm09jaxrf/9jrpyYl86Pxp0Q5LRE7AnwqL+feHN3NG/ijuvHJ+XF2T0B0lhYAlJhjffc8CmlvbWfXENhITjA8smxL3/7BEhjp353tP7+L7z+xi6dQx/Oz9i0lLjr+J5a4Cm2g2szQz22Bmm8xsq5l9pZs2HzOz18ys0MxeMLO5QcUTTUmJCXz/2kWsOC2POx7bymf+UNjrVdIiEj2NLW3c+vtCvv/MLt69OJ9ff2QpYzJSoh3WoAjsOgUL/RTOcPdaM0sGXgA+5e7rItqMdPfq8PLlwL+5+8rjbTeWr1PoTVu78+Nnd/O9p3cyacwIfnjdIs7Iz452WCISVlnfzJ7SWr725+1sPFjJF1bO4WMXxkcZi75epxDY8JGHsk1t+Gly+OFd2lRHPM3o+nq8SUwwbr1kFstmjOVTv9vI1T99kc+/bQ4fPn8aCQlD/x+dyFDh7hysqGf93gpePXiM3SW17C2r6yyAmZacwM+uP5OV88ZHOdLBF+icgpklAq8AM4Efu/v6btp8ArgNSAEuDjKeWHHW1DH85VMX8IVHNvP1v2xnb1kdd145Ly5+jYjEqqqGFv625QjP7y5jw75yjlY3AZA9IpnZ47J42+njmJ6TyfTcDObnjyIvKy3KEUfHoJS5MLNs4FHgFnff0kObfwXe5u43dPPazcDNAJMnT1584MCBIMMdNO7Ot/76Oj97bg9ffPscPnrhjGiHJBJXmlvbWf16CY9uLOaZ7SU0t7WTl5XK2dPHcva0MZw9bQwz8zKHxQ+yqA8fRXL3SjNbDawEuk0KwO+Bn/bw/ruBuyE0pxBEjNFgZnz+badSdKyebzy5g8ljRvD2+cPvcFVkoB0sr+f+F/fzPxuLqKxvYWxGCv969mSuOnMi8yeOGhZJ4GQFlhTMLBdoCSeEdGAF8K0ubWa5+67w038BdjHMJCQY33nPAoorG/j0HwoZn53OwkmafBY5Ue7OS3vLuW/tfp7efpREM1bOO4WrF+dz/swckhNV1acvgjxSGA88EJ5XSAAecvcnzGwVUODujwGfNLMVQAtwDPinoaPhIC05kXs+sIQrf7KWjzzwMo/+23lxW1dFZKBV1jfzp8LD/G7DQXYcqWFMRgqfXD6T68+ZwriRw3NeoD9UOjuG7C6p4cqfvMgpI9O4670LmTdxVLRDEolJbe3Oml2lPFxQxFPbjtLc1s7pE0Zyw7KpXL5wwrC4yOxE9XVOQUkhxry4p4yP/voVahpbuWROHrdcMkvDSTJsuTt7SuvYeriK/WX17C+vY19ZHXtLa6lubGX0iGTetWgi71k8ibkTRkY73JimpDCEVTW08MCL+7l37T4q61u4YFYOt711Nosmj452aCKBampt40B5Pev3VbBubznr91ZQVhs6ddQMJoxKZ2rOCKaOzeCCWTlcPGdc3N35LChKCnGgtqmV36w7wD1r9lJR38xHzp/GZy89VYfGEvO2FFfx/K4yEhMgJTGB1OREUhITcKCuqZXaplbqm1upa2rjaHUjhysbOFzVSGlNU+c2xo9K4+xpYzhn+lgWTs5m6tgM/dvvByWFOFLX1Mo3ntzOb9YdZFZeJv99zULm52u+QWJLQ3Mbj28+zG/XH2TTocpe2ycmGBkpieRmpTIhO50Jo9KZkJ1O/uh0zpo6hklj0nXq6ABSUohDz+0s5QsPb6astolPXjyTTyyfqdPsJCra2p3DlQ3sK6tjf3kd29+o4YnNh6lpbGVWXibvO3syVyycSEpSAk2t7TS3ttPU2oZhZKQmkpGaRGpSgr70B5GSQpyqqm/hy49v5dGNxZw6LotVV5zO2dPHRjssiXPuzutHa3h621Ge2l7C9sPVNLe1d76ekZLIJaeN4/pzpnDW1NH6so9BSgpx7qltR/nyY1sprmzgioUT+M/LTtM52TIgWtraeaOykaJj9RQda2DbG9U8vf0oRccaAFgwKZtzpo1hem4GU8dmMC0ng9ysVCWCGKekMAw0NLfx09W7+dmavSSHK7DecO5UTcZJnzQ0t7H9SDW7j9ayq6SGnUdr2V1SyxtVDbRHfC2kJiVw/swcVswdxyVz8sjTj48hSUlhGDlQXsdXHt/GP3aUMCYjhfefM4UPLJvC2MzUaIcmMaat3Vm7u4xHNxbz1y1HaGhpA0Jf/DNyM5k1LpMpYzPIHx2a8J00egSnjErT3FUcUFIYhtbtLeeeNXt5ZkcJqUkJXL04n5sumM60nIxohyZR1NbubCqq5MnX3uBPhYcpqWliZFoS/3LGBJafmsupp2SRP3oEibqnR1yLqSqpMjjOmT6Wc6aPZXdJDb94fh8PFxTxh5cPcf3Zk/n0itmMHia3E5RQPaDndpby7I4SnttZyrH6FpITjYtOzeOqRRNZPidPw4zSLR0pxLGSmka+//QufrfhIJmpSdx6ySw+sGyqrgAdZB1DNkXHGjha3UhJTROlNY00tzn5o9OZPGZE52PSmBGMSk8+oe03t7az82gNhYcqOx97SmtxhzEZKVw0O5eL5uRx4axcRo04sW1L/NDwkXR6/UgNX/vzNp7fVca0nAzevTif2eOymD0uU8MGAXv9SA1feGQzhREXc43NSCE3K5XkxAQOHaunsr7lf70ne0QyU8ZmMGXMCCaOTqe1rZ3aplZqm9qobWyhtqmVqoYWqhtCf3bMC3Rse+GkbBZOyub8WTmckZ+tz1cAJQXpwt1ZvbOUbz25gx1HajrXd0wwTs/NYHpuJjNyM5iek8mMvAxGpGh08WQ1t7bz42d385PVu8lKS+ZLl53GshljyclM/acjterGFg5V1HOwvJ6DFfUcCC/vL6/jjapGUpMSyEhNIjPiMSo9mZHp4T/TkpmSk8GiSdnkj9ZVwNI9JQXpUXVjC7tLajtPRdxVUsve0jqKjtV3noqYnpzIe8+axIfPn6Z7O5yA9nZn/b4K7nhsCzuP1nLFwgnc/o65J30mmLvrS14GhCaapUcj05I5c/JozuxSdbWjQuXe0lqe2lbCb9cf4Fcv7eey+eP56FtmqN5SD8prm3h+VxmrXy9hza4yKuqaGT8qjXtvXMLFc8b1a9tKCDLYdKQgPTpS1ch9a/fx4PqD1DS1snTaGD547lTeOnccScP8vPX65laefO0IDxUcYsP+CtxD4/lvmZ3LhbNzWTF3HJmp+s0lsUPDRzJgahpb+P2GQzzw0n6KjjUwYVQa1y+bwrVnTWbMMDrN1d3ZeKiSPxYU8fimw9Q2tTJl7AjetXAiF8/JY/7EUSRoUldilJKCDLi2ducfO0q4/8V9rN1dTkpiAmfkj2LBpGwWTMpmYX523JU7bm93CsMXfj255QhFxxpIT07ksvnjuWZJPkunjYmr/kr8UlKQQO06WsNDBYd49WAlW4qraGoNVczMyUzhnOljOX9mDufNzBmyk9SlNU3c8/xeHt90mDeqGklONM6fmcPb54/n7fNOIStN5/vL0BL1pGBmacAaIJXQhPbD7n5Hlza3AR8BWoFS4EPufuB421VSiD0tbaGLpzYdqqJgfwVr95RxtDp0B63JY0ZwyWl5XLNkEqeNj/176FY1tHD3mj3c+8J+mtvaWX5qHv9yxilcPGfcCV9UJhJLYiEpGJDh7rVmlgy8AHzK3ddFtFkOrHf3ejP7OHCRu7/3eNtVUoh9oZut17J2dznP7ypjzc5SmtvaOSN/FNcsmcQ7F0yIuS/YhuY27n9xPz97bg9VDS28c8EEbnvrbNWNkrgR9aTQJZgRhJLCx919fQ9tFgE/cvfzjrctJYWh51hdM49uLOahgkPsOFJDWnICN5w7lU8sn8nIKA/DtLc7j206zDef3MGR6kaWn5rL5952KqdP0Om3El9iIimYWSLwCjAT+LG7f+E4bX8EHHH3r3Xz2s3AzQCTJ09efODAcUeYJEa5O68VV3H/2v08WljM6BEpfGbFLK5bOjkqp7i+evAYqx7fRuGhSuZPHMX/fcdclk4bM+hxiAyGmEgKEcFkA48Ct7j7lm5evx74JHChuzcdb1s6UogPW4qr+Nqft7FubwUz8zL51CWzmDUuk7ysNEaPSA7sjJ6WtnZePXCMBzcc5E+Fh8nLSuWai+r2AAAJWklEQVTzK+dw1aKJOp1U4lpMJQUAM7sDqHP373RZvwL4IaGEUNLbdpQU4oe789S2o9z5l+3sL6/vXJ+caORmpjI2M5WR6UmMTAvV98nOSOZdCyee8IR1aU1TZxnpNbtKqWlsJTUpgZsumM7HL5pBhi4yk2Eg6mUuzCwXaHH3SjNLB1YA3+rSZhHwc2BlXxKCxBcz49LTT+GiU/PYVFQZKitd3URJTRMlNY0cq2umurGVo9W1VDe0cKy+mXvW7OXapZP57Ftn91hPyN3Z/kYNz2w/yjM7SthUVIk75GWlctm88Syfk8t5M3N0WqlIN4L8iTQeeCA8r5AAPOTuT5jZKqDA3R8Dvg1kAn8MDxccdPfLA4xJYlBKUgJnTe19LL+qvoXvPbOTX710gMc3HeZT4ftDNDS3se2Nara/Uc3Ww9Ws21tOceWbN5m/bcVsls/J4/QJI3WhmUgvdPGaDDm7S2r46hPbeW5nKZmpSdQ2tXa+lpuVyqJJ2VxyWh7L5+SRl6WbzItADAwfiQRlZl4WD3xoKc/uKOGvW44wLTeDueNHctr4keRmnVyJahEJUVKQIWv5nNDRgIgMnOFd/1hERP4XJQUREemkpCAiIp2UFEREpJOSgoiIdFJSEBGRTkoKIiLSSUlBREQ6DbkyF2ZWCkTeUGEUUNWlWV/WRT7vaTkHKOtnyD3Fc6Ltenqtt772td8D0deB6GdPr+szjU5f+9rP3tr2ta/6TI+/3J++TnH33F5bufuQfgB3n8y6yOfHWS4IKsYTbdfTa7319QT63e++DkQ/9ZnGVl/72s+B6qs+08Hp6/Ee8TB89PhJrnu8D8sDpa/bPF67nl7rra997fdAGIh+9vS6PtOen8fCZ9pb2772VZ/p8ZcDN+SGjwaTmRV4H6oKxoPh0tfh0k8YPn0dLv2EwelrPBwpBOnuaAcwiIZLX4dLP2H49HW49BMGoa86UhARkU46UhARkU7DJimY2b1mVmJmW07ivYvN7DUz221mP7CIezqa2S1m9rqZbTWz/xrYqE9cEP00sy+bWbGZFYYflw185CcuqM80/PrnzMzNLGfgIj45AX2mXzWzzeHP8+9mNmHgIz9xAfX122a2I9zfR80se+AjP+FYg+jne8LfQ+1mdvLzDkGf3hQrD+AtwJnAlpN47wZgGWDAk8Dbw+uXA08DqeHneXHazy8Dn4t23wajr+HXJgF/I3Q9TE489hMYGdHmVuBn0e5ngH29FEgKL38L+Fac9vM04FRgNbDkZGMbNkcK7r4GqIhcZ2YzzOyvZvaKmT1vZnO6vs/MxhP6D/SSh/7mfwW8K/zyx4FvuntTeB8lwfaidwH1MyYF2Ne7gM8DMTHhFkQ/3b06omkG8d3Xv7t7x4281wH5wfaidwH1c7u7v97f2IZNUujB3cAt7r4Y+Bzwk27aTASKIp4XhdcBzAYuMLP1ZvacmZ0VaLQnr7/9BPhk+PD7XjMbHVyo/davvprZ5UCxu28KOtB+6vdnamZfN7NDwPuA2wOMtb8G4t9vhw8R+nUdiwaynydt2N6j2cwygXOBP0YMJ3d313frZl3Hr6okYDRwDnAW8JCZTQ9n8JgwQP38KfDV8POvAt8l9J8rpvS3r2Y2AvgSoeGGmDVAnynu/iXgS2b2ReCTwB0DHGq/DVRfw9v6EtAK/HYgYxwIA9nP/hq2SYHQUVKluy+MXGlmicAr4aePEfpCjDzczAcOh5eLgP8JJ4ENZtZOqDZJaZCBn6B+99Pdj0a87x7giSAD7of+9nUGMA3YFP6PmQ+8amZL3f1IwLGfiIH4txvpQeDPxGBSYID6amY3AO8ALomlH20RBvozPXnRnnAZzAcwlYiJHeBF4D3hZQMW9PC+lwkdDXRM7FwWXv8xYFV4eTZwiPC1H3HWz/ERbT4D/D7afQyqr13a7CcGJpoD+kxnRbS5BXg42n0MsK8rgW1AbrT7FmQ/I15fTT8mmqP+FzOIH8DvgDeAFkK/8D9M6FfhX4FN4X80t/fw3iXAFmAP8KOOL34gBfhN+LVXgYvjtJ+/Bl4DNhP6tTJ+sPoz2H3t0iYmkkJAn+kj4fWbCdXWmRjtfgbY192EfrAVhh9RP9MqoH5eGd5WE3AU+NvJxKYrmkVEpNNwP/tIREQiKCmIiEgnJQUREemkpCAiIp2UFEREpJOSggx5ZlY7yPv7hZnNHaBttYUrlW4xs8d7q+BpZtlm9m8DsW+R7uiUVBnyzKzW3TMHcHtJ/mYBtUBFxm5mDwA73f3rx2k/FXjC3ecNRnwy/OhIQeKSmeWa2SNm9nL4cV54/VIze9HMNob/PDW8/kYz+6OZPQ783cwuMrPVZvZwuBb/byPq1q/uqFdvZrXhwnKbzGydmY0Lr58Rfv6yma3q49HMS7xZmC/TzJ4xs1ctVDv/inCbbwIzwkcX3w63/ffwfjab2VcG8K9RhiElBYlX3wfucvezgKuBX4TX7wDe4u6LCFUGvTPiPcuAG9z94vDzRcCngbnAdOC8bvaTAaxz9wXAGuCmiP1/P7z/XmvThGvcXELoinGARuBKdz+T0H07vhtOSv8B7HH3he7+72Z2KTALWAosBBab2Vt6259IT4ZzQTyJbyuAuREVJ0eaWRYwCnjAzGYRqi6ZHPGep9w9ssb9BncvAjCzQkK1al7osp9m3iwQ+Arw1vDyMt68R8ODwHd6iDM9YtuvAE+F1xtwZ/gLvp3QEcS4bt5/afixMfw8k1CSWNPD/kSOS0lB4lUCsMzdGyJXmtkPgWfd/crw+PzqiJfrumyjKWK5je7/v7T4mxNzPbU5ngZ3X2hmowgll08APyB0j4NcYLG7t5jZfiCtm/cb8A13//kJ7lekWxo+knj1d0L3CADAzDpKEo8CisPLNwa4/3WEhq0Aru2tsbtXEbot5ufMLJlQnCXhhLAcmBJuWgNkRbz1b8CHwvX4MbOJZpY3QH2QYUhJQeLBCDMrinjcRugLdkl48nUboTLnAP8FfMPM1gKJAcb0aeA2M9sAjAeqenuDu28kVCHzWkI3glliZgWEjhp2hNuUA2vDp7B+293/Tmh46iUzew14mP+dNEROiE5JFQlA+C5uDe7uZnYtcJ27X9Hb+0SiTXMKIsFYDPwofMZQJTF4+1KR7uhIQUREOmlOQUREOikpiIhIJyUFERHppKQgIiKdlBRERKSTkoKIiHT6/6ITitjGmIACAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"find_lr()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Although we can see that the loss is increasing as we go above a learning rate of, `1e-4` we will choose a learning rate of `1e-3` since that generally seems to work best when initially training a random set of weights at the head of a pretrained model."
]
},
{
"cell_type": "code",
"execution_count": 202,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>epoch</th>\n",
" <th>train_loss</th>\n",
" <th>valid_loss</th>\n",
" <th>time</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>0.073462</td>\n",
" <td>0.065989</td>\n",
" <td>04:13</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"learn.fit_one_cycle(1, 1e-3, moms=(0.8, 0.7))"
]
},
{
"cell_type": "code",
"execution_count": 203,
"metadata": {},
"outputs": [],
"source": [
"learn.save('first')"
]
},
{
"cell_type": "code",
"execution_count": 204,
"metadata": {
"collapsed": true
},
"outputs": [
{
"data": {
"text/plain": [
"RNNLearner(data=TextClasDataBunch;\n",
"\n",
"Train: LabelList (143614 items)\n",
"x: TextList\n",
"xxbos xxmaj explanation \n",
" xxmaj why the edits made under my username xxmaj hardcore xxmaj metallica xxmaj fan were reverted ? xxmaj they were n't vandalisms , just closure on some gas after i voted at xxmaj new xxmaj york xxmaj dolls xxup fac . xxmaj and please do n't remove the template from the talk page since i 'm retired xxunk,xxbos xxmaj xxunk ! xxmaj he matches this background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ),xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about the formatting than the actual info .,xxbos \" \n",
" xxmaj more \n",
" i ca n't make any real suggestions on improvement - i wondered if the section statistics should be later on , or a subsection of \" \" types of accidents \" \" xxup -i think the references may need tidying so that they are all in the exact same format ie date format etc . i can do that later on , if no - one else does first - if you have any preferences for formatting style on references or want to do it yourself please let me know . \n",
" \n",
" xxmaj there appears to be a backlog on articles for review so i guess there may be a delay until a reviewer turns up . xxmaj it 's listed in the relevant form eg xxmaj wikipedia : xxmaj xxunk # xxmaj transport \",xxbos xxmaj you , sir , are my hero . xxmaj any chance you remember what page that 's on ?\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Valid: LabelList (15957 items)\n",
"x: TextList\n",
"xxbos \" \n",
" \n",
" xxmaj sockpuppetry case \n",
" \n",
" xxmaj you have been accused of sockpuppetry . xxmaj please refer to xxmaj wikipedia : xxmaj sockpuppet investigations / xxmaj xxunk for evidence . xxmaj please make sure you make yourself familiar with notes for the suspect before editing the evidence page . 77 \",xxbos xxmaj welcome ! \n",
" \n",
" xxmaj hello , xxmaj xxunk , and welcome to xxmaj wikipedia ! xxmaj thank you for your contributions . i hope you like the place and decide to stay . xxmaj here are a few good links for newcomers : \n",
" xxmaj the five pillars of xxmaj wikipedia \n",
" xxmaj how to edit a page \n",
" xxmaj help pages \n",
" xxmaj tutorial \n",
" xxmaj how to write a great article \n",
" xxmaj manual of xxmaj style \n",
" i hope you enjoy editing here and being a xxmaj wikipedian ! xxmaj please sign your name on talk pages using four tildes ( xxrep 4 ~ ) ; this will automatically produce your name and the date . xxmaj if you need help , check out xxmaj wikipedia : xxmaj where to ask a question , ask me on my talk page , or place { { helpme } } on your talk page and someone will show up shortly to answer your questions . xxmaj again , welcome !,xxbos \" \n",
" \n",
" xxmaj agree with xxmaj user : xxmaj xxunk : \" \" this is not an article about racial or ethnic purity but a list of xxmaj americans that have roots in xxmaj estonia . \" \" \",xxbos xxmaj taking xxmaj xxunk as an example , in the xxmaj discography section the album information for each of their 3 albums is pretty much the same thing that can be found on each individual album 's page . xxmaj the albums are already linked to . xxmaj so what i am suggesting is removing what is duplicative and migrating anything that does not appear for each individual album onto that respective album 's article page . xxmaj in the end , you would be left with a xxmaj discography section listing all 3 albums . \n",
" \n",
" i also noticed the xxmaj notes on selected pieces section ; this section be taken out of the xxmaj xxunk article and appear in the pages for all 3 albums containing only those notes relevant for that particular album . xxmaj as it appears right now , the reader can not tell where a song appears unless he / she scrolls up and searches . xxmaj it would be more relevant to have the specific information pertaining to songs appearing on particular albums on the respective album 's pages .,xxbos xxmaj if someone wants to boil that plot down , go for it .\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Test: None, model=SequentialRNN(\n",
" (0): MultiBatchEncoder(\n",
" (module): AWD_LSTM(\n",
" (encoder): Embedding(60004, 400, padding_idx=1)\n",
" (encoder_dp): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (rnns): ModuleList(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (2): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" )\n",
" (input_dp): RNNDropout()\n",
" (hidden_dps): ModuleList(\n",
" (0): RNNDropout()\n",
" (1): RNNDropout()\n",
" (2): RNNDropout()\n",
" )\n",
" )\n",
" )\n",
" (1): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
"), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of BCEWithLogitsLoss(), metrics=[], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
"learn: RNNLearner(data=TextClasDataBunch;\n",
"\n",
"Train: LabelList (143614 items)\n",
"x: TextList\n",
"xxbos xxmaj explanation \n",
" xxmaj why the edits made under my username xxmaj hardcore xxmaj metallica xxmaj fan were reverted ? xxmaj they were n't vandalisms , just closure on some gas after i voted at xxmaj new xxmaj york xxmaj dolls xxup fac . xxmaj and please do n't remove the template from the talk page since i 'm retired xxunk,xxbos xxmaj xxunk ! xxmaj he matches this background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ),xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about the formatting than the actual info .,xxbos \" \n",
" xxmaj more \n",
" i ca n't make any real suggestions on improvement - i wondered if the section statistics should be later on , or a subsection of \" \" types of accidents \" \" xxup -i think the references may need tidying so that they are all in the exact same format ie date format etc . i can do that later on , if no - one else does first - if you have any preferences for formatting style on references or want to do it yourself please let me know . \n",
" \n",
" xxmaj there appears to be a backlog on articles for review so i guess there may be a delay until a reviewer turns up . xxmaj it 's listed in the relevant form eg xxmaj wikipedia : xxmaj xxunk # xxmaj transport \",xxbos xxmaj you , sir , are my hero . xxmaj any chance you remember what page that 's on ?\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Valid: LabelList (15957 items)\n",
"x: TextList\n",
"xxbos \" \n",
" \n",
" xxmaj sockpuppetry case \n",
" \n",
" xxmaj you have been accused of sockpuppetry . xxmaj please refer to xxmaj wikipedia : xxmaj sockpuppet investigations / xxmaj xxunk for evidence . xxmaj please make sure you make yourself familiar with notes for the suspect before editing the evidence page . 77 \",xxbos xxmaj welcome ! \n",
" \n",
" xxmaj hello , xxmaj xxunk , and welcome to xxmaj wikipedia ! xxmaj thank you for your contributions . i hope you like the place and decide to stay . xxmaj here are a few good links for newcomers : \n",
" xxmaj the five pillars of xxmaj wikipedia \n",
" xxmaj how to edit a page \n",
" xxmaj help pages \n",
" xxmaj tutorial \n",
" xxmaj how to write a great article \n",
" xxmaj manual of xxmaj style \n",
" i hope you enjoy editing here and being a xxmaj wikipedian ! xxmaj please sign your name on talk pages using four tildes ( xxrep 4 ~ ) ; this will automatically produce your name and the date . xxmaj if you need help , check out xxmaj wikipedia : xxmaj where to ask a question , ask me on my talk page , or place { { helpme } } on your talk page and someone will show up shortly to answer your questions . xxmaj again , welcome !,xxbos \" \n",
" \n",
" xxmaj agree with xxmaj user : xxmaj xxunk : \" \" this is not an article about racial or ethnic purity but a list of xxmaj americans that have roots in xxmaj estonia . \" \" \",xxbos xxmaj taking xxmaj xxunk as an example , in the xxmaj discography section the album information for each of their 3 albums is pretty much the same thing that can be found on each individual album 's page . xxmaj the albums are already linked to . xxmaj so what i am suggesting is removing what is duplicative and migrating anything that does not appear for each individual album onto that respective album 's article page . xxmaj in the end , you would be left with a xxmaj discography section listing all 3 albums . \n",
" \n",
" i also noticed the xxmaj notes on selected pieces section ; this section be taken out of the xxmaj xxunk article and appear in the pages for all 3 albums containing only those notes relevant for that particular album . xxmaj as it appears right now , the reader can not tell where a song appears unless he / she scrolls up and searches . xxmaj it would be more relevant to have the specific information pertaining to songs appearing on particular albums on the respective album 's pages .,xxbos xxmaj if someone wants to boil that plot down , go for it .\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Test: None, model=SequentialRNN(\n",
" (0): MultiBatchEncoder(\n",
" (module): AWD_LSTM(\n",
" (encoder): Embedding(60004, 400, padding_idx=1)\n",
" (encoder_dp): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (rnns): ModuleList(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (2): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" )\n",
" (input_dp): RNNDropout()\n",
" (hidden_dps): ModuleList(\n",
" (0): RNNDropout()\n",
" (1): RNNDropout()\n",
" (2): RNNDropout()\n",
" )\n",
" )\n",
" )\n",
" (1): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
"), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of BCEWithLogitsLoss(), metrics=[], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[...], layer_groups=[Sequential(\n",
" (0): Embedding(60004, 400, padding_idx=1)\n",
" (1): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
")], add_time=True, silent=None)\n",
"alpha: 2.0\n",
"beta: 1.0], layer_groups=[Sequential(\n",
" (0): Embedding(60004, 400, padding_idx=1)\n",
" (1): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
")], add_time=True, silent=None)"
]
},
"execution_count": 204,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn.load('first')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"*We will now unfreeze and fine tune the last 2 layers instead of just the last layer.*"
]
},
{
"cell_type": "code",
"execution_count": 205,
"metadata": {},
"outputs": [],
"source": [
"learn.freeze_to(-2)"
]
},
{
"cell_type": "code",
"execution_count": 206,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>epoch</th>\n",
" <th>train_loss</th>\n",
" <th>valid_loss</th>\n",
" <th>time</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>0.072802</td>\n",
" <td>0.061296</td>\n",
" <td>05:32</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"learn.fit_one_cycle(1, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7))"
]
},
{
"cell_type": "code",
"execution_count": 236,
"metadata": {},
"outputs": [],
"source": [
"learn.save('second')"
]
},
{
"cell_type": "code",
"execution_count": 237,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"RNNLearner(data=TextClasDataBunch;\n",
"\n",
"Train: LabelList (143614 items)\n",
"x: TextList\n",
"xxbos xxmaj explanation \n",
" xxmaj why the edits made under my username xxmaj hardcore xxmaj metallica xxmaj fan were reverted ? xxmaj they were n't vandalisms , just closure on some gas after i voted at xxmaj new xxmaj york xxmaj dolls xxup fac . xxmaj and please do n't remove the template from the talk page since i 'm retired xxunk,xxbos xxmaj xxunk ! xxmaj he matches this background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ),xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about the formatting than the actual info .,xxbos \" \n",
" xxmaj more \n",
" i ca n't make any real suggestions on improvement - i wondered if the section statistics should be later on , or a subsection of \" \" types of accidents \" \" xxup -i think the references may need tidying so that they are all in the exact same format ie date format etc . i can do that later on , if no - one else does first - if you have any preferences for formatting style on references or want to do it yourself please let me know . \n",
" \n",
" xxmaj there appears to be a backlog on articles for review so i guess there may be a delay until a reviewer turns up . xxmaj it 's listed in the relevant form eg xxmaj wikipedia : xxmaj xxunk # xxmaj transport \",xxbos xxmaj you , sir , are my hero . xxmaj any chance you remember what page that 's on ?\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Valid: LabelList (15957 items)\n",
"x: TextList\n",
"xxbos \" \n",
" \n",
" xxmaj sockpuppetry case \n",
" \n",
" xxmaj you have been accused of sockpuppetry . xxmaj please refer to xxmaj wikipedia : xxmaj sockpuppet investigations / xxmaj xxunk for evidence . xxmaj please make sure you make yourself familiar with notes for the suspect before editing the evidence page . 77 \",xxbos xxmaj welcome ! \n",
" \n",
" xxmaj hello , xxmaj xxunk , and welcome to xxmaj wikipedia ! xxmaj thank you for your contributions . i hope you like the place and decide to stay . xxmaj here are a few good links for newcomers : \n",
" xxmaj the five pillars of xxmaj wikipedia \n",
" xxmaj how to edit a page \n",
" xxmaj help pages \n",
" xxmaj tutorial \n",
" xxmaj how to write a great article \n",
" xxmaj manual of xxmaj style \n",
" i hope you enjoy editing here and being a xxmaj wikipedian ! xxmaj please sign your name on talk pages using four tildes ( xxrep 4 ~ ) ; this will automatically produce your name and the date . xxmaj if you need help , check out xxmaj wikipedia : xxmaj where to ask a question , ask me on my talk page , or place { { helpme } } on your talk page and someone will show up shortly to answer your questions . xxmaj again , welcome !,xxbos \" \n",
" \n",
" xxmaj agree with xxmaj user : xxmaj xxunk : \" \" this is not an article about racial or ethnic purity but a list of xxmaj americans that have roots in xxmaj estonia . \" \" \",xxbos xxmaj taking xxmaj xxunk as an example , in the xxmaj discography section the album information for each of their 3 albums is pretty much the same thing that can be found on each individual album 's page . xxmaj the albums are already linked to . xxmaj so what i am suggesting is removing what is duplicative and migrating anything that does not appear for each individual album onto that respective album 's article page . xxmaj in the end , you would be left with a xxmaj discography section listing all 3 albums . \n",
" \n",
" i also noticed the xxmaj notes on selected pieces section ; this section be taken out of the xxmaj xxunk article and appear in the pages for all 3 albums containing only those notes relevant for that particular album . xxmaj as it appears right now , the reader can not tell where a song appears unless he / she scrolls up and searches . xxmaj it would be more relevant to have the specific information pertaining to songs appearing on particular albums on the respective album 's pages .,xxbos xxmaj if someone wants to boil that plot down , go for it .\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Test: None, model=SequentialRNN(\n",
" (0): MultiBatchEncoder(\n",
" (module): AWD_LSTM(\n",
" (encoder): Embedding(60004, 400, padding_idx=1)\n",
" (encoder_dp): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (rnns): ModuleList(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (2): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" )\n",
" (input_dp): RNNDropout()\n",
" (hidden_dps): ModuleList(\n",
" (0): RNNDropout()\n",
" (1): RNNDropout()\n",
" (2): RNNDropout()\n",
" )\n",
" )\n",
" )\n",
" (1): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
"), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of BCEWithLogitsLoss(), metrics=[functools.partial(<function accuracy_thresh at 0x7faa93d3c268>, thresh=0.4)], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
"learn: RNNLearner(data=TextClasDataBunch;\n",
"\n",
"Train: LabelList (143614 items)\n",
"x: TextList\n",
"xxbos xxmaj explanation \n",
" xxmaj why the edits made under my username xxmaj hardcore xxmaj metallica xxmaj fan were reverted ? xxmaj they were n't vandalisms , just closure on some gas after i voted at xxmaj new xxmaj york xxmaj dolls xxup fac . xxmaj and please do n't remove the template from the talk page since i 'm retired xxunk,xxbos xxmaj xxunk ! xxmaj he matches this background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ),xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about the formatting than the actual info .,xxbos \" \n",
" xxmaj more \n",
" i ca n't make any real suggestions on improvement - i wondered if the section statistics should be later on , or a subsection of \" \" types of accidents \" \" xxup -i think the references may need tidying so that they are all in the exact same format ie date format etc . i can do that later on , if no - one else does first - if you have any preferences for formatting style on references or want to do it yourself please let me know . \n",
" \n",
" xxmaj there appears to be a backlog on articles for review so i guess there may be a delay until a reviewer turns up . xxmaj it 's listed in the relevant form eg xxmaj wikipedia : xxmaj xxunk # xxmaj transport \",xxbos xxmaj you , sir , are my hero . xxmaj any chance you remember what page that 's on ?\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Valid: LabelList (15957 items)\n",
"x: TextList\n",
"xxbos \" \n",
" \n",
" xxmaj sockpuppetry case \n",
" \n",
" xxmaj you have been accused of sockpuppetry . xxmaj please refer to xxmaj wikipedia : xxmaj sockpuppet investigations / xxmaj xxunk for evidence . xxmaj please make sure you make yourself familiar with notes for the suspect before editing the evidence page . 77 \",xxbos xxmaj welcome ! \n",
" \n",
" xxmaj hello , xxmaj xxunk , and welcome to xxmaj wikipedia ! xxmaj thank you for your contributions . i hope you like the place and decide to stay . xxmaj here are a few good links for newcomers : \n",
" xxmaj the five pillars of xxmaj wikipedia \n",
" xxmaj how to edit a page \n",
" xxmaj help pages \n",
" xxmaj tutorial \n",
" xxmaj how to write a great article \n",
" xxmaj manual of xxmaj style \n",
" i hope you enjoy editing here and being a xxmaj wikipedian ! xxmaj please sign your name on talk pages using four tildes ( xxrep 4 ~ ) ; this will automatically produce your name and the date . xxmaj if you need help , check out xxmaj wikipedia : xxmaj where to ask a question , ask me on my talk page , or place { { helpme } } on your talk page and someone will show up shortly to answer your questions . xxmaj again , welcome !,xxbos \" \n",
" \n",
" xxmaj agree with xxmaj user : xxmaj xxunk : \" \" this is not an article about racial or ethnic purity but a list of xxmaj americans that have roots in xxmaj estonia . \" \" \",xxbos xxmaj taking xxmaj xxunk as an example , in the xxmaj discography section the album information for each of their 3 albums is pretty much the same thing that can be found on each individual album 's page . xxmaj the albums are already linked to . xxmaj so what i am suggesting is removing what is duplicative and migrating anything that does not appear for each individual album onto that respective album 's article page . xxmaj in the end , you would be left with a xxmaj discography section listing all 3 albums . \n",
" \n",
" i also noticed the xxmaj notes on selected pieces section ; this section be taken out of the xxmaj xxunk article and appear in the pages for all 3 albums containing only those notes relevant for that particular album . xxmaj as it appears right now , the reader can not tell where a song appears unless he / she scrolls up and searches . xxmaj it would be more relevant to have the specific information pertaining to songs appearing on particular albums on the respective album 's pages .,xxbos xxmaj if someone wants to boil that plot down , go for it .\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Test: None, model=SequentialRNN(\n",
" (0): MultiBatchEncoder(\n",
" (module): AWD_LSTM(\n",
" (encoder): Embedding(60004, 400, padding_idx=1)\n",
" (encoder_dp): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (rnns): ModuleList(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (2): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" )\n",
" (input_dp): RNNDropout()\n",
" (hidden_dps): ModuleList(\n",
" (0): RNNDropout()\n",
" (1): RNNDropout()\n",
" (2): RNNDropout()\n",
" )\n",
" )\n",
" )\n",
" (1): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
"), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of BCEWithLogitsLoss(), metrics=[functools.partial(<function accuracy_thresh at 0x7faa93d3c268>, thresh=0.4)], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[...], layer_groups=[Sequential(\n",
" (0): Embedding(60004, 400, padding_idx=1)\n",
" (1): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
")], add_time=True, silent=None)\n",
"alpha: 2.0\n",
"beta: 1.0], layer_groups=[Sequential(\n",
" (0): Embedding(60004, 400, padding_idx=1)\n",
" (1): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
")], add_time=True, silent=None)"
]
},
"execution_count": 237,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn.load('second')"
]
},
{
"cell_type": "code",
"execution_count": 233,
"metadata": {},
"outputs": [],
"source": [
"learn.validate()"
]
},
{
"cell_type": "code",
"execution_count": 238,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>epoch</th>\n",
" <th>train_loss</th>\n",
" <th>valid_loss</th>\n",
" <th>accuracy_thresh</th>\n",
" <th>time</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>0.053653</td>\n",
" <td>0.054283</td>\n",
" <td>0.978756</td>\n",
" <td>09:50</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"learn.freeze_to(-3)\n",
"learn.fit_one_cycle(1, slice(5e-4/(2.6**4),5e-4), moms=(0.8,0.7))"
]
},
{
"cell_type": "code",
"execution_count": 239,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"RNNLearner(data=TextClasDataBunch;\n",
"\n",
"Train: LabelList (143614 items)\n",
"x: TextList\n",
"xxbos xxmaj explanation \n",
" xxmaj why the edits made under my username xxmaj hardcore xxmaj metallica xxmaj fan were reverted ? xxmaj they were n't vandalisms , just closure on some gas after i voted at xxmaj new xxmaj york xxmaj dolls xxup fac . xxmaj and please do n't remove the template from the talk page since i 'm retired xxunk,xxbos xxmaj xxunk ! xxmaj he matches this background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ),xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about the formatting than the actual info .,xxbos \" \n",
" xxmaj more \n",
" i ca n't make any real suggestions on improvement - i wondered if the section statistics should be later on , or a subsection of \" \" types of accidents \" \" xxup -i think the references may need tidying so that they are all in the exact same format ie date format etc . i can do that later on , if no - one else does first - if you have any preferences for formatting style on references or want to do it yourself please let me know . \n",
" \n",
" xxmaj there appears to be a backlog on articles for review so i guess there may be a delay until a reviewer turns up . xxmaj it 's listed in the relevant form eg xxmaj wikipedia : xxmaj xxunk # xxmaj transport \",xxbos xxmaj you , sir , are my hero . xxmaj any chance you remember what page that 's on ?\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Valid: LabelList (15957 items)\n",
"x: TextList\n",
"xxbos \" \n",
" \n",
" xxmaj sockpuppetry case \n",
" \n",
" xxmaj you have been accused of sockpuppetry . xxmaj please refer to xxmaj wikipedia : xxmaj sockpuppet investigations / xxmaj xxunk for evidence . xxmaj please make sure you make yourself familiar with notes for the suspect before editing the evidence page . 77 \",xxbos xxmaj welcome ! \n",
" \n",
" xxmaj hello , xxmaj xxunk , and welcome to xxmaj wikipedia ! xxmaj thank you for your contributions . i hope you like the place and decide to stay . xxmaj here are a few good links for newcomers : \n",
" xxmaj the five pillars of xxmaj wikipedia \n",
" xxmaj how to edit a page \n",
" xxmaj help pages \n",
" xxmaj tutorial \n",
" xxmaj how to write a great article \n",
" xxmaj manual of xxmaj style \n",
" i hope you enjoy editing here and being a xxmaj wikipedian ! xxmaj please sign your name on talk pages using four tildes ( xxrep 4 ~ ) ; this will automatically produce your name and the date . xxmaj if you need help , check out xxmaj wikipedia : xxmaj where to ask a question , ask me on my talk page , or place { { helpme } } on your talk page and someone will show up shortly to answer your questions . xxmaj again , welcome !,xxbos \" \n",
" \n",
" xxmaj agree with xxmaj user : xxmaj xxunk : \" \" this is not an article about racial or ethnic purity but a list of xxmaj americans that have roots in xxmaj estonia . \" \" \",xxbos xxmaj taking xxmaj xxunk as an example , in the xxmaj discography section the album information for each of their 3 albums is pretty much the same thing that can be found on each individual album 's page . xxmaj the albums are already linked to . xxmaj so what i am suggesting is removing what is duplicative and migrating anything that does not appear for each individual album onto that respective album 's article page . xxmaj in the end , you would be left with a xxmaj discography section listing all 3 albums . \n",
" \n",
" i also noticed the xxmaj notes on selected pieces section ; this section be taken out of the xxmaj xxunk article and appear in the pages for all 3 albums containing only those notes relevant for that particular album . xxmaj as it appears right now , the reader can not tell where a song appears unless he / she scrolls up and searches . xxmaj it would be more relevant to have the specific information pertaining to songs appearing on particular albums on the respective album 's pages .,xxbos xxmaj if someone wants to boil that plot down , go for it .\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Test: None, model=SequentialRNN(\n",
" (0): MultiBatchEncoder(\n",
" (module): AWD_LSTM(\n",
" (encoder): Embedding(60004, 400, padding_idx=1)\n",
" (encoder_dp): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (rnns): ModuleList(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (2): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" )\n",
" (input_dp): RNNDropout()\n",
" (hidden_dps): ModuleList(\n",
" (0): RNNDropout()\n",
" (1): RNNDropout()\n",
" (2): RNNDropout()\n",
" )\n",
" )\n",
" )\n",
" (1): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
"), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of BCEWithLogitsLoss(), metrics=[functools.partial(<function accuracy_thresh at 0x7faa93d3c268>, thresh=0.4)], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
"learn: RNNLearner(data=TextClasDataBunch;\n",
"\n",
"Train: LabelList (143614 items)\n",
"x: TextList\n",
"xxbos xxmaj explanation \n",
" xxmaj why the edits made under my username xxmaj hardcore xxmaj metallica xxmaj fan were reverted ? xxmaj they were n't vandalisms , just closure on some gas after i voted at xxmaj new xxmaj york xxmaj dolls xxup fac . xxmaj and please do n't remove the template from the talk page since i 'm retired xxunk,xxbos xxmaj xxunk ! xxmaj he matches this background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ),xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about the formatting than the actual info .,xxbos \" \n",
" xxmaj more \n",
" i ca n't make any real suggestions on improvement - i wondered if the section statistics should be later on , or a subsection of \" \" types of accidents \" \" xxup -i think the references may need tidying so that they are all in the exact same format ie date format etc . i can do that later on , if no - one else does first - if you have any preferences for formatting style on references or want to do it yourself please let me know . \n",
" \n",
" xxmaj there appears to be a backlog on articles for review so i guess there may be a delay until a reviewer turns up . xxmaj it 's listed in the relevant form eg xxmaj wikipedia : xxmaj xxunk # xxmaj transport \",xxbos xxmaj you , sir , are my hero . xxmaj any chance you remember what page that 's on ?\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Valid: LabelList (15957 items)\n",
"x: TextList\n",
"xxbos \" \n",
" \n",
" xxmaj sockpuppetry case \n",
" \n",
" xxmaj you have been accused of sockpuppetry . xxmaj please refer to xxmaj wikipedia : xxmaj sockpuppet investigations / xxmaj xxunk for evidence . xxmaj please make sure you make yourself familiar with notes for the suspect before editing the evidence page . 77 \",xxbos xxmaj welcome ! \n",
" \n",
" xxmaj hello , xxmaj xxunk , and welcome to xxmaj wikipedia ! xxmaj thank you for your contributions . i hope you like the place and decide to stay . xxmaj here are a few good links for newcomers : \n",
" xxmaj the five pillars of xxmaj wikipedia \n",
" xxmaj how to edit a page \n",
" xxmaj help pages \n",
" xxmaj tutorial \n",
" xxmaj how to write a great article \n",
" xxmaj manual of xxmaj style \n",
" i hope you enjoy editing here and being a xxmaj wikipedian ! xxmaj please sign your name on talk pages using four tildes ( xxrep 4 ~ ) ; this will automatically produce your name and the date . xxmaj if you need help , check out xxmaj wikipedia : xxmaj where to ask a question , ask me on my talk page , or place { { helpme } } on your talk page and someone will show up shortly to answer your questions . xxmaj again , welcome !,xxbos \" \n",
" \n",
" xxmaj agree with xxmaj user : xxmaj xxunk : \" \" this is not an article about racial or ethnic purity but a list of xxmaj americans that have roots in xxmaj estonia . \" \" \",xxbos xxmaj taking xxmaj xxunk as an example , in the xxmaj discography section the album information for each of their 3 albums is pretty much the same thing that can be found on each individual album 's page . xxmaj the albums are already linked to . xxmaj so what i am suggesting is removing what is duplicative and migrating anything that does not appear for each individual album onto that respective album 's article page . xxmaj in the end , you would be left with a xxmaj discography section listing all 3 albums . \n",
" \n",
" i also noticed the xxmaj notes on selected pieces section ; this section be taken out of the xxmaj xxunk article and appear in the pages for all 3 albums containing only those notes relevant for that particular album . xxmaj as it appears right now , the reader can not tell where a song appears unless he / she scrolls up and searches . xxmaj it would be more relevant to have the specific information pertaining to songs appearing on particular albums on the respective album 's pages .,xxbos xxmaj if someone wants to boil that plot down , go for it .\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Test: None, model=SequentialRNN(\n",
" (0): MultiBatchEncoder(\n",
" (module): AWD_LSTM(\n",
" (encoder): Embedding(60004, 400, padding_idx=1)\n",
" (encoder_dp): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (rnns): ModuleList(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (2): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" )\n",
" (input_dp): RNNDropout()\n",
" (hidden_dps): ModuleList(\n",
" (0): RNNDropout()\n",
" (1): RNNDropout()\n",
" (2): RNNDropout()\n",
" )\n",
" )\n",
" )\n",
" (1): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
"), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of BCEWithLogitsLoss(), metrics=[functools.partial(<function accuracy_thresh at 0x7faa93d3c268>, thresh=0.4)], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[...], layer_groups=[Sequential(\n",
" (0): Embedding(60004, 400, padding_idx=1)\n",
" (1): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
")], add_time=True, silent=None)\n",
"alpha: 2.0\n",
"beta: 1.0], layer_groups=[Sequential(\n",
" (0): Embedding(60004, 400, padding_idx=1)\n",
" (1): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
")], add_time=True, silent=None)"
]
},
"execution_count": 239,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn.save('third')\n",
"learn.load('third')"
]
},
{
"cell_type": "code",
"execution_count": 240,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(MultiCategory toxic;severe_toxic;obscene;insult,\n",
" tensor([1., 1., 1., 0., 1., 0.]),\n",
" tensor([1.0000, 0.9644, 0.9999, 0.2910, 0.9989, 0.2105]))"
]
},
"execution_count": 240,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn.predict(\"I will kill you. You are fat. Fuck off\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will now unfreeze the model and train it for a few epochs"
]
},
{
"cell_type": "code",
"execution_count": 241,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>epoch</th>\n",
" <th>train_loss</th>\n",
" <th>valid_loss</th>\n",
" <th>accuracy_thresh</th>\n",
" <th>time</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>0.055374</td>\n",
" <td>0.049385</td>\n",
" <td>0.980823</td>\n",
" <td>13:01</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>0.050902</td>\n",
" <td>0.047490</td>\n",
" <td>0.981314</td>\n",
" <td>11:02</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>0.050022</td>\n",
" <td>0.046927</td>\n",
" <td>0.981711</td>\n",
" <td>10:57</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"learn.unfreeze()\n",
"learn.fit_one_cycle(3, slice(1e-3/(2.6**4),1e-4), moms=(0.8,0.7))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"learn.save('unfrozen_four')"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"RNNLearner(data=TextClasDataBunch;\n",
"\n",
"Train: LabelList (143614 items)\n",
"x: TextList\n",
"xxbos xxmaj explanation \n",
" xxmaj why the edits made under my username xxmaj hardcore xxmaj metallica xxmaj fan were reverted ? xxmaj they were n't vandalisms , just closure on some gas after i voted at xxmaj new xxmaj york xxmaj dolls xxup fac . xxmaj and please do n't remove the template from the talk page since i 'm retired xxunk,xxbos xxmaj xxunk ! xxmaj he matches this background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ),xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about the formatting than the actual info .,xxbos \" \n",
" xxmaj more \n",
" i ca n't make any real suggestions on improvement - i wondered if the section statistics should be later on , or a subsection of \" \" types of accidents \" \" xxup -i think the references may need tidying so that they are all in the exact same format ie date format etc . i can do that later on , if no - one else does first - if you have any preferences for formatting style on references or want to do it yourself please let me know . \n",
" \n",
" xxmaj there appears to be a backlog on articles for review so i guess there may be a delay until a reviewer turns up . xxmaj it 's listed in the relevant form eg xxmaj wikipedia : xxmaj xxunk # xxmaj transport \",xxbos xxmaj you , sir , are my hero . xxmaj any chance you remember what page that 's on ?\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Valid: LabelList (15957 items)\n",
"x: TextList\n",
"xxbos \" \n",
" \n",
" xxmaj sockpuppetry case \n",
" \n",
" xxmaj you have been accused of sockpuppetry . xxmaj please refer to xxmaj wikipedia : xxmaj sockpuppet investigations / xxmaj xxunk for evidence . xxmaj please make sure you make yourself familiar with notes for the suspect before editing the evidence page . 77 \",xxbos xxmaj welcome ! \n",
" \n",
" xxmaj hello , xxmaj xxunk , and welcome to xxmaj wikipedia ! xxmaj thank you for your contributions . i hope you like the place and decide to stay . xxmaj here are a few good links for newcomers : \n",
" xxmaj the five pillars of xxmaj wikipedia \n",
" xxmaj how to edit a page \n",
" xxmaj help pages \n",
" xxmaj tutorial \n",
" xxmaj how to write a great article \n",
" xxmaj manual of xxmaj style \n",
" i hope you enjoy editing here and being a xxmaj wikipedian ! xxmaj please sign your name on talk pages using four tildes ( xxrep 4 ~ ) ; this will automatically produce your name and the date . xxmaj if you need help , check out xxmaj wikipedia : xxmaj where to ask a question , ask me on my talk page , or place { { helpme } } on your talk page and someone will show up shortly to answer your questions . xxmaj again , welcome !,xxbos \" \n",
" \n",
" xxmaj agree with xxmaj user : xxmaj xxunk : \" \" this is not an article about racial or ethnic purity but a list of xxmaj americans that have roots in xxmaj estonia . \" \" \",xxbos xxmaj taking xxmaj xxunk as an example , in the xxmaj discography section the album information for each of their 3 albums is pretty much the same thing that can be found on each individual album 's page . xxmaj the albums are already linked to . xxmaj so what i am suggesting is removing what is duplicative and migrating anything that does not appear for each individual album onto that respective album 's article page . xxmaj in the end , you would be left with a xxmaj discography section listing all 3 albums . \n",
" \n",
" i also noticed the xxmaj notes on selected pieces section ; this section be taken out of the xxmaj xxunk article and appear in the pages for all 3 albums containing only those notes relevant for that particular album . xxmaj as it appears right now , the reader can not tell where a song appears unless he / she scrolls up and searches . xxmaj it would be more relevant to have the specific information pertaining to songs appearing on particular albums on the respective album 's pages .,xxbos xxmaj if someone wants to boil that plot down , go for it .\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Test: None, model=SequentialRNN(\n",
" (0): MultiBatchEncoder(\n",
" (module): AWD_LSTM(\n",
" (encoder): Embedding(60004, 400, padding_idx=1)\n",
" (encoder_dp): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (rnns): ModuleList(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (2): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" )\n",
" (input_dp): RNNDropout()\n",
" (hidden_dps): ModuleList(\n",
" (0): RNNDropout()\n",
" (1): RNNDropout()\n",
" (2): RNNDropout()\n",
" )\n",
" )\n",
" )\n",
" (1): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
"), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of BCEWithLogitsLoss(), metrics=[], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
"learn: RNNLearner(data=TextClasDataBunch;\n",
"\n",
"Train: LabelList (143614 items)\n",
"x: TextList\n",
"xxbos xxmaj explanation \n",
" xxmaj why the edits made under my username xxmaj hardcore xxmaj metallica xxmaj fan were reverted ? xxmaj they were n't vandalisms , just closure on some gas after i voted at xxmaj new xxmaj york xxmaj dolls xxup fac . xxmaj and please do n't remove the template from the talk page since i 'm retired xxunk,xxbos xxmaj xxunk ! xxmaj he matches this background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ),xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about the formatting than the actual info .,xxbos \" \n",
" xxmaj more \n",
" i ca n't make any real suggestions on improvement - i wondered if the section statistics should be later on , or a subsection of \" \" types of accidents \" \" xxup -i think the references may need tidying so that they are all in the exact same format ie date format etc . i can do that later on , if no - one else does first - if you have any preferences for formatting style on references or want to do it yourself please let me know . \n",
" \n",
" xxmaj there appears to be a backlog on articles for review so i guess there may be a delay until a reviewer turns up . xxmaj it 's listed in the relevant form eg xxmaj wikipedia : xxmaj xxunk # xxmaj transport \",xxbos xxmaj you , sir , are my hero . xxmaj any chance you remember what page that 's on ?\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Valid: LabelList (15957 items)\n",
"x: TextList\n",
"xxbos \" \n",
" \n",
" xxmaj sockpuppetry case \n",
" \n",
" xxmaj you have been accused of sockpuppetry . xxmaj please refer to xxmaj wikipedia : xxmaj sockpuppet investigations / xxmaj xxunk for evidence . xxmaj please make sure you make yourself familiar with notes for the suspect before editing the evidence page . 77 \",xxbos xxmaj welcome ! \n",
" \n",
" xxmaj hello , xxmaj xxunk , and welcome to xxmaj wikipedia ! xxmaj thank you for your contributions . i hope you like the place and decide to stay . xxmaj here are a few good links for newcomers : \n",
" xxmaj the five pillars of xxmaj wikipedia \n",
" xxmaj how to edit a page \n",
" xxmaj help pages \n",
" xxmaj tutorial \n",
" xxmaj how to write a great article \n",
" xxmaj manual of xxmaj style \n",
" i hope you enjoy editing here and being a xxmaj wikipedian ! xxmaj please sign your name on talk pages using four tildes ( xxrep 4 ~ ) ; this will automatically produce your name and the date . xxmaj if you need help , check out xxmaj wikipedia : xxmaj where to ask a question , ask me on my talk page , or place { { helpme } } on your talk page and someone will show up shortly to answer your questions . xxmaj again , welcome !,xxbos \" \n",
" \n",
" xxmaj agree with xxmaj user : xxmaj xxunk : \" \" this is not an article about racial or ethnic purity but a list of xxmaj americans that have roots in xxmaj estonia . \" \" \",xxbos xxmaj taking xxmaj xxunk as an example , in the xxmaj discography section the album information for each of their 3 albums is pretty much the same thing that can be found on each individual album 's page . xxmaj the albums are already linked to . xxmaj so what i am suggesting is removing what is duplicative and migrating anything that does not appear for each individual album onto that respective album 's article page . xxmaj in the end , you would be left with a xxmaj discography section listing all 3 albums . \n",
" \n",
" i also noticed the xxmaj notes on selected pieces section ; this section be taken out of the xxmaj xxunk article and appear in the pages for all 3 albums containing only those notes relevant for that particular album . xxmaj as it appears right now , the reader can not tell where a song appears unless he / she scrolls up and searches . xxmaj it would be more relevant to have the specific information pertaining to songs appearing on particular albums on the respective album 's pages .,xxbos xxmaj if someone wants to boil that plot down , go for it .\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Test: None, model=SequentialRNN(\n",
" (0): MultiBatchEncoder(\n",
" (module): AWD_LSTM(\n",
" (encoder): Embedding(60004, 400, padding_idx=1)\n",
" (encoder_dp): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (rnns): ModuleList(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (2): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" )\n",
" (input_dp): RNNDropout()\n",
" (hidden_dps): ModuleList(\n",
" (0): RNNDropout()\n",
" (1): RNNDropout()\n",
" (2): RNNDropout()\n",
" )\n",
" )\n",
" )\n",
" (1): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
"), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of BCEWithLogitsLoss(), metrics=[], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[...], layer_groups=[Sequential(\n",
" (0): Embedding(60004, 400, padding_idx=1)\n",
" (1): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
")], add_time=True, silent=None)\n",
"alpha: 2.0\n",
"beta: 1.0], layer_groups=[Sequential(\n",
" (0): Embedding(60004, 400, padding_idx=1)\n",
" (1): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
")], add_time=True, silent=None)"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn.load('unfrozen_four')"
]
},
{
"cell_type": "code",
"execution_count": 243,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(MultiCategory toxic;severe_toxic;obscene;threat;insult,\n",
" tensor([1., 1., 1., 1., 1., 0.]),\n",
" tensor([0.9999, 0.9392, 0.9997, 0.6798, 0.9904, 0.1193]))"
]
},
"execution_count": 243,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn.predict(\"I will kill you. You are fat. Fuck off\")"
]
},
{
"cell_type": "code",
"execution_count": 254,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(MultiCategory toxic;obscene;threat,\n",
" tensor([1., 0., 1., 1., 0., 0.]),\n",
" tensor([0.8747, 0.2716, 0.6994, 0.6640, 0.3426, 0.0494]))"
]
},
"execution_count": 254,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn.predict(\"I am gonna kick you\")"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.9851895132990093"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"roc_score()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Training a bit more"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xl8VNXd+PHPNzsJJJCNLUACBBCULQHZ3KBad7Rgq7WtWx9r6/K0drOPT/21tk9bu2gXra1VW9S6glas1l1RRPZVlCUJW9iyAdmXSb6/P+ZOGMJkn0uSyff9es2LmXPPvXMOA/Odc84954iqYowxxgRbWFcXwBhjTGiyAGOMMcYVFmCMMca4wgKMMcYYV1iAMcYY4woLMMYYY1xhAcYYY4wrLMAYY4xxhQUYY4wxrojo6gJ0peTkZE1PT+/qYhhjTI+ybt26IlVNaS1frw4w6enprF27tquLYYwxPYqI7GlLPle7yETkQhHZLiI5InJXgOPRIvKcc3yViKT7HZsoIh+LyFYR2SIiMSISKyKvisg2J/1XfvmvF5FCEdnoPL7uZt2MMca0zLUAIyLhwEPARcB44BoRGd8k203AEVUdDTwA3OecGwE8BdyiqhOAc4E655zfquo4YAowW0Qu8rvec6o62Xk86lLVjDHGtIGbLZjpQI6q5qlqLfAsML9JnvnAIuf5YmCeiAhwAbBZVTcBqGqxqtaraqWqvuek1QLrgTQX62CMMaaD3AwwQ4F9fq/znbSAeVTVAxwDkoAxgIrIGyKyXkR+0PTiItIfuAx4xy95gYhsFpHFIjIseFUxxhjTXm4GGAmQ1nTzmebyRABzgGudP68UkXmNJ3m70J4B/qiqeU7yK0C6qk4E3uZ4y+jENxS5WUTWisjawsLC9tTHGGNMO7gZYPIB/1ZEGnCguTxO0EgASpz0ZapapKqVwGvAVL/zHgF2qurvfQlON1qN8/JvQFagQqnqI6qararZKSmt3mVnjDGmg9wMMGuATBHJEJEo4GpgaZM8S4HrnOcLgXfVu8XmG8BE566xCOAc4FMAEfk53kD0bf8Lichgv5eXA58FuT7GGGPawbV5MKrqEZHb8AaLcOBxVd0qIvcCa1V1KfAY8KSI5OBtuVztnHtERO7HG6QUeE1VXxWRNOBuYBuw3ns/AA86d4zdISKXAx7nWte7VbfuSlWprK0nLtqdj3Xn4TIOl9YwJzPZlesbY0KLeBsMvVN2draG0kTL17Yc5HsvbOLtO89hSP8+Qb/+bU+v5/3thWy453wiw22VIWN6KxFZp6rZreWzb4kQsq+kksrael7asN+V6x8uraa8xsPGfUddub4xJrRYgAkhFbX1ACxZl48bLdOCMu89FB/uLAr6tY0xoccCTAipqPEAkFdUwfq9wW1lqCoFpd4As3yn3d5tjGmdBZgQUlnroV9MBH0iw1m8Lj+o1y6v8VBVV098TAQb9x3lWFVd6ycZY3o1CzAhpKKmnqS4KC46fRD/3nyA6rr6oF37sNN6uWzSEBoUPs4tDtq1jTGhyQJMCKms9RAXHcGCrDTKqj28+enhoF27oKwagAsmDCIuKpwPrZvMGNMKCzAhpLzGQ1xUBDNHJjEkIYYlQewmK3QG+If2j2HmqCSW59hAvzGmZRZgQkhlbT2x0eGEhQlfmJrGhzsLOXSsOijX9g3wp/SLYc7oZPYUV7K3uDIo1zbGhCYLMCGkwmnBACzISqNBCdqcmIKyaqIjwoiPieCsMd413D7MsW4yY0zzLMCEkIqaeuKiwwHISI4je8QAlqwPzpyYgrIaUuOjERFGJscxJCGG5TYfxhjTAgswIaSi1kNs1PF1yBZkpZFTUM6m/GOdvnZBaQ2p/WIAEBHmZCbzUU4R9Q29d6khY0zLLMCEiOMLXYY3pl0ycTDREWEsXrevhTPbpqCsmtR+0Y2vz8pMobTaw+Z8WzbGGBOYBZgQUeNpoL5BT1hJOT4mkgtPH8TSjZ2fE1NQVnNCgJk9OhkRrJvMGNMsCzAhwrdMTFzUiUv1X5U1jNJqD291Yk5MdV09ZdUeUuNjGtMS46KYMCTe1iUzxjTLAkyIqHQWuoyNCj8hfdaoJIb278PzazveTea7Rdm/BQPebrL1e49Q7gQ3Y4zxZwEmRFTUOi2YJpuNhYUJC7LSWJ5TxIGjVR26tm8Wv38LBuCs0cl4GpSVtmyMMSYACzAhoqLG24IJtJvlVVlpqNLhmf2+ZfqbtmCy0gcQExnG258Fb0kaY0zosAATIo6PwYSfdGxYYiwzRibywrp8GjpwW3FBqdOCaRJgoiPCuXLKUJasz7dZ/caYk1iACRGVThdZbNTJLRiAL2YPY29JJat3l7T72gVlNUSECQNio0469t/zxhAmwv1vbW/3dY0xoc0CTIg43kV2cgsG4KLTB9M3OoIX1ra/m+xwaQ0p/aIJC5OTjg1KiOHGORm8vOkAWw90fkKnMSZ0WIAJEZXNDPL79IkK57JJg3lty8F23/XVdJJlU7ecM4r4mEh+/bq1Yowxx7kaYETkQhHZLiI5InJXgOPRIvKcc3yViKT7HZsoIh+LyFYR2SIiMU56lvM6R0T+KCLipCeKyFsistP5c4Cbdetuyn0tmGa6yAAWZg2jqq6eVzcfaNe1C8tqSOkX0+zxhD6RfOvcUSzbUWgbkRljGrkWYEQkHHgIuAgYD1wjIuObZLsJOKKqo4EHgPuccyOAp4BbVHUCcC7g26P3YeBmINN5XOik3wW8o6qZwDvO616jstZDmEBMZPMf6dTh/RmVEtfubjLfQpctuW5WOoMTYvjV69uCsrimMabnc7MFMx3IUdU8Va0FngXmN8kzH1jkPF8MzHNaJBcAm1V1E4CqFqtqvYgMBuJV9WP1fos9AVwR4FqL/NJ7hYqaeuKiInAadAGJCFdlD2PtniPkFpa36bq1ngZKKmpb7CIDiIkM5zufG8OmfUd5/ZND7Sq7MSY0uRlghgL+08fznbSAeVTVAxwDkoAxgIrIGyKyXkR+4Jff/+e3/zUHqupB51oHgdQg1qXbq6jxENvMAL+/L0wZSniYsLiNc2KKyn1zYJrvImu89tShjE7ty2/e3I6nvqFN1zfGhC43A0ygn9JN+06ayxMBzAGudf68UkTmtfGaLRdK5GYRWSsiawsLQ2fDrIpaT4vjLz6p8TGcOyaFF9fnt2mp/eYmWQYSER7GDz4/lrzCiqBtdGaM6bncDDD5wDC/12lA09HlxjzOuEsCUOKkL1PVIlWtBF4Dpjrpac1c87DThYbzZ0GgQqnqI6qararZKSkpnahe9+LbLrktFmalcbi0hg93th5gGydZtjIG43P++IGMG9SPx5bvsrEYY3o5NwPMGiBTRDJEJAq4GljaJM9S4Drn+ULgXWds5Q1goojEOoHnHOBTp+urTERmOGM1XwNeDnCt6/zSewX/7ZJbM++0gQyIjWxTN9nxFkzrXWTgHee5cU4G2w6V8VGO3VFmTG/mWoBxxlRuwxssPgOeV9WtInKviFzuZHsMSBKRHOBOnDu/VPUIcD/eILURWK+qrzrnfBN4FMgBcoH/OOm/As4XkZ3A+c7rXqOi1tPsHJimoiLCmD95KG9+ephjlXUt5i0oq0EEkvuePIu/OZdPGkJy3ygeW57X5nOMMaGnbd9IHaSqr+Ht3vJPu8fveTVwVTPnPoX3VuWm6WuB0wOkFwPzOlnkHquypp7YpLZ1kYG3m+wfK3bzyuYDfGXGiGbzFZZVkxQXTUR423+LxESG85UZI/j92zvJKShndGrfNp9rjAkdNpM/RLR1kN9nwpB4xg3qxwutdJMVlNa0aYC/qa/MGEFURBh//2hXu881xoQGCzAhorKmvs1dZOAdK1mYlcamfUfZebis2XxtmWQZSHLfaK6YPIQl6/M5UlHb7vONMT2fBZgQoKrOGEzbu8gArpgylIgwYfH65lsxra1D1pIb52RQXdfA06v3duh8Y0zPZgEmBFTXNdCgzS/V35zkvtGcNy6VF9fvDzgxsr5BKSqvbfMdZE2NGxTPnNHJPPHxbmo9NvHSmN7GAkwI8G2X3LedLRjwDvYXltXw4c6ik44VV9RQ36Ad6iLzuWlOBodLa3hty8EOX8MY0zNZgAkBvt0s29uCAThvbCqJcVEB58QUlLZ9Fn9zzhmTwsiUOB5dnmcTL43pZSzAhIDWNhtrSVREGFdMHspbnx7maOWJg/GFziTLlpbqb01YmHDDrHQ+2V/KZwebv5nAGBN6LMCEgNa2S27Nwqw0ausbTlrGv6DMWSamEy0YgAsmDALggzYsTWOMCR0WYEJARa2vBdOxADN+SDwzRiby+Ee7ThiM93WRpXQywAyMj2HswH5tWvvMGBM6LMCEAN8YTEe6yHy+cc4oDh6r5pVNx9cjLSirIaFPJDGRHb+uz9ljklmz60hja8sYE/oswISAxgDTwS4ygHPHpDB2YD8e+eD4YHxn5sA0dfaYFGrrG1iVVxKU6xljuj8LMCGg0ukii43qeEtDRPjGOSPZfriM93d4u7I6Oos/kGnpiURHhLFsh3WTGdNbWIAJAb55MB0dg/G5bNIQhiTE8NdluYBvHbKO30HmLyYynDNHJtlAvzG9iAWYEFBR4yE8TIiO6NzHGRkexo1zMliZV8LGfUcpLOvYQpfNOTszmbzCCvKPVAbtmsaY7ssCTAioqKknNioc7x5snXP19OH0i4ng169vo7a+gdT44LRgwDvpEgi4aoAxJvRYgAkBlbUe+naye8ynb3QEX50xghW53t0og9mCGZ3al8EJMXxg4zDG9AoWYEJARW19pwb4m7p+djpRzgZjwQwwIsJZmckszykKuLimMSa0WIAJARU1bd8uuS1S+8WwIGso4J0kGUxnj0mhrNrDpvyjQb2uMab7cXXLZHNqVNYEtwUD8L0LxjJ+cDwjkmKDet05o5MJE/hgRxFZIxKDem1jTPdiLZgQUBHEMRifpL7RfHVmelBuHPDXPzaKiWn97XZlY3oBCzAhoKLG0+GFLrvC2ZnJbNp3lGOVdV1dFGOMi1wNMCJyoYhsF5EcEbkrwPFoEXnOOb5KRNKd9HQRqRKRjc7jL056P7+0jSJSJCK/d45dLyKFfse+7mbdupOK2vpOrUN2qp09JoUGheU5XXu78nvbC7j5ibUUl9d0aTmMCVWu/ewVkXDgIeB8IB9YIyJLVfVTv2w3AUdUdbSIXA3cB3zJOZarqpP9r6mqZUBjmoisA170y/Kcqt4W/Np0b5U1nk6tQ3aqTR7Wn34xEXy4s5BLJg4+5e/f0KA8+F4OD7y9A1UYkRTL3ZeMP+XlMCbUudmCmQ7kqGqeqtYCzwLzm+SZDyxyni8G5kkbO/1FJBNIBT4MUnl7pIYGpbKuntggj8G4KSI8jNmjklm2o/CU73JZWl3HzU+u4/63djB/0hAunTiYJ1fuadz7xhgTPG4GmKHAPr/X+U5awDyq6gGOAUnOsQwR2SAiy0TkrADXvwZvi8X/G2qBiGwWkcUiMixQoUTkZhFZKyJrCwt7/kBzVV09qhAX5LvI3Db3tFQOHqtm64HSU/aeOw+XccWDH/H+9gJ+ctl4HvjSZL53wVjq6pW/Lss7ZeUwprdwM8AEaok0/bnaXJ6DwHBVnQLcCTwtIvFN8l0NPOP3+hUgXVUnAm9zvGV04sVVH1HVbFXNTklJaUM1ujffQpc9qQUDMHdcKiLw9meHT8n7HauqY8HDKyit9vDPr5/J9bMzEBHSk+O4cspQnlq5h4JSa8UYE0xuBph8wL8VkQYcaC6PiEQACUCJqtaoajGAqq4DcoExvpNEZBIQ4RzDyVesqr7R2r8BWcGtTvdUWeNdqr9vDxrkB0juG83U4QN469NTE2A+zi2itNrDn66Zwpkjk044dvvc0XgalD+/n3tKymJMb+FmgFkDZIpIhohE4W1xLG2SZylwnfN8IfCuqqqIpDg3CSAiI4FMwL8P4xpObL0gIv6jxZcDnwWtJt1YYwumBw3y+5w/fiBbD5Ry4GiV6+/14c4i4qLCyU4fcNKxEUlxLJg6lKdX7+XQMWvFGBMsrgUYZ0zlNuANvF/2z6vqVhG5V0Qud7I9BiSJSA7erjDfrcxnA5tFZBPewf9bVNV/K8Qv0iTAAHeIyFbnnDuA692oV3dT4bRgetJdZD6fO20gAO+cgm6y5TlFzBiZRGR44H/yt8/NpKFBefj9HNfLYkxv4eq3kqq+BrzWJO0ev+fVwFUBzlsCLGnhuiMDpP0I+FFnytsTHR+D6VldZOBdXXlkchxvfnqYr85Md+199pVUsqe4kutnNf8ewxJjWZiVxjOr9/GNc0YxpH8f18pjTG9hM/l7uONjMD2vBQPwufEDWZlXTFm1e7P6ffvPnJWZ3GK+W88bTYMqf7ZWjDFBYQGmh6uo8Y3B9LwWDHi7yerqlQ92uDer/6OcIgbFxzAqpW+L+YYlxrJgahrPr82n0mkZGmM6zgJMD+frIuuJYzAAWSMGMCA2krc+PeTK9esblI9yi5iTmdymhTsvmzSEWk8DK3KKXSmPMb2JBZgerrLWGeTvoV1k4WHC3HEDeXdbAXUubEK29cAxjlbWtdo95jM9I5G4qHDe2VYQ9LIY09tYgOnhKmo8RIYLURE996M8f/xASqs9rN19JOjX9o2/zB7dtgATFRHGWZkpvLet4JQvY2NMqOm530ohYvWuEr7014/ZVVTRofN72lL9gZyVmUxURJgrky6X7yzitMHxJPdt+9bPc09L5VBpNZ8ePHXL2BgTiizAdLGVecWs2lXCwodXsCX/WLvPr6it73HrkDUVFx3B7FFJvPXZoaC2Gqpq61m350ibu8d8zh3rXULoPesmM6ZTLMB0seLyGvpEhhMTGc7Vj3zM8p3tu5uqstbTY8df/J0/fhD7SqrYcbg8aNdctauY2voG5rSxe8wntV8Mk9ISbBzGmE6yANPFiipqGdw/hhe/NYthibHc8I/V/Htz0yXbmldR07OW6m/OvNNSAYJ6N9nynUVERYQxPSOx3efOHTeQjfuO2mZkxnSCBZguVlJeS3JcNAPjY3juGzOZMmwAtz+zgWdW723T+RU1nh7fRQYwMD6G04fGNw7KB8PynCKmpQ8gJrL9fz9zx6WiCu9v7/lbOhjTVSzAdLHiihoS46IASOgTyRM3TWdGRhL3vb4NTxtu262ore/xg/w+MzKS2LDvKDWe+k5fq7Cshm2Hytp891hTE4bEk9ovmnetm8yYDrMA08WKy2tJ6hvV+DomMpyvzhzB0co6Nuw72ur5lbWeHrdUf3OmZyRS62lg07723+zQ1Ec5zvIwozu2509YmDB3XCof7Ch0ZX6OMb2BBZguVN+gHKmsJSku6oT0szKTiQgT3vms9V/PFTWekBiDAW+AEYFVeZ2fRf/hziIGxEYyYUjTferabu64VMpqPKzZXdJ6ZmPMSSzAdKGjlbU0KCQ1maPRLyaSM0cm8u621ueFVNT0/NuUffrHRjF2YD9Wd/ILvcZTz3vbC5iTmUJYWOvLwzRn9uhkosLDeLcNgd4YczILMF2ouKIW4IQuMp+54way43A5+0oqmz2/vkGpqqsPiduUfc7MSGTdniOd6pZ6Y+thSipquSorrVNliYuOYMaoJN7dbgHGmI6wANOFisu9ASYx7uQAM2+c97bdlgaZq+p67mZjzTlzZBKVtfVs2d/xcZinV+1heGJsu+e/BDJvXCp5hRXs7uBKC8b0ZhZgulBxhXeORaBlTNKT4xiZEtfiZL/GpfpDZJAfYFq6d87K6l0d6ybLLSxnZV4JV08f1qnuMZ+5TqB/Y6s7qz0bE8oswHQhXwum6SC/z7xxqazMLW4MJE350kOpBZPSL5pRKXEdHuh/dvVeIsKEhZ3sHvMZlhjLGUMT+OV/tnHRHz7kT+/sJKegLCjXNibUWYDpQsUVtYh4B7cDmTtuILX1DSzPCTz5sKcv1d+cM0cmsXb3Eeob2rcuWXVdPYvX5XPBhIGk9osJWnn+ccM07rl0PH2jw7n/7R187v4POP/+Zew8bIHGmJZYgOlCxeU1JMZGEd5MV052+gD6xUQ0exfT8RZM6HSRgXegv6zGw2ftXM34ja2HOFJZx5enjwhqeZL6RnPjnAxeuGUWK380j3vnTyD/SBVPrdwT1PcxJtS4GmBE5EIR2S4iOSJyV4Dj0SLynHN8lYikO+npIlIlIhudx1/8znnfuabvWGpL1+rOistrAw7w+0SGh3HOmBTe3V5AQ4Bf877dLENlHoyPb+2wle3sJvvnqr0MT4xl1qgkN4oFeJe0+drMdLJGDGCNC/vXGBNKXAswIhIOPARcBIwHrhGR8U2y3QQcUdXRwAPAfX7HclV1svO4pcl51/odK2jDtbqlkoragLco+5t3WiqFZTV8cuDku6oqanx3kYVWC2ZwQh+GJ8ayqh0D/TkF5azeVcI104cHZXC/NVkjBrDtUCll1XWuv5cxPZWbLZjpQI6q5qlqLfAsML9JnvnAIuf5YmCetGXj9MCCea1Toqii5qRJlk2dMyaVMCHgrP5KpwUTamMw4O0mW7O7JGDLLZBnVu8lMly4Kjs4g/utmZaeSIPChr2tL+djTG/lZoAZCuzze53vpAXMo6oe4Bjg69/IEJENIrJMRM5qct7fne6xH/sFkZau1S0Vl5+8TExTiXFRTBk+IOB8mOMtmBAMMCOTOFpZx4423LFVXVfPkvX5XDB+ULt2ruyMycP7Eyawdo91kxnTHDcDTKDWQ9Ofo83lOQgMV9UpwJ3A0yLiW1TqWlU9AzjLeXy1He+HiNwsImtFZG1hYdctxV5X38CxqjqS4lr/Qpw7LpUt+49xuLT6hPRQnAfjc6YzDrMqr/Vusn9vPsjRyjq+fOZwt4vVqG90BOOHxLPW1ikzplluBph8YJjf6zSg6U5ajXlEJAJIAEpUtUZViwFUdR2QC4xxXu93/iwDnsbbFdfstZoWSlUfUdVsVc1OSenYSrvBcKSFZWKa8m3G1bQVU1FbT1REGJHhoXczYNqAPgxJiGl1wuVHOUX8+F+fcNrgeGaOPLUN1uwRiWzYe9RWWzamGW5+M60BMkUkQ0SigKuBpU3yLAWuc54vBN5VVRWRFOcmAURkJJAJ5IlIhIgkO+mRwKXAJy1dy6W6dVpRK5Ms/Y0d2I9hiX14dfPBE9Ira0Njs7FARITpGYms2lVMcx/je9sKuOEfaxieGMuiG6edksF9f9npA6iqq2/37dTG9BauBRhnHOQ24A3gM+B5Vd0qIveKyOVOtseAJBHJwdsV5ruV+Wxgs4hswjtgf4uqlgDRwBsishnYCOwH/tbKtbqlksYWTOtdZCLCF6ak8VFuEfuPVjWml9d4QmazsUDOHJlEUXktuYUnrwP2ny0HufnJtYwZ2Jdnb54R1ImVbZU9wtuNZ7crGxOYq99Oqvoa8FqTtHv8nlcDVwU4bwmwJEB6BZDVzHsFvFZ35VuHrC1dZAALs9L4wzs7eXFdPrfPywSgsqaeuBAcf/GZ4XR5zX9wOdnpicwYmcSMkYnkFVbwgyWbmZSWwN9vmE5Cn8guKd+ghBjSBvRh3Z4SbpqT0SVlMKY7a1OAEZFRQL6q1ojIucBE4AlVtXs0O6g9XWTgXRNr5sgkFq/P57a5oxERKmo9IXmLsk9GchyPXZfN+9sLWZlXzH2vb2s8NnNkEo9el93l9Z+WnsjynCJUlW5+V7wxp1xb/3cuAbJFZDTerqileAfYL3arYKGupKKGiDAhPqbtv76vyk7jzuc3sXpXSeOy9qF4i7K/eacNZN5pAwEoKq9h9a4SCkqruXr6cGIiu771ljViAC9t2M/ekkpGJMV1dXGM6VbaOgbT4IypXAn8XlW/Awx2r1ihr7i8lgFxUe0amL7w9EH0jY7ghXX5gLNdcogO8geS3Deai88YzPWzM7pFcIHj2wustXEYY07S1gBTJyLX4L1L699OWtd0fIeIojZMsmwqNiqCS84YzGtbDlJR4wn5LrKeIDO1L/ExEazdY/NhjGmqrQHmBmAm8H+quktEMoCn3CtW6CupqOnQrPOrstOorK3ntS0HQ36QvycICxOyRgywFowxAbQpwKjqp6p6h6o+IyIDgH6q+iuXyxbSiitaXkm5OVkjBjAyOY4X1uV7WzAhPgbTE2SnJ7KzoJyjlbVdXRRjupU2BRhnifx4EUkENuFdC+x+d4sW2orLW19JORARYUFWGqt3lVBd1xDS82B6iuwRAwBYZ+uSGXOCtnaRJahqKfAF4O+qmgV8zr1ihbbqunrKazwdXphxwdQ0fPcGWBdZ15s0rD+R4WITLo1poq0BJkJEBgNf5Pggv+kg3yz+jnSRgXeC31mZ3nXUbJC/68VEhnP60ARb+NKYJtoaYO7Fu+RLrqqucdYH2+lesUJbcTsnWQayMMu774kFmO5hWnoim/OPUV1X39VFMabbaOsg/wuqOlFVv+m8zlPVBe4WreerrqsPuGHW8WViOr53yYWnD+KeS8dz7tiuWxHaHDdjZCK19Q08+mFeVxfFmG6jrYP8aSLykogUiMhhEVkiIqdm68Aeqr5Bmfe7ZTz0Xs5Jx4LRgokMD+PGORntWgnAuOe8sanMnzyE3765g5c37u/q4hjTLbS1i+zveJeHGYJ358hXnDTTjE/2H2P/0So+2HnypmbtXejSdH8iwq8XTuTMjES+/8JmVuUVd3WRjOlybQ0wKar6d1X1OI9/ANY304KPcosA2Jx/jFrPiRtSFVfUEhUeRl8bPwkp0RHhPPLVbIYl9uHmJ9eRW1je1UUypku1NcAUichXRCTceXwFsJ9oLViRU4wI1HgaTtqQyjcHxlbfDT0JsZH844bpRIYL1/99NUXlNV1dJGO6TFsDzI14b1E+BBzEu2PkDW4Vqqerrqtnze4SLjp9EAAb9p44P6K4vMa6x0LYsMRYHr1uGoVlNXztsdXsPFzW1UUypku09S6yvap6uaqmqGqqql6Bd9KlCWD9niPUeBpYMDWNwQkxrN974rY5JRW1JMZ1/A4y0/1NHtafh6/N4sCxKi7+44f87s3tdguz6XU6s2XynUErRYj5KLeI8DDvnvJThvdnfZMWTFF5LcmduIPp3PRTAAAgAElEQVTM9AznjUvl7TvP4bKJQ/jTuzlc9IcP+SinqKuLZcwp05kAYwMIzfgop5jJw/rTLyaSqcMHkH+kioKy6sbjxRXWRdZbJPeN5v4vTeapm86kQZVrH13FH9+xOcqmd+hMgDl5BqHhWFUdm/OPMnuUdz/5KcO9CyFucLrJKms9VNc1WBdZLzMnM5k3vn02548fyMPv51JaXdfVRTLGdS0GGBEpE5HSAI8yvHNiTBOr8oppUJg1OhmACUPiiQyXxgDTOMnSWjC9TkxkOHfMzaSqrp4XnV1JjQllLQYYVe2nqvEBHv1UtdVJHCJyoYhsF5EcEbkrwPFoEXnOOb5KRNKd9HQRqRKRjc7jL056rIi8KiLbRGSriPzK71rXi0ih3zlfb+9fRjCsyC2mT2Q4U4b3B7xfKhOGJDSOw/huW+3MLH7Tc52RlsCkYf15atVeVK0TwIS2znSRtUhEwoGHgIuA8cA1IjK+SbabgCOqOhp4ALjP71iuqk52Hrf4pf9WVccBU4DZInKR37Hn/M55NOiVaoPlOUVMy0gkOuL4MvpThvdnc/5R6uobGldS7sw6ZKZn++qMEeQUlLMyz1ZfNqHNtQADTAdynIUxa4FngflN8swHFjnPFwPzpIXZh6paqarvOc9rgfVAt1kT7XBpNTkF5Y3jLz5Thw+guq6B7YfKgrIOmenZLp04mIQ+kTy1ck9XF8UYV7kZYIYC+/xe5ztpAfOoqgc4Bvi+nTNEZIOILBORs5peXET6A5cB7/glLxCRzSKyWESGBSqUiNwsImtFZG1h4cnrhHXGCmd5mNnO+IvPVGfHw/V7j1Bk65D1ejGR4XwxO403th6ioLS69ROM6aHcDDCBWiJNO52by3MQGK6qU/DOt3laROIbTxKJAJ4B/qiqvvXRXwHSVXUi8DbHW0YnXlz1EVXNVtXslJTgLqe2fGcxA2IjGT84/oT0IQkxpPaLZv2eI5SU19InMty2Ou7lvnzmCDwNyrNr9rWe2Zgeys0Akw/4tyLSgAPN5XGCRgJQoqo1qloMoKrrgFxgjN95jwA7VfX3vgRVLVZV38JPfwOygliXVqkqK3KLmDkqibCwE+OmiDB1+AA27DtKcUWttV4MGclxnJWZzNOr9uKpb2j9BGN6IDcDzBogU0QyRCQKuBrvkv/+lgLXOc8XAu+qqopIinOTAM7umZlAnvP653gD0bf9L+Rs6exzOfBZkOvTol1FFRw8Vn1S95jP1BH92VNcyY7DZTb+YgDvYP+h0mre2VZwQvqRilre2HqIOgs8podzLcA4Yyq34d1q+TPgeVXdKiL3isjlTrbHgCQRycHbFea7lflsYLOIbMI7+H+LqpY4m5zdjfeutPVNbke+w7l1eRNwB3C9W3UL5KNc7+LSs0cFDjC+CZdbD5TaHWQGgLnjUhmcENM42J9TUMb/vLSFmb96h288uY7bnl5Pjadz65cVlFbzf69+SmGZrepsTj1XBwJU9TXgtSZp9/g9rwauCnDeEmBJgPR8mlmiRlV/BPyok0XusFV5xQxJiGFEUmzA42cMTSAiTPA0qLVgDAAR4WF8efpwfvfWDr78t5WsyC0mKiKMKycPZXD/GH7/9k5ufmIdf/lKFn2iwlu/YBPVdfX81xNr2ZR/jE/2l/LU188kPMxWeDKnjptdZL1KQVkNwxJjm93jJSYynPFDvIP/iTYGYxxfmj6MPpHh7Cwo57vnj+Hju+Zy38KJfPtzY7hvwRl8sLOQ6/++mvIaT7uuq6r8YPFmNuUf44vZaXycV8wDb+1wqRbGBGa3MgVJWbWHof37tJhn6vABbM4/RrKtQ2Ycqf1i+OAH5xHfJ+KEybkAX5o2nJjIcO58fhNfeXQVi26YTnRkGPlHqthXUsm+I5UMT4zlnDEpJ/2w+fP7uSzddIDvf34st543GkF48L0cstIHcN7Y1DaVTVVtUzzTKRZggqS0qo7TBvdrMc+U4f35xwpItC4y4yelX/M/OOZPHkqfyHBue3oDM375DlUB9pSZlj6Auy4aR9aIRADe3HqI37yxncsnDeFb544C4KfzJ7B5/zG+89xGXr3jrFZ/DB06Vs1Vf13BjbMzuGF2RidqZ3ozCzBBUlZdR3xMZIt5zspMYXpGItnpA05RqUwouGDCIBbdOJ2lm/YzKL4Pw5P6MDwxlqH9Y3ln22F+//ZOFjz8MReMH8gXpg7lzuc3MSktgV8vnNjYAomJDOfP107lsj8t59Z/ruf5b8wkKiJwD7mnvoE7nt3AvpIqHnovh2ume1tSxrSXBZggaGhQymo8xMe0/NeZGBfF89+YeYpKZULJzFFJzGyyBBHAtWeO4MopQ3l8+S7+siyPNz89zMD4aB75WvZJQSEjOY7fLJzIN/+5nv979VN+cvmEgF1gf3xnJ6t3lXD1tGE8u2Yfr2w6wFXZARfGMKZFFmCCoLzWgyrE92m5BWOMG2KjIrhtbibXTB/O06v2cv6EgQyMjwmY96IzBnPTnAweW76LY1V1/PILE0+4Q+2jnCL+9F4OV2Wl8csvnMGGvUd5bPkuFmal2XiMaTe7iywIyqq9d/j0a6UFY4ybkvpGc/u8TMYNim8x390Xn8Z3zx/Dy5sOsODhFewrqQSgsKyG/352I6NS+vLT+d7WzY1z0tl2qIyPnXlexrSHBZggKK3y7k7Y2hiMMd1BWJhw+7xMHr9uGvuOVHLZg8tZtqOQ7zy3kbLqOh768tTGtfLmTx5KUlwUj3+0q4tLbXoiCzBBcLwFYwHG9BznjUvlldvmMLBfDNc9vprlOUX89PIJjB10/G7ImMhwrp0xgne2FbCrqKILS2t6IgswQdDYguljXWSmZ0lPjuPFb83iS9nDuHF2Bl+advJg/ldmDCcyLIy/WyvGtJN9IwZBWY03wFgLxvREcdER3LdwYrPHU/vFcNmkIbywNp/vnj+WhFj7d27axlowQVBa5e0ia+02ZWN6qpvmZFBVV8+za/ae0vfdkn+MlXl2g0FPZQEmCHxdZNaCMaFq/JB4Zo5MYtGK3VS0c120jlJV/vu5DXzzqXW2dUEPZQEmCMpqPMREhjU7M9qYUPBfZ2dw4Fg1k376JgseXsFv39jORzlFVAdYviYYNu47Sl5hBUcq61i+s8iV9zDusm/EICitan2ZGGN6urnjBvLszTP4r7NHUt+gPLwsl2sfXcV5v32f0uq6oL/fkvX5xESGkdAnkn9t3B/06xv32aBBEJRVe2ySpekVZoxMYsZI75I1ZdV1vP3ZYb7z3CaeX7OPr581MmjvU+Op55VNB/n8hEHERkXwrw37qaz1NM7PMT2DtWCCoLS6zpaJMb1Ov5hIrpySxrT0ASz6eDf1DRq0a7/zWQHHqupYMDWNKyYPoaqunrc+PRww7/6jVVz9yMds2HskaO9vgsMCTBCUVntsgN/0WjfOzmBfSRVvfxY4AHTE4nX5DIqPYfboZKalJzIkIYaXNx4ImPd3b25nZV4J331hk2vjQaZjLMAEQVlVnd2ibHqt88cPZGj/Pjy+PDgTMQvLali2o5Arpw4lPEwICxMumzyED3YUUlJRe0Lezw6W8tKG/cwcmUReYQUPvpsTlDKY4LAAEwTWRWZ6s4jwMK6flc6qXSV8sv9Yp6/38sb91DcoC6amNabNnzQUT4Py6paDJ+T9zRvb6RcdwV++ksUXpg7lL8ty+fRAaafLYILD1QAjIheKyHYRyRGRuwIcjxaR55zjq0Qk3UlPF5EqEdnoPP7id06WiGxxzvmjOGuIi0iiiLwlIjudP0/Zrl6lNshverkvThtGbFQ4f/9od6euo6osXpfPpGH9GZ3atzH9tMH9yEzty8sbjt9NtjKvmHe3FfCt80aTEBvJjy8ZT//YSH64ZDMemzfTLbgWYEQkHHgIuAgYD1wjIuObZLsJOKKqo4EHgPv8juWq6mTncYtf+sPAzUCm87jQSb8LeEdVM4F3nNeuq66rp9bTYLcpm14toU8kV2Wl8cqmAxSUVXf4OlsPlLLtUBkLs9JOSBcRrpgylLV7jrCvpBJV5Vf/2cag+Biun5UOwIC4KH5y+QS27D9mqz93E262YKYDOaqap6q1wLPA/CZ55gOLnOeLgXnSwq5GIjIYiFfVj1VVgSeAKwJca5Ffuqt8KynbGIzp7a6fnUFtfQP/XNnx5WSWrM8nKjyMyyYOPunY5ZOGAPDK5gO8sfUwG/cd5TvnZ56wc+clZwzmc6cN5P63drCn2FZ/7mpuBpihwD6/1/lOWsA8quoBjgG+fWEzRGSDiCwTkbP88uc3c82BqnrQudZBIDVYFWmJb4KZjcGY3i4jOY5541L556o9Hbqbq66+gaUbD/C58an0j4066fiwxFiyRgzgpfX7+fUb2xiVEnfCOA14Wzo/v+J0IsPCuGvJlqDeOm3az80AE6gl0vTTbi7PQWC4qk4B7gSeFpH4Nl6z5UKJ3Cwia0VkbWFhYXtODch2szTmuBvnZFBUXssrmwLfUtyStz89THFF7UlBw9/8yUPYWVBOXmEFP7hwHBHhJ3+FDUqI4ceXjufjvGK+v3iTBZku5GaAyQf8N5dIA5r+q2vMIyIRQAJQoqo1qloMoKrrgFxgjJPf/1+f/zUPO11ovq60gkCFUtVHVDVbVbNTUlI6UT0v283SmONmjUpi7MB+PPx+brtaMcXlNfy/pVsZmRLH2WOa/395yRmDCQ8Tpg7vzwXjBzab74vThvHd88fw4vr9/OjFzTRYkOkSbgaYNUCmiGSISBRwNbC0SZ6lwHXO84XAu6qqIpLi3CSAiIzEO5if53R9lYnIDGes5mvAywGudZ1fuqsax2Csi8wYRIS7LzmNvKK2z0lpaFC++8ImjlbV8eA1U4kM0CrxSeobzePXT+MPV0+hheFaAG6fl8kd8zJ5fm0+//vyJ3iHbc2p5Fq/jqp6ROQ24A0gHHhcVbeKyL3AWlVdCjwGPCkiOUAJ3iAEcDZwr4h4gHrgFlUtcY59E/gH0Af4j/MA+BXwvIjcBOwFrnKrbv58YzDWRWaM19ljUlgwNY2/LMvlkomDOW1wfIv5H1u+i/e3F/Kz+RMYP6TlvADntNDCaeo7n8vEU9/An9/PJSJM+OnlE1oNTCZ4XP1WVNXXgNeapN3j97yaAIFAVZcAS5q55lrg9ADpxcC8Tha53ayLzJiT/fjS01i2o4AfLtnMi9+cFXCsBLxL8t/3+jY+P2EgX5kxIujlEBG+//mxeBqURz7IIzoijLsvaTpbwrjFZvJ3Ulm1h/AwITYqvPXMxvQS/WO9c1I25zc/J6W0uo7bn1nPwPgYfr1gkmstCxHhRxeN47qZI/jbh7t4Ye2+1k8yQWEBppNKq+voFxNhzW5jmmhpTkqNp54fvbiFA0er+eM1k0mIdbcHQET48aXjmTUqibv/9Qmb84+6+n7GywJMJ9leMMYE5j8n5UcvbuFIRS0vbcjn1n+uJ+tnb/Pq5oPcef4YskYknpLyRISH8eCXp5LSN5pbnlxHcXnNKXnf3swCTCfZbpbGNG9QQgx3XTyOFbnFTP35W3znuU2s3l3CZZOG8I8bpvGtc0ed0vIkxkXx169mUVxRy61Pr7c1y1xmP707qazaYwHGmBZcM204e4oriQwXzh8/iIlDEwgL67ou5dOHJvDLL5zBnc9v4pf/2caPL7VBf7dYgOmk0uo6hifGdnUxjOm2wsKE/7n4tK4uxgm+MDWNzfnHeGz5LrJGDODiM05e+8x0nnWRdVJple0FY0xPdPclpzWuOmDcYQGmk2yQ35ieKTI8jC9NG8aW/cfYcbisq4sTkizAdEJ9g1JWY2MwxvRU8ycPISJMWLIuv/XMpt0swHRCeY2tpGxMT5bUN5pzx6by0ob9p+SOspV5xVz0hw/ZVdQ79qqxANMJjcvE2BiMMT3WwqyhFJTVsDynyPX3emz5Lj47WMptT6/v0J45PY0FmE6w3SyN6fnOG5dK/9hIlqzf7+r7HKmo5f3tBUwe1p+tB0r5xWufufp+3YEFmE5o3M3SxmCM6bGiI8K5fNIQ3tx6qPH/tBte3XKQunrl51ecztfnZPDEx3t4bctB196vO7AA0wnWRWZMaFgwNY0aTwOvbnbvC//ljfvJTO3LhCHx/ODCcUwa1p8fLt580jptocQCTCfYdsnGhIaJaQmMTu3r2t1k+0oqWbP7CFdMGYqIEBURxoPXTEEEbnt6AzWe0ByPsQDTCdZFZkxoEBG+MHUoa/ccYbcLd3i9vNE7vjN/8pDGtGGJsfzmqkls2X+MX762Lejv2R1YgOkEXwumr7VgjOnxrpwyFBF4cX1wWzGqyksb9jM9I5G0AScuK/X5CYO4cXYG/1ixm9c/Cb3xGAswnVBaVUdsVHiLe4gbY3qGwQl9mDM6mRc37KehQVvM+3FucZsDwif7S8ktrODKKUMDHr/ronFMSkvg+4s3s7e4st3l7s7sm7ETbJkYY0LLgqlp5B+p4tk1ze96uWRdPl95bBW3Pb2hTQHhpQ37iQoP4+LTAy+oGRXh3adGgFufXt+p8ZhjlXXkFpZ3+PxgswDTCaXVtheMMaHkwtMHMWNkIv/z0hbuefkTaj0nzu5/fPkuvvvCJrJGDCA8TPj92ztavJ6nvoGlmw4wd1xqi7t2+o/H/OLVjs+P+ckrWzn//mU8/H4uqoFbYccq6/jff205JasJWIDphLJqj92ibEwIiYkM58mbzmycp3L1Ix9z6Fg1qsoDb+3g3n9/yucnDOSJG6dz/ax0Xtq4n50tLJS5IreYovIarpgypNk8Pp+fMIib5mSwqIPzY1SVD3cWEhsVwX2vb+PWp9c3LmflO7500wHm3b+MZ1bvY1Vecbvfo71cDTAicqGIbBeRHBG5K8DxaBF5zjm+SkTSmxwfLiLlIvI95/VYEdno9ygVkW87x34iIvv9jl3sZt3A24KxLjJjQktkeBj/e+l4HvzyFLYdKuPSP33Ifz+7kT+8s5OFWWk89OWpxESGc8s5o4iLiuD+t5pvxfxrw37iYyI4d2xqm977h52YH7PjcDlF5bXcc+l47r74NF7/5BBXPvQReYXl7Cup5IZ/rOGOZzYwOCGGl2+dzdXTh7fr+h3hWoARkXDgIeAiYDxwjYg03TruJuCIqo4GHgDua3L8AeA/vhequl1VJ6vqZCALqARe8s/vO66qrwW3Riez7ZKNCV2XThzCy7fOJr5PJEs3HeDrczL49YKJRDg39QyIi+KmORn855NDbMk/dtL5BaXVvL71EJdMHExMZHib3tM3PyYsTPjy31ax7VBpm8v7kbOW2qzRSfzX2SN56qYzKa6oZf6DH3HBAx+welcJ91w6nn/dOpvThya0+bqd4WYLZjqQo6p5qloLPAvMb5JnPrDIeb4YmCciAiAiVwB5wNZmrj8PyFXVPUEveRvZIL8xoS1zYD+W3jaH526ewd2XnHbSVs9fPyuD/rGR/PbN7Sek5xaWc+WfVwDw1Rnp7XrPYYmx/PPrZ+JpaGDhwx/z/vaCNp23IreYEUmxjbdCzxqdzCu3z+G0IfGcPSaZt+48hxvnZBB+CrerdjPADAX8b8XId9IC5lFVD3AMSBKROOCHwE9buP7VwDNN0m4Tkc0i8riIDOhM4Vujqt5BfhuDMSak9Y2O4MyRSTi/fU/QLyaSb54zimU7Clm9qwSADXuPsPDhFVTX1fPszTMYPyS+3e95+tAE/nXrbIYlxnLTorU8ubLl39Ge+gZW5RUza1TSCelD+/fh+W/M5K9fzWZo/z7tLkdnuRlgAoXJprc1NJfnp3i7uwLebyciUcDlwAt+yQ8Do4DJwEHgd82ce7OIrBWRtYWFhS3XoAU1ngbq6tVaMMb0cl+bmU5Kv2h++8Z23t12mGv+tpL4PpEs+eYsJqb17/B1Byf04YVbZnLOmBR+/K9P+Nm/P6W+mfk5nxwopazGw6xRyR1+Pze4GWDygWF+r9OAA83lEZEIIAEoAc4Efi0iu4FvA/8jIrf5nXcRsF5VD/sSVPWwqtaragPwN7xddCdR1UdUNVtVs1NSUjpcucaFLm0MxpherU9UOLfPHc3q3SXctGgtman9WPLNWaQnx3X62n2jI/jb17K5flY6jy3fxVPNtGRW5HrHX2aMTAp4vKu4GWDWAJkikuG0OK4GljbJsxS4znm+EHhXvc5S1XRVTQd+D/xCVR/0O+8amnSPiYj/LKYrgU+CV5WTlfr2grEuMmN6vaunDWfcoH6cNzaVZ2+eQXLf6KBdOzxM+MnlE5g8rD+LVuwOuMrAipxixg7sR0q/4L1vMLgWYJwxlduAN4DPgOdVdauI3CsilzvZHsM75pID3AmcdCtzUyISC5wPvNjk0K9FZIuIbAbOA74TpKoE5Fvo0rrIjDFREWG8dsdZPH79NOKi3flOuH5WOnlFFXzYZOfNGk89a3aXMGt092q9ALj67ejcKvxak7R7/J5XA1e1co2fNHldCZz0N6mqX+1MWdvLusiMMf6a3mEWbBefMZifv/oZi1bs5pwxx7v31+85So2noduNv4DN5O8w2y7ZGHMqRUWE8eXpw3hve8EJkzA/zi0iTODMkYldWLrALMB0UONeMDYGY4w5Ra6dMYJwEZ74+Phg/4rcYs5I698te1MswHSQ7WZpjDnVBsbHcOHpg3h+7T4qaz1U1HjYuO8os0d1v/EXsADTYaVVdUSECX3auASEMcYEw/Wz0imr9vDShv2s3l2Cp0G75fgLWIDpMN9KyoFm9xpjjFuyRgxgwpB4Fq3YzYqcIqLCw8ga4erCJR1mAaaDbCVlY0xXEBGum5nOjsPlPLt6H1NH9KdPVPfsSbEA00Fl1Z5uOahmjAl9l08ewoDYyG65PIw/CzAdVFplLRhjTNeIiQznS9O8+7nM7oYTLH3sG7KDSqvrGJnct6uLYYzppW49bxSjUuKYOrx7jr+ABZgOs71gjDFdqV9MJFdlD2s9YxeyLrIOKq2yvWCMMaYlFmA6wFPfQEVtvbVgjDGmBRZgOqC8xrcOmbVgjDGmORZgOqDM9oIxxphWWYDpgGNVtheMMca0xgJMBzSupGxdZMYY0ywLMB1gKykbY0zrLMB0gG83ywQbgzHGmGZZgOkAa8EYY0zrLMB0QNqAPnx+wkD6RluAMcaY5tg3ZAdcMGEQF0wY1NXFMMaYbs3VFoyIXCgi20UkR0TuCnA8WkSec46vEpH0JseHi0i5iHzPL223iGwRkY0istYvPVFE3hKRnc6f3XcFOGOM6QVcCzAiEg48BFwEjAeuEZHxTbLdBBxR1dHAA8B9TY4/APwnwOXPU9XJqprtl3YX8I6qZgLvOK+NMcZ0ETdbMNOBHFXNU9Va4FlgfpM884FFzvPFwDxx9iAWkSuAPGBrG9/P/1qLgCs6UXZjjDGd5GaAGQrs83ud76QFzKOqHuAYkCQiccAPgZ8GuK4Cb4rIOhG52S99oKoedK51EEgNVCgRuVlE1orI2sLCwg5UyxhjTFu4GWAkQJq2Mc9PgQdUtTzA8dmqOhVv19utInJ2ewqlqo+oaraqZqekpLTnVGOMMe3g5l1k+YD/bjhpwIFm8uSLSASQAJQAZwILReTXQH+gQUSqVfVBVT0AoKoFIvIS3q64D4DDIjJYVQ+KyGCgwMW6GWOMaYWbLZg1QKaIZIhIFHA1sLRJnqXAdc7zhcC76nWWqqarajrwe+AXqvqgiMSJSD8ApxvtAuCTANe6DnjZrYoZY4xpnWstGFX1iMhtwBtAOPC4qm4VkXuBtaq6FHgMeFJEcvC2XK5u5bIDgZec+wAigKdV9XXn2K+A50XkJmAvcFXQK2WMMabNRLXpsEjvISKFwB6/pAS8NxrQzjT/1809TwaKOlnk5srTkbzNHWupbq297sr6BqOugdK642fbWr6e8Nme6n/HgdLssz3xeXvqOkJVWx/EVlV7OA/gkY6k+b9u4flat8rYkbzNHWupbu2s+ymtbzDq2lM+29by9YTP9lT/O7bPtvW6B6uu/g9bi+xEr3Qw7ZU2PA+W9lyzpbzNHWupbq297sr6BqOugdK642fbWr6e8Nme6n/HgdLss23be3ZYr+4iO5VEZK2euPJASOtN9e1NdYXeVV+ra+dYC+bUeaSrC3CK9ab69qa6Qu+qr9W1E6wFY4wxxhXWgjHGGOMKCzAdICKPi0iBiHzSeu6Tzs1ythvIEZE/+hb3dI7d7mxvsNVZxaDLuVFXEfmJiOx3tlzYKCIXB7/kHePWZ+sc/56IqIgkB6/EHefSZ/szEdnsfK5visiQ4Je8Y1yq729EZJtT55dEpH/wS95+LtX1Kue7qUFE2jZWE+zb0nrDAzgbmAp80oFzVwMz8a7D9h/gIif9POBtINp5ndrV9XSxrj8BvtfVdTtV9XWODcM76XgPkNzV9XTxs433y3MH8JeurqfL9b0AiHCe3wfc19X1dLGupwFjgfeB7LZcy1owHaCqH+BdeaCRiIwSkdedVZ4/FJFxTc9z1kiLV9WP1fuJPcHxbQW+CfxKVWuc9+gWa6m5VNduy8X6PgD8gJMXfO0ybtRVVUv9ssYR+vV9U70rwQOsxLvmYpdzqa6fqer29pTDAkzwPALcrqpZwPeAPwfIMxTvAp8+/lsYjAHOEu/OnstEZJqrpe2cztYV4DanW+Fx6f67j3aqviJyObBfVTe5XdAg6PRnKyL/JyL7gGuBe1wsazAE49+yz40E3iCxuwhmXdvEzdWUew0R6QvMAl7w63aPDpQ1QJrvF14EMACYAUzDu67aSOdXRLcRpLo+DPzMef0z4Hd4/3N2O52tr4jEAnfj7Urp1oL02aKqdwN3i8iPgNuA/xfkogZFsOrrXOtuwAP8M5hlDJZg1rU9LMAERxhwVFUn+yeKd9vodc7LpXi/WP2b0P5bGOQDLzoBZbWINOBdG6i77YrW6bqq6mG/8/4G/NvNAndSZ+s7CsgANow/JhwAAARWSURBVDn/sdOA9SIyXVUPuVz29grGv2N/TwOv0k0DDEGqr4hcB1wKzOtuPwj9BPuzbZuuHozqqQ8gHb8BNGAFcJXzXIBJzZy3Bm8rxTeAdrGTfgtwr/N8DN6dPqWr6+lSXQf75fkO8GxX19HN+jbJs5tuMsjv0meb6ZfndmBxV9fR5fpeCHwKpHR13dyuq9/x92njIH+X/yX0xAfwDHAQqMPb8rgJ76/U14FNzj+4e5o5NxvvHja5wIO+IAJEAU85x9YDc7u6ni7W9UlgC7AZ76+mwaeqPl1R3yZ5uk2AcemzXeKkb8a7xtXQrq6ny/XNwftjcKPz6BZ3zblU1yuda9UAh4E3WiuHzeQ3xhjjCruLzBhjjCsswBhjjHGFBRhjjDGusABjjDHGFRZgjDHGuMICjDF+RKT8FL/foyIyPkjXqndWMf5ERF5pbWVfEekvIt8KxnsbE4jdpmyMHxEpV9W+QbxehB5fDNFV/mUXkUXADlX9vxbypwP/VtXTT0X5TO9jLRhjWiEiKSKyRETWOI/ZTvp0EVkhIhucP8c66deLyAsi8grwpoicKyLvi8hiZ++Qf/rtsfG+b28NESl3ForcJCIrRWSgkz7Keb1GRO5tYyvrY44vttlXRN4RkfXi3edjvpPnV8Aop9XzGyfv95332SwiPw3iX6PphSzAGNO6PwAPqOo0YAHwqJO+DThbVafgXTX4F37nzASuU9W5zuspwLeB8cBIYHaA94kDVqrqJOAD4L/83v8Pzvu3ui6Us77UPLyrJABUA1eq6lS8+w79zglwdwG5qjpZVb8vIhcAmcB0YDKQJSJnt/Z+xjTHFrs0pnWfA8b7rUIbLyL9gARgkYhk4l1xNtLvnLdU1X8/jtWqmg8gIhvxrhO1vMn71HJ84c91wPnO85kc31vmaeC3zZSzj9+11wFvOekC/MIJFg14WzYDA5x/gfPY4LzuizfgfNDM+xnTIgswxrQuDJipqlX+iSLyJ+A9Vb3SGc943+9wRZNr1Pg9ryfw/706PT4o2lyellSp6mQRScAbqG4F/oh3X5YUIEtV60RkNxAT4HwB/n9794/SQBCGYfz5EBtB7ASx8BDmAp7ANnbiCSR4h4C2Nt7BC5g+RESwEMQDaCMIBgvLz2JGSERYAg4GfH6wsMUMs9vsy/xhv2FmXiw4rvQjl8ikbiNKXRMAIuLrl+cbwHO9P2w4/jVlaQ6g39U4M6eUcsUnEbFKec6XGi57wE5t+g6sz3S9Ao5q7RAiYjsiNn/pHfQPGTDSvLWIeJq5BpSPda9ufD9QSisAnALDiBgDKw2f6RgYRMQNsAVMuzpk5h3lr7l9ShGsXkTcUmYzj7XNKzCux5rPMnNEWYKbRMQ9cMl8AEkL8ZiytORqVcyPzMyI6AMHmbnf1U/6a+7BSMtvFzivJ7/eWNLy0tJ3zmAkSU24ByNJasKAkSQ1YcBIkpowYCRJTRgwkqQmDBhJUhOfkgFgAX1Dd3gAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"find_lr()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <div>\n",
" <style>\n",
" /* Turns off some styling */\n",
" progress {\n",
" /* gets rid of default border in Firefox and Opera. */\n",
" border: none;\n",
" /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
" background-size: auto;\n",
" }\n",
" .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
" background: #F44336;\n",
" }\n",
" </style>\n",
" <progress value='4' class='' max='5', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
" 80.00% [4/5 18:46<04:41]\n",
" </div>\n",
" \n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>epoch</th>\n",
" <th>train_loss</th>\n",
" <th>valid_loss</th>\n",
" <th>accuracy_thresh</th>\n",
" <th>time</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>0.053492</td>\n",
" <td>0.046831</td>\n",
" <td>0.980980</td>\n",
" <td>04:24</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>0.052653</td>\n",
" <td>0.046132</td>\n",
" <td>0.982369</td>\n",
" <td>04:00</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>0.052927</td>\n",
" <td>0.044149</td>\n",
" <td>0.982818</td>\n",
" <td>05:31</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>0.046456</td>\n",
" <td>0.044377</td>\n",
" <td>0.982432</td>\n",
" <td>04:49</td>\n",
" </tr>\n",
" </tbody>\n",
"</table><p>\n",
"\n",
" <div>\n",
" <style>\n",
" /* Turns off some styling */\n",
" progress {\n",
" /* gets rid of default border in Firefox and Opera. */\n",
" border: none;\n",
" /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
" background-size: auto;\n",
" }\n",
" .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
" background: #F44336;\n",
" }\n",
" </style>\n",
" <progress value='76' class='' max='333', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
" 22.82% [76/333 00:11<00:40]\n",
" </div>\n",
" "
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"learn.fit_one_cycle(5, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7))"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [],
"source": [
"learn.save('unfrozen_2')"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"RNNLearner(data=TextClasDataBunch;\n",
"\n",
"Train: LabelList (143614 items)\n",
"x: TextList\n",
"xxbos xxmaj explanation \n",
" xxmaj why the edits made under my username xxmaj hardcore xxmaj metallica xxmaj fan were reverted ? xxmaj they were n't vandalisms , just closure on some gas after i voted at xxmaj new xxmaj york xxmaj dolls xxup fac . xxmaj and please do n't remove the template from the talk page since i 'm retired xxunk,xxbos xxmaj xxunk ! xxmaj he matches this background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ),xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about the formatting than the actual info .,xxbos \" \n",
" xxmaj more \n",
" i ca n't make any real suggestions on improvement - i wondered if the section statistics should be later on , or a subsection of \" \" types of accidents \" \" xxup -i think the references may need tidying so that they are all in the exact same format ie date format etc . i can do that later on , if no - one else does first - if you have any preferences for formatting style on references or want to do it yourself please let me know . \n",
" \n",
" xxmaj there appears to be a backlog on articles for review so i guess there may be a delay until a reviewer turns up . xxmaj it 's listed in the relevant form eg xxmaj wikipedia : xxmaj xxunk # xxmaj transport \",xxbos xxmaj you , sir , are my hero . xxmaj any chance you remember what page that 's on ?\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Valid: LabelList (15957 items)\n",
"x: TextList\n",
"xxbos \" \n",
" \n",
" xxmaj sockpuppetry case \n",
" \n",
" xxmaj you have been accused of sockpuppetry . xxmaj please refer to xxmaj wikipedia : xxmaj sockpuppet investigations / xxmaj xxunk for evidence . xxmaj please make sure you make yourself familiar with notes for the suspect before editing the evidence page . 77 \",xxbos xxmaj welcome ! \n",
" \n",
" xxmaj hello , xxmaj xxunk , and welcome to xxmaj wikipedia ! xxmaj thank you for your contributions . i hope you like the place and decide to stay . xxmaj here are a few good links for newcomers : \n",
" xxmaj the five pillars of xxmaj wikipedia \n",
" xxmaj how to edit a page \n",
" xxmaj help pages \n",
" xxmaj tutorial \n",
" xxmaj how to write a great article \n",
" xxmaj manual of xxmaj style \n",
" i hope you enjoy editing here and being a xxmaj wikipedian ! xxmaj please sign your name on talk pages using four tildes ( xxrep 4 ~ ) ; this will automatically produce your name and the date . xxmaj if you need help , check out xxmaj wikipedia : xxmaj where to ask a question , ask me on my talk page , or place { { helpme } } on your talk page and someone will show up shortly to answer your questions . xxmaj again , welcome !,xxbos \" \n",
" \n",
" xxmaj agree with xxmaj user : xxmaj xxunk : \" \" this is not an article about racial or ethnic purity but a list of xxmaj americans that have roots in xxmaj estonia . \" \" \",xxbos xxmaj taking xxmaj xxunk as an example , in the xxmaj discography section the album information for each of their 3 albums is pretty much the same thing that can be found on each individual album 's page . xxmaj the albums are already linked to . xxmaj so what i am suggesting is removing what is duplicative and migrating anything that does not appear for each individual album onto that respective album 's article page . xxmaj in the end , you would be left with a xxmaj discography section listing all 3 albums . \n",
" \n",
" i also noticed the xxmaj notes on selected pieces section ; this section be taken out of the xxmaj xxunk article and appear in the pages for all 3 albums containing only those notes relevant for that particular album . xxmaj as it appears right now , the reader can not tell where a song appears unless he / she scrolls up and searches . xxmaj it would be more relevant to have the specific information pertaining to songs appearing on particular albums on the respective album 's pages .,xxbos xxmaj if someone wants to boil that plot down , go for it .\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Test: None, model=SequentialRNN(\n",
" (0): MultiBatchEncoder(\n",
" (module): AWD_LSTM(\n",
" (encoder): Embedding(60004, 400, padding_idx=1)\n",
" (encoder_dp): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (rnns): ModuleList(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (2): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" )\n",
" (input_dp): RNNDropout()\n",
" (hidden_dps): ModuleList(\n",
" (0): RNNDropout()\n",
" (1): RNNDropout()\n",
" (2): RNNDropout()\n",
" )\n",
" )\n",
" )\n",
" (1): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
"), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of BCEWithLogitsLoss(), metrics=[functools.partial(<function accuracy_thresh at 0x7f4d9682e1e0>, thresh=0.4)], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
"learn: RNNLearner(data=TextClasDataBunch;\n",
"\n",
"Train: LabelList (143614 items)\n",
"x: TextList\n",
"xxbos xxmaj explanation \n",
" xxmaj why the edits made under my username xxmaj hardcore xxmaj metallica xxmaj fan were reverted ? xxmaj they were n't vandalisms , just closure on some gas after i voted at xxmaj new xxmaj york xxmaj dolls xxup fac . xxmaj and please do n't remove the template from the talk page since i 'm retired xxunk,xxbos xxmaj xxunk ! xxmaj he matches this background colour i 'm seemingly stuck with . xxmaj thanks . ( talk ) 21:51 , xxmaj january 11 , 2016 ( xxup utc ),xxbos xxmaj hey man , i 'm really not trying to edit war . xxmaj it 's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page . xxmaj he seems to care more about the formatting than the actual info .,xxbos \" \n",
" xxmaj more \n",
" i ca n't make any real suggestions on improvement - i wondered if the section statistics should be later on , or a subsection of \" \" types of accidents \" \" xxup -i think the references may need tidying so that they are all in the exact same format ie date format etc . i can do that later on , if no - one else does first - if you have any preferences for formatting style on references or want to do it yourself please let me know . \n",
" \n",
" xxmaj there appears to be a backlog on articles for review so i guess there may be a delay until a reviewer turns up . xxmaj it 's listed in the relevant form eg xxmaj wikipedia : xxmaj xxunk # xxmaj transport \",xxbos xxmaj you , sir , are my hero . xxmaj any chance you remember what page that 's on ?\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Valid: LabelList (15957 items)\n",
"x: TextList\n",
"xxbos \" \n",
" \n",
" xxmaj sockpuppetry case \n",
" \n",
" xxmaj you have been accused of sockpuppetry . xxmaj please refer to xxmaj wikipedia : xxmaj sockpuppet investigations / xxmaj xxunk for evidence . xxmaj please make sure you make yourself familiar with notes for the suspect before editing the evidence page . 77 \",xxbos xxmaj welcome ! \n",
" \n",
" xxmaj hello , xxmaj xxunk , and welcome to xxmaj wikipedia ! xxmaj thank you for your contributions . i hope you like the place and decide to stay . xxmaj here are a few good links for newcomers : \n",
" xxmaj the five pillars of xxmaj wikipedia \n",
" xxmaj how to edit a page \n",
" xxmaj help pages \n",
" xxmaj tutorial \n",
" xxmaj how to write a great article \n",
" xxmaj manual of xxmaj style \n",
" i hope you enjoy editing here and being a xxmaj wikipedian ! xxmaj please sign your name on talk pages using four tildes ( xxrep 4 ~ ) ; this will automatically produce your name and the date . xxmaj if you need help , check out xxmaj wikipedia : xxmaj where to ask a question , ask me on my talk page , or place { { helpme } } on your talk page and someone will show up shortly to answer your questions . xxmaj again , welcome !,xxbos \" \n",
" \n",
" xxmaj agree with xxmaj user : xxmaj xxunk : \" \" this is not an article about racial or ethnic purity but a list of xxmaj americans that have roots in xxmaj estonia . \" \" \",xxbos xxmaj taking xxmaj xxunk as an example , in the xxmaj discography section the album information for each of their 3 albums is pretty much the same thing that can be found on each individual album 's page . xxmaj the albums are already linked to . xxmaj so what i am suggesting is removing what is duplicative and migrating anything that does not appear for each individual album onto that respective album 's article page . xxmaj in the end , you would be left with a xxmaj discography section listing all 3 albums . \n",
" \n",
" i also noticed the xxmaj notes on selected pieces section ; this section be taken out of the xxmaj xxunk article and appear in the pages for all 3 albums containing only those notes relevant for that particular album . xxmaj as it appears right now , the reader can not tell where a song appears unless he / she scrolls up and searches . xxmaj it would be more relevant to have the specific information pertaining to songs appearing on particular albums on the respective album 's pages .,xxbos xxmaj if someone wants to boil that plot down , go for it .\n",
"y: MultiCategoryList\n",
",,,,\n",
"Path: /home/akash/personal_projects/kaggle/ToxicComments/data;\n",
"\n",
"Test: None, model=SequentialRNN(\n",
" (0): MultiBatchEncoder(\n",
" (module): AWD_LSTM(\n",
" (encoder): Embedding(60004, 400, padding_idx=1)\n",
" (encoder_dp): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
" (rnns): ModuleList(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (2): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" )\n",
" (input_dp): RNNDropout()\n",
" (hidden_dps): ModuleList(\n",
" (0): RNNDropout()\n",
" (1): RNNDropout()\n",
" (2): RNNDropout()\n",
" )\n",
" )\n",
" )\n",
" (1): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
"), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of BCEWithLogitsLoss(), metrics=[functools.partial(<function accuracy_thresh at 0x7f4d9682e1e0>, thresh=0.4)], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/home/akash/personal_projects/kaggle/ToxicComments/data'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[...], layer_groups=[Sequential(\n",
" (0): Embedding(60004, 400, padding_idx=1)\n",
" (1): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
")], add_time=True, silent=None)\n",
"alpha: 2.0\n",
"beta: 1.0], layer_groups=[Sequential(\n",
" (0): Embedding(60004, 400, padding_idx=1)\n",
" (1): EmbeddingDropout(\n",
" (emb): Embedding(60004, 400, padding_idx=1)\n",
" )\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(400, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 1150, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): WeightDropout(\n",
" (module): LSTM(1150, 400, batch_first=True)\n",
" )\n",
" (1): RNNDropout()\n",
"), Sequential(\n",
" (0): PoolingLinearClassifier(\n",
" (layers): Sequential(\n",
" (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (1): Dropout(p=0.2)\n",
" (2): Linear(in_features=1200, out_features=50, bias=True)\n",
" (3): ReLU(inplace)\n",
" (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (5): Dropout(p=0.1)\n",
" (6): Linear(in_features=50, out_features=6, bias=True)\n",
" )\n",
" )\n",
")], add_time=True, silent=None)"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn.load('unfrozen_2')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Evaluating The Model"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>text</th>\n",
" <th>target</th>\n",
" <th>prediction</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>xxbos xxup suck xxup my xxup cock d xxup suck xxup my xxup cock d xxup suck xxup my xxup cock d xxup suck xxup my xxup cock d xxup suck xxup my xxup cock d xxup suck xxup my xxup cock d xxup suck xxup my xxup cock d xxup suck xxup my xxup cock d xxup suck xxup my xxup cock d xxup suck xxup my xxup cock</td>\n",
" <td>toxic;severe_toxic;obscene;insult</td>\n",
" <td>toxic;severe_toxic;obscene;insult</td>\n",
" </tr>\n",
" <tr>\n",
" <td>xxbos xxup damn xxup you u xxup cunt xxup damn xxup you u xxup cunt xxup damn xxup you u xxup cunt xxup damn xxup you u xxup cunt xxup damn xxup you u xxup cunt xxup damn xxup you u xxup cunt xxup damn xxup you u xxup cunt xxup damn xxup you u xxup cunt xxup damn xxup you u xxup cunt xxup damn xxup you u xxup</td>\n",
" <td>toxic;severe_toxic;obscene;insult</td>\n",
" <td>toxic;severe_toxic;obscene;insult;identity_hate</td>\n",
" </tr>\n",
" <tr>\n",
" <td>xxbos do go fuck off bastard \\n xxmaj do xxmaj yyou xxmaj have a life ? \\n go fuck off bastard and yank your cock through your ass . i hate you and hope you go away forever . lame is you fuck your mom . die die die and all that crap . this is for xxunk xxunk \\n ass . i ass . i ass</td>\n",
" <td>toxic;severe_toxic;obscene;threat;insult</td>\n",
" <td>toxic;severe_toxic;obscene;threat;insult</td>\n",
" </tr>\n",
" <tr>\n",
" <td>xxbos xxup shut xxup the xxup fuck xxup up ! \\n xxup shut xxup the xxup fuck xxup up ! \\n xxup shut xxup the xxup fuck xxup up ! \\n xxup shut xxup the xxup fuck xxup up ! \\n xxup shut xxup the xxup fuck xxup up ! \\n xxup shut xxup the xxup fuck xxup up ! \\n xxup shut xxup</td>\n",
" <td>toxic;severe_toxic;obscene</td>\n",
" <td>toxic;severe_toxic;obscene;insult</td>\n",
" </tr>\n",
" <tr>\n",
" <td>xxbos xxup wikipedia xxup loves xxup me . xxup they xxup like xxup to xxup censor xxup me . xxup communism xxup is xxup censorship . xxup wikipedia xxup is xxup communism . \\n xxup wikipedia xxup loves xxup me . xxup they xxup like xxup to xxup censor xxup me . xxup communism xxup is xxup censorship . xxup wikipedia xxup is xxup communism . \\n xxup</td>\n",
" <td>toxic;severe_toxic</td>\n",
" <td>toxic</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"learn.show_results()"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.9871003646219655"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"roc_score()"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[0.0441175, tensor(0.9826)]"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn.validate()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.6"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment