Skip to content

Instantly share code, notes, and snippets.

@i4kimura

i4kimura/ゼロから作るディープラーニング② 6章 ゲート付きRNN.ipynb

Last active August 11, 2018 17:17
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save i4kimura/469436243ec82757d3e98c01d3174c2d to your computer and use it in GitHub Desktop.
Save i4kimura/469436243ec82757d3e98c01d3174c2d to your computer and use it in GitHub Desktop.
Download ZIP
ゼロから作るディープラーニング② 第6章 ゲート付きRNN
Display the source blob
Display the rendered blob
Raw
ゼロから作るディープラーニング② 6章 ゲート付きRNN.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 6章 ゲート付きRNN\n",
"\n",
"5章のRNNは比較的シンプルなRNNであり、実際にはRNNといえばLSTMやGRUであることが多い。\n",
"- シンプルな5章のRNNは「エルマン(Elman)」と呼ばれる。\n",
"\n",
"## 6.1 RNNの問題点\n",
"\n",
"BPTT(Backpropagation Through Time)において、勾配消失もしくは勾配爆発が起きることが問題になる。\n",
"\n",
"### 6.1.1 RNNの復習\n",
"\n",
"過去の情報を記録される$h_t$はRNNの **隠れ状態(hidden state)** と呼ばれる。\n",
"\n",
"### 6.1.2 勾配消失もしくは勾配爆発\n",
"\n",
"RNNが学習するとき、遠いコンテキストに学習結果を伝える必要がある。\n",
"シンプルなRNNを使用すると、時間をさかのぼるに従って\n",
"- 勾配が小さくなってしまう(勾配消失)\n",
"- 勾配が大きくなってしまう(勾配爆発)\n",
"という状態が発生する。\n",
"\n",
"これは、RNNの逆伝搬において$\\tanh$が使用されており、この$\\tanh$を通過するたびに勾配は弱められることになる。\n",
"さらに、MatMulノードを通過するたびに勾配の大きさの変化は大きくなり、勾配爆発または勾配消失が発生する。"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"\n",
"N = 2 # ミニバッチサイズ\n",
"H = 3 # 隠れ状態ベクトルの次元数\n",
"T = 20 # 時系列データの長さ\n",
"\n",
"dh = np.ones((N, H))\n",
"np.random.seed(3) # 再現性のため乱数のシードを固定\n",
"Wh = np.random.randn(H, H)\n",
"\n",
"norm_list = []\n",
"for t in range(T):\n",
" dh = np.dot(dh, Wh.T)\n",
" norm = np.sqrt(np.sum(dh**2)) / N\n",
" norm_list.append(norm)\n",
" \n",
"plt.plot( norm)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 6.1.4 勾配爆発への対策\n",
"\n",
"**勾配クリッピング(gradients clipping)**と呼ばれる手法。しきい値を超えた場合は、値を小さくする。\n",
"\n",
"```python\n",
"def clip_grads(grads, max_norm):\n",
" total_norm = 0\n",
" for grad in grads:\n",
" total_norm += np.sum(grad ** 2)\n",
" total_norm = np.sqrt(total_norm)\n",
"\n",
" rate = max_norm / (total_norm + 1e-6)\n",
" if rate < 1:\n",
" for grad in grads:\n",
" grad *= rate\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6.2 勾配消失とLSTM\n",
"\n",
"勾配消失を解決するためには、「ゲート付きRNN」を使用する必要がある。\n",
"\n",
"### 6.2.1 LSTMのインタフェース\n",
"\n",
"LSTMのインタフェースでは、$\\tanh(\\bf h_{t-1}W_h+x_tW_x+b)$という計算を、ひとつの$\\tanh$というノードで表す。\n",
"\n",
"- outputゲート\n",
"- forgetゲート\n",
"- 新しい記憶セル\n",
"- inputゲート\n",
"\n",
"を挿入する。"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6.3 LSTMの実装\n",
"\n",
"```python\n",
"class LSTM:\n",
"...\n",
" def forward(self, x, h_prev, c_prev):\n",
" Wx, Wh, b = self.params\n",
" N, H = h_prev.shape\n",
"\n",
" A = np.dot(x, Wx) + np.dot(h_prev, Wh) + b\n",
"\n",
" f = A[:, :H]\n",
" g = A[:, H:2*H]\n",
" i = A[:, 2*H:3*H]\n",
" o = A[:, 3*H:]\n",
"\n",
" f = sigmoid(f)\n",
" g = np.tanh(g)\n",
" i = sigmoid(i)\n",
" o = sigmoid(o)\n",
"\n",
" c_next = f * c_prev + g * i\n",
" h_next = o * np.tanh(c_next)\n",
"\n",
" self.cache = (x, h_prev, c_prev, i, f, g, o, c_next)\n",
" return h_next, c_next\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6.4 LSTMを使った言語モデル\n",
"\n",
"これまでのTimeRNNレイヤだった場所に、TimeLSTMを挿入する。"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"| epoch 1 | iter 1 / 1327 | time 0[s] | perplexity 10001.53\n",
"| epoch 1 | iter 21 / 1327 | time 4[s] | perplexity 3369.27\n",
"| epoch 1 | iter 41 / 1327 | time 8[s] | perplexity 1270.10\n",
"| epoch 1 | iter 61 / 1327 | time 12[s] | perplexity 985.67\n",
"| epoch 1 | iter 81 / 1327 | time 16[s] | perplexity 794.73\n",
"| epoch 1 | iter 101 / 1327 | time 20[s] | perplexity 680.45\n",
"| epoch 1 | iter 121 / 1327 | time 24[s] | perplexity 658.45\n",
"| epoch 1 | iter 141 / 1327 | time 29[s] | perplexity 619.76\n",
"| epoch 1 | iter 161 / 1327 | time 32[s] | perplexity 606.06\n",
"| epoch 1 | iter 181 / 1327 | time 36[s] | perplexity 584.49\n",
"| epoch 1 | iter 201 / 1327 | time 40[s] | perplexity 521.27\n",
"| epoch 1 | iter 221 / 1327 | time 44[s] | perplexity 503.49\n",
"| epoch 1 | iter 241 / 1327 | time 49[s] | perplexity 450.23\n",
"| epoch 1 | iter 261 / 1327 | time 53[s] | perplexity 478.97\n",
"| epoch 1 | iter 281 / 1327 | time 57[s] | perplexity 448.23\n",
"| epoch 1 | iter 301 / 1327 | time 61[s] | perplexity 393.59\n",
"| epoch 1 | iter 321 / 1327 | time 65[s] | perplexity 358.08\n",
"| epoch 1 | iter 341 / 1327 | time 69[s] | perplexity 404.20\n",
"| epoch 1 | iter 361 / 1327 | time 73[s] | perplexity 413.65\n",
"| epoch 1 | iter 381 / 1327 | time 77[s] | perplexity 344.77\n",
"| epoch 1 | iter 401 / 1327 | time 81[s] | perplexity 360.71\n",
"| epoch 1 | iter 421 / 1327 | time 85[s] | perplexity 348.61\n",
"| epoch 1 | iter 441 / 1327 | time 89[s] | perplexity 333.02\n",
"| epoch 1 | iter 461 / 1327 | time 93[s] | perplexity 326.15\n",
"| epoch 1 | iter 481 / 1327 | time 97[s] | perplexity 309.48\n",
"| epoch 1 | iter 501 / 1327 | time 101[s] | perplexity 320.86\n",
"| epoch 1 | iter 521 / 1327 | time 105[s] | perplexity 304.07\n",
"| epoch 1 | iter 541 / 1327 | time 109[s] | perplexity 317.89\n",
"| epoch 1 | iter 561 / 1327 | time 113[s] | perplexity 289.84\n",
"| epoch 1 | iter 581 / 1327 | time 118[s] | perplexity 260.17\n",
"| epoch 1 | iter 601 / 1327 | time 125[s] | perplexity 336.76\n",
"| epoch 1 | iter 621 / 1327 | time 131[s] | perplexity 317.84\n",
"| epoch 1 | iter 641 / 1327 | time 136[s] | perplexity 283.07\n",
"| epoch 1 | iter 661 / 1327 | time 140[s] | perplexity 271.58\n",
"| epoch 1 | iter 681 / 1327 | time 146[s] | perplexity 230.88\n",
"| epoch 1 | iter 701 / 1327 | time 152[s] | perplexity 250.57\n",
"| epoch 1 | iter 721 / 1327 | time 157[s] | perplexity 259.89\n",
"| epoch 1 | iter 741 / 1327 | time 161[s] | perplexity 221.75\n",
"| epoch 1 | iter 761 / 1327 | time 165[s] | perplexity 234.70\n",
"| epoch 1 | iter 781 / 1327 | time 169[s] | perplexity 219.92\n",
"| epoch 1 | iter 801 / 1327 | time 173[s] | perplexity 240.31\n",
"| epoch 1 | iter 821 / 1327 | time 177[s] | perplexity 224.74\n",
"| epoch 1 | iter 841 / 1327 | time 181[s] | perplexity 229.62\n",
"| epoch 1 | iter 861 / 1327 | time 186[s] | perplexity 222.55\n",
"| epoch 1 | iter 881 / 1327 | time 191[s] | perplexity 205.47\n",
"| epoch 1 | iter 901 / 1327 | time 196[s] | perplexity 256.40\n",
"| epoch 1 | iter 921 / 1327 | time 201[s] | perplexity 229.65\n",
"| epoch 1 | iter 941 / 1327 | time 207[s] | perplexity 229.20\n",
"| epoch 1 | iter 961 / 1327 | time 212[s] | perplexity 245.76\n",
"| epoch 1 | iter 981 / 1327 | time 218[s] | perplexity 230.29\n",
"| epoch 1 | iter 1001 / 1327 | time 223[s] | perplexity 193.78\n",
"| epoch 1 | iter 1021 / 1327 | time 228[s] | perplexity 226.22\n",
"| epoch 1 | iter 1041 / 1327 | time 233[s] | perplexity 209.86\n",
"| epoch 1 | iter 1061 / 1327 | time 238[s] | perplexity 198.70\n",
"| epoch 1 | iter 1081 / 1327 | time 243[s] | perplexity 168.97\n",
"| epoch 1 | iter 1101 / 1327 | time 248[s] | perplexity 192.36\n",
"| epoch 1 | iter 1121 / 1327 | time 253[s] | perplexity 229.63\n",
"| epoch 1 | iter 1141 / 1327 | time 258[s] | perplexity 206.56\n",
"| epoch 1 | iter 1161 / 1327 | time 262[s] | perplexity 199.81\n",
"| epoch 1 | iter 1181 / 1327 | time 267[s] | perplexity 191.34\n",
"| epoch 1 | iter 1201 / 1327 | time 272[s] | perplexity 163.47\n",
"| epoch 1 | iter 1221 / 1327 | time 277[s] | perplexity 161.37\n",
"| epoch 1 | iter 1241 / 1327 | time 281[s] | perplexity 187.98\n",
"| epoch 1 | iter 1261 / 1327 | time 286[s] | perplexity 172.95\n",
"| epoch 1 | iter 1281 / 1327 | time 291[s] | perplexity 180.20\n",
"| epoch 1 | iter 1301 / 1327 | time 296[s] | perplexity 222.08\n",
"| epoch 1 | iter 1321 / 1327 | time 301[s] | perplexity 210.05\n",
"| epoch 2 | iter 1 / 1327 | time 302[s] | perplexity 223.75\n",
"| epoch 2 | iter 21 / 1327 | time 307[s] | perplexity 203.74\n",
"| epoch 2 | iter 41 / 1327 | time 312[s] | perplexity 189.99\n",
"| epoch 2 | iter 61 / 1327 | time 317[s] | perplexity 177.51\n",
"| epoch 2 | iter 81 / 1327 | time 322[s] | perplexity 160.12\n",
"| epoch 2 | iter 101 / 1327 | time 327[s] | perplexity 152.91\n",
"| epoch 2 | iter 121 / 1327 | time 332[s] | perplexity 160.44\n",
"| epoch 2 | iter 141 / 1327 | time 337[s] | perplexity 178.88\n",
"| epoch 2 | iter 161 / 1327 | time 341[s] | perplexity 193.38\n",
"| epoch 2 | iter 181 / 1327 | time 346[s] | perplexity 199.88\n",
"| epoch 2 | iter 201 / 1327 | time 350[s] | perplexity 184.36\n",
"| epoch 2 | iter 221 / 1327 | time 354[s] | perplexity 183.61\n",
"| epoch 2 | iter 241 / 1327 | time 359[s] | perplexity 177.71\n",
"| epoch 2 | iter 261 / 1327 | time 364[s] | perplexity 185.46\n",
"| epoch 2 | iter 281 / 1327 | time 368[s] | perplexity 185.07\n",
"| epoch 2 | iter 301 / 1327 | time 373[s] | perplexity 166.49\n",
"| epoch 2 | iter 321 / 1327 | time 377[s] | perplexity 139.45\n",
"| epoch 2 | iter 341 / 1327 | time 382[s] | perplexity 173.59\n",
"| epoch 2 | iter 361 / 1327 | time 386[s] | perplexity 196.56\n",
"| epoch 2 | iter 381 / 1327 | time 390[s] | perplexity 152.65\n",
"| epoch 2 | iter 401 / 1327 | time 395[s] | perplexity 167.52\n",
"| epoch 2 | iter 421 / 1327 | time 399[s] | perplexity 153.49\n",
"| epoch 2 | iter 441 / 1327 | time 404[s] | perplexity 162.79\n",
"| epoch 2 | iter 461 / 1327 | time 408[s] | perplexity 158.78\n",
"| epoch 2 | iter 481 / 1327 | time 412[s] | perplexity 156.82\n",
"| epoch 2 | iter 501 / 1327 | time 417[s] | perplexity 168.73\n",
"| epoch 2 | iter 521 / 1327 | time 421[s] | perplexity 174.60\n",
"| epoch 2 | iter 541 / 1327 | time 426[s] | perplexity 175.13\n",
"| epoch 2 | iter 561 / 1327 | time 430[s] | perplexity 154.33\n",
"| epoch 2 | iter 581 / 1327 | time 435[s] | perplexity 138.94\n",
"| epoch 2 | iter 601 / 1327 | time 439[s] | perplexity 190.57\n",
"| epoch 2 | iter 621 / 1327 | time 444[s] | perplexity 181.79\n",
"| epoch 2 | iter 641 / 1327 | time 448[s] | perplexity 164.16\n",
"| epoch 2 | iter 661 / 1327 | time 452[s] | perplexity 154.69\n",
"| epoch 2 | iter 681 / 1327 | time 457[s] | perplexity 129.25\n",
"| epoch 2 | iter 701 / 1327 | time 461[s] | perplexity 149.68\n",
"| epoch 2 | iter 721 / 1327 | time 466[s] | perplexity 160.68\n",
"| epoch 2 | iter 741 / 1327 | time 470[s] | perplexity 132.86\n",
"| epoch 2 | iter 761 / 1327 | time 475[s] | perplexity 130.31\n",
"| epoch 2 | iter 781 / 1327 | time 479[s] | perplexity 135.17\n",
"| epoch 2 | iter 801 / 1327 | time 484[s] | perplexity 147.12\n",
"| epoch 2 | iter 821 / 1327 | time 488[s] | perplexity 143.79\n",
"| epoch 2 | iter 841 / 1327 | time 493[s] | perplexity 143.31\n",
"| epoch 2 | iter 861 / 1327 | time 498[s] | perplexity 144.79\n",
"| epoch 2 | iter 881 / 1327 | time 503[s] | perplexity 131.04\n",
"| epoch 2 | iter 901 / 1327 | time 508[s] | perplexity 165.02\n",
"| epoch 2 | iter 921 / 1327 | time 513[s] | perplexity 148.06\n",
"| epoch 2 | iter 941 / 1327 | time 518[s] | perplexity 153.83\n",
"| epoch 2 | iter 961 / 1327 | time 523[s] | perplexity 165.04\n",
"| epoch 2 | iter 981 / 1327 | time 528[s] | perplexity 153.31\n",
"| epoch 2 | iter 1001 / 1327 | time 533[s] | perplexity 132.15\n",
"| epoch 2 | iter 1021 / 1327 | time 538[s] | perplexity 156.56\n",
"| epoch 2 | iter 1041 / 1327 | time 543[s] | perplexity 141.93\n",
"| epoch 2 | iter 1061 / 1327 | time 548[s] | perplexity 128.13\n",
"| epoch 2 | iter 1081 / 1327 | time 554[s] | perplexity 110.03\n",
"| epoch 2 | iter 1101 / 1327 | time 559[s] | perplexity 119.79\n",
"| epoch 2 | iter 1121 / 1327 | time 563[s] | perplexity 152.99\n",
"| epoch 2 | iter 1141 / 1327 | time 568[s] | perplexity 141.46\n",
"| epoch 2 | iter 1161 / 1327 | time 572[s] | perplexity 133.02\n",
"| epoch 2 | iter 1181 / 1327 | time 577[s] | perplexity 133.30\n",
"| epoch 2 | iter 1201 / 1327 | time 582[s] | perplexity 112.68\n",
"| epoch 2 | iter 1221 / 1327 | time 587[s] | perplexity 109.20\n",
"| epoch 2 | iter 1241 / 1327 | time 591[s] | perplexity 130.53\n",
"| epoch 2 | iter 1261 / 1327 | time 596[s] | perplexity 124.27\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"| epoch 2 | iter 1281 / 1327 | time 600[s] | perplexity 122.96\n",
"| epoch 2 | iter 1301 / 1327 | time 605[s] | perplexity 157.10\n",
"| epoch 2 | iter 1321 / 1327 | time 609[s] | perplexity 153.77\n",
"| epoch 3 | iter 1 / 1327 | time 611[s] | perplexity 159.04\n",
"| epoch 3 | iter 21 / 1327 | time 615[s] | perplexity 144.00\n",
"| epoch 3 | iter 41 / 1327 | time 620[s] | perplexity 135.04\n",
"| epoch 3 | iter 61 / 1327 | time 625[s] | perplexity 126.73\n",
"| epoch 3 | iter 81 / 1327 | time 629[s] | perplexity 116.52\n",
"| epoch 3 | iter 101 / 1327 | time 634[s] | perplexity 105.99\n",
"| epoch 3 | iter 121 / 1327 | time 638[s] | perplexity 116.02\n",
"| epoch 3 | iter 141 / 1327 | time 643[s] | perplexity 126.64\n",
"| epoch 3 | iter 161 / 1327 | time 647[s] | perplexity 142.06\n",
"| epoch 3 | iter 181 / 1327 | time 652[s] | perplexity 148.62\n",
"| epoch 3 | iter 201 / 1327 | time 656[s] | perplexity 141.37\n",
"| epoch 3 | iter 221 / 1327 | time 660[s] | perplexity 140.41\n",
"| epoch 3 | iter 241 / 1327 | time 665[s] | perplexity 135.36\n",
"| epoch 3 | iter 261 / 1327 | time 670[s] | perplexity 139.09\n",
"| epoch 3 | iter 281 / 1327 | time 674[s] | perplexity 141.31\n",
"| epoch 3 | iter 301 / 1327 | time 679[s] | perplexity 123.63\n",
"| epoch 3 | iter 321 / 1327 | time 683[s] | perplexity 101.11\n",
"| epoch 3 | iter 341 / 1327 | time 687[s] | perplexity 123.65\n",
"| epoch 3 | iter 361 / 1327 | time 692[s] | perplexity 151.25\n",
"| epoch 3 | iter 381 / 1327 | time 696[s] | perplexity 114.11\n",
"| epoch 3 | iter 401 / 1327 | time 701[s] | perplexity 129.45\n",
"| epoch 3 | iter 421 / 1327 | time 705[s] | perplexity 113.48\n",
"| epoch 3 | iter 441 / 1327 | time 710[s] | perplexity 123.47\n",
"| epoch 3 | iter 461 / 1327 | time 714[s] | perplexity 119.60\n",
"| epoch 3 | iter 481 / 1327 | time 719[s] | perplexity 118.64\n",
"| epoch 3 | iter 501 / 1327 | time 723[s] | perplexity 128.42\n",
"| epoch 3 | iter 521 / 1327 | time 728[s] | perplexity 138.72\n",
"| epoch 3 | iter 541 / 1327 | time 732[s] | perplexity 135.22\n",
"| epoch 3 | iter 561 / 1327 | time 737[s] | perplexity 117.48\n",
"| epoch 3 | iter 581 / 1327 | time 742[s] | perplexity 105.71\n",
"| epoch 3 | iter 601 / 1327 | time 746[s] | perplexity 147.57\n",
"| epoch 3 | iter 621 / 1327 | time 751[s] | perplexity 141.62\n",
"| epoch 3 | iter 641 / 1327 | time 755[s] | perplexity 129.66\n",
"| epoch 3 | iter 661 / 1327 | time 759[s] | perplexity 120.43\n",
"| epoch 3 | iter 681 / 1327 | time 764[s] | perplexity 99.94\n",
"| epoch 3 | iter 701 / 1327 | time 768[s] | perplexity 118.26\n",
"| epoch 3 | iter 721 / 1327 | time 773[s] | perplexity 126.15\n",
"| epoch 3 | iter 741 / 1327 | time 778[s] | perplexity 106.56\n",
"| epoch 3 | iter 761 / 1327 | time 782[s] | perplexity 103.61\n",
"| epoch 3 | iter 781 / 1327 | time 787[s] | perplexity 103.21\n",
"| epoch 3 | iter 801 / 1327 | time 791[s] | perplexity 114.00\n",
"| epoch 3 | iter 821 / 1327 | time 796[s] | perplexity 115.18\n",
"| epoch 3 | iter 841 / 1327 | time 800[s] | perplexity 114.29\n",
"| epoch 3 | iter 861 / 1327 | time 805[s] | perplexity 118.70\n",
"| epoch 3 | iter 881 / 1327 | time 809[s] | perplexity 106.59\n",
"| epoch 3 | iter 901 / 1327 | time 814[s] | perplexity 130.60\n",
"| epoch 3 | iter 921 / 1327 | time 818[s] | perplexity 119.56\n",
"| epoch 3 | iter 941 / 1327 | time 822[s] | perplexity 126.76\n",
"| epoch 3 | iter 961 / 1327 | time 827[s] | perplexity 132.62\n",
"| epoch 3 | iter 981 / 1327 | time 831[s] | perplexity 122.90\n",
"| epoch 3 | iter 1001 / 1327 | time 836[s] | perplexity 109.57\n",
"| epoch 3 | iter 1021 / 1327 | time 842[s] | perplexity 128.46\n",
"| epoch 3 | iter 1041 / 1327 | time 848[s] | perplexity 118.44\n",
"| epoch 3 | iter 1061 / 1327 | time 853[s] | perplexity 102.22\n",
"| epoch 3 | iter 1081 / 1327 | time 857[s] | perplexity 87.85\n",
"| epoch 3 | iter 1101 / 1327 | time 864[s] | perplexity 95.52\n",
"| epoch 3 | iter 1121 / 1327 | time 869[s] | perplexity 120.50\n",
"| epoch 3 | iter 1141 / 1327 | time 874[s] | perplexity 114.41\n",
"| epoch 3 | iter 1161 / 1327 | time 880[s] | perplexity 107.33\n",
"| epoch 3 | iter 1181 / 1327 | time 885[s] | perplexity 110.84\n",
"| epoch 3 | iter 1201 / 1327 | time 890[s] | perplexity 93.96\n",
"| epoch 3 | iter 1221 / 1327 | time 895[s] | perplexity 88.58\n",
"| epoch 3 | iter 1241 / 1327 | time 900[s] | perplexity 105.31\n",
"| epoch 3 | iter 1261 / 1327 | time 904[s] | perplexity 105.53\n",
"| epoch 3 | iter 1281 / 1327 | time 909[s] | perplexity 100.72\n",
"| epoch 3 | iter 1301 / 1327 | time 914[s] | perplexity 130.49\n",
"| epoch 3 | iter 1321 / 1327 | time 919[s] | perplexity 127.11\n",
"| epoch 4 | iter 1 / 1327 | time 920[s] | perplexity 132.73\n",
"| epoch 4 | iter 21 / 1327 | time 925[s] | perplexity 121.07\n",
"| epoch 4 | iter 41 / 1327 | time 929[s] | perplexity 106.88\n",
"| epoch 4 | iter 61 / 1327 | time 934[s] | perplexity 106.00\n",
"| epoch 4 | iter 81 / 1327 | time 939[s] | perplexity 95.54\n",
"| epoch 4 | iter 101 / 1327 | time 944[s] | perplexity 86.23\n",
"| epoch 4 | iter 121 / 1327 | time 949[s] | perplexity 94.56\n",
"| epoch 4 | iter 141 / 1327 | time 953[s] | perplexity 103.08\n",
"| epoch 4 | iter 161 / 1327 | time 958[s] | perplexity 118.39\n",
"| epoch 4 | iter 181 / 1327 | time 963[s] | perplexity 127.76\n",
"| epoch 4 | iter 201 / 1327 | time 967[s] | perplexity 119.95\n",
"| epoch 4 | iter 221 / 1327 | time 971[s] | perplexity 121.68\n",
"| epoch 4 | iter 241 / 1327 | time 976[s] | perplexity 114.74\n",
"| epoch 4 | iter 261 / 1327 | time 981[s] | perplexity 114.72\n",
"| epoch 4 | iter 281 / 1327 | time 985[s] | perplexity 120.67\n",
"| epoch 4 | iter 301 / 1327 | time 990[s] | perplexity 103.55\n",
"| epoch 4 | iter 321 / 1327 | time 995[s] | perplexity 83.56\n",
"| epoch 4 | iter 341 / 1327 | time 999[s] | perplexity 100.04\n",
"| epoch 4 | iter 361 / 1327 | time 1004[s] | perplexity 127.70\n",
"| epoch 4 | iter 381 / 1327 | time 1009[s] | perplexity 96.72\n",
"| epoch 4 | iter 401 / 1327 | time 1013[s] | perplexity 109.85\n",
"| epoch 4 | iter 421 / 1327 | time 1018[s] | perplexity 94.10\n",
"| epoch 4 | iter 441 / 1327 | time 1022[s] | perplexity 102.51\n",
"| epoch 4 | iter 461 / 1327 | time 1027[s] | perplexity 99.95\n",
"| epoch 4 | iter 481 / 1327 | time 1032[s] | perplexity 101.84\n",
"| epoch 4 | iter 501 / 1327 | time 1036[s] | perplexity 108.18\n",
"| epoch 4 | iter 521 / 1327 | time 1041[s] | perplexity 117.74\n",
"| epoch 4 | iter 541 / 1327 | time 1045[s] | perplexity 111.75\n",
"| epoch 4 | iter 561 / 1327 | time 1050[s] | perplexity 101.40\n",
"| epoch 4 | iter 581 / 1327 | time 1055[s] | perplexity 89.76\n",
"| epoch 4 | iter 601 / 1327 | time 1060[s] | perplexity 126.00\n",
"| epoch 4 | iter 621 / 1327 | time 1064[s] | perplexity 120.89\n",
"| epoch 4 | iter 641 / 1327 | time 1069[s] | perplexity 109.96\n",
"| epoch 4 | iter 661 / 1327 | time 1074[s] | perplexity 102.95\n",
"| epoch 4 | iter 681 / 1327 | time 1079[s] | perplexity 84.85\n",
"| epoch 4 | iter 701 / 1327 | time 1084[s] | perplexity 101.62\n",
"| epoch 4 | iter 721 / 1327 | time 1089[s] | perplexity 107.78\n",
"| epoch 4 | iter 741 / 1327 | time 1094[s] | perplexity 95.20\n",
"| epoch 4 | iter 761 / 1327 | time 1098[s] | perplexity 88.72\n",
"| epoch 4 | iter 781 / 1327 | time 1103[s] | perplexity 87.57\n",
"| epoch 4 | iter 801 / 1327 | time 1108[s] | perplexity 97.64\n",
"| epoch 4 | iter 821 / 1327 | time 1112[s] | perplexity 102.00\n",
"| epoch 4 | iter 841 / 1327 | time 1116[s] | perplexity 98.01\n",
"| epoch 4 | iter 861 / 1327 | time 1121[s] | perplexity 103.25\n",
"| epoch 4 | iter 881 / 1327 | time 1125[s] | perplexity 92.42\n",
"| epoch 4 | iter 901 / 1327 | time 1130[s] | perplexity 114.35\n",
"| epoch 4 | iter 921 / 1327 | time 1134[s] | perplexity 104.32\n",
"| epoch 4 | iter 941 / 1327 | time 1139[s] | perplexity 112.19\n",
"| epoch 4 | iter 961 / 1327 | time 1143[s] | perplexity 112.16\n",
"| epoch 4 | iter 981 / 1327 | time 1148[s] | perplexity 106.37\n",
"| epoch 4 | iter 1001 / 1327 | time 1152[s] | perplexity 97.07\n",
"| epoch 4 | iter 1021 / 1327 | time 1157[s] | perplexity 112.89\n",
"| epoch 4 | iter 1041 / 1327 | time 1161[s] | perplexity 103.65\n",
"| epoch 4 | iter 1061 / 1327 | time 1166[s] | perplexity 88.32\n",
"| epoch 4 | iter 1081 / 1327 | time 1170[s] | perplexity 77.75\n",
"| epoch 4 | iter 1101 / 1327 | time 1175[s] | perplexity 79.79\n",
"| epoch 4 | iter 1121 / 1327 | time 1179[s] | perplexity 102.84\n",
"| epoch 4 | iter 1141 / 1327 | time 1183[s] | perplexity 99.15\n",
"| epoch 4 | iter 1161 / 1327 | time 1188[s] | perplexity 91.90\n",
"| epoch 4 | iter 1181 / 1327 | time 1192[s] | perplexity 95.42\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"| epoch 4 | iter 1201 / 1327 | time 1197[s] | perplexity 83.07\n",
"| epoch 4 | iter 1221 / 1327 | time 1201[s] | perplexity 76.08\n",
"| epoch 4 | iter 1241 / 1327 | time 1206[s] | perplexity 91.78\n",
"| epoch 4 | iter 1261 / 1327 | time 1210[s] | perplexity 94.18\n",
"| epoch 4 | iter 1281 / 1327 | time 1215[s] | perplexity 88.94\n",
"| epoch 4 | iter 1301 / 1327 | time 1219[s] | perplexity 111.88\n",
"| epoch 4 | iter 1321 / 1327 | time 1223[s] | perplexity 110.80\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8XGeV8PHfmS6NerVkuXcncUmcHlKdQEiDLCUsC2FhCbCBpWXZAC8s7Lvsy9JhqYGwSSihJSEhhCROcZzqGnfHstzVe++a5/3j3juakUbSSNZY8uh8Px99NHPnzui5Hn/mzHnKecQYg1JKKTWUa6oboJRSanrSAKGUUiomDRBKKaVi0gChlFIqJg0QSimlYtIAoZRSKqaEBggROSYie0Rkp4hss4/liMgGETlk/862j4uI/EBEykRkt4icm8i2KaWUGt3pyCCuMsasMcass+/fDTxrjFkCPGvfB7geWGL/3AH85DS0TSml1AimoovpFuB++/b9wNsijj9gLK8BWSJSNAXtU0opBXgS/PoGeFpEDPAzY8w9QKExpsp+vBootG/PBk5GPLfcPlYVcQwRuQMrwyAYDJ63fPnyBDZ/UGlNGwGvm7k5qafl7ymlVKJs37693hiTP9Z5iQ4QlxljKkSkANggIm9EPmiMMXbwiJsdZO4BWLdundm2bdvktXYUN//wJbJTfdz/wQtOy99TSqlEEZHj8ZyX0C4mY0yF/bsWeAS4AKhxuo7s37X26RXAnIinl9jHpoWgz0Nnb/9UN0MppU6bhAUIEQmKSLpzG7gO2As8Btxun3Y78Kh9+zHg/fZspouAloiuqCkX9Ltp7xmY6mYopdRpk8gupkLgERFx/s5vjTFPishW4A8i8iHgOPAu+/wngLcCZUAn8I8JbNu4Bf2aQSilZpaEBQhjzBFgdYzjDcA1MY4b4M5EtedUBf0eOno0QCilZg5dSR2noM9NuwYIpdQMogEiTkG/h+6+EAMh3WBJKTUzaICIU9Bn9cbVt/fQ0N4zxa1RSqnE0wARp6DfChBffGQv77t3yxS3RimlEi/RC+WSRtDvBmBvRQtt3X1T3BqllEo8DRBxcrqYqlu7AejuGyDgdU9lk5RSKqG0iylOTheTo6Gjd4paopRSp4cGiDg5XUwOHahWSiU7DRBxGpZBtGsGoZRKbhog4pQ2JEDUawahlEpyGiDilOob0sWkYxBKqSSnASJOziwmR6MGCKVUktMAESeXS8JZRE7Qp11MSqmkp+sgxiHVziJKslN0kFoplfQ0QIxDmt9Nqs9NbtBHnWYQSqkkp11M45Dq85Cf7ic3za8ZhFIq6WkGMQ7/cNE8Al4XB2vaaGjvxRiDvWOeUkolHc0gxuHvL5zLreeWkBHw0jsQoqc/NNVNUkqphNEAMQHOornO3oEpbolSSiWOBogJcKa76h7VSqlkpgFiApy6TB29GiCUUslLA8QEhANEj3YxKaWSlwaICQjaXUydmkEopZKYBogJcFZU6xiEUiqZaYCYAGfzIO1iUkolMw0QExAMT3PVDEIplbw0QEyAU/q7XTMIpVQS0wAxAQGvC5doBqGUSm4aICZARAj6PDoGoZRKahogJijV79YMQimV1DRATFDQ56Fdp7kqpZKYBogJsjKIwS6mu/64iw37a6awRUopNbk0QEyQNQZhZRAnGjr50/Zy7vztjilulVJKTR4NEBMU9HvCxfpeOFQHwNnFGVPZJKWUmlQaICYo1eem057FtKnUChBFmSlT2SSllJpUCQ8QIuIWkddF5HH7/gIR2SwiZSLyexHx2cf99v0y+/H5iW7bqUizM4i+gRCvHm4AoLtPp70qpZLH6cggPgkciLj/38B3jTGLgSbgQ/bxDwFN9vHv2udNW6k+D509A9S394RnM3VpgFBKJZGEBggRKQFuAH5h3xfgauBP9in3A2+zb99i38d+/Br7/Gkp6HfT0dtPa9fgVFfNIJRSySTRGcT3gM8BIft+LtBsjHE+VcuB2fbt2cBJAPvxFvv8KCJyh4hsE5FtdXV1iWz7qIJ+DyED9e09APjcLrr6QmM8SymlzhwJCxAiciNQa4zZPpmva4y5xxizzhizLj8/fzJfelycTYOqW7oByE/3awahlEoqngS+9qXAzSLyViAAZADfB7JExGNnCSVAhX1+BTAHKBcRD5AJNCSwfafE2TSounUwQDjBQimlkkHCMghjzOeNMSXGmPnAbcBzxpj3As8D77BPux141L79mH0f+/HnjDEmUe07VZkpXgDKmzoBO4Po1wxCKZU8pmIdxL8BnxGRMqwxhnvt4/cCufbxzwB3T0Hb4pYdtALEycYuwAoQXb0aIJRSySORXUxhxpiNwEb79hHgghjndAPvPB3tmQxZqT4ATjR2IgJ5QR89/SFCIYPLNW0nXymlVNx0JfUEZdsBorK5izSfh1R7G1LtZlJKJQsNEBOUmeJFBPpDhrSAhxSvNaupW6e6KqWShAaICXK7hIyANQ6RHvAQ8Fr/lLqaWimVLDRAnILsVCdAeAnYGYQOVCulkoUGiFPgDFSn+SO7mDRAKKWSgwaIUzCYQXhI8WmAUEolFw0Qp8CZyWSNQdhdTBoglFJJQgPEKciMGINI0TEIpVSS0QBxCrIjxiA0g1BKJRsNEKcg1hhEj66DUEolCQ0QpyArPAbhJeCx/il/9dpxvv30walsllJKTQoNEKcgsovJySD2VLRwz6Yj9A1oJqGUOrNpgDgF58zO5OrlBZw7N4uAxx0+3tMf4o2qtilsmVJKnToNEKcgM9XLLz9wPgUZgWEVXHeebJqiViml1OTQAJEgr59onuomKKXUKdEAkQCrSzLZeVIDhFLqzKYBIgHWryjkSH0H7T39U90UpZSaMA0QCbB0VjoAZbXtU9wSpZSauNOy5ehM8fP3ryPoc1OUlQJAaU0ba+ZkTXGrlFJqYjRATKJrVxYCMBAy+D0uDtXoVFel1JlLu5gSwO0SFuWnUVozehdTKGTYeLAWY8xpaplSSsVPA0SCLC1MGzODeGxXJR/43638buvJ09QqpZSKnwaIBFlSmE5lSzctXX0jnuPMctqpayaUUtOQBogEuXBBDgAb9teMeE5milUNtqat+7S0SSmlxkMDRIKcNy+bBXlB/rBt5O6jnn6roF91iwYIpdT0owEiQUSEd5xXwpajjRxv6Ih5jrN/dU2rBgil1PSjASKB1q+wpr2OVJfJySCaOvt0JpNSatrRAJFA8/NScQkcrmvn+YO1VDR3RT3eHbE9aW1bz+lunlJKjUoDRAL5PW7m5QbZVd7CP92/jZ9uPExX70B49lJPRIA43tA5Vc1USqmYNEAk2KL8IC8dqmMgZCirbeeLj+zhg/+7FYDu/sFd5040aoBQSk0vGiASbFF+GiF7eOFwXTubjzZypN5aYd3TN4DH3miosUO7mJRS04vWYkqwRQVp4dvOOIMI9A+E6O4LkZvmo7Gjl8aOkRfUKaXUVNAMIsEW5VsBoigzED5mDDR29tLdP0DA6yYr1UdzZ+9UNVEppWLSAJFgy2els6wwnTuvWhx1vL6tl+6+AfweF9mpXpo0QCilppmEBQgRCYjIFhHZJSL7ROSr9vEFIrJZRMpE5Pci4rOP++37Zfbj8xPVttMp6Pfw1Kcv593nzwmPNwDUt/fQ3RcKZxBNndrFpJSaXhKZQfQAVxtjVgNrgLeIyEXAfwPfNcYsBpqAD9nnfwhoso9/1z4vaXjdLpYUprO00Opyqm/voad/gIDHTU6qj6YOzSCUUtNLwgKEsTgbInjtHwNcDfzJPn4/8Db79i32fezHrxGRwa/cSeAXt6/j5+9fB0Bdm5VB+L0usoNezSCUUtNOXAFCRLaLyJ0ikj2eFxcRt4jsBGqBDcBhoNkY02+fUg7Mtm/PBk4C2I+3ALkxXvMOEdkmItvq6urG05wpNzsrhbk5qfg9LruLaQC/Z3CQWsttKKWmk3gziHcDxcBWEfmdiLw5nm/3xpgBY8waoAS4AFg+8aaGX/MeY8w6Y8y6/Pz8U325005EyE/3U9/eS09/iIDXRU6qj/6QCa+wVkqp6SCuAGGMKTPGfBFYCvwW+CVwXES+KiI5cTy/GXgeuBjIEhFn/UUJUGHfrgDmANiPZwIN47iWM0Zemt8ag+hzprla+0I06VoIpdQ0EvcYhIisAr4NfBN4CHgn0Ao8N8L5+SKSZd9OAa4FDmAFinfYp90OPGrffsy+j/34cyZJ+1zy0vzWGISdQWSn+gB0qqtSalqJayW1iGwHmoF7gbuNMU5diM0icukITysC7hcRN1Yg+oMx5nER2Q/8TkT+E3jdfk3s378SkTKgEbhtQld0BshP97HzZBPdfdYspuygnUFogFBKTSPxltp4pzHmSOQBEVlgjDlqjLk11hOMMbuBtTGOH8Eajxh6vBsrK0l6+Wl+Gu1prX7NIJRS01S8XUx/ivOYikNeup+QgZDByiCcAKFjEEqpaWTUDEJElgNnAZkiEpkpZACB2M9SY8lL84dvB7xuMlK8uMRaPKeUUtPFWF1My4AbgSzgpojjbcCHE9WoZBcdIFy4XcLSwnT2VLRMYauUUiraqAHCGPMo8KiIXGyMefU0tSnp5acPBgi/xw3AefOyeWxnJQMhg9uVVAvIlVJnqLG6mD5njPkG8Pci8p6hjxtj/iVhLUtieWm+8G2/1xoGOm9eNr/ZfIJDtW0sn5UxVU1TSqmwsbqYDti/tyW6ITNJmt+D3+OyV1JbGcS6edZ6w23HmjRAKKWmhbG6mP5i3/y9PQ01TETyEtaqJOeU2yhv6goHiDk5KeSl+dl5spl/uGjeFLdQKaXin+a6xS7VDYCI/B3wSmKaNDM4A9V+j/UWiAizswLUtelMJqXU9BDvQrn3Ar8UkY1YRftyscp2qwlyAoSTQQBkpHhp7da1EEqp6SGuAGGM2SMiXwN+hTXF9XJjTHlCW5bk8tOtgeqAdzCJy0jxUtHcNVVNUkqpKPHWYroXWASswqro+riI/I8x5keJbFwyy3cyCE9EBhHw0tqlJb+VUtNDvGMQe4Cr7NpLTwEXAucmrlnJrygrBYC0wGCMzkjx0NrVhzGGx3dXcuF/PUN338BUNVEpNcPF28X0PRFJEZG5xpiDxpgWBveSVhPw9rWzmZeTGrWqOjPFS+9AiJ7+EB//7esA1Lb2MDc3dcTXMcbw1L5qrlpeEF50p5RSkyHeLUdvAnYCT9r314jIY4lsWLILeN1csjh6pnBGwCr77VR6BWjpGn3Q+nBdBx/99Q6e3Fs9+Y1USs1o8XYxfQWrRHczgDFmJ7AwQW2asTJTrADxUll9+Fhz1+glwFvsx6taukc9TymlxiveANFndytFCk12Y2a6DDtAPL67KnxsrAyivccao6ht1fUTSqnJFW+A2Ccifw+4RWSJiPwPulBu0jkZxI7jTaT6rPGE5s4xAkS3Neuptk0zCKXU5Io3QHwCa1+IHuBBrL2oP5WoRs1UGfaMpvaefs6enQmMnUF09DgBwsog7n3pKPdsOpzAViqlZop4ZzF1Al+0f1SCOBkEwJKCNHaXN8fRxWQFCKdEx282H6dvIMQdly9KXEOVUjPCWOW+/wKYkR43xtw86S2awdIDgwFibk4qmSlemiP2qd5+vIm8NB/zcoPhY04GUdPaTd9AiBMNnfSHDB09/QT98VZSUUqp4cb6BPnWaWmFAsDncZHiddPVN8DcnFSyUnxRYxCf+O0OVpVk8dP3nRc+1t5rBYjO3gH2V7bSH7LieVltO6vnZJ3eC1BKJZWxyn2/4NwWER+wHCujOGiMGX3+pZqQzBQvXX0DzMlJJTPVG+5i6u4boLKlG7+3Lep8J4MAeOVwQ/h2aU2bBgil1CmJd6HcDcBh4AfAD4EyEbk+kQ2bqTJSrJg9N9fqYjrW0MHF/+9ZHt1ZAcCxho6o8hvOLCaAV49YAcLndlFaEx1IlFJqvOLtpP42Vi2mMgARWQT8Ffhboho2U2WmeMlO9ZIR8JKV4qXGXt/w2y0nATDG6j5yZjm19wzgc7voHQjx6uF6ijIDZKf6KK1pn7JrUEolh3inubY5wcF2BKvst5pkC/PSOKfE6hqKnNW062Rz+Pb3njnEFx7ZA1hdTPPsWk19A4bFBWksLUyjrHbkAPHX3VV848k3EtF8pVQSiTeD2CYiTwB/wBqDeCewVURuBTDGPJyg9s04//n2swkZa6A5K9Ub9ZjP7cJgeOZADQDvu2geHb39FGelcPnSfF453MAN5xRxpL6Dho4eQiHDnoqWYWMRT+ypYuPBWv71zcsQkdNzYUqpM068GUQAqAGuAK4E6oAU4CbgxoS0bIbyul3hqqyZqdamQs5n+OzsFBbkWVNc3S7hoe3ltPf0kxbw8KUbV/K3T76J2y6YS1aql+6+EE/uq+aWH73MgarWqL/R3NVLR+8Ard3x7z1xsrGT/gGtrqLUTDJmBiEibmC3Mea7p6E9KoLTxfSmJflsKq2jJDuFa5YX0NjRyxvVbfx5ZyUAab7otzErxQos+yutwHCgqpUVRRnhx52ps1UtXVHdWEO1dffx191VXLOikKu/vZGvve0c3nX+nMm7QKXUtDZmBmGMGQDecxraooZYlB/E6xY+cMk8AObkpPKBSxfwmeuWsX5lIfXtPdS39wxbEOd0TR2pt8Yhho5HhANE8+j1m77/zCHufngP33+2lL4BwxvVOuyk1EwS7xjEyyLyQ+D3QIdz0BizIyGtUgCcVZzJvq++BY9LuHxpPlctKwg/tig/LXw7zR+9UZATIA7XWm/V0ADhrK2obBl9/2una+uh7dYU25NNnRO4CqXUmSreALHG/v0fEccMcPXkNkcN5fNYSd4DH7wg6vii/MFyG8MyCLuL6WiDHSDqBgNE30AoXL9prAwixWsFni573cXJRg0QSs0k8RbruyrRDVHjk5XqIzfoo6GjN2pfa+sxK4Po7bcGlY83dNLbH8LncUUV/xsrgxg6iH2ysRNjjM58UmqGiHcldaGI3Csif7PvrxQR3ZN6ii20s4i0IRlEtj37CaxuooGQ4bidTUTWdhorg2iNCCb56X46egeitkNVSiW3eKe53gc8BRTb90vR/SCm3MI8axwiOGQWU8DrCndNnVVszV46aJfecLYozQh4qBozgxgMELeunQ3AyabRn6OUSh7xBog8Y8wfsLcZNcb0AwOjP0Ul2qICK4MYOgYhImTb3UwXL8wl4HWx7VgTMJhBrCjKoLKlm4HQiNXcae3q56KFOWz5wjXcem4JACd0HEKpGSPeANEhIrnYe0OIyEXA0D2qo4jIHBF5XkT2i8g+EfmkfTxHRDaIyCH7d7Z9XETkByJSJiK7ReTcU7iuGWF1SRYiUJwVGPaYM1BdkB5g7Zxsth1vBAYDxJXLCujtD/FSWf2Ir9/a3UdGwEtBRoA5OSmADlQrNZPEGyA+AzwGLBSRl4EHsLYhHU0/8FljzErgIuBOEVkJ3A08a4xZAjxr3we4Hlhi/9wB/GQ8FzITXbgwl21fXB+1gZAj084gcoI+zp+fzf7KVtp7+mm2xxX+7rzZZKd6+f3WEyO+fmtXHxn2QrpUn4fCDP+oNZ6UUskl3gCxH3gE2IpVcuPnWOMQIzLGVDnrJIwxbcABYDZwC3C/fdr9wNvs27cADxjLa0CWiBSN41pmpNw0f8zj2ZEBYkEOIQOvn2iipbMXEcgL+nn72hI27K8ZceC5tbufjIhd7s4uzmRPxaiJI195bB//+/LRCV6NUmo6iTdAPIC1WdB/Af8DLAV+Fe8fEZH5wFpgM1BojKmyH6oGCu3bs4GTEU8rt48Nfa07RGSbiGyrq6uLtwkzjtPFlBP0scYu1renooXmrj4yU7y4XMKt586mb8DwrF38L1K/vV4ishTHOSWZHK5rj9qkKNKx+g7ue+UYj7xeMWb7alu72X68aSKXppQ6TeINEGcbY/7JGPO8/fNh4Kx4nigiacBDwKeMMVFV44wxhlH2vI7FGHOPMWadMWZdfn7+eJ46o2RFZBDpAS9pfg/1bb00d/aRZX/on1WcQXFmgKf3Dw8QzmI6ZwMjgHNmZ2IM7B9S/M/xoN1ddaimndAog98AP3q+jA/et3X8F6aUOm3iDRA77IFpAETkQmDbWE8SES9WcPhNREnwGqfryP5dax+vACIrwZXYx9QElGSnEPC6yLO7oHLTfNS391gZRLhKrLB+ZSEvHqqjqzd6Ulprlx0gIrqYzrE3KdpTPrybyRjDQ9vLCXhddPUNUD7GdNgTjZ20dPVphVilprF4A8R5wCsickxEjgGvAueLyB4R2R3rCWItt70XOGCM+U7EQ48Bt9u3bwcejTj+fns200VAS0RXlBqnd50/h6c/dQUpPqtcRl6an4aOHurbesiJ2Gdi/YpCuvtCbD3WGPV8Zw1ERkQXU0FGgIJ0P3tjjENUNHdR397LDedYS2XG2vK0qsVapNc+QnfVSP7fEwd4eEf5uJ6jlJqYeGsxvWUCr30p8D5gj4jstI99Afg68Ad7JfZx4F32Y08AbwXKgE7gHyfwN5XN73Ez195pDiA36ON4Qye1bd2smTu4gZCzGttZNPfwjnL8Hnd4kDtjSBmPc2bHHqh+o8oKCDetLuKhHeWU1raxfmXhsPMclc3W32vr7icrYuX3WH626QhAeF2GUipx4q3FdHy8L2yMeQkYqWjPNTHON8Cd4/07Kj556X5ePdxAW08/xZmD6yacLqi6Nmvv659sPIzX7eJfrlkMRGcQAGfPzuS5g7V09PRHLdB7o9oal1g3P4eizACldmnwtu4+/ue5MvLT/Nx2wRzSA17ae/rDdZ4iV2uPR3lTJyXZqWOfqJSasHgzCHWGywv6aLO7c4oyU8LHA143GQFPOEDUtvXQ1TdAY8fwLiaAVSWDA9Xnz88JHz9Q1cbcnFTS/B6Wz0rn8d1VGGDNnCzusb/1p/rdvPfCeVQ1D45PtI1jV7u+iPGK596o5f0Xz4/7uUqp8Yt3DEKd4fLSB9dLFGVGr7zOT/dT195Dd98ALV199PaH2FPRDMTuYoLhA9UHqltZPisdgH+/6SzevnY2j+6s5AfPHmJZYToBr4ujdVbBwMqWwSKBkQUBxxI5vfb5N2pHOVMpNRk0QMwQucGIAJGVEvVYfrqfuraecBYB8Pwbdfg9rmGFAGMNVHf1DnCsvoPl9ram8/OC/OfbzyY/3U9TZx83rS5iXk6QYw1WmY6JZhCRA9qlNbqiW6lE0wAxQ+SlDQ4ED88gAtS19VDbNvjNvrq1m/UrC3G5hg8jnTM7k72VgwHiaH0HIQNLCgZ3ufN73Hzgkvm4BG5YVcz8vNRwyfHIDKJtjDEIYwydvVZg6OixpuIuLUyjormL7j6tF6lUImmAmCGckhw5QR8Bb/QWpflpVgZR22plEB47KLx9zbCF7AAsLkzjWH1nuBKs88G/IC+6JtRHr1jE05++ggV5QebnBjne2EkoZKhq7iI3aAWs0TKIlw7Vs/b/buCcrzzNiYbOcAZxtt3NdayhY8TnKqVOnQaIGSLfDhCzMoZXfnU2A3K2KL1wYQ65QR9XLIu9Un1hXpDegRAV9mI4p+toXm70rCK3S1hsZxXzcoP09oeobu2mpq0nvJBvpFlMxhi+/uQBevtDDIQMb1S3hscgnHEQZ0xDKZUYGiBmiIwUD163xCwNnm8PYO+raMXjEr777jX86WOX4HXH/u+xMN/60D9cb40DHKvvIC/NKukxkvl28DjW0EFTRy85QR8ZAe+IGcTmo43srWjlzqus6bYnm7qGBYgj9RoglEokDRAzhIhwzuzMcOG+SAVOgKhsIS/NT0F6YFh3USTnMecb/LGGDubHKDkeaZ79nGP1nTR29JId9JEe8FBW286HH9hGc2d0RdmHtpeTEfDwocsWEPS5KW8a7GIqSA9QmOHnqAYIpRJKA8QM8vA/X8rHr14y7LiTQRxr6KQwI3b58Ei5QR8ZAQ9H7AzieENnzD0pIhVlBPC6hRONnTR19pKTamUc2443sWF/DS+XNUSdv+NEExcsyCHgdVOSnUp5RAYR9LtZkBeccID4w7aTPL67ErA2QPr8w3vo6dcBb6WG0gChKIwYl4in7IWIsDA/jaP1HXT1DlDd2s2CvNFXNbtcQn6an5NNnXT2DoQzCMe+yha+/OheXjlcT0tnH4frOlg7NxuwCg+WN3XRYRcUDPo9LC5Io7S6bULF/u598Sg/e8FavPfQjnIe3HKC7ce09LhSQ2mAUOQEfXzyGiuzmJ2dMsbZloV5QY7UdYRnEo2VQQDkZwQ4ZBfxy071Ra3S/svuSh549Tif/N1OXjhk7fOx1u4OK8lOobzR6mLyuAS/x8XFC/No6+lnV7m1oK9/IMTHf7uD7ccbGUtjZy9H6toxxoT36p7svSn6B0JaqVad8TRAKAA+fe1SHv/EZfzrdcviOv+s2ZlUtXTzwKvHAKsEx1gK0v0cscctslO9Uau0TzZaM6IaO3r5wsN7EIFVdoCYk5NKW08/lc1dBP0eRITLFufhEnih1NpTe19lK4/vrmLjwdE3kTLG0NzZS0fvABXNXew4YQUG5/epOlbfQXffAB/99Q4+eP+YFfGVmtY0QKiws2dnkh2Mr7LqjauKcAk8uOUkq0oy48ogCtL99NtrJ7KDg7OeSuysZVlhOv/9d6voHQixsiiDNLsYoPP4weq28LHMVC+r52SxqdQKCE658sjV4LG09/TTN2C14fHdVXT2DpAT9PH6yWaMMdzyw5cmvGVqKGS48lsbWf6lJ3nmQA3bjjWOuXGSUtOZBgg1IYUZAS5dnAfAzauL43pOfkQ9qJygj3T7w/7ta60FeVcuy+cd55Xw7Geu4Kf/cF74XKdq6xvVbQT9g4v8Llucx67yZrr7BthyNL4A0dQxuO7i91utHW7ff/E8mjv72FfZyq7yFl4ojc5CBkLWZkhjdRkNXdPR2Tv2xklKTWcaINSEvf/i+WSmeLkpzgBRkB45GO5lXl6QoM+q8PqWs2bxznXWhoJzclKZkzM46B055TayxPi83CDGQHVLN9vsMYS69tEDRGPEdNqj9R2cMzuT9SusfSs22WMf+yujt1TdfKSBz/6O5RihAAAgAElEQVRxV8ytWSM1dVoBYnVJJv9ij+kcHGPjpHh19w3w+Yd3jxkAlZpMGiDUhF27spBd/35d1Cyo0RREZBDZqT5uPKeI175wDbMyA/z0feeFV10PFfR7mJNjdTOlRQQIZ1X45qMNNHb0EvC6xs4g7ADhlBP58OULw8HImclU29ZDQ0SgcYLOWAPZjR3Wa3/62qXccflCAA5Wx96/23GysZN7XzqKtR3KyHaXt/DglpO8crh+1POGMsZozSo1YRog1GlTYK+xSA948LpduFwy6urrSMsKrVLikdVlnTUbrx2xupfWzsmmrq0n3O//5N5qbvnRy9RHfNg32R/iFy/KZUFekLeePYvMFC/pAU84CwFrf4uhzxkrQDjnZaf6SPN7KMlO4Y3q0TOI72wo5f8+vp/XTzaPep4T+FrGUR4d4KcvHGH5l54c9/OUAg0Q6jRyupiyx7HFqGOJHSD83sH/sgV2BrHNntp67rws+kOGZvvD8K97qth1spmP/Xo79718lBt+8CJP7LG2Of/uu9fw+Ccuw2OXE5mdlRL1IXqgavCbf6PddbSvsmXUb+NOdpJjD/QvK0wfdW/u5s5e/mq355EdFaNef51dabe5c3wf9M6+GX+wx1uUGg8NEOq0yU3zIULcM6UiORlEZBdSRsBDitfNycYuAl4XK+z9KJxzXj/RRHFmgNdPNPOVv+xnX2Urzxyoxe0ScoO+qPEMZyA8PeBhVkYgKkA4mUHfgIm5H3f4PDtAONd3TkkmZbXtw8qIOB7eUUFvf4izijP4y+5KqiPKoJfVtnEsYqV4rX1N4w0QTvfZfa8c03UZatw0QKjTxut2kZPqIzs1vm6lSEsKrfGJiojNhkQk3M1Ukp0azlCcvS3Km7r44GULeP3L1/LUpy7nvHnWyuzsVC8i0ftcOFNpZ2UEWFGUzv6oDKKXPDu4vXRo5DGAps4+vG4h6LNmWl2+NJ+QgU0jPGfz0QYW5AX50o0rae3q49L/fo4XSut45PVy1n9nEx+8b2v4XCfoNXfFDjaOQzVtUVuzOjOrKpq72D1KcFMqFg0Q6rS6aXUxVy0rGPfzFtkVZJ0psQ6nm2lOdkp4Gm1dezc7T1h9+mvnZpEe8LJsVjpnF1sZRqwurnCAyAywoiiDstr2cH2mxvZe5ucGWTcvmw2jzGRq6uglO9UXDj6rS7LITvWy8WDs7VFLa9pZPiudixbm8uxnr2ROdgpfeHgPd/1xN2BVq3UGr52B8pZRMoia1m7e8v0XueuPu8LPa+3qC88Cc/5NlIqXBgh1Wn3l5rO4/ZL5435ewOvm4H++JVwSxOHMZJqTkzoYINp62H68Ca9bOKt4cIW3cztWF5cTIIrsANEfMpTVWsUImzqt6rPXrixkf1UrJxs7Y7ax0S5j7nC7hDctyeeFg3XDFsx19w1wvKEjPLayIC/Ip69dSkVzF4Xp/vB1OlNnBzOIkQPEy2X1DIQMj+6s5NGdVjHC1u5+FhekWV1tYwyEKzWUBgh1xvB73MO6hga7mFII+tykeN3sqWjld1tP8qYl+VG7560MZxDDu7icMQiri8k6z5nJ1NjRS27Qx7UrZwFw8w9f4icbDw97jebOPrKGvPZVy/Np6OiN2qIVoKy2nZAZHFsBuGlVMf9y9WLuef+6cFn2o3bF3MExiJG7mF4uayA71UtxZoBnDliZTmtXHxkBL2vnZvP6JJQTMcbw801HeGOM6bvjVd3SzX/8Zb9OyZ1mNECoM1phuIspFRFh/cpC/rKrkpauPj573dKoc5cWpuN1S9S3fMf8vCDZqV7OKcliQV6QgNfFgapWjDHhDGJBXpCPXrEIt8vFs/YH8JG6dv790b309odo7Owd9tqXL8lHhGE1og7VWsFn2azBtR8ul/CZ65Zx9uzMcLfQkboOBkImvC6jpauP2rbu8D7dDmMMrxyu5+JFuSwpTA+XQm/t6iMjxcPauVmUN3VF7Tser7Ladj7/8B76BkJsP97E1544EK6Ge6pePFTH954p5bsbSvnly0d57UjD2E9Sp40GCHVGm2vP0nF2ufvWO1fxrnUlfOzKRVHdSwA+j4vvvXstH7x0wbDXSfN7eP3L13HtykLcLmFZYTr7K1vDtZty7HGLu69fzpXL8sMlNJ7cV839rx7nb3uraO7sHTa+kZvmZ9XszGHjEAer2/G6ZcQaViXZKXhcwtH6Dho7egkZa4ZVc2cfb//RK3zrqdKo84/Wd1DV0s0li/JYkBfkWH0H/QMh2nr6wxkEEK5eOx5P76/mwS0n2HWymZ+/aAWGFw/VY4xhw/4avvTnveN+Tcd/PfEG33vmEL/fZk3D3THGWhNjDH/aXh61tmUkO040jTrNWI1NA4Q6o61fUcgj/3wJy2bZ6yQ8br7xjtX821uWxzz/hlVF4X7/0SwtTKesrj1cuyly3KIkO4Watm56+gdrLd33yjGaOvtiDoBfsayAnSebw9NlwVphvTAvbcRtXT1uF3NzUzla3xEef1hamE5/yFgzksqjxxNePmx98750cR4L84PWHuN2FpGR4mV1SSYZAQ8vRGQyv9tygu89Ex1oYnGm3z7yegVP769hQV6Q+vYe3qhu49GdFfzqteNRU3THIzPFmmrsdQtFmQG2j9ENdrCmjbv+uIsfPz+8i2+oux/azdf/9sa42tPbH+KuP+6irFYDC2iAUGc4l0vC344n0/y8IHVtPZxssgakc4KDYwtzslMxBiqauqiwA8TrJ5oZCJmYA+BXLnOmu1ofzgMhw7bjTZw7b/j2r5GcPTecbqElEaVIDtW280pZPffZlWdfPVxPcWaA+bmpg7OW7EHpjIAHj9tlDZiX1mGMIRQyfPeZUn792vEx/y2q7A//3245gTHwzXesAqwpv8cbrH+fl8vGVwLE0dDey/oVhTx/15WsX1HITvvfcSROgHtiT9WYlXKrW7rHXbvqSH07f9puTTNWGiCUimmh/SHrdHnkBAfrSDkznsqbuihv6uQquwotwIpZw7MTZ7qr8+G2r7KFtu5+LlqYO2ob5uYEKW/qDH87XxqR+bR09fF//ryXr/xlPxv21/Dq4QYuXpSHiIQDhLOZkrMx0xVL86lu7eZgTRs7TjRR09pDfXtveK/vkTh/3xi4YEEO6+bnsDAvyOajDeENo14eUiOqrLYtrg/nho5eCjP8lGSncu68LDp6B0YdAN90qA6XQHVrN+/9xWZ+vLEMgK7egagpyN19A7R294frY8UrchrxtmNjbz6V7DRAKBXDfPtDdov9IZET0XXkrE4+0dhJRXMXi/LT+NY7V3Ps6zdwiV0CPZLbJVy+1Pr2HgoZXrW7gy4eI0AUZwXsD8w2XALLhwSfI3YX0ice3EFTZx+XLrZerzgzBZ/Hxa6T1sypDLve1WVLrLZtPtIYLvEBjDht95XD9WzYX0NVS3e4NPut9jqUVSWZbD7SSFt3PyLwSllDVMHB9d/ZxPlfe2bU6+sfCNHU2UtemhV818yxMsE95bEX9HX29rP1aBPvPn8uAa+LV4808KPnyujs7eeHzx/iww9sC09NdoJTfXvPqIUQe/oH6OodnDnVFBEgHhqj/MlMoAFCqRjm24PHL5XVk53qDWcNYM2c8rqF3eXNdPeF4tqm9cplg9NdXz3SwKL8YHiR30hmZ1mvu/VYI4UZAfLsdR6R5c//821nc+XSAhbmB7liaT5gdbvNz00NlwXJsPv5izIDpHjdnGjs5KVD9eEpwidiBIjGjl4+9usd3P3Qburbe3jvRfP44ltX8DY7QJw9O5M2O/O4bHEe1a3dVNqZRmQX0WjZQGNnL8ZAXpoVfOflpBL0uaPKnETaebKZ3oEQ151VyB8/cgnfv20NHb0DPLyjgt9sPgEwGCDsQeye/hCdvSNPnb3rj7t5x09fCQeRFnul+rLC9PAmVDOZBgilYkjxuZmVEcAYuGRxHi7X4PoLt0sozkoJV5F11lCM5k1LrA/vFw7WsfVoIxcvGj17ACiyA8SBqlaKMgNk2V1Fly3OIz3gwed28XfnlvDT953Hc5+9kty0wW6wsyNmcDkZhIhQkp3C8YZOjjd0hle0n7DHEYwxfO+ZUlZ95Smu+ObztHT10WB30SzIS+XDly8Mrys5e/bg6193lrU+xPlwjiyVft/Lx0a8vvo267WdDMLlEpYXZURV0o1UalfGPasog3NKMrl5dTHzc1P5j7/sD9eoOlwXnUFY7YndzdTbH+K5AzXsq2zlJXsMxckgrl1ZSFlt+7i7qMD6d/zOhtKkmEGlAUKpEczPsz743xSj22hOdmr4m7fzTX80eWl+lham8cBrx+noHeDihcNfc6jiLCvDCBkozkohJ+jjwgU5XLuykDVzsrhkcS4pPnfM566dNzhw74xBgNU9tu14I70DIVbPySIj4Alfx96KVr73zCHWzM1mdUlW1E6BszKjr9FZdAhw9XIr0DgBorrVyiSyUr38ZVclnb39w3bbA2josD7EIwPbiqL08PqToQ7WtJOV6g2vmBcR/v2ms7jurELuvGpR1J7nkQGiviP2WMjrJ5ro6B1ABO59yRrsb+7sw+d2ccUyK6BPJIuoa+/hB88e4pcvHeWBV4/x9z9/jW89dRCwamKt/urTE94DvbtvgO9sKOXlsvox9xCZDJ6xT1FqZlqQF+S1I43hvvtI7z5/TvhbZzxdTAAXLsjlV/asoYsW5ox5fl7Qj8/toncgxOysFDxuF7//yMUAnDsvGxnluefOHZwhlR5RtXZOdgrP2SXA5+WmMi83yHE7QGw+ao2NfOPvVjErM0Bvf4gn91XT2x+iKDO6Oywj4GV+bio9/SGKMwNkp3rDAaKm1fpAvvPKxXztiQNc/o2N1Lf3sO+rb46qoOusZXC6mABWFGXw69dOUN7UFbWrIEBpTRtLC9OjVtNftbyAq+wAteN4M0eGrDwHeHpfDc+/Uctnr1sW9XqbDtXhdgnvvXAuv37tOC1dfTR39pKV6mVVSSY+j4vNRxp5s50hxauy2QqQLx6q5697qhgIGV453MANq4ooq22npauPx3dVce4EZt9tPtrID549BMDnr1/OR65YNO7XGA/NIJQawbvWzeGT1yyJ2YV00+piblljdXFkpsRXndaZtbSsMD3qW/NIXC6hyM4iiodkKWl+T9SH7VCRJTwiu8cir2VBXpC5OanhQeotRxuZl5vKLDsY+DyucIHDWZnDx0veuW4ON68pRkRYXJAWXjvgZBA3rS6mJDslHAicb/cOp+sn8t9ipV3mZNOhuqhvyMYYSqvboq5rqIX51rRgYwx1bT04ceQXLx7hf54rG7Zp0vNv1LFmThbXn11EyMDWo43hcil+j5tLFuXyt71V4TGVY/UdfOB/t0StZ4nFmfpc0dxFW3c/X3v72aT63Pz8xSMcsoPoxtLYBRzHUmlXM77ruqVxb/V7KhIWIETklyJSKyJ7I47liMgGETlk/862j4uI/EBEykRkt4icm6h2KRWvtXOz+fS1S0d8/Pu3reXZz14Z9+tdaGcN8Yw/OJxv7kO/wY/FM8ICPGfrVr/HRWF6gMUFaRxv6OD5g7VsPdbIBfOjM5tLF+cxKyMQlYU47rxqMZ+/fgUAiwvSBzOIlm7cLiE/3c9d1y0Lz746XNeOMYa7/riL67//It9/5hA+t4uMwOBrL5+VwayMAF98ZC8/jqh3VdXSTVtPP0sLY29LC9Zq+pauPho7eqlr6wlPNOi3P+AjF7/tOtnM/qpWblpVxNq5Wfg81qyops5esuwZa+84r4Sqlu7wrLOfbTrMxoN14cxxJJURJenTAx5uOKeYd62bw2M7K8PTpo/UdYw4e2w0Vc1duAQ+esWiYV8aEiGRGcR9wFuGHLsbeNYYswR41r4PcD2wxP65A/hJAtul1KRxu0br6ImWl+bnf//xfD5+9eK4n+N8CEzkw+CRf76En73vvKhjTgYxPzeIyyX805sWsHxWBh+6bytNnX1cOGTq7SeuXsKTn3rTsCKJQy0uSKOps4+G9h6qW7vJT/PjdglvWzubRz9+KS6x6lZVtnTzp+3lnGjooK2nH4OJeu0Un5unPnU5a+dm8fjuwam4B+0B6qVjZBBgTf+ta+8JF3B07K9s5cEtJ2jt7uPXrx0n1efm1vNKCHjdnDc3m1cPN9DS1ReeDLB+RSEZAQ9/2n6S5s5eHnndmva6a0hV3I6efk42dvJGdSu33fMqeytbSPN7mJebyo2rivF5XNywqoj+kOGlsvrwgkdn4SRYa2O++dQboy4SBKho7qYwIzDiF4DJlrAxCGPMJhGZP+TwLcCV9u37gY3Av9nHHzBWTvmaiGSJSJExpgqlksh498JwBsDjGQgfKtYKc6df3xmATw94eeBDF3D/K8eob+/h2pWFUef7PC58nrF3AHQ+9Epr2qlp7aYwIuPxe9zMzUnlcF0Hh+0s40s3ruTuh/fQNzD8AzEz1cvVywr4zjOl9piAj5fK6vG5XZw1O3PY+Q4nYzje0EldazeL8/PITfPTYX9T/8FzZdS19fCTjYcpb+rkPRfMDc/wunhRLt99ppQ0n4fVJdb4TcDr5vKl+ew40cxf91TR3RciP93P7oh1GsYYPnjfVnaVN7N8VgY7TzbjdQsL89L448cuxu+xPsjXzskiPeChrbufa1cWUtPazf7Kwem8P9l4mMd3V5Hq83DnVSN/gahq6Rp3NnkqTvcgdWHEh3414PxvnA1Ebppbbh8bFiBE5A6sLIO5c+cmrqVKTQO3XTCX4qyUCW3TGktmipeF+cHw7npgZTZDB3DHa3mR9c3+QFUr1S3d4W/zjkX5aRyuaw9PQ12/spCv9ofweWJ/E75oUS5mgzUoe93KQjbsr+GSxbmkjTLuMjsrBRGrRHptWw+FGX5ygj5ONHZSnBmgsqWbzBQvtW3d3Ly6mM+/dUX4uevmZ2MMtPX0R5VsX5gX5Ik9Vewpb7G7i4r4/daT9A+E8LhdPLqzks1HG3HJYGmTvgHD7OyUcPABq8vvssV5/G1vNUsL01k2K3q/cme45bsbSrl5dfGwAXpHVUt31AyyRJuyQWo7Wxj3PC1jzD3GmHXGmHX5+fkJaJlS08fsrBTec8HkfhF69jNX8OE3LZzU1yxID5CX5rMCRGt3uAy7Y1FBGkfqOyitaSczxUtu0Mftl8wf8dpWlWRaq6UPN3Cwpo0TjZ1ct3L02UQ+j4vizBReOlRPf8iwKD+N/HQ/szIC4QkCb1tTzO5/fzPfu21tVLBZXZIV7i7Milg1Pz8vSMhY5doX5aexek4mXX0DlNmB7hcvHWFFUQbfu20tF8zPCY/hOFOUIznTgVcUZbCkMJ3SmvbwQHxlSxcL8oL0hwwP76jgyb1Vw3YPNMZQ2dw1oWxyok53BlHjdB2JSBHgDOVXAHMiziuxjymlJtlY4wkTtXxWBhtL62jr7g939zgW5gXp7Q+xqbSORfnBMdvg97hZNy+Hl8vqyQ1a+4GvXzl291xJdkq4PMrigjQ+V7KMlq4+ttplzq87a1bMrCXo97B8Vjr7KlujMgin5Ep1azeXLs4LT03dVFrHrIwA+ypb+dQ1S7l5dTE3ry7mJxsPs+VYI7OzhmcAt55bwqKCNJbNSmdZYTq/7TphZzoBqpqt1y/M8POTF8ro7gtx13VL+fjVSzDG8OONh/nb3ip6Ykw5TqTTnUE8Btxu374deDTi+Pvt2UwXAS06/qDUmWVFUXp4gdqbz47+tn/JojxcQrh2VTyuWJrPodp2frf1JGvmZFGQPvYH49yc1HB3zaKCNJYUprNufg5vXzubT69fyoULRl5/4nS7Re44uCAi0C0qCDIv1+qee3DLSV493IAxhGtgweAMtfm5wwOE2yXhAOMMth+sbqN/IERtWzfFWQFuXVtCd18IGOyy+tmmI3zzqYPsrbDGLIoyT18Gkchprg8CrwLLRKRcRD4EfB24VkQOAevt+wBPAEeAMuDnwD8nql1KqcRwtmo9b172sG6Qubmp3LLGquO0qCDOAGGvZq5o7hqze8nh9N3PyghEdSHNygzwyfVLRp39MxggBruYsoO+8DoXJ7C998K5HK3v4PvPHiLV52ZVyeCixDVzsvjjRy8Olx8ZiTNdt7SmjZq2nvBq+ZvXFPOlG1eyfoW1h4izKdPqOYN/IykGqY0x7xnhoWtinGuAOxPVFqVU4p1jzzC6eYQFXB+/ejEvlNZxwSjf4iMtKUijKDNAVUs3151VOPYTGNxhcHGcQSjS9WcX0XnrAOuGrAWZnxdk18nm8Gu+9ZwifvhcGW9Ut3HlsvxhXVbnzx/7+nLT/OSn+9lX2Rr+8C/KDBDwuvnQZQvwuYVnDtRS3tRFaU0bt6wp5ss3ruCbTx1kyShrQSabltpQSk2KJYXpPPSxi8PTRIdalJ/G9v+zPu4xEBHhljWz2XqsMe5uKSeDWJQfeyvX0fg8rpiD5gtyU9lX0RIOPgGvm7996k08s7+WFUVj7044kgvm57D5SANX2plS5FoXJ2g8vb+Gtu5+lhamc968HH53x8UT/nsToQFCKTVpzps3+rfn8Q6Q33197K1jR7IwL4jf44rqkjlVH7h0AefNy47aHtbvcXPDqqJTet2LFubw1z1VbDlqDapHdh0tn5WBz+Pid1usMuYTyYgmgwYIpVTSyA76ePFzV4VLiE+GNXOyWDOJAcfhrFp/fHcV6X4P6RHrJnweF1cty+epfdYueaOtIE8kLdanlEoqBRmBqAKF09WSgjRygj5auvq4esXwKbzvOM+a+Z8T9E1qwBsPzSCUUmoKiAj/+uZlNHb08pHLhy9cvHJZPnlpvinrXgINEEopNWVGWyXvdbv4xe3nk+KNvSnU6aABQimlpqlEjH2Mh45BKKWUikkDhFJKqZg0QCillIpJA4RSSqmYNEAopZSKSQOEUkqpmDRAKKWUikkDhFJKqZg0QCillIpJA4RSSqmYNEAopZSKSQOEUkqpmDRAKKWUikkDhFJKqZg0QCillIpJA4RSSqmYNEAopZSKSQOEUkqpmDRAKKWUikkDhFJKqZg0QCillIpJA4RSSqmYNEAopZSKSQOEUkqpmDRAKKWUikkDhFJKqZg0QCillIpJA4RSSqmYplWAEJG3iMhBESkTkbunuj1KKTWTTZsAISJu4EfA9cBK4D0isnJqW6WUUjPXtAkQwAVAmTHmiDGmF/gdcMsUt0kppWYsz1Q3IMJs4GTE/XLgwqEnicgdwB323XYROTjBv5cH1E/wuWcKvcbkoNeYHKbTNc6L56TpFCDiYoy5B7jnVF9HRLYZY9ZNQpOmLb3G5KDXmBzOxGucTl1MFcCciPsl9jGllFJTYDoFiK3AEhFZICI+4DbgsSluk1JKzVjTpovJGNMvIh8HngLcwC+NMfsS+CdPuZvqDKDXmBz0GpPDGXeNYoyZ6jYopZSahqZTF5NSSqlpRAOEUkqpmGZkgEjWkh4ickxE9ojIThHZZh/LEZENInLI/p091e0cDxH5pYjUisjeiGMxr0ksP7Df190icu7UtTx+I1zjV0Skwn4vd4rIWyMe+7x9jQdF5M1T0+rxEZE5IvK8iOwXkX0i8kn7eNK8l6Nc45n7XhpjZtQP1gD4YWAh4AN2ASunul2TdG3HgLwhx74B3G3fvhv476lu5ziv6XLgXGDvWNcEvBX4GyDARcDmqW7/KVzjV4C7Ypy70v4/6wcW2P+X3VN9DXFcYxFwrn07HSi1ryVp3stRrvGMfS9nYgYx00p63ALcb9++H3jbFLZl3Iwxm4DGIYdHuqZbgAeM5TUgS0SKTk9LJ26EaxzJLcDvjDE9xpijQBnW/+lpzRhTZYzZYd9uAw5gVU9ImvdylGscybR/L2digIhV0mO0N/FMYoCnRWS7XZIEoNAYU2XfrgYKp6Zpk2qka0q29/bjdvfKLyO6Bs/4axSR+cBaYDNJ+l4OuUY4Q9/LmRggktllxphzsSri3ikil0c+aKy8NqnmNSfjNdl+AiwC1gBVwLentjmTQ0TSgIeATxljWiMfS5b3MsY1nrHv5UwMEElb0sMYU2H/rgUewUpXa5zU3P5dO3UtnDQjXVPSvLfGmBpjzIAxJgT8nMGuhzP2GkXEi/XB+RtjzMP24aR6L2Nd45n8Xs7EAJGUJT1EJCgi6c5t4DpgL9a13W6fdjvw6NS0cFKNdE2PAe+3Z8BcBLREdF+cUYb0t78d670E6xpvExG/iCwAlgBbTnf7xktEBLgXOGCM+U7EQ0nzXo50jWf0eznVo+RT8YM1Q6IUa9bAF6e6PZN0TQuxZkTsAvY51wXkAs8Ch4BngJypbus4r+tBrLS8D6uP9kMjXRPWjJcf2e/rHmDdVLf/FK7xV/Y17Mb6ICmKOP+L9jUeBK6f6vbHeY2XYXUf7QZ22j9vTab3cpRrPGPfSy21oZRSKqaZ2MWklFIqDhoglFJKxaQBQimlVEwaIJRSSsWkAUIppVRMGiDUGU9EXrF/zxeRv5/k1/5CrL+VKCLyNhH58hjnfFNE3rBLNzwiIlkRjw2rDioiPhHZJCLTZgdJdWbQAKHOeMaYS+yb84FxBYg4PjSjAkTE30qUzwE/HuOcDcDZxphVWOt5Pg8gIiuxFn6eBbwF+LGIuI1VlPJZ4N0Ja7VKShog1BlPRNrtm18H3mTX3P+0iLjtb9tb7W/bH7HPv1JEXhSRx4D99rE/20UO9zmFDkXk60CK/Xq/ifxb9grfb4rIXrH24Hh3xGtvFJE/2d/yf2OvsEVEvm7vFbBbRL4V4zqWAj3GmHr7/qMi8n779kecNhhjnjbG9NtPew2rRAOMXh30z8B7J+GfW80gmnKqZHI3Vt39GwHsD/oWY8z5IuIHXhaRp+1zz8X6Fn7Uvv9BY0yjiKQAW0XkIWPM3SLycWPMmhh/61as4murgTz7OZvsx9ZifYuvBF4GLhWRA1hlFpYbY0xkt1CES4EdEffvsFFadeUAAAIvSURBVNt8FPgs1r4IQ30Q+L19ezZWwHBEVgfdC5wf4/lKjUgzCJXMrsOq57MTq+xyLla9G4AtEcEB4F9EZBfWB+yciPNGchnwoLGKsNUALzD4AbzFGFNurOJsO7G6vlqAbuBeEbkV6IzxmkVAnXPHft0vA88DnzXGRO0ZISJfBPqB34zRVowxA0CvU69LqXhoBqGSmQCfMMY8FXVQ5EqgY8j99cDFxphOEdkIBE7h7/ZE3B4APMaYfhG5ALgGeAfwceDqIc/rAjKHHDsHaACKh1zDB4AbgWvMYL2csaqD+rGClFJx0QxCJZM2rK0eHU8BH7NLMCMiS+1Kt0NlAk12cFhOdFdOn/P8IV4E3m2Pc+RjbRs6YiVOe4+ATGPME8CnsbqmhjoALI54zgVYe3usBe6yK34iIm/BGsy+2RgTmYmMWB1URHKBemNM30htVGoozSBUMtkNDNhdRfcB38fq3tlhDxTXEXvL1SeBj9rjBAeJ7se/B9gtIjuMMZGDvI8AF2NVzzXA54wx1XaAiSUdeFREAliZzWdinLMJ+LbdVh/W3gH/aIypFJHPAr8UkauBH2JlAxvs8e/XjDEfNcbsE5E/YA289wN32l1LAFcBfx2hbUrFpNVclZpGROT7wF+MMc9M8us+DNxtjCmdzNdVyU27mJSaXv4LSJ3MFxRrY6w/a3BQ46UZhFJKqZg0g1BKKRWTBgillFIxaYBQSikVkwYIpZRSMWmAUEopFdP/B9JCq05oPXuJAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"evaluating perplexity ...\n",
"234 / 235\n",
"test perplexity: 135.81750561235523\n"
]
}
],
"source": [
"# coding: utf-8\n",
"import sys\n",
"sys.path.append('..')\n",
"from common.optimizer import SGD\n",
"from common.trainer import RnnlmTrainer\n",
"from common.util import eval_perplexity\n",
"from dataset import ptb\n",
"from rnnlm import Rnnlm\n",
"\n",
"\n",
"# ハイパーパラメータの設定\n",
"batch_size = 20\n",
"wordvec_size = 100\n",
"hidden_size = 100 # RNNの隠れ状態ベクトルの要素数\n",
"time_size = 35 # RNNを展開するサイズ\n",
"lr = 20.0\n",
"max_epoch = 4\n",
"max_grad = 0.25\n",
"\n",
"# 学習データの読み込み\n",
"corpus, word_to_id, id_to_word = ptb.load_data('train')\n",
"corpus_test, _, _ = ptb.load_data('test')\n",
"vocab_size = len(word_to_id)\n",
"xs = corpus[:-1]\n",
"ts = corpus[1:]\n",
"\n",
"# モデルの生成\n",
"model = Rnnlm(vocab_size, wordvec_size, hidden_size)\n",
"optimizer = SGD(lr)\n",
"trainer = RnnlmTrainer(model, optimizer)\n",
"\n",
"# 勾配クリッピングを適用して学習\n",
"trainer.fit(xs, ts, max_epoch, batch_size, time_size, max_grad,\n",
" eval_interval=20)\n",
"trainer.plot(ylim=(0, 500))\n",
"\n",
"# テストデータで評価\n",
"model.reset_state()\n",
"ppl_test = eval_perplexity(model, corpus_test)\n",
"print('test perplexity: ', ppl_test)\n",
"\n",
"# パラメータの保存\n",
"model.save_params()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6.5 RNNLMのさらなる改善\n",
"\n",
"- LSTIMレイヤの多層化\n",
"- Dropoutによる過学習の抑制\n",
"- 重み共有"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"| epoch 1 | iter 1 / 1327 | time 8[s] | perplexity 10000.15\n",
"| epoch 1 | iter 21 / 1327 | time 93[s] | perplexity 4234.82\n",
"| epoch 1 | iter 41 / 1327 | time 178[s] | perplexity 1896.71\n",
"| epoch 1 | iter 61 / 1327 | time 272[s] | perplexity 1280.45\n",
"| epoch 1 | iter 81 / 1327 | time 332[s] | perplexity 1023.16\n",
"| epoch 1 | iter 101 / 1327 | time 394[s] | perplexity 831.45\n",
"| epoch 1 | iter 121 / 1327 | time 458[s] | perplexity 807.89\n",
"| epoch 1 | iter 141 / 1327 | time 522[s] | perplexity 720.43\n",
"| epoch 1 | iter 161 / 1327 | time 587[s] | perplexity 689.12\n",
"| epoch 1 | iter 181 / 1327 | time 651[s] | perplexity 679.70\n",
"| epoch 1 | iter 201 / 1327 | time 716[s] | perplexity 602.49\n",
"| epoch 1 | iter 221 / 1327 | time 781[s] | perplexity 567.63\n",
"| epoch 1 | iter 241 / 1327 | time 844[s] | perplexity 528.17\n",
"| epoch 1 | iter 261 / 1327 | time 909[s] | perplexity 538.42\n",
"| epoch 1 | iter 281 / 1327 | time 975[s] | perplexity 521.44\n",
"| epoch 1 | iter 301 / 1327 | time 1043[s] | perplexity 449.47\n",
"| epoch 1 | iter 321 / 1327 | time 1112[s] | perplexity 399.01\n",
"| epoch 1 | iter 341 / 1327 | time 1180[s] | perplexity 452.80\n",
"| epoch 1 | iter 361 / 1327 | time 1247[s] | perplexity 460.58\n",
"| epoch 1 | iter 381 / 1327 | time 1312[s] | perplexity 383.68\n",
"| epoch 1 | iter 401 / 1327 | time 1379[s] | perplexity 404.59\n",
"| epoch 1 | iter 421 / 1327 | time 1445[s] | perplexity 394.29\n",
"| epoch 1 | iter 441 / 1327 | time 1509[s] | perplexity 375.40\n",
"| epoch 1 | iter 461 / 1327 | time 1574[s] | perplexity 373.20\n",
"| epoch 1 | iter 481 / 1327 | time 1639[s] | perplexity 344.29\n",
"| epoch 1 | iter 501 / 1327 | time 1682[s] | perplexity 355.09\n",
"| epoch 1 | iter 521 / 1327 | time 1722[s] | perplexity 345.81\n",
"| epoch 1 | iter 541 / 1327 | time 1761[s] | perplexity 364.67\n",
"| epoch 1 | iter 561 / 1327 | time 1798[s] | perplexity 323.76\n",
"| epoch 1 | iter 581 / 1327 | time 1836[s] | perplexity 293.38\n",
"| epoch 1 | iter 601 / 1327 | time 1873[s] | perplexity 377.00\n",
"| epoch 1 | iter 621 / 1327 | time 1911[s] | perplexity 346.04\n",
"| epoch 1 | iter 641 / 1327 | time 1948[s] | perplexity 315.85\n",
"| epoch 1 | iter 661 / 1327 | time 1985[s] | perplexity 307.43\n",
"| epoch 1 | iter 681 / 1327 | time 2023[s] | perplexity 257.17\n",
"| epoch 1 | iter 701 / 1327 | time 2062[s] | perplexity 281.79\n",
"| epoch 1 | iter 721 / 1327 | time 2101[s] | perplexity 289.17\n",
"| epoch 1 | iter 741 / 1327 | time 2140[s] | perplexity 249.45\n",
"| epoch 1 | iter 761 / 1327 | time 2179[s] | perplexity 258.53\n",
"| epoch 1 | iter 781 / 1327 | time 2216[s] | perplexity 244.86\n",
"| epoch 1 | iter 801 / 1327 | time 2254[s] | perplexity 269.02\n",
"| epoch 1 | iter 821 / 1327 | time 2291[s] | perplexity 248.95\n",
"| epoch 1 | iter 841 / 1327 | time 2328[s] | perplexity 254.58\n",
"| epoch 1 | iter 861 / 1327 | time 2367[s] | perplexity 249.46\n",
"| epoch 1 | iter 881 / 1327 | time 2405[s] | perplexity 230.01\n",
"| epoch 1 | iter 901 / 1327 | time 2442[s] | perplexity 280.32\n",
"| epoch 1 | iter 921 / 1327 | time 2480[s] | perplexity 253.94\n",
"| epoch 1 | iter 941 / 1327 | time 2516[s] | perplexity 257.12\n",
"| epoch 1 | iter 961 / 1327 | time 2553[s] | perplexity 275.56\n",
"| epoch 1 | iter 981 / 1327 | time 2590[s] | perplexity 256.14\n",
"| epoch 1 | iter 1001 / 1327 | time 2627[s] | perplexity 215.38\n",
"| epoch 1 | iter 1021 / 1327 | time 2664[s] | perplexity 251.41\n",
"| epoch 1 | iter 1041 / 1327 | time 2702[s] | perplexity 228.57\n",
"| epoch 1 | iter 1061 / 1327 | time 2739[s] | perplexity 218.85\n",
"| epoch 1 | iter 1081 / 1327 | time 2776[s] | perplexity 188.14\n",
"| epoch 1 | iter 1101 / 1327 | time 2813[s] | perplexity 215.89\n",
"| epoch 1 | iter 1121 / 1327 | time 2854[s] | perplexity 255.60\n",
"| epoch 1 | iter 1141 / 1327 | time 2897[s] | perplexity 229.44\n",
"| epoch 1 | iter 1161 / 1327 | time 2941[s] | perplexity 221.68\n",
"| epoch 1 | iter 1181 / 1327 | time 2981[s] | perplexity 210.42\n",
"| epoch 1 | iter 1201 / 1327 | time 3020[s] | perplexity 181.10\n",
"| epoch 1 | iter 1221 / 1327 | time 3065[s] | perplexity 177.75\n",
"| epoch 1 | iter 1241 / 1327 | time 3110[s] | perplexity 209.31\n",
"| epoch 1 | iter 1261 / 1327 | time 3154[s] | perplexity 191.01\n",
"| epoch 1 | iter 1281 / 1327 | time 3203[s] | perplexity 199.19\n",
"| epoch 1 | iter 1301 / 1327 | time 3251[s] | perplexity 246.91\n",
"| epoch 1 | iter 1321 / 1327 | time 3299[s] | perplexity 234.73\n",
"evaluating perplexity ...\n",
"209 / 210\n",
"valid perplexity: 196.80691468962846\n",
"--------------------------------------------------\n",
"| epoch 2 | iter 1 / 1327 | time 2[s] | perplexity 291.16\n",
"| epoch 2 | iter 21 / 1327 | time 46[s] | perplexity 230.42\n",
"| epoch 2 | iter 41 / 1327 | time 90[s] | perplexity 210.98\n",
"| epoch 2 | iter 61 / 1327 | time 137[s] | perplexity 195.20\n",
"| epoch 2 | iter 81 / 1327 | time 182[s] | perplexity 179.33\n",
"| epoch 2 | iter 101 / 1327 | time 234[s] | perplexity 168.96\n",
"| epoch 2 | iter 121 / 1327 | time 318[s] | perplexity 179.30\n",
"| epoch 2 | iter 141 / 1327 | time 366[s] | perplexity 198.83\n",
"| epoch 2 | iter 161 / 1327 | time 411[s] | perplexity 215.86\n",
"| epoch 2 | iter 181 / 1327 | time 456[s] | perplexity 222.62\n",
"| epoch 2 | iter 201 / 1327 | time 501[s] | perplexity 206.98\n",
"| epoch 2 | iter 221 / 1327 | time 547[s] | perplexity 204.71\n",
"| epoch 2 | iter 241 / 1327 | time 625[s] | perplexity 197.85\n",
"| epoch 2 | iter 261 / 1327 | time 702[s] | perplexity 213.96\n",
"| epoch 2 | iter 281 / 1327 | time 755[s] | perplexity 205.79\n",
"| epoch 2 | iter 301 / 1327 | time 808[s] | perplexity 186.50\n",
"| epoch 2 | iter 321 / 1327 | time 859[s] | perplexity 152.48\n",
"| epoch 2 | iter 341 / 1327 | time 907[s] | perplexity 198.20\n",
"| epoch 2 | iter 361 / 1327 | time 957[s] | perplexity 215.05\n",
"| epoch 2 | iter 381 / 1327 | time 1006[s] | perplexity 170.60\n",
"| epoch 2 | iter 401 / 1327 | time 1067[s] | perplexity 193.21\n",
"| epoch 2 | iter 421 / 1327 | time 1135[s] | perplexity 176.80\n",
"| epoch 2 | iter 441 / 1327 | time 1201[s] | perplexity 180.86\n",
"| epoch 2 | iter 461 / 1327 | time 1267[s] | perplexity 182.34\n",
"| epoch 2 | iter 481 / 1327 | time 1332[s] | perplexity 174.90\n",
"| epoch 2 | iter 501 / 1327 | time 1381[s] | perplexity 192.26\n",
"| epoch 2 | iter 521 / 1327 | time 1449[s] | perplexity 189.70\n",
"| epoch 2 | iter 541 / 1327 | time 1541[s] | perplexity 202.00\n",
"| epoch 2 | iter 561 / 1327 | time 1641[s] | perplexity 171.94\n",
"| epoch 2 | iter 581 / 1327 | time 1726[s] | perplexity 158.15\n",
"| epoch 2 | iter 601 / 1327 | time 1818[s] | perplexity 215.57\n",
"| epoch 2 | iter 621 / 1327 | time 1908[s] | perplexity 201.54\n",
"| epoch 2 | iter 641 / 1327 | time 2002[s] | perplexity 185.20\n",
"| epoch 2 | iter 661 / 1327 | time 2090[s] | perplexity 174.48\n",
"| epoch 2 | iter 681 / 1327 | time 2174[s] | perplexity 146.61\n",
"| epoch 2 | iter 701 / 1327 | time 2228[s] | perplexity 171.08\n",
"| epoch 2 | iter 721 / 1327 | time 2284[s] | perplexity 176.13\n",
"| epoch 2 | iter 741 / 1327 | time 2349[s] | perplexity 150.93\n",
"| epoch 2 | iter 761 / 1327 | time 2402[s] | perplexity 149.54\n",
"| epoch 2 | iter 781 / 1327 | time 2482[s] | perplexity 149.39\n",
"| epoch 2 | iter 801 / 1327 | time 2536[s] | perplexity 168.56\n",
"| epoch 2 | iter 821 / 1327 | time 2626[s] | perplexity 161.91\n",
"| epoch 2 | iter 841 / 1327 | time 2680[s] | perplexity 165.24\n",
"| epoch 2 | iter 861 / 1327 | time 2735[s] | perplexity 161.05\n",
"| epoch 2 | iter 881 / 1327 | time 2787[s] | perplexity 149.73\n",
"| epoch 2 | iter 901 / 1327 | time 2843[s] | perplexity 188.35\n",
"| epoch 2 | iter 921 / 1327 | time 2922[s] | perplexity 165.84\n",
"| epoch 2 | iter 941 / 1327 | time 2982[s] | perplexity 168.55\n",
"| epoch 2 | iter 961 / 1327 | time 3047[s] | perplexity 186.29\n",
"| epoch 2 | iter 981 / 1327 | time 3108[s] | perplexity 174.66\n",
"| epoch 2 | iter 1001 / 1327 | time 3162[s] | perplexity 149.82\n",
"| epoch 2 | iter 1021 / 1327 | time 3223[s] | perplexity 174.38\n",
"| epoch 2 | iter 1041 / 1327 | time 3286[s] | perplexity 157.34\n",
"| epoch 2 | iter 1061 / 1327 | time 3348[s] | perplexity 148.87\n",
"| epoch 2 | iter 1081 / 1327 | time 3407[s] | perplexity 123.84\n",
"| epoch 2 | iter 1101 / 1327 | time 3458[s] | perplexity 137.86\n",
"| epoch 2 | iter 1121 / 1327 | time 3511[s] | perplexity 172.72\n",
"| epoch 2 | iter 1141 / 1327 | time 3566[s] | perplexity 164.43\n",
"| epoch 2 | iter 1161 / 1327 | time 3616[s] | perplexity 147.04\n",
"| epoch 2 | iter 1181 / 1327 | time 3666[s] | perplexity 148.45\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"| epoch 2 | iter 1201 / 1327 | time 3715[s] | perplexity 127.46\n",
"| epoch 2 | iter 1221 / 1327 | time 3769[s] | perplexity 125.83\n",
"| epoch 2 | iter 1241 / 1327 | time 3822[s] | perplexity 146.89\n",
"| epoch 2 | iter 1261 / 1327 | time 3871[s] | perplexity 137.78\n",
"| epoch 2 | iter 1281 / 1327 | time 3923[s] | perplexity 140.56\n",
"| epoch 2 | iter 1301 / 1327 | time 3973[s] | perplexity 178.40\n",
"| epoch 2 | iter 1321 / 1327 | time 4023[s] | perplexity 171.81\n",
"evaluating perplexity ...\n",
"209 / 210\n",
"valid perplexity: 145.6046890226078\n",
"--------------------------------------------------\n",
"| epoch 3 | iter 1 / 1327 | time 6[s] | perplexity 222.54\n",
"| epoch 3 | iter 21 / 1327 | time 124[s] | perplexity 161.39\n",
"| epoch 3 | iter 41 / 1327 | time 243[s] | perplexity 152.21\n",
"| epoch 3 | iter 61 / 1327 | time 295[s] | perplexity 142.97\n",
"| epoch 3 | iter 81 / 1327 | time 344[s] | perplexity 128.21\n",
"| epoch 3 | iter 101 / 1327 | time 393[s] | perplexity 122.36\n",
"| epoch 3 | iter 121 / 1327 | time 441[s] | perplexity 133.44\n",
"| epoch 3 | iter 141 / 1327 | time 492[s] | perplexity 146.84\n",
"| epoch 3 | iter 161 / 1327 | time 541[s] | perplexity 162.45\n",
"| epoch 3 | iter 181 / 1327 | time 596[s] | perplexity 169.60\n",
"| epoch 3 | iter 201 / 1327 | time 647[s] | perplexity 157.57\n",
"| epoch 3 | iter 221 / 1327 | time 695[s] | perplexity 155.70\n",
"| epoch 3 | iter 241 / 1327 | time 742[s] | perplexity 152.59\n",
"| epoch 3 | iter 261 / 1327 | time 797[s] | perplexity 163.07\n",
"| epoch 3 | iter 281 / 1327 | time 846[s] | perplexity 156.43\n",
"| epoch 3 | iter 301 / 1327 | time 906[s] | perplexity 138.56\n",
"| epoch 3 | iter 321 / 1327 | time 970[s] | perplexity 112.04\n",
"| epoch 3 | iter 341 / 1327 | time 1036[s] | perplexity 151.57\n",
"| epoch 3 | iter 361 / 1327 | time 1098[s] | perplexity 165.21\n",
"| epoch 3 | iter 381 / 1327 | time 1159[s] | perplexity 131.16\n",
"| epoch 3 | iter 401 / 1327 | time 1215[s] | perplexity 149.15\n",
"| epoch 3 | iter 421 / 1327 | time 1264[s] | perplexity 130.95\n",
"| epoch 3 | iter 441 / 1327 | time 1322[s] | perplexity 140.28\n",
"| epoch 3 | iter 461 / 1327 | time 1375[s] | perplexity 138.78\n",
"| epoch 3 | iter 481 / 1327 | time 1429[s] | perplexity 134.81\n",
"| epoch 3 | iter 501 / 1327 | time 1480[s] | perplexity 149.98\n",
"| epoch 3 | iter 521 / 1327 | time 1534[s] | perplexity 152.35\n",
"| epoch 3 | iter 541 / 1327 | time 1583[s] | perplexity 157.25\n",
"| epoch 3 | iter 561 / 1327 | time 1633[s] | perplexity 134.19\n",
"| epoch 3 | iter 581 / 1327 | time 1682[s] | perplexity 123.06\n",
"| epoch 3 | iter 601 / 1327 | time 1735[s] | perplexity 169.27\n",
"| epoch 3 | iter 621 / 1327 | time 1788[s] | perplexity 160.02\n",
"| epoch 3 | iter 641 / 1327 | time 1838[s] | perplexity 146.95\n",
"| epoch 3 | iter 661 / 1327 | time 1888[s] | perplexity 136.56\n",
"| epoch 3 | iter 681 / 1327 | time 1939[s] | perplexity 117.16\n",
"| epoch 3 | iter 701 / 1327 | time 1991[s] | perplexity 136.33\n",
"| epoch 3 | iter 721 / 1327 | time 2043[s] | perplexity 139.97\n",
"| epoch 3 | iter 741 / 1327 | time 2094[s] | perplexity 121.00\n",
"| epoch 3 | iter 761 / 1327 | time 2154[s] | perplexity 115.03\n",
"| epoch 3 | iter 781 / 1327 | time 2216[s] | perplexity 122.38\n",
"| epoch 3 | iter 801 / 1327 | time 2270[s] | perplexity 135.23\n",
"| epoch 3 | iter 821 / 1327 | time 2326[s] | perplexity 133.33\n",
"| epoch 3 | iter 841 / 1327 | time 2405[s] | perplexity 134.35\n",
"| epoch 3 | iter 861 / 1327 | time 2457[s] | perplexity 130.02\n"
]
}
],
"source": [
"# coding: utf-8\n",
"import sys\n",
"sys.path.append('..')\n",
"from common import config\n",
"# GPUで実行する場合は下記のコメントアウトを消去（要cupy）\n",
"# ==============================================\n",
"# config.GPU = True\n",
"# ==============================================\n",
"from common.optimizer import SGD\n",
"from common.trainer import RnnlmTrainer\n",
"from common.util import eval_perplexity, to_gpu\n",
"from dataset import ptb\n",
"from better_rnnlm import BetterRnnlm\n",
"\n",
"\n",
"# ハイパーパラメータの設定\n",
"batch_size = 20\n",
"wordvec_size = 650\n",
"hidden_size = 650\n",
"time_size = 35\n",
"lr = 20.0\n",
"max_epoch = 40\n",
"max_grad = 0.25\n",
"dropout = 0.5\n",
"\n",
"# 学習データの読み込み\n",
"corpus, word_to_id, id_to_word = ptb.load_data('train')\n",
"corpus_val, _, _ = ptb.load_data('val')\n",
"corpus_test, _, _ = ptb.load_data('test')\n",
"\n",
"if config.GPU:\n",
" corpus = to_gpu(corpus)\n",
" corpus_val = to_gpu(corpus_val)\n",
" corpus_test = to_gpu(corpus_test)\n",
"\n",
"vocab_size = len(word_to_id)\n",
"xs = corpus[:-1]\n",
"ts = corpus[1:]\n",
"\n",
"model = BetterRnnlm(vocab_size, wordvec_size, hidden_size, dropout)\n",
"optimizer = SGD(lr)\n",
"trainer = RnnlmTrainer(model, optimizer)\n",
"\n",
"best_ppl = float('inf')\n",
"for epoch in range(max_epoch):\n",
" trainer.fit(xs, ts, max_epoch=1, batch_size=batch_size,\n",
" time_size=time_size, max_grad=max_grad)\n",
"\n",
" model.reset_state()\n",
" ppl = eval_perplexity(model, corpus_val)\n",
" print('valid perplexity: ', ppl)\n",
"\n",
" if best_ppl > ppl:\n",
" best_ppl = ppl\n",
" model.save_params()\n",
" else:\n",
" lr /= 4.0\n",
" optimizer.lr = lr\n",
"\n",
" model.reset_state()\n",
" print('-' * 50)\n",
"\n",
"\n",
"# テストデータでの評価\n",
"model.reset_state()\n",
"ppl_test = eval_perplexity(model, corpus_test)\n",
"print('test perplexity: ', ppl_test)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6.6 まとめ\n",
"\n",
"- 単純なRNNの勾配喪失・勾配爆発の問題を解決するために、以下のテクニックを導入する。\n",
" - 勾配クリッピング\n",
" - LSTM/GRUなどのゲート付きRNN\n",
"- LSTMに使用されるゲート\n",
" - inputゲート\n",
" - forgetゲート\n",
" - outputゲート\n",
" - それぞれに重みがついている。sigmoid関数で0.0から1.0までの値が使われる。\n",
"- 言語モデルの実装\n",
" - LSTMレイヤの多層化\n",
" - Dropout\n",
" - 重み共有\n",
"- RNNの正則化は重要なテーマであり、Dropoutベースの様々な手法が提案されている。\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment