Skip to content

Instantly share code, notes, and snippets.

@Naetmul

Naetmul/Tensorflow-Keras ADAM with lr=0.0002.txt

Created November 2, 2018 05:45
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Naetmul/218a6f9e4f1523d24bea5ac02c1b450d to your computer and use it in GitHub Desktop.
Save Naetmul/218a6f9e4f1523d24bea5ac02c1b450d to your computer and use it in GitHub Desktop.
Download ZIP
Adam is also affected by the initial learning rate (lr).
Raw
Tensorflow-Keras ADAM with lr=0.0002.txt
Vocabulary size 2544
Sequence length 20
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embedding (Embedding) (None, 20, 50) 127200
_________________________________________________________________
lstm (LSTM) (None, 20, 100) 60400
_________________________________________________________________
lstm_1 (LSTM) (None, 100) 80400
_________________________________________________________________
dense (Dense) (None, 100) 10100
_________________________________________________________________
dense_1 (Dense) (None, 2544) 256944
=================================================================
Total params: 535,044
Trainable params: 535,044
Non-trainable params: 0
_________________________________________________________________
None
Epoch 1/100
2382813/2382813 [==============================] - 1513s 635us/step - loss: 4.1767 - acc: 0.2949
Epoch 2/100
2382813/2382813 [==============================] - 1506s 632us/step - loss: 3.8160 - acc: 0.3248
Epoch 3/100
2382813/2382813 [==============================] - 1434s 602us/step - loss: 3.6767 - acc: 0.3376
Epoch 4/100
2382813/2382813 [==============================] - 1475s 619us/step - loss: 3.5847 - acc: 0.3467
Epoch 5/100
2382813/2382813 [==============================] - 1416s 594us/step - loss: 3.5137 - acc: 0.3541
Epoch 6/100
2382813/2382813 [==============================] - 1424s 598us/step - loss: 3.4575 - acc: 0.3601
Epoch 7/100
2382813/2382813 [==============================] - 1414s 594us/step - loss: 3.4117 - acc: 0.3652
Epoch 8/100
2382813/2382813 [==============================] - 1499s 629us/step - loss: 3.3732 - acc: 0.3692
Epoch 9/100
2382813/2382813 [==============================] - 1464s 614us/step - loss: 3.3412 - acc: 0.3724
Epoch 10/100
2382813/2382813 [==============================] - 1512s 635us/step - loss: 3.3141 - acc: 0.3749
Epoch 11/100
2382813/2382813 [==============================] - 1439s 604us/step - loss: 3.2908 - acc: 0.3774
Epoch 12/100
2382813/2382813 [==============================] - 1441s 605us/step - loss: 3.2710 - acc: 0.3793
Epoch 13/100
2382813/2382813 [==============================] - 1459s 612us/step - loss: 3.2534 - acc: 0.3811
Epoch 14/100
2382813/2382813 [==============================] - 1428s 599us/step - loss: 3.2379 - acc: 0.3827
Epoch 15/100
2382813/2382813 [==============================] - 1440s 604us/step - loss: 3.2238 - acc: 0.3845
Epoch 16/100
2382813/2382813 [==============================] - 1415s 594us/step - loss: 3.2109 - acc: 0.3858
Epoch 17/100
2382813/2382813 [==============================] - 1432s 601us/step - loss: 3.1992 - acc: 0.3871
Epoch 18/100
2382813/2382813 [==============================] - 1414s 593us/step - loss: 3.1885 - acc: 0.3883
Epoch 19/100
2382813/2382813 [==============================] - 1411s 592us/step - loss: 3.1787 - acc: 0.3894
Epoch 20/100
2382813/2382813 [==============================] - 1414s 593us/step - loss: 3.1696 - acc: 0.3905
Epoch 21/100
2382813/2382813 [==============================] - 1410s 592us/step - loss: 3.1610 - acc: 0.3913
Epoch 22/100
2382813/2382813 [==============================] - 1417s 595us/step - loss: 3.1533 - acc: 0.3921
Epoch 23/100
2382813/2382813 [==============================] - 1410s 592us/step - loss: 3.1457 - acc: 0.3931
Epoch 24/100
2382813/2382813 [==============================] - 1417s 595us/step - loss: 3.1387 - acc: 0.3939
Epoch 25/100
2382813/2382813 [==============================] - 1406s 590us/step - loss: 3.1322 - acc: 0.3946
Epoch 26/100
2382813/2382813 [==============================] - 1416s 594us/step - loss: 3.1259 - acc: 0.3954
Epoch 27/100
2382813/2382813 [==============================] - 1415s 594us/step - loss: 3.1199 - acc: 0.3962
Epoch 28/100
2382813/2382813 [==============================] - 1416s 594us/step - loss: 3.1142 - acc: 0.3967
Epoch 29/100
2382813/2382813 [==============================] - 1407s 590us/step - loss: 3.1091 - acc: 0.3975
Epoch 30/100
2382813/2382813 [==============================] - 1438s 603us/step - loss: 3.1039 - acc: 0.3981
Epoch 31/100
2382813/2382813 [==============================] - 1411s 592us/step - loss: 3.0991 - acc: 0.3987
Epoch 32/100
2382813/2382813 [==============================] - 1417s 595us/step - loss: 3.0944 - acc: 0.3992
Epoch 33/100
2382813/2382813 [==============================] - 1416s 594us/step - loss: 3.0899 - acc: 0.3998
Epoch 34/100
2382813/2382813 [==============================] - 1413s 593us/step - loss: 3.0857 - acc: 0.4003
Epoch 35/100
2382813/2382813 [==============================] - 1415s 594us/step - loss: 3.0815 - acc: 0.4009
Epoch 36/100
2382813/2382813 [==============================] - 1413s 593us/step - loss: 3.0776 - acc: 0.4012
Epoch 37/100
2382813/2382813 [==============================] - 1415s 594us/step - loss: 3.0738 - acc: 0.4018
Epoch 38/100
2382813/2382813 [==============================] - 1416s 594us/step - loss: 3.0701 - acc: 0.4020
Epoch 39/100
2382813/2382813 [==============================] - 1425s 598us/step - loss: 3.0665 - acc: 0.4028
Epoch 40/100
2382813/2382813 [==============================] - 1407s 590us/step - loss: 3.0631 - acc: 0.4029
Epoch 41/100
2382813/2382813 [==============================] - 1414s 594us/step - loss: 3.0597 - acc: 0.4035
Epoch 42/100
2382813/2382813 [==============================] - 1410s 592us/step - loss: 3.0565 - acc: 0.4040
Epoch 43/100
2382813/2382813 [==============================] - 1444s 606us/step - loss: 3.0533 - acc: 0.4045
Epoch 44/100
2382813/2382813 [==============================] - 1470s 617us/step - loss: 3.0504 - acc: 0.4047
Epoch 45/100
2382813/2382813 [==============================] - 1503s 631us/step - loss: 3.0473 - acc: 0.4051
Epoch 46/100
2382813/2382813 [==============================] - 1442s 605us/step - loss: 3.0445 - acc: 0.4052
Epoch 47/100
2382813/2382813 [==============================] - 1482s 622us/step - loss: 3.0417 - acc: 0.4057
Epoch 48/100
2382813/2382813 [==============================] - 1434s 602us/step - loss: 3.0392 - acc: 0.4062
Epoch 49/100
2382813/2382813 [==============================] - 1435s 602us/step - loss: 3.0368 - acc: 0.4061
Epoch 50/100
2382813/2382813 [==============================] - 1431s 600us/step - loss: 3.0341 - acc: 0.4065
Epoch 51/100
2382813/2382813 [==============================] - 1441s 605us/step - loss: 3.0318 - acc: 0.4069
Epoch 52/100
2382813/2382813 [==============================] - 1504s 631us/step - loss: 3.0295 - acc: 0.4071
Epoch 53/100
2382813/2382813 [==============================] - 1490s 625us/step - loss: 3.0271 - acc: 0.4076
Epoch 54/100
2382813/2382813 [==============================] - 1532s 643us/step - loss: 3.0249 - acc: 0.4077
Epoch 55/100
2382813/2382813 [==============================] - 1510s 634us/step - loss: 3.0227 - acc: 0.4080
Epoch 56/100
2382813/2382813 [==============================] - 1549s 650us/step - loss: 3.0206 - acc: 0.4083
Epoch 57/100
2382813/2382813 [==============================] - 1537s 645us/step - loss: 3.0186 - acc: 0.4085
Epoch 58/100
2382813/2382813 [==============================] - 1525s 640us/step - loss: 3.0166 - acc: 0.4087
Epoch 59/100
2382813/2382813 [==============================] - 1511s 634us/step - loss: 3.0147 - acc: 0.4089
Epoch 60/100
2382813/2382813 [==============================] - 1530s 642us/step - loss: 3.0129 - acc: 0.4092
Epoch 61/100
2382813/2382813 [==============================] - 1499s 629us/step - loss: 3.0111 - acc: 0.4094
Epoch 62/100
2382813/2382813 [==============================] - 1466s 615us/step - loss: 3.0093 - acc: 0.4097
Epoch 63/100
2382813/2382813 [==============================] - 1462s 613us/step - loss: 3.0076 - acc: 0.4098
Epoch 64/100
2382813/2382813 [==============================] - 1482s 622us/step - loss: 3.0060 - acc: 0.4099
Epoch 65/100
2382813/2382813 [==============================] - 1454s 610us/step - loss: 3.0042 - acc: 0.4102
Epoch 66/100
2382813/2382813 [==============================] - 1449s 608us/step - loss: 3.0026 - acc: 0.4104
Epoch 67/100
2382813/2382813 [==============================] - 1425s 598us/step - loss: 3.0010 - acc: 0.4106
Epoch 68/100
2382813/2382813 [==============================] - 1432s 601us/step - loss: 2.9993 - acc: 0.4106
Epoch 69/100
2382813/2382813 [==============================] - 1428s 599us/step - loss: 2.9979 - acc: 0.4110
Epoch 70/100
2382813/2382813 [==============================] - 1427s 599us/step - loss: 2.9963 - acc: 0.4111
Epoch 71/100
2382813/2382813 [==============================] - 1422s 597us/step - loss: 2.9948 - acc: 0.4113
Epoch 72/100
2382813/2382813 [==============================] - 1434s 602us/step - loss: 2.9936 - acc: 0.4114
Epoch 73/100
2382813/2382813 [==============================] - 1427s 599us/step - loss: 2.9922 - acc: 0.4117
Epoch 74/100
2382813/2382813 [==============================] - 1424s 597us/step - loss: 2.9909 - acc: 0.4118
Epoch 75/100
2382813/2382813 [==============================] - 1434s 602us/step - loss: 2.9895 - acc: 0.4121
Epoch 76/100
2382813/2382813 [==============================] - 1429s 600us/step - loss: 2.9882 - acc: 0.4121
Epoch 77/100
2382813/2382813 [==============================] - 1429s 600us/step - loss: 2.9871 - acc: 0.4123
Epoch 78/100
2382813/2382813 [==============================] - 1425s 598us/step - loss: 2.9858 - acc: 0.4123
Epoch 79/100
2382813/2382813 [==============================] - 1423s 597us/step - loss: 2.9847 - acc: 0.4125
Epoch 80/100
2382813/2382813 [==============================] - 1424s 598us/step - loss: 2.9838 - acc: 0.4127
Epoch 81/100
2382813/2382813 [==============================] - 1430s 600us/step - loss: 2.9826 - acc: 0.4128
Epoch 82/100
2382813/2382813 [==============================] - 1422s 597us/step - loss: 2.9814 - acc: 0.4131
Epoch 83/100
2382813/2382813 [==============================] - 1426s 598us/step - loss: 2.9802 - acc: 0.4132
Epoch 84/100
2382813/2382813 [==============================] - 1425s 598us/step - loss: 2.9793 - acc: 0.4130
Epoch 85/100
2382813/2382813 [==============================] - 1432s 601us/step - loss: 2.9782 - acc: 0.4134
Epoch 86/100
2382813/2382813 [==============================] - 1425s 598us/step - loss: 2.9774 - acc: 0.4134
Epoch 87/100
2382813/2382813 [==============================] - 1432s 601us/step - loss: 2.9765 - acc: 0.4134
Epoch 88/100
2382813/2382813 [==============================] - 1420s 596us/step - loss: 2.9755 - acc: 0.4136
Epoch 89/100
2382813/2382813 [==============================] - 1430s 600us/step - loss: 2.9745 - acc: 0.4137
Epoch 90/100
2382813/2382813 [==============================] - 1427s 599us/step - loss: 2.9735 - acc: 0.4140
Epoch 91/100
2382813/2382813 [==============================] - 1458s 612us/step - loss: 2.9725 - acc: 0.4140
Epoch 92/100
2382813/2382813 [==============================] - 1447s 607us/step - loss: 2.9717 - acc: 0.4142
Epoch 93/100
2382813/2382813 [==============================] - 1448s 608us/step - loss: 2.9709 - acc: 0.4144
Epoch 94/100
2382813/2382813 [==============================] - 1443s 606us/step - loss: 2.9702 - acc: 0.4143
Epoch 95/100
2382813/2382813 [==============================] - 1435s 602us/step - loss: 2.9693 - acc: 0.4144
Epoch 96/100
2382813/2382813 [==============================] - 1444s 606us/step - loss: 2.9682 - acc: 0.4146
Epoch 97/100
2382813/2382813 [==============================] - 1435s 602us/step - loss: 2.9675 - acc: 0.4145
Epoch 98/100
2382813/2382813 [==============================] - 1441s 605us/step - loss: 2.9668 - acc: 0.4148
Epoch 99/100
2382813/2382813 [==============================] - 1443s 605us/step - loss: 2.9660 - acc: 0.4150
Epoch 100/100
2382813/2382813 [==============================] - 1436s 602us/step - loss: 2.9652 - acc: 0.4150
Raw
Tensorflow-Keras ADAM with lr=0.001.txt
Vocabulary size 2544
Sequence length 20
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embedding (Embedding) (None, 20, 50) 127200
_________________________________________________________________
lstm (LSTM) (None, 20, 100) 60400
_________________________________________________________________
lstm_1 (LSTM) (None, 100) 80400
_________________________________________________________________
dense (Dense) (None, 100) 10100
_________________________________________________________________
dense_1 (Dense) (None, 2544) 256944
=================================================================
Total params: 535,044
Trainable params: 535,044
Non-trainable params: 0
_________________________________________________________________
None
Epoch 1/100
2382813/2382813 [==============================] - 1494s 627us/step - loss: 3.8172 - acc: 0.3246
Epoch 2/100
2382813/2382813 [==============================] - 1424s 598us/step - loss: 3.4649 - acc: 0.3593
Epoch 3/100
2382813/2382813 [==============================] - 1451s 609us/step - loss: 3.3591 - acc: 0.3698
Epoch 4/100
2382813/2382813 [==============================] - 1511s 634us/step - loss: 3.3018 - acc: 0.3758
Epoch 5/100
2382813/2382813 [==============================] - 1455s 611us/step - loss: 3.2621 - acc: 0.3802
Epoch 6/100
2382813/2382813 [==============================] - 1439s 604us/step - loss: 3.2342 - acc: 0.3834
Epoch 7/100
2382813/2382813 [==============================] - 1422s 597us/step - loss: 3.2131 - acc: 0.3860
Epoch 8/100
2382813/2382813 [==============================] - 1461s 613us/step - loss: 3.1970 - acc: 0.3879
Epoch 9/100
2382813/2382813 [==============================] - 1465s 615us/step - loss: 3.1841 - acc: 0.3894
Epoch 10/100
2382813/2382813 [==============================] - 1447s 607us/step - loss: 3.1751 - acc: 0.3907
Epoch 11/100
2382813/2382813 [==============================] - 1420s 596us/step - loss: 3.1652 - acc: 0.3916
Epoch 12/100
2382813/2382813 [==============================] - 1422s 597us/step - loss: 3.1580 - acc: 0.3927
Epoch 13/100
2382813/2382813 [==============================] - 1426s 599us/step - loss: 3.1519 - acc: 0.3934
Epoch 14/100
2382813/2382813 [==============================] - 1474s 619us/step - loss: 3.1463 - acc: 0.3942
Epoch 15/100
2382813/2382813 [==============================] - 1454s 610us/step - loss: 3.1418 - acc: 0.3948
Epoch 16/100
2382813/2382813 [==============================] - 1430s 600us/step - loss: 3.1378 - acc: 0.3954
Epoch 17/100
2382813/2382813 [==============================] - 1412s 593us/step - loss: 3.1345 - acc: 0.3957
Epoch 18/100
2382813/2382813 [==============================] - 1422s 597us/step - loss: 3.1313 - acc: 0.3962
Epoch 19/100
2382813/2382813 [==============================] - 1419s 595us/step - loss: 3.1290 - acc: 0.3965
Epoch 20/100
2382813/2382813 [==============================] - 1421s 596us/step - loss: 3.1262 - acc: 0.3971
Epoch 21/100
2382813/2382813 [==============================] - 1424s 598us/step - loss: 3.1251 - acc: 0.3971
Epoch 22/100
2382813/2382813 [==============================] - 1416s 594us/step - loss: 3.1230 - acc: 0.3975
Epoch 23/100
2382813/2382813 [==============================] - 1433s 601us/step - loss: 3.1214 - acc: 0.3976
Epoch 24/100
2382813/2382813 [==============================] - 1425s 598us/step - loss: 3.1207 - acc: 0.3979
Epoch 25/100
2382813/2382813 [==============================] - 1422s 597us/step - loss: 3.1192 - acc: 0.3983
Epoch 26/100
2382813/2382813 [==============================] - 1414s 594us/step - loss: 3.1183 - acc: 0.3983
Epoch 27/100
2382813/2382813 [==============================] - 1456s 611us/step - loss: 3.1177 - acc: 0.3983
Epoch 28/100
2382813/2382813 [==============================] - 1417s 594us/step - loss: 3.1166 - acc: 0.3986
Epoch 29/100
2382813/2382813 [==============================] - 1426s 599us/step - loss: 3.1163 - acc: 0.3988
Epoch 30/100
2382813/2382813 [==============================] - 1419s 596us/step - loss: 3.1155 - acc: 0.3988
Epoch 31/100
2382813/2382813 [==============================] - 1426s 598us/step - loss: 3.1152 - acc: 0.3991
Epoch 32/100
2382813/2382813 [==============================] - 1420s 596us/step - loss: 3.1141 - acc: 0.3988
Epoch 33/100
2382813/2382813 [==============================] - 1443s 605us/step - loss: 3.1140 - acc: 0.3992
Epoch 34/100
2382813/2382813 [==============================] - 1428s 599us/step - loss: 3.1138 - acc: 0.3993
Epoch 35/100
2382813/2382813 [==============================] - 1437s 603us/step - loss: 3.1131 - acc: 0.3995
Epoch 36/100
2382813/2382813 [==============================] - 1429s 600us/step - loss: 3.1137 - acc: 0.3991
Epoch 37/100
2382813/2382813 [==============================] - 1449s 608us/step - loss: 3.1141 - acc: 0.3992
Epoch 38/100
2382813/2382813 [==============================] - 1572s 660us/step - loss: 3.1144 - acc: 0.3994
Epoch 39/100
2382813/2382813 [==============================] - 1479s 621us/step - loss: 3.1140 - acc: 0.3993
Epoch 40/100
2382813/2382813 [==============================] - 1480s 621us/step - loss: 3.1142 - acc: 0.3994
Epoch 41/100
2382813/2382813 [==============================] - 1487s 624us/step - loss: 3.1145 - acc: 0.3995
Epoch 42/100
2382813/2382813 [==============================] - 1469s 617us/step - loss: 3.1147 - acc: 0.3996
Epoch 43/100
2382813/2382813 [==============================] - 1472s 618us/step - loss: 3.1156 - acc: 0.3993
Epoch 44/100
2382813/2382813 [==============================] - 1465s 615us/step - loss: 3.1160 - acc: 0.3994
Epoch 45/100
2382813/2382813 [==============================] - 1477s 620us/step - loss: 3.1159 - acc: 0.3994
Epoch 46/100
2382813/2382813 [==============================] - 1506s 632us/step - loss: 3.1170 - acc: 0.3993
Epoch 47/100
2382813/2382813 [==============================] - 1532s 643us/step - loss: 3.1173 - acc: 0.3993
Epoch 48/100
2382813/2382813 [==============================] - 1496s 628us/step - loss: 3.1179 - acc: 0.3993
Epoch 49/100
2382813/2382813 [==============================] - 1529s 642us/step - loss: 3.1176 - acc: 0.3994
Epoch 50/100
2382813/2382813 [==============================] - 1607s 674us/step - loss: 3.1180 - acc: 0.3993
Epoch 51/100
2382813/2382813 [==============================] - 1548s 650us/step - loss: 3.1183 - acc: 0.3994
Epoch 52/100
2382813/2382813 [==============================] - 1515s 636us/step - loss: 3.1193 - acc: 0.3992
Epoch 53/100
2382813/2382813 [==============================] - 1531s 642us/step - loss: 3.1197 - acc: 0.3994
Epoch 54/100
2382813/2382813 [==============================] - 1631s 685us/step - loss: 3.1210 - acc: 0.3991
Ctrl+C
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment