Comparison of Chainer's mnist example between in FP32 mode and in FP16 mode

mnist_fp16

$ CHAINER_DTYPE=float16 python train_mnist.py -d 0
Device: @cupy:0
# unit: 1000
# Minibatch-size: 100
# epoch: 20

epoch       main/loss   validation/main/loss  main/accuracy  validation/main/accuracy  elapsed_time
1           nan         nan                   0.0994271      0.0980225                 3.91818
2           nan         nan                   0.0997917      0.0980225                 6.22553
3           nan         nan                   0.0995833      0.0980225                 8.72424
4           nan         nan                   0.099375       0.0980225                 11.3325
5           nan         nan                   0.0997396      0.0980225                 13.7157
6           nan         nan                   0.0995312      0.0980225                 16.2174
7           nan         nan                   0.0994271      0.0980225                 18.8041
8           nan         nan                   0.0996354      0.0980225                 21.3289
9           nan         nan                   0.0996875      0.0980225                 23.8468
10          nan         nan                   0.099375       0.0980225                 26.3605
11          nan         nan                   0.0995833      0.0980225                 28.8567
12          nan         nan                   0.0993229      0.0980225                 31.3275
13          nan         nan                   0.0996875      0.0980225                 33.886
14          nan         nan                   0.0996875      0.0980225                 36.3637
15          nan         nan                   0.0996875      0.0980225                 38.6991
16          nan         nan                   0.0994792      0.0980225                 40.981
17          nan         nan                   0.0991667      0.0980225                 43.2793
18          nan         nan                   0.0992187      0.0980225                 45.6158
19          nan         nan                   0.0997917      0.0980225                 47.8968
20          nan         nan                   0.0998437      0.0980225                 50.1865

mnist_fp32

$ python train_mnist.py -d 0
Device: @cupy:0
# unit: 1000
# Minibatch-size: 100
# epoch: 20

Downloading from http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz...
Downloading from http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz...
Downloading from http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz...
Downloading from http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz...
epoch       main/loss   validation/main/loss  main/accuracy  validation/main/accuracy  elapsed_time
1           0.191721    0.103644              0.942867       0.966                     3.72396
2           0.0727625   0.0803097             0.977783       0.9745                    6.0102
3           0.0488638   0.0803536             0.984132       0.974                     8.31127
4           0.0353504   0.0675346             0.988282       0.9814                    10.617
5           0.0268242   0.0791875             0.991048       0.9788                    12.9174
6           0.0236898   0.0881506             0.992232       0.9778                    15.253
7           0.0250604   0.0709666             0.991898       0.9806                    17.5925
8           0.0176728   0.082598              0.994231       0.9785                    19.9265
9           0.0146536   0.0784669             0.995482       0.9802                    22.207
10          0.015635    0.0938949             0.994982       0.9796                    24.4942
11          0.0143474   0.0830709             0.995849       0.9803                    26.7645
12          0.0112423   0.097842              0.996365       0.9798                    29.0801
13          0.0117989   0.0894112             0.996432       0.9828                    31.4166
14          0.0116941   0.112915              0.996582       0.9801                    33.6981
15          0.0136293   0.0840091             0.995899       0.9823                    35.9836
16          0.00918777  0.0957944             0.997099       0.9824                    38.2678
17          0.00915884  0.120189              0.997132       0.9762                    40.564
18          0.0102279   0.0965553             0.997149       0.9837                    42.8879
19          0.00831765  0.100924              0.997699       0.9816                    45.2091
20          0.0102448   0.111097              0.997066       0.9817                    47.5284

runtime_info

$ python
Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import chainer
>>> chainer.print_runtime_info()
Platform: Linux-4.15.0-50-generic-x86_64-with-debian-buster-sid
Chainer: 6.0.0
NumPy: 1.16.4
CuPy:
  CuPy Version          : 6.0.0
  CUDA Root             : /usr/local/cuda
  CUDA Build Version    : 10000
  CUDA Driver Version   : 10000
  CUDA Runtime Version  : 10000
  cuDNN Build Version   : 7500
  cuDNN Version         : 7500
  NCCL Build Version    : 2402
  NCCL Runtime Version  : 2402
iDeep: 2.0.0.post3
>>>