ChainerMN with nvidia-docker on p3

00.README.md

Links

• https://github.com/mazgi/docker-cuda-cv/tree/615706f87e0555cdac40c6eeb5eea11b63751029

10.single-node-single-gpu.md

[user@p3-mazgi] $ nvidia-smi
Fri Dec  1 09:11:28 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.90                 Driver Version: 384.90                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   39C    P0    37W / 300W |    460MiB / 16152MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       526      C   python3                                      450MiB |
+-----------------------------------------------------------------------------+
[user@p3-mazgi] $ git clone --depth=1 https://github.com/chainer/chainermn.git
Cloning into 'chainermn'...
remote: Counting objects: 126, done.
remote: Compressing objects: 100% (122/122), done.
remote: Total 126 (delta 8), reused 46 (delta 1), pack-reused 0
Receiving objects: 100% (126/126), 1.96 MiB | 914.00 KiB/s, done.
Resolving deltas: 100% (8/8), done.
Checking connectivity... done.
[user@p3-mazgi] $ docker pull mazgi/cuda-cv:9.0-cudnn7-devel-ubuntu16.04
9.0-cudnn7-devel-ubuntu16.04: Pulling from mazgi/cuda-cv
Digest: sha256:8503b454b522497633a19e410c0b37c064cf54125eca3d16469a731e69a7e244
Status: Image is up to date for mazgi/cuda-cv:9.0-cudnn7-devel-ubuntu16.04
[user@p3-mazgi] $ nvidia-docker run -v $(pwd):/mnt mazgi/cuda-cv:9.0-cudnn7-devel-ubuntu16.04 mpiexec --allow-run-as-root -n $(nvidia-smi -L | wc -l) python3 /mnt/chainermn/examples/mnist/train_mnist.py --gpu
==========================================
Num process (COMM_WORLD): 1
Using GPUs
Using hierarchical communicator
Num unit: 1000
Num Minibatch-size: 100
Num epoch: 20
==========================================
epoch       main/loss   validation/main/loss  main/accuracy  validation/main/accuracy  elapsed_time
Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/ZEYECI64A342DLUO2ZVF2CX25W:/var/lib/docker/overlay2/l/5YWR2W6LW2AA7BOZCKIMHXW2S7:/var/lib/docker/overlay2/l/2XV32RLGHQDXSIJZEX7QW6SM3H:/var/lib/docker/overlay2/l/BXTPWC5HK2APQZ6L4TWTSQ7IGC:/var/lib/docker/overlay2/l/GSMVJ7HEQANHQNNPLYWAUCLUZJ:/var/lib/docker/overlay2/l/34HBE6ON2CD3MJQUJC2DE2ZJQ4:/var/lib/docker/overlay2/l/K53R4BHBI7GEP2YPF43XCBNILL:/var/lib/docker/overlay2/l/3HEPWIXHVM25NHCXNGUOY3Y4M5:/var/lib/docker/overlay2/l/ICHLUN7H5VVWC'
Unexpected end of /proc/mounts line `BL3EGEYTHVAKI:/var/lib/docker/overlay2/l/DUBW7YAU55EXPAHLJEMSNUGGRB:/var/lib/docker/overlay2/l/M5AXNMY4HLIMVTJDOLZ4NKCIGL:/var/lib/docker/overlay2/l/WMCADCGE5KV73OMLJUDISI6QBU:/var/lib/docker/overlay2/l/5W6WFWBYDWG6AXFUNSOZHOVAXA:/var/lib/docker/overlay2/l/AHLOVSBW3JMY34GUDN57FJ54YA:/var/lib/docker/overlay2/l/MRAGHIGHI3SK6SWYLUF6YMV23R:/var/lib/docker/overlay2/l/B2DP3URUYG5247YIO4RDCHBKM3:/var/lib/docker/overlay2/l/VBU75G2K3DTUVAGMUWME2OZ6H3:/var/lib/docker/overlay2/l/MUNKARMOL4O3RAELM3QA4ZG5N4,upperdir=/v'
1           0.196025    0.100757              0.939467       0.9688                    16.9047       
2           0.075174    0.0786887             0.976416       0.9755                    21.6053       
3           0.0486721   0.0715596             0.984515       0.9797                    26.3054       
4           0.0365207   0.0714404             0.988349       0.9793                    31.0116       
5           0.0298104   0.0816801             0.989948       0.9785                    35.7117       
6           0.0242309   0.0750142             0.991965       0.9814                    40.4067       
7           0.020712    0.0924447             0.993532       0.9785                    45.1009       
8           0.0196817   0.0881265             0.993681       0.978                     49.7997       
9           0.0158144   0.0804658             0.994815       0.9814                    54.496        
10          0.0154907   0.0938101             0.994882       0.9799                    59.205        
11          0.0128462   0.096795              0.995965       0.9809                    63.8997       
12          0.0138647   0.0838137             0.995765       0.9807                    68.6034       
13          0.0139337   0.0927002             0.995765       0.9812                    73.3142       
14          0.00941665  0.0829678             0.997116       0.9832                    78.0221       
15          0.0111692   0.136958              0.996566       0.9746                    82.7157       
16          0.00998178  0.0934425             0.997249       0.9824                    87.4262       
17          0.00874116  0.116233              0.997316       0.9838                    92.1265       
18          0.00979822  0.120985              0.997216       0.9812                    96.8448       
19          0.0090636   0.122347              0.997633       0.9818                    101.542       
20          0.00953769  0.117708              0.997049       0.9823                    106.233       
/usr/local/lib/python3.5/dist-packages/cupy/core/fusion.py:659: FutureWarning: cupy.core.fusion is experimental. The interface can change in the future.
  util.experimental('cupy.core.fusion')
Downloading from http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz...
Downloading from http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz...
Downloading from http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz...
Downloading from http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz...

11.single-node-multi-gpus.md

$ nvidia-smi 
Fri Dec  1 09:16:26 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.90                 Driver Version: 384.90                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:00:17.0 Off |                    0 |
| N/A   45C    P0    56W / 300W |     10MiB / 16152MiB |     10%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  Off  | 00000000:00:18.0 Off |                    0 |
| N/A   39C    P0    40W / 300W |     10MiB / 16152MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  Off  | 00000000:00:19.0 Off |                    0 |
| N/A   40C    P0    42W / 300W |     10MiB / 16152MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  Off  | 00000000:00:1A.0 Off |                    0 |
| N/A   42C    P0    41W / 300W |     10MiB / 16152MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-SXM2...  Off  | 00000000:00:1B.0 Off |                    0 |
| N/A   44C    P0    44W / 300W |     10MiB / 16152MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-SXM2...  Off  | 00000000:00:1C.0 Off |                    0 |
| N/A   41C    P0    42W / 300W |     10MiB / 16152MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-SXM2...  Off  | 00000000:00:1D.0 Off |                    0 |
| N/A   39C    P0    41W / 300W |     10MiB / 16152MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-SXM2...  Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   42C    P0    55W / 300W |     10MiB / 16152MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
$ git clone --depth=1 https://github.com/chainer/chainermn.git
Cloning into 'chainermn'...
remote: Counting objects: 126, done.
remote: Compressing objects: 100% (122/122), done.
remote: Total 126 (delta 8), reused 46 (delta 1), pack-reused 0
Receiving objects: 100% (126/126), 1.96 MiB | 1.47 MiB/s, done.
Resolving deltas: 100% (8/8), done.
Checking connectivity... done.
$ docker pull mazgi/cuda-cv:9.0-cudnn7-devel-ubuntu16.04
9.0-cudnn7-devel-ubuntu16.04: Pulling from mazgi/cuda-cv
660c48dd555d: Pull complete 
4c7380416e78: Pull complete 
421e436b5f80: Pull complete 
e4ce6c3651b3: Pull complete 
be588e74bd34: Pull complete 
f597507b3c37: Pull complete 
9c5d4127a23d: Pull complete 
398bf259fcdc: Pull complete 
4f4092762618: Pull complete 
94130a21e154: Pull complete 
777c330a57a0: Pull complete 
869e77002d74: Pull complete 
2ea753515b1e: Pull complete 
98cc76538f16: Pull complete 
9f379402e346: Pull complete 
ea1f40a9dc63: Pull complete 
315cfc91391e: Pull complete 
Digest: sha256:8503b454b522497633a19e410c0b37c064cf54125eca3d16469a731e69a7e244
Status: Downloaded newer image for mazgi/cuda-cv:9.0-cudnn7-devel-ubuntu16.04
$ nvidia-docker run -v $(pwd):/mnt mazgi/cuda-cv:9.0-cudnn7-devel-ubuntu16.04 mpiexec --allow-run-as-root -n $(nvidia-smi -L | wc -l) python3 /mnt/chainermn/examples/mnist/train_mnist.py --gpu
==========================================
Num process (COMM_WORLD): 8
Using GPUs
Using hierarchical communicator
Num unit: 1000
Num Minibatch-size: 100
Num epoch: 20
==========================================
epoch       main/loss   validation/main/loss  main/accuracy  validation/main/accuracy  elapsed_time
Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/BPK22XVXRMMTC4BKKOMOR7ZAI2:/var/lib/docker/overlay2/l/33N3JY5BIS6BU553FPQ4ZQKCG3:/var/lib/docker/overlay2/l/FRQ3NCT75NVSIGYMWNTEE5USOX:/var/lib/docker/overlay2/l/YHATTH7VMCXC6N3N5Q44TJ4Y3L:/var/lib/docker/overlay2/l/IFZQAKFIRRDE52LUAQ6CXPQGCD:/var/lib/docker/overlay2/l/LKQNRGHQ54FISASSKCICFD267U:/var/lib/docker/overlay2/l/DRLEYZBURZLTL3XQ2RGUSZ6ZLD:/var/lib/docker/overlay2/l/GW453AJ6NRFMTW54T77CVKFHOT:/var/lib/docker/overlay2/l/N7Y4YGYFSMROV'
Unexpected end of /proc/mounts line `6J5LYBYFFUDN3:/var/lib/docker/overlay2/l/SM77AQWNWR6SLRV5M2NDGHTO5Q:/var/lib/docker/overlay2/l/SGPH3NDKZX3K6GX76GIEBDF3AD:/var/lib/docker/overlay2/l/AQUHUBJZTAWIYIGQACVI6RYRIY:/var/lib/docker/overlay2/l/WUR7WLCVSN342XTP24R3MG3KOG:/var/lib/docker/overlay2/l/O7N65L3RAOVJMB2SCHFZHUVVRI:/var/lib/docker/overlay2/l/RXRELJLOCHEO4V56W5PGHPBXXG:/var/lib/docker/overlay2/l/5NVZ3WHCB5UPKSYVDTIC7VJ7CX:/var/lib/docker/overlay2/l/X4LTBNC3HBU5S6EPJ374XN4UMN:/var/lib/docker/overlay2/l/QQKCWK5WH7GJMTMCDJATXG3AW7,upperdir=/v'
Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/BPK22XVXRMMTC4BKKOMOR7ZAI2:/var/lib/docker/overlay2/l/33N3JY5BIS6BU553FPQ4ZQKCG3:/var/lib/docker/overlay2/l/FRQ3NCT75NVSIGYMWNTEE5USOX:/var/lib/docker/overlay2/l/YHATTH7VMCXC6N3N5Q44TJ4Y3L:/var/lib/docker/overlay2/l/IFZQAKFIRRDE52LUAQ6CXPQGCD:/var/lib/docker/overlay2/l/LKQNRGHQ54FISASSKCICFD267U:/var/lib/docker/overlay2/l/DRLEYZBURZLTL3XQ2RGUSZ6ZLD:/var/lib/docker/overlay2/l/GW453AJ6NRFMTW54T77CVKFHOT:/var/lib/docker/overlay2/l/N7Y4YGYFSMROV'
Unexpected end of /proc/mounts line `6J5LYBYFFUDN3:/var/lib/docker/overlay2/l/SM77AQWNWR6SLRV5M2NDGHTO5Q:/var/lib/docker/overlay2/l/SGPH3NDKZX3K6GX76GIEBDF3AD:/var/lib/docker/overlay2/l/AQUHUBJZTAWIYIGQACVI6RYRIY:/var/lib/docker/overlay2/l/WUR7WLCVSN342XTP24R3MG3KOG:/var/lib/docker/overlay2/l/O7N65L3RAOVJMB2SCHFZHUVVRI:/var/lib/docker/overlay2/l/RXRELJLOCHEO4V56W5PGHPBXXG:/var/lib/docker/overlay2/l/5NVZ3WHCB5UPKSYVDTIC7VJ7CX:/var/lib/docker/overlay2/l/X4LTBNC3HBU5S6EPJ374XN4UMN:/var/lib/docker/overlay2/l/QQKCWK5WH7GJMTMCDJATXG3AW7,upperdir=/v'
Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/BPK22XVXRMMTC4BKKOMOR7ZAI2:/var/lib/docker/overlay2/l/33N3JY5BIS6BU553FPQ4ZQKCG3:/var/lib/docker/overlay2/l/FRQ3NCT75NVSIGYMWNTEE5USOX:/var/lib/docker/overlay2/l/YHATTH7VMCXC6N3N5Q44TJ4Y3L:/var/lib/docker/overlay2/l/IFZQAKFIRRDE52LUAQ6CXPQGCD:/var/lib/docker/overlay2/l/LKQNRGHQ54FISASSKCICFD267U:/var/lib/docker/overlay2/l/DRLEYZBURZLTL3XQ2RGUSZ6ZLD:/var/lib/docker/overlay2/l/GW453AJ6NRFMTW54T77CVKFHOT:/var/lib/docker/overlay2/l/N7Y4YGYFSMROV'
Unexpected end of /proc/mounts line `6J5LYBYFFUDN3:/var/lib/docker/overlay2/l/SM77AQWNWR6SLRV5M2NDGHTO5Q:/var/lib/docker/overlay2/l/SGPH3NDKZX3K6GX76GIEBDF3AD:/var/lib/docker/overlay2/l/AQUHUBJZTAWIYIGQACVI6RYRIY:/var/lib/docker/overlay2/l/WUR7WLCVSN342XTP24R3MG3KOG:/var/lib/docker/overlay2/l/O7N65L3RAOVJMB2SCHFZHUVVRI:/var/lib/docker/overlay2/l/RXRELJLOCHEO4V56W5PGHPBXXG:/var/lib/docker/overlay2/l/5NVZ3WHCB5UPKSYVDTIC7VJ7CX:/var/lib/docker/overlay2/l/X4LTBNC3HBU5S6EPJ374XN4UMN:/var/lib/docker/overlay2/l/QQKCWK5WH7GJMTMCDJATXG3AW7,upperdir=/v'
Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/BPK22XVXRMMTC4BKKOMOR7ZAI2:/var/lib/docker/overlay2/l/33N3JY5BIS6BU553FPQ4ZQKCG3:/var/lib/docker/overlay2/l/FRQ3NCT75NVSIGYMWNTEE5USOX:/var/lib/docker/overlay2/l/YHATTH7VMCXC6N3N5Q44TJ4Y3L:/var/lib/docker/overlay2/l/IFZQAKFIRRDE52LUAQ6CXPQGCD:/var/lib/docker/overlay2/l/LKQNRGHQ54FISASSKCICFD267U:/var/lib/docker/overlay2/l/DRLEYZBURZLTL3XQ2RGUSZ6ZLD:/var/lib/docker/overlay2/l/GW453AJ6NRFMTW54T77CVKFHOT:/var/lib/docker/overlay2/l/N7Y4YGYFSMROV'
Unexpected end of /proc/mounts line `6J5LYBYFFUDN3:/var/lib/docker/overlay2/l/SM77AQWNWR6SLRV5M2NDGHTO5Q:/var/lib/docker/overlay2/l/SGPH3NDKZX3K6GX76GIEBDF3AD:/var/lib/docker/overlay2/l/AQUHUBJZTAWIYIGQACVI6RYRIY:/var/lib/docker/overlay2/l/WUR7WLCVSN342XTP24R3MG3KOG:/var/lib/docker/overlay2/l/O7N65L3RAOVJMB2SCHFZHUVVRI:/var/lib/docker/overlay2/l/RXRELJLOCHEO4V56W5PGHPBXXG:/var/lib/docker/overlay2/l/5NVZ3WHCB5UPKSYVDTIC7VJ7CX:/var/lib/docker/overlay2/l/X4LTBNC3HBU5S6EPJ374XN4UMN:/var/lib/docker/overlay2/l/QQKCWK5WH7GJMTMCDJATXG3AW7,upperdir=/v'
Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/BPK22XVXRMMTC4BKKOMOR7ZAI2:/var/lib/docker/overlay2/l/33N3JY5BIS6BU553FPQ4ZQKCG3:/var/lib/docker/overlay2/l/FRQ3NCT75NVSIGYMWNTEE5USOX:/var/lib/docker/overlay2/l/YHATTH7VMCXC6N3N5Q44TJ4Y3L:/var/lib/docker/overlay2/l/IFZQAKFIRRDE52LUAQ6CXPQGCD:/var/lib/docker/overlay2/l/LKQNRGHQ54FISASSKCICFD267U:/var/lib/docker/overlay2/l/DRLEYZBURZLTL3XQ2RGUSZ6ZLD:/var/lib/docker/overlay2/l/GW453AJ6NRFMTW54T77CVKFHOT:/var/lib/docker/overlay2/l/N7Y4YGYFSMROV'
Unexpected end of /proc/mounts line `6J5LYBYFFUDN3:/var/lib/docker/overlay2/l/SM77AQWNWR6SLRV5M2NDGHTO5Q:/var/lib/docker/overlay2/l/SGPH3NDKZX3K6GX76GIEBDF3AD:/var/lib/docker/overlay2/l/AQUHUBJZTAWIYIGQACVI6RYRIY:/var/lib/docker/overlay2/l/WUR7WLCVSN342XTP24R3MG3KOG:/var/lib/docker/overlay2/l/O7N65L3RAOVJMB2SCHFZHUVVRI:/var/lib/docker/overlay2/l/RXRELJLOCHEO4V56W5PGHPBXXG:/var/lib/docker/overlay2/l/5NVZ3WHCB5UPKSYVDTIC7VJ7CX:/var/lib/docker/overlay2/l/X4LTBNC3HBU5S6EPJ374XN4UMN:/var/lib/docker/overlay2/l/QQKCWK5WH7GJMTMCDJATXG3AW7,upperdir=/v'
Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/BPK22XVXRMMTC4BKKOMOR7ZAI2:/var/lib/docker/overlay2/l/33N3JY5BIS6BU553FPQ4ZQKCG3:/var/lib/docker/overlay2/l/FRQ3NCT75NVSIGYMWNTEE5USOX:/var/lib/docker/overlay2/l/YHATTH7VMCXC6N3N5Q44TJ4Y3L:/var/lib/docker/overlay2/l/IFZQAKFIRRDE52LUAQ6CXPQGCD:/var/lib/docker/overlay2/l/LKQNRGHQ54FISASSKCICFD267U:/var/lib/docker/overlay2/l/DRLEYZBURZLTL3XQ2RGUSZ6ZLD:/var/lib/docker/overlay2/l/GW453AJ6NRFMTW54T77CVKFHOT:/var/lib/docker/overlay2/l/N7Y4YGYFSMROV'
Unexpected end of /proc/mounts line `6J5LYBYFFUDN3:/var/lib/docker/overlay2/l/SM77AQWNWR6SLRV5M2NDGHTO5Q:/var/lib/docker/overlay2/l/SGPH3NDKZX3K6GX76GIEBDF3AD:/var/lib/docker/overlay2/l/AQUHUBJZTAWIYIGQACVI6RYRIY:/var/lib/docker/overlay2/l/WUR7WLCVSN342XTP24R3MG3KOG:/var/lib/docker/overlay2/l/O7N65L3RAOVJMB2SCHFZHUVVRI:/var/lib/docker/overlay2/l/RXRELJLOCHEO4V56W5PGHPBXXG:/var/lib/docker/overlay2/l/5NVZ3WHCB5UPKSYVDTIC7VJ7CX:/var/lib/docker/overlay2/l/X4LTBNC3HBU5S6EPJ374XN4UMN:/var/lib/docker/overlay2/l/QQKCWK5WH7GJMTMCDJATXG3AW7,upperdir=/v'
Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/BPK22XVXRMMTC4BKKOMOR7ZAI2:/var/lib/docker/overlay2/l/33N3JY5BIS6BU553FPQ4ZQKCG3:/var/lib/docker/overlay2/l/FRQ3NCT75NVSIGYMWNTEE5USOX:/var/lib/docker/overlay2/l/YHATTH7VMCXC6N3N5Q44TJ4Y3L:/var/lib/docker/overlay2/l/IFZQAKFIRRDE52LUAQ6CXPQGCD:/var/lib/docker/overlay2/l/LKQNRGHQ54FISASSKCICFD267U:/var/lib/docker/overlay2/l/DRLEYZBURZLTL3XQ2RGUSZ6ZLD:/var/lib/docker/overlay2/l/GW453AJ6NRFMTW54T77CVKFHOT:/var/lib/docker/overlay2/l/N7Y4YGYFSMROV'
Unexpected end of /proc/mounts line `6J5LYBYFFUDN3:/var/lib/docker/overlay2/l/SM77AQWNWR6SLRV5M2NDGHTO5Q:/var/lib/docker/overlay2/l/SGPH3NDKZX3K6GX76GIEBDF3AD:/var/lib/docker/overlay2/l/AQUHUBJZTAWIYIGQACVI6RYRIY:/var/lib/docker/overlay2/l/WUR7WLCVSN342XTP24R3MG3KOG:/var/lib/docker/overlay2/l/O7N65L3RAOVJMB2SCHFZHUVVRI:/var/lib/docker/overlay2/l/RXRELJLOCHEO4V56W5PGHPBXXG:/var/lib/docker/overlay2/l/5NVZ3WHCB5UPKSYVDTIC7VJ7CX:/var/lib/docker/overlay2/l/X4LTBNC3HBU5S6EPJ374XN4UMN:/var/lib/docker/overlay2/l/QQKCWK5WH7GJMTMCDJATXG3AW7,upperdir=/v'
Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/BPK22XVXRMMTC4BKKOMOR7ZAI2:/var/lib/docker/overlay2/l/33N3JY5BIS6BU553FPQ4ZQKCG3:/var/lib/docker/overlay2/l/FRQ3NCT75NVSIGYMWNTEE5USOX:/var/lib/docker/overlay2/l/YHATTH7VMCXC6N3N5Q44TJ4Y3L:/var/lib/docker/overlay2/l/IFZQAKFIRRDE52LUAQ6CXPQGCD:/var/lib/docker/overlay2/l/LKQNRGHQ54FISASSKCICFD267U:/var/lib/docker/overlay2/l/DRLEYZBURZLTL3XQ2RGUSZ6ZLD:/var/lib/docker/overlay2/l/GW453AJ6NRFMTW54T77CVKFHOT:/var/lib/docker/overlay2/l/N7Y4YGYFSMROV'
Unexpected end of /proc/mounts line `6J5LYBYFFUDN3:/var/lib/docker/overlay2/l/SM77AQWNWR6SLRV5M2NDGHTO5Q:/var/lib/docker/overlay2/l/SGPH3NDKZX3K6GX76GIEBDF3AD:/var/lib/docker/overlay2/l/AQUHUBJZTAWIYIGQACVI6RYRIY:/var/lib/docker/overlay2/l/WUR7WLCVSN342XTP24R3MG3KOG:/var/lib/docker/overlay2/l/O7N65L3RAOVJMB2SCHFZHUVVRI:/var/lib/docker/overlay2/l/RXRELJLOCHEO4V56W5PGHPBXXG:/var/lib/docker/overlay2/l/5NVZ3WHCB5UPKSYVDTIC7VJ7CX:/var/lib/docker/overlay2/l/X4LTBNC3HBU5S6EPJ374XN4UMN:/var/lib/docker/overlay2/l/QQKCWK5WH7GJMTMCDJATXG3AW7,upperdir=/v'
1           0.386521    0.150009              0.886133       0.954423                  16.1751       
2           0.117418    0.100159              0.965467       0.968365                  16.8099       
3           0.0712852   0.0723796             0.979734       0.978654                  17.4238       
4           0.046616    0.0722298             0.985867       0.9775                    18.033        
5           0.0322429   0.0623279             0.989733       0.98125                   18.644        
6           0.0216789   0.0647544             0.994          0.980192                  19.2536       
7           0.0137309   0.0687912             0.995733       0.979135                  19.8645       
8           0.00891079  0.0595985             0.997733       0.982404                  20.4756       
9           0.00643663  0.0626981             0.9992         0.982115                  21.0857       
10          0.0042832   0.0637832             0.998933       0.983173                  21.6961       
11          0.00407322  0.0680919             0.999333       0.980962                  22.3072       
12          0.00408843  0.0741623             0.999467       0.979327                  22.916        
13          0.00284694  0.0721324             0.999733       0.982308                  23.5269       
14          0.00180912  0.0674107             0.999733       0.983365                  24.1363       
15          0.00108169  0.0696619             0.999867       0.983269                  24.7486       
16          0.00194914  0.0897867             0.999467       0.979038                  25.3588       
17          0.00245598  0.0777146             0.999733       0.982692                  25.9686       
18          0.00154039  0.0778656             0.999733       0.981731                  26.5771       
19          0.00654551  0.102835              0.998533       0.978654                  27.1868       
/usr/local/lib/python3.5/dist-packages/cupy/core/fusion.py:659: FutureWarning: cupy.core.fusion is experimental. The interface can change in the future.
  util.experimental('cupy.core.fusion')
/usr/local/lib/python3.5/dist-packages/cupy/core/fusion.py:659: FutureWarning: cupy.core.fusion is experimental. The interface can change in the future.
  util.experimental('cupy.core.fusion')
/usr/local/lib/python3.5/dist-packages/cupy/core/fusion.py:659: FutureWarning: cupy.core.fusion is experimental. The interface can change in the future.
  util.experimental('cupy.core.fusion')
/usr/local/lib/python3.5/dist-packages/cupy/core/fusion.py:659: FutureWarning: cupy.core.fusion is experimental. The interface can change in the future.
  util.experimental('cupy.core.fusion')
/usr/local/lib/python3.5/dist-packages/cupy/core/fusion.py:659: FutureWarning: cupy.core.fusion is experimental. The interface can change in the future.
  util.experimental('cupy.core.fusion')
/usr/local/lib/python3.5/dist-packages/cupy/core/fusion.py:659: FutureWarning: cupy.core.fusion is experimental. The interface can change in the future.
  util.experimental('cupy.core.fusion')
/usr/local/lib/python3.5/dist-packages/cupy/core/fusion.py:659: FutureWarning: cupy.core.fusion is experimental. The interface can change in the future.
  util.experimental('cupy.core.fusion')
20          0.0149654   0.0887706             0.995334       0.979615                  27.7961       
/usr/local/lib/python3.5/dist-packages/cupy/core/fusion.py:659: FutureWarning: cupy.core.fusion is experimental. The interface can change in the future.
  util.experimental('cupy.core.fusion')
Downloading from http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz...
Downloading from http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz...
Downloading from http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz...
Downloading from http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz...