Skip to content

Instantly share code, notes, and snippets.

@ProGamerGov
Last active October 22, 2017 01:23
Show Gist options
  • Save ProGamerGov/8f1d07d866700b159c4b15c4e2fda868 to your computer and use it in GitHub Desktop.
Save ProGamerGov/8f1d07d866700b159c4b15c4e2fda868 to your computer and use it in GitHub Desktop.
GPU Memory Usage In Neural-Style with 8 Nvidia Tesla K80 graphics cards
Content Image size: 1051x976
-content_layers relu4_2
-style_layers relu1_1,relu2_1,relu3_1,relu4_1,relu5_1
-optimizer lbfgs (unless specified)
No -cudnn_autotune (unless specified)
Each Nvidia Tesla K80 graphics card used, has 11439MiB of Memory.
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2115 C /home/ubuntu/torch/install/bin/luajit 6491MiB |
| 1 2115 C /home/ubuntu/torch/install/bin/luajit 1763MiB |
| 2 2115 C /home/ubuntu/torch/install/bin/luajit 2313MiB |
| 3 2115 C /home/ubuntu/torch/install/bin/luajit 1510MiB |
| 4 2115 C /home/ubuntu/torch/install/bin/luajit 977MiB |
| 5 2115 C /home/ubuntu/torch/install/bin/luajit 1527MiB |
| 6 2115 C /home/ubuntu/torch/install/bin/luajit 842MiB |
| 7 2115 C /home/ubuntu/torch/install/bin/luajit 2097MiB |
+-----------------------------------------------------------------------------+
-image_size 1536 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 2,3,4,6,8,11,12 -cudnn_autotune
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2302 C /home/ubuntu/torch/install/bin/luajit 7589MiB |
| 1 2302 C /home/ubuntu/torch/install/bin/luajit 2042MiB |
| 2 2302 C /home/ubuntu/torch/install/bin/luajit 2685MiB |
| 3 2302 C /home/ubuntu/torch/install/bin/luajit 1743MiB |
| 4 2302 C /home/ubuntu/torch/install/bin/luajit 1117MiB |
| 5 2302 C /home/ubuntu/torch/install/bin/luajit 1762MiB |
| 6 2302 C /home/ubuntu/torch/install/bin/luajit 958MiB |
| 7 2302 C /home/ubuntu/torch/install/bin/luajit 2416MiB |
+-----------------------------------------------------------------------------+
-image_size 1664 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 2,3,4,6,8,11,12 -cudnn_autotune
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2452 C /home/ubuntu/torch/install/bin/luajit 10042MiB |
| 1 2452 C /home/ubuntu/torch/install/bin/luajit 2665MiB |
| 2 2452 C /home/ubuntu/torch/install/bin/luajit 3515MiB |
| 3 2452 C /home/ubuntu/torch/install/bin/luajit 2261MiB |
| 4 2452 C /home/ubuntu/torch/install/bin/luajit 1428MiB |
| 5 2452 C /home/ubuntu/torch/install/bin/luajit 2280MiB |
| 6 2452 C /home/ubuntu/torch/install/bin/luajit 1218MiB |
| 7 2452 C /home/ubuntu/torch/install/bin/luajit 3121MiB |
+-----------------------------------------------------------------------------+
-image_size 1920 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 2,3,4,6,8,11,12 -cudnn_autotune
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2599 C /home/ubuntu/torch/install/bin/luajit 11402MiB |
| 1 2599 C /home/ubuntu/torch/install/bin/luajit 3011MiB |
| 2 2599 C /home/ubuntu/torch/install/bin/luajit 3976MiB |
| 3 2599 C /home/ubuntu/torch/install/bin/luajit 2549MiB |
| 4 2599 C /home/ubuntu/torch/install/bin/luajit 1601MiB |
| 5 2599 C /home/ubuntu/torch/install/bin/luajit 2568MiB |
| 6 2599 C /home/ubuntu/torch/install/bin/luajit 1362MiB |
| 7 2599 C /home/ubuntu/torch/install/bin/luajit 3513MiB |
+-----------------------------------------------------------------------------+
-image_size 2048 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 2,3,4,6,8,11,12
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2950 C /home/ubuntu/torch/install/bin/luajit 3317MiB |
| 1 2950 C /home/ubuntu/torch/install/bin/luajit 4181MiB |
| 2 2950 C /home/ubuntu/torch/install/bin/luajit 5537MiB |
| 3 2950 C /home/ubuntu/torch/install/bin/luajit 3525MiB |
| 4 2950 C /home/ubuntu/torch/install/bin/luajit 2186MiB |
| 5 2950 C /home/ubuntu/torch/install/bin/luajit 3543MiB |
| 6 2950 C /home/ubuntu/torch/install/bin/luajit 1849MiB |
| 7 2950 C /home/ubuntu/torch/install/bin/luajit 5226MiB |
+-----------------------------------------------------------------------------+
-image_size 2432 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 2,3,4,6,8,11,12 -optimizer adam -cudnn_autotune
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 3104 C /home/ubuntu/torch/install/bin/luajit 3657MiB |
| 1 3104 C /home/ubuntu/torch/install/bin/luajit 4616MiB |
| 2 3104 C /home/ubuntu/torch/install/bin/luajit 6116MiB |
| 3 3104 C /home/ubuntu/torch/install/bin/luajit 3887MiB |
| 4 3104 C /home/ubuntu/torch/install/bin/luajit 2404MiB |
| 5 3104 C /home/ubuntu/torch/install/bin/luajit 3906MiB |
| 6 3104 C /home/ubuntu/torch/install/bin/luajit 2031MiB |
| 7 3104 C /home/ubuntu/torch/install/bin/luajit 5341MiB |
+-----------------------------------------------------------------------------+
-image_size 2560 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 2,3,4,6,8,11,12 -optimizer adam -cudnn_autotune
All the loss values for every iteration are the same?
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 3314 C /home/ubuntu/torch/install/bin/luajit 4389MiB |
| 1 3314 C /home/ubuntu/torch/install/bin/luajit 5553MiB |
| 2 3314 C /home/ubuntu/torch/install/bin/luajit 7365MiB |
| 3 3314 C /home/ubuntu/torch/install/bin/luajit 4668MiB |
| 4 3314 C /home/ubuntu/torch/install/bin/luajit 2873MiB |
| 5 3314 C /home/ubuntu/torch/install/bin/luajit 4686MiB |
| 6 3314 C /home/ubuntu/torch/install/bin/luajit 2421MiB |
| 7 3314 C /home/ubuntu/torch/install/bin/luajit 6398MiB |
+-----------------------------------------------------------------------------+
-image_size 2816 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 2,3,4,6,8,11,12 -optimizer adam -cudnn_autotune
Content Image size: 1920x1200
-content_layers relu4_2
-style_layers relu1_1,relu2_1,relu3_1,relu4_1,relu5_1
-optimizer adam
No -cudnn_autotune
Each Nvidia Tesla K80 graphics card used, has 11439MiB of Memory.
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2389 C /home/ubuntu/torch/install/bin/luajit 3724MiB |
| 1 2389 C /home/ubuntu/torch/install/bin/luajit 4599MiB |
| 2 2389 C /home/ubuntu/torch/install/bin/luajit 6307MiB |
| 3 2389 C /home/ubuntu/torch/install/bin/luajit 3240MiB |
| 4 2389 C /home/ubuntu/torch/install/bin/luajit 4007MiB |
| 5 2389 C /home/ubuntu/torch/install/bin/luajit 9920MiB |
| 6 2389 C /home/ubuntu/torch/install/bin/luajit 145MiB |
| 7 2389 C /home/ubuntu/torch/install/bin/luajit 145MiB |
+-----------------------------------------------------------------------------+
-image_size 3000 -gpu 0,1,2,3,4,5 -multigpu_strategy 2,3,4,5,6
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2499 C /home/ubuntu/torch/install/bin/luajit 8179MiB |
| 1 2499 C /home/ubuntu/torch/install/bin/luajit 4007MiB |
| 2 2499 C /home/ubuntu/torch/install/bin/luajit 2476MiB |
| 3 2499 C /home/ubuntu/torch/install/bin/luajit 2379MiB |
| 4 2499 C /home/ubuntu/torch/install/bin/luajit 3244MiB |
| 5 2499 C /home/ubuntu/torch/install/bin/luajit 1707MiB |
| 6 2499 C /home/ubuntu/torch/install/bin/luajit 2090MiB |
| 7 2499 C /home/ubuntu/torch/install/bin/luajit 5395MiB |
+-----------------------------------------------------------------------------+
-image_size 3000 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 4,6,8,9,10,11,12
+-----------------------------------------------------------------------------+
| Memory Error |
+-----------------------------------------------------------------------------+
-image_size 3584 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 4,6,8,9,10,11,12
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2806 C /home/ubuntu/torch/install/bin/luajit 7213MiB |
| 1 2806 C /home/ubuntu/torch/install/bin/luajit 10022MiB |
| 2 2806 C /home/ubuntu/torch/install/bin/luajit 3458MiB |
| 3 2806 C /home/ubuntu/torch/install/bin/luajit 3326MiB |
| 4 2806 C /home/ubuntu/torch/install/bin/luajit 4553MiB |
| 5 2806 C /home/ubuntu/torch/install/bin/luajit 2361MiB |
| 6 2806 C /home/ubuntu/torch/install/bin/luajit 2909MiB |
| 7 2806 C /home/ubuntu/torch/install/bin/luajit 7579MiB |
+-----------------------------------------------------------------------------+
-image_size 3584 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 3,6,8,9,10,11,12
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2953 C /home/ubuntu/torch/install/bin/luajit 7213MiB |
| 1 2953 C /home/ubuntu/torch/install/bin/luajit 8927MiB |
| 2 2953 C /home/ubuntu/torch/install/bin/luajit 7834MiB |
| 3 2953 C /home/ubuntu/torch/install/bin/luajit 3326MiB |
| 4 2953 C /home/ubuntu/torch/install/bin/luajit 4553MiB |
| 5 2953 C /home/ubuntu/torch/install/bin/luajit 2361MiB |
| 6 2953 C /home/ubuntu/torch/install/bin/luajit 2909MiB |
| 7 2953 C /home/ubuntu/torch/install/bin/luajit 7579MiB |
+-----------------------------------------------------------------------------+
-image_size 3584 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 3,5,8,9,10,11,12
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 3094 C /home/ubuntu/torch/install/bin/luajit 5239MiB |
| 1 3094 C /home/ubuntu/torch/install/bin/luajit 10903MiB |
| 2 3094 C /home/ubuntu/torch/install/bin/luajit 5644MiB |
| 3 3094 C /home/ubuntu/torch/install/bin/luajit 3458MiB |
| 4 3094 C /home/ubuntu/torch/install/bin/luajit 5547MiB |
| 5 3094 C /home/ubuntu/torch/install/bin/luajit 2361MiB |
| 6 3094 C /home/ubuntu/torch/install/bin/luajit 2909MiB |
| 7 3094 C /home/ubuntu/torch/install/bin/luajit 7579MiB |
+-----------------------------------------------------------------------------+
-image_size 3584 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 2,4,6,8,10,11,12
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 3246 C /home/ubuntu/torch/install/bin/luajit 5239MiB |
| 1 3246 C /home/ubuntu/torch/install/bin/luajit 6496MiB |
| 2 3246 C /home/ubuntu/torch/install/bin/luajit 8927MiB |
| 3 3246 C /home/ubuntu/torch/install/bin/luajit 5644MiB |
| 4 3246 C /home/ubuntu/torch/install/bin/luajit 3458MiB |
| 5 3246 C /home/ubuntu/torch/install/bin/luajit 5547MiB |
| 6 3246 C /home/ubuntu/torch/install/bin/luajit 2909MiB |
| 7 3246 C /home/ubuntu/torch/install/bin/luajit 7579MiB |
+-----------------------------------------------------------------------------+
-image_size 3584 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 2,3,4,6,8,11,12
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 3355 C /home/ubuntu/torch/install/bin/luajit 5989MiB |
| 1 3355 C /home/ubuntu/torch/install/bin/luajit 7434MiB |
| 2 3355 C /home/ubuntu/torch/install/bin/luajit 10224MiB |
| 3 3355 C /home/ubuntu/torch/install/bin/luajit 6454MiB |
| 4 3355 C /home/ubuntu/torch/install/bin/luajit 3944MiB |
| 5 3355 C /home/ubuntu/torch/install/bin/luajit 6340MiB |
| 6 3355 C /home/ubuntu/torch/install/bin/luajit 3314MiB |
| 7 3355 C /home/ubuntu/torch/install/bin/luajit 8657MiB |
+-----------------------------------------------------------------------------+
-image_size 3840 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 2,3,4,6,8,11,12
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 3595 C /home/ubuntu/torch/install/bin/luajit 6484MiB |
| 1 3595 C /home/ubuntu/torch/install/bin/luajit 8053MiB |
| 2 3595 C /home/ubuntu/torch/install/bin/luajit 11078MiB |
| 3 3595 C /home/ubuntu/torch/install/bin/luajit 6989MiB |
| 4 3595 C /home/ubuntu/torch/install/bin/luajit 4265MiB |
| 5 3595 C /home/ubuntu/torch/install/bin/luajit 6865MiB |
| 6 3595 C /home/ubuntu/torch/install/bin/luajit 3581MiB |
| 7 3595 C /home/ubuntu/torch/install/bin/luajit 9378MiB |
+-----------------------------------------------------------------------------+
-image_size 4000 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 2,3,4,6,8,11,12
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 3688 C /home/ubuntu/torch/install/bin/luajit 6586MiB |
| 1 3688 C /home/ubuntu/torch/install/bin/luajit 8180MiB |
| 2 3688 C /home/ubuntu/torch/install/bin/luajit 11254MiB |
| 3 3688 C /home/ubuntu/torch/install/bin/luajit 7098MiB |
| 4 3688 C /home/ubuntu/torch/install/bin/luajit 4330MiB |
| 5 3688 C /home/ubuntu/torch/install/bin/luajit 6971MiB |
| 6 3688 C /home/ubuntu/torch/install/bin/luajit 3636MiB |
| 7 3688 C /home/ubuntu/torch/install/bin/luajit 9521MiB |
+-----------------------------------------------------------------------------+
-image_size 4032 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 2,3,4,6,8,11,12
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2535 C /home/ubuntu/torch/install/bin/luajit 4672MiB |
| 1 2535 C /home/ubuntu/torch/install/bin/luajit 6207MiB |
| 2 2535 C /home/ubuntu/torch/install/bin/luajit 4550MiB |
| 3 2535 C /home/ubuntu/torch/install/bin/luajit 1534MiB |
| 4 2535 C /home/ubuntu/torch/install/bin/luajit 2364MiB |
| 5 2535 C /home/ubuntu/torch/install/bin/luajit 1564MiB |
| 6 2535 C /home/ubuntu/torch/install/bin/luajit 1051MiB |
| 7 2535 C /home/ubuntu/torch/install/bin/luajit 288MiB |
+-----------------------------------------------------------------------------+
-image_size 2432 -gpu 0,1,2,3,4,5,6,7 -multigpu_strategy 3,6,12,15,20,26,31
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 3497 C /home/ubuntu/torch/install/bin/luajit 4672MiB |
| 1 3497 C /home/ubuntu/torch/install/bin/luajit 6207MiB |
| 2 3497 C /home/ubuntu/torch/install/bin/luajit 4550MiB |
| 3 3497 C /home/ubuntu/torch/install/bin/luajit 1534MiB |
| 4 3497 C /home/ubuntu/torch/install/bin/luajit 2364MiB |
| 5 3497 C /home/ubuntu/torch/install/bin/luajit 1564MiB |
| 6 3497 C /home/ubuntu/torch/install/bin/luajit 1112MiB |
| 7 3497 C /home/ubuntu/torch/install/bin/luajit 145MiB |
+-----------------------------------------------------------------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 3732 C /home/ubuntu/torch/install/bin/luajit 4672MiB |
| 1 3732 C /home/ubuntu/torch/install/bin/luajit 6207MiB |
| 2 3732 C /home/ubuntu/torch/install/bin/luajit 4550MiB |
| 3 3732 C /home/ubuntu/torch/install/bin/luajit 1534MiB |
| 4 3732 C /home/ubuntu/torch/install/bin/luajit 2364MiB |
| 5 3732 C /home/ubuntu/torch/install/bin/luajit 2151MiB |
| 6 3732 C /home/ubuntu/torch/install/bin/luajit 145MiB |
| 7 3732 C /home/ubuntu/torch/install/bin/luajit 145MiB |
+-----------------------------------------------------------------------------+
-image_size 2432 -gpu 0,1,2,3,4,5 -multigpu_strategy 3,6,12,15,20
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 4107 C /home/ubuntu/torch/install/bin/luajit 145MiB |
| 1 4107 C /home/ubuntu/torch/install/bin/luajit 8025MiB |
| 2 4107 C /home/ubuntu/torch/install/bin/luajit 4550MiB |
| 3 4107 C /home/ubuntu/torch/install/bin/luajit 1534MiB |
| 4 4107 C /home/ubuntu/torch/install/bin/luajit 2364MiB |
| 5 4107 C /home/ubuntu/torch/install/bin/luajit 1564MiB |
| 6 4107 C /home/ubuntu/torch/install/bin/luajit 1112MiB |
| 7 4107 C /home/ubuntu/torch/install/bin/luajit 145MiB |
+-----------------------------------------------------------------------------+
-image_size 2432 -gpu 1,2,3,4,5,6 -multigpu_strategy 6,12,15,20,26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment