Created
May 28, 2019 14:27
-
-
Save maxhgerlach/2c7030198b86da5093c56272ede114b6 to your computer and use it in GitHub Desktop.
Log of hanging Horovod shutdown
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.361660: I horovod/common/operations.cc:1106] [6]: MaxG Shutdown A | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.361653: I horovod/common/operations.cc:1106] [4]: MaxG Shutdown A | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.361644: I horovod/common/operations.cc:1106] [7]: MaxG Shutdown A | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.361649: I horovod/common/operations.cc:1106] [3]: MaxG Shutdown A | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.361646: I horovod/common/operations.cc:1106] [1]: MaxG Shutdown A | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.361652: I horovod/common/operations.cc:1106] [2]: MaxG Shutdown A | |
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.361660: I horovod/common/operations.cc:1106] [0]: MaxG Shutdown A | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.361700: I horovod/common/operations.cc:1106] [5]: MaxG Shutdown A | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379057: I horovod/common/operations.cc:1115] [2]: MaxG Shutdown B | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379097: I horovod/common/operations.cc:1134] [2]: MaxG Shutdown C | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379107: I horovod/common/operations.cc:1141] [2]: MaxG Shutdown D | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379098: I horovod/common/operations.cc:1115] [3]: MaxG Shutdown B | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379136: I horovod/common/operations.cc:1134] [3]: MaxG Shutdown C | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379146: I horovod/common/operations.cc:1141] [3]: MaxG Shutdown D | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379203: I horovod/common/operations.cc:1148] [2]: MaxG Shutdown E | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379241: I horovod/common/operations.cc:1148] [3]: MaxG Shutdown E | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379246: I horovod/common/operations.cc:1154] [2]: MaxG Shutdown F | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379264: I horovod/common/operations.cc:1160] [2]: MaxG Shutdown G | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379284: I horovod/common/operations.cc:1154] [3]: MaxG Shutdown F | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379276: I horovod/common/operations.cc:1166] [2]: MaxG Shutdown H | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379291: I horovod/common/operations.cc:1172] [2]: MaxG Shutdown I | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379305: I horovod/common/operations.cc:1160] [3]: MaxG Shutdown G | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379301: I horovod/common/operations.cc:1176] [2]: MaxG Shutdown J | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379309: I horovod/common/operations.cc:1185] [2]: ~~ using default MPI errhandler | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379316: I horovod/common/operations.cc:1166] [3]: MaxG Shutdown H | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379318: I horovod/common/operations.cc:1189] [2]: MaxG Shutdown J.a | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379328: I horovod/common/operations.cc:1193] [2]: MaxG MPI_Finalized returned 0 | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379336: I horovod/common/operations.cc:1195] [2]: MaxG Shutdown J.b | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379340: I horovod/common/operations.cc:1172] [3]: MaxG Shutdown I | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379353: I horovod/common/operations.cc:1176] [3]: MaxG Shutdown J | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379361: I horovod/common/operations.cc:1185] [3]: ~~ using default MPI errhandler | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379371: I horovod/common/operations.cc:1189] [3]: MaxG Shutdown J.a | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379393: I horovod/common/operations.cc:1193] [3]: MaxG MPI_Finalized returned 0 | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379402: I horovod/common/operations.cc:1195] [3]: MaxG Shutdown J.b | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379606: I horovod/common/operations.cc:1115] [6]: MaxG Shutdown B | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379659: I horovod/common/operations.cc:1134] [6]: MaxG Shutdown C | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379669: I horovod/common/operations.cc:1141] [6]: MaxG Shutdown D | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.379742: I horovod/common/operations.cc:1115] [7]: MaxG Shutdown B | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.379786: I horovod/common/operations.cc:1134] [7]: MaxG Shutdown C | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.379794: I horovod/common/operations.cc:1141] [7]: MaxG Shutdown D | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379764: I horovod/common/operations.cc:1148] [6]: MaxG Shutdown E | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379811: I horovod/common/operations.cc:1154] [6]: MaxG Shutdown F | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379831: I horovod/common/operations.cc:1160] [6]: MaxG Shutdown G | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379843: I horovod/common/operations.cc:1166] [6]: MaxG Shutdown H | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379858: I horovod/common/operations.cc:1172] [6]: MaxG Shutdown I | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379869: I horovod/common/operations.cc:1176] [6]: MaxG Shutdown J | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379876: I horovod/common/operations.cc:1185] [6]: ~~ using default MPI errhandler | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.379799: I horovod/common/operations.cc:1115] [4]: MaxG Shutdown B | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.379840: I horovod/common/operations.cc:1134] [4]: MaxG Shutdown C | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.379850: I horovod/common/operations.cc:1141] [4]: MaxG Shutdown D | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.379905: I horovod/common/operations.cc:1148] [7]: MaxG Shutdown E | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379884: I horovod/common/operations.cc:1189] [6]: MaxG Shutdown J.a | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379894: I horovod/common/operations.cc:1193] [6]: MaxG MPI_Finalized returned 0 | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379902: I horovod/common/operations.cc:1195] [6]: MaxG Shutdown J.b | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.379940: I horovod/common/operations.cc:1115] [5]: MaxG Shutdown B | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.379968: I horovod/common/operations.cc:1134] [5]: MaxG Shutdown C | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.379955: I horovod/common/operations.cc:1154] [7]: MaxG Shutdown F | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.379975: I horovod/common/operations.cc:1160] [7]: MaxG Shutdown G | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.379948: I horovod/common/operations.cc:1148] [4]: MaxG Shutdown E | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.379977: I horovod/common/operations.cc:1141] [5]: MaxG Shutdown D | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.379988: I horovod/common/operations.cc:1166] [7]: MaxG Shutdown H | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.380010: I horovod/common/operations.cc:1172] [7]: MaxG Shutdown I | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.380028: I horovod/common/operations.cc:1176] [7]: MaxG Shutdown J | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.380037: I horovod/common/operations.cc:1185] [7]: ~~ using default MPI errhandler | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.380044: I horovod/common/operations.cc:1189] [7]: MaxG Shutdown J.a | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.380059: I horovod/common/operations.cc:1193] [7]: MaxG MPI_Finalized returned 0 | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.379959: I horovod/common/operations.cc:1115] [1]: MaxG Shutdown B | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.379992: I horovod/common/operations.cc:1134] [1]: MaxG Shutdown C | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.379999: I horovod/common/operations.cc:1141] [1]: MaxG Shutdown D | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380011: I horovod/common/operations.cc:1154] [4]: MaxG Shutdown F | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380030: I horovod/common/operations.cc:1160] [4]: MaxG Shutdown G | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380045: I horovod/common/operations.cc:1166] [4]: MaxG Shutdown H | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380064: I horovod/common/operations.cc:1172] [4]: MaxG Shutdown I | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380079: I horovod/common/operations.cc:1176] [4]: MaxG Shutdown J | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380088: I horovod/common/operations.cc:1185] [4]: ~~ using default MPI errhandler | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380093: I horovod/common/operations.cc:1148] [1]: MaxG Shutdown E | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380131: I horovod/common/operations.cc:1154] [1]: MaxG Shutdown F | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380146: I horovod/common/operations.cc:1160] [1]: MaxG Shutdown G | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380098: I horovod/common/operations.cc:1148] [5]: MaxG Shutdown E | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.380068: I horovod/common/operations.cc:1195] [7]: MaxG Shutdown J.b | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380096: I horovod/common/operations.cc:1189] [4]: MaxG Shutdown J.a | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380106: I horovod/common/operations.cc:1193] [4]: MaxG MPI_Finalized returned 0 | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380115: I horovod/common/operations.cc:1195] [4]: MaxG Shutdown J.b | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380156: I horovod/common/operations.cc:1166] [1]: MaxG Shutdown H | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380172: I horovod/common/operations.cc:1172] [1]: MaxG Shutdown I | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380184: I horovod/common/operations.cc:1176] [1]: MaxG Shutdown J | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380159: I horovod/common/operations.cc:1154] [5]: MaxG Shutdown F | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380180: I horovod/common/operations.cc:1160] [5]: MaxG Shutdown G | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380190: I horovod/common/operations.cc:1166] [5]: MaxG Shutdown H | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380191: I horovod/common/operations.cc:1185] [1]: ~~ using default MPI errhandler | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380199: I horovod/common/operations.cc:1189] [1]: MaxG Shutdown J.a | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380213: I horovod/common/operations.cc:1193] [1]: MaxG MPI_Finalized returned 0 | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380209: I horovod/common/operations.cc:1172] [5]: MaxG Shutdown I | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380219: I horovod/common/operations.cc:1195] [1]: MaxG Shutdown J.b | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380224: I horovod/common/operations.cc:1176] [5]: MaxG Shutdown J | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380232: I horovod/common/operations.cc:1185] [5]: ~~ using default MPI errhandler | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380240: I horovod/common/operations.cc:1189] [5]: MaxG Shutdown J.a | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380250: I horovod/common/operations.cc:1193] [5]: MaxG MPI_Finalized returned 0 | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380259: I horovod/common/operations.cc:1195] [5]: MaxG Shutdown J.b | |
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380403: I horovod/common/operations.cc:1115] [0]: MaxG Shutdown B | |
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380446: I horovod/common/operations.cc:1134] [0]: MaxG Shutdown C | |
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380458: I horovod/common/operations.cc:1141] [0]: MaxG Shutdown D | |
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380594: I horovod/common/operations.cc:1148] [0]: MaxG Shutdown E | |
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380645: I horovod/common/operations.cc:1154] [0]: MaxG Shutdown F | |
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380662: I horovod/common/operations.cc:1160] [0]: MaxG Shutdown G | |
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380673: I horovod/common/operations.cc:1166] [0]: MaxG Shutdown H | |
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380692: I horovod/common/operations.cc:1172] [0]: MaxG Shutdown I | |
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380706: I horovod/common/operations.cc:1176] [0]: MaxG Shutdown J | |
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380714: I horovod/common/operations.cc:1185] [0]: ~~ using default MPI errhandler | |
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380721: I horovod/common/operations.cc:1189] [0]: MaxG Shutdown J.a | |
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380736: I horovod/common/operations.cc:1193] [0]: MaxG MPI_Finalized returned 0 | |
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380744: I horovod/common/operations.cc:1195] [0]: MaxG Shutdown J.b | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.428706: I horovod/common/operations.cc:1200] [6]: MaxG MPI_Finalize returned 0 | |
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.428757: I horovod/common/operations.cc:1205] [6]: MaxG Shutdown K | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.428702: I horovod/common/operations.cc:1200] [5]: MaxG MPI_Finalize returned 0 | |
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.428752: I horovod/common/operations.cc:1205] [5]: MaxG Shutdown K | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.428703: I horovod/common/operations.cc:1200] [4]: MaxG MPI_Finalize returned 0 | |
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.428754: I horovod/common/operations.cc:1205] [4]: MaxG Shutdown K | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.429920: I horovod/common/operations.cc:1200] [7]: MaxG MPI_Finalize returned 0 | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.429914: I horovod/common/operations.cc:1200] [1]: MaxG MPI_Finalize returned 0 | |
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.429959: I horovod/common/operations.cc:1205] [7]: MaxG Shutdown K | |
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.429956: I horovod/common/operations.cc:1205] [1]: MaxG Shutdown K | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.431074: I horovod/common/operations.cc:1200] [2]: MaxG MPI_Finalize returned 0 | |
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.431114: I horovod/common/operations.cc:1205] [2]: MaxG Shutdown K | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.432082: I horovod/common/operations.cc:1200] [3]: MaxG MPI_Finalize returned 0 | |
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.432126: I horovod/common/operations.cc:1205] [3]: MaxG Shutdown K |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment