Skip to content

Instantly share code, notes, and snippets.

@maxhgerlach

maxhgerlach/gist:2c7030198b86da5093c56272ede114b6

Created May 28, 2019 14:27
Show Gist options
  • Save maxhgerlach/2c7030198b86da5093c56272ede114b6 to your computer and use it in GitHub Desktop.
Save maxhgerlach/2c7030198b86da5093c56272ede114b6 to your computer and use it in GitHub Desktop.
Download ZIP
Log of hanging Horovod shutdown
Raw
gistfile1.txt
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.361660: I horovod/common/operations.cc:1106] [6]: MaxG Shutdown A
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.361653: I horovod/common/operations.cc:1106] [4]: MaxG Shutdown A
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.361644: I horovod/common/operations.cc:1106] [7]: MaxG Shutdown A
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.361649: I horovod/common/operations.cc:1106] [3]: MaxG Shutdown A
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.361646: I horovod/common/operations.cc:1106] [1]: MaxG Shutdown A
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.361652: I horovod/common/operations.cc:1106] [2]: MaxG Shutdown A
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.361660: I horovod/common/operations.cc:1106] [0]: MaxG Shutdown A
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.361700: I horovod/common/operations.cc:1106] [5]: MaxG Shutdown A
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379057: I horovod/common/operations.cc:1115] [2]: MaxG Shutdown B
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379097: I horovod/common/operations.cc:1134] [2]: MaxG Shutdown C
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379107: I horovod/common/operations.cc:1141] [2]: MaxG Shutdown D
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379098: I horovod/common/operations.cc:1115] [3]: MaxG Shutdown B
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379136: I horovod/common/operations.cc:1134] [3]: MaxG Shutdown C
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379146: I horovod/common/operations.cc:1141] [3]: MaxG Shutdown D
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379203: I horovod/common/operations.cc:1148] [2]: MaxG Shutdown E
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379241: I horovod/common/operations.cc:1148] [3]: MaxG Shutdown E
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379246: I horovod/common/operations.cc:1154] [2]: MaxG Shutdown F
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379264: I horovod/common/operations.cc:1160] [2]: MaxG Shutdown G
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379284: I horovod/common/operations.cc:1154] [3]: MaxG Shutdown F
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379276: I horovod/common/operations.cc:1166] [2]: MaxG Shutdown H
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379291: I horovod/common/operations.cc:1172] [2]: MaxG Shutdown I
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379305: I horovod/common/operations.cc:1160] [3]: MaxG Shutdown G
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379301: I horovod/common/operations.cc:1176] [2]: MaxG Shutdown J
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379309: I horovod/common/operations.cc:1185] [2]: ~~ using default MPI errhandler
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379316: I horovod/common/operations.cc:1166] [3]: MaxG Shutdown H
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379318: I horovod/common/operations.cc:1189] [2]: MaxG Shutdown J.a
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379328: I horovod/common/operations.cc:1193] [2]: MaxG MPI_Finalized returned 0
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.379336: I horovod/common/operations.cc:1195] [2]: MaxG Shutdown J.b
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379340: I horovod/common/operations.cc:1172] [3]: MaxG Shutdown I
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379353: I horovod/common/operations.cc:1176] [3]: MaxG Shutdown J
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379361: I horovod/common/operations.cc:1185] [3]: ~~ using default MPI errhandler
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379371: I horovod/common/operations.cc:1189] [3]: MaxG Shutdown J.a
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379393: I horovod/common/operations.cc:1193] [3]: MaxG MPI_Finalized returned 0
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.379402: I horovod/common/operations.cc:1195] [3]: MaxG Shutdown J.b
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379606: I horovod/common/operations.cc:1115] [6]: MaxG Shutdown B
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379659: I horovod/common/operations.cc:1134] [6]: MaxG Shutdown C
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379669: I horovod/common/operations.cc:1141] [6]: MaxG Shutdown D
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.379742: I horovod/common/operations.cc:1115] [7]: MaxG Shutdown B
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.379786: I horovod/common/operations.cc:1134] [7]: MaxG Shutdown C
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.379794: I horovod/common/operations.cc:1141] [7]: MaxG Shutdown D
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379764: I horovod/common/operations.cc:1148] [6]: MaxG Shutdown E
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379811: I horovod/common/operations.cc:1154] [6]: MaxG Shutdown F
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379831: I horovod/common/operations.cc:1160] [6]: MaxG Shutdown G
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379843: I horovod/common/operations.cc:1166] [6]: MaxG Shutdown H
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379858: I horovod/common/operations.cc:1172] [6]: MaxG Shutdown I
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379869: I horovod/common/operations.cc:1176] [6]: MaxG Shutdown J
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379876: I horovod/common/operations.cc:1185] [6]: ~~ using default MPI errhandler
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.379799: I horovod/common/operations.cc:1115] [4]: MaxG Shutdown B
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.379840: I horovod/common/operations.cc:1134] [4]: MaxG Shutdown C
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.379850: I horovod/common/operations.cc:1141] [4]: MaxG Shutdown D
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.379905: I horovod/common/operations.cc:1148] [7]: MaxG Shutdown E
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379884: I horovod/common/operations.cc:1189] [6]: MaxG Shutdown J.a
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379894: I horovod/common/operations.cc:1193] [6]: MaxG MPI_Finalized returned 0
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.379902: I horovod/common/operations.cc:1195] [6]: MaxG Shutdown J.b
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.379940: I horovod/common/operations.cc:1115] [5]: MaxG Shutdown B
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.379968: I horovod/common/operations.cc:1134] [5]: MaxG Shutdown C
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.379955: I horovod/common/operations.cc:1154] [7]: MaxG Shutdown F
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.379975: I horovod/common/operations.cc:1160] [7]: MaxG Shutdown G
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.379948: I horovod/common/operations.cc:1148] [4]: MaxG Shutdown E
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.379977: I horovod/common/operations.cc:1141] [5]: MaxG Shutdown D
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.379988: I horovod/common/operations.cc:1166] [7]: MaxG Shutdown H
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.380010: I horovod/common/operations.cc:1172] [7]: MaxG Shutdown I
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.380028: I horovod/common/operations.cc:1176] [7]: MaxG Shutdown J
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.380037: I horovod/common/operations.cc:1185] [7]: ~~ using default MPI errhandler
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.380044: I horovod/common/operations.cc:1189] [7]: MaxG Shutdown J.a
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.380059: I horovod/common/operations.cc:1193] [7]: MaxG MPI_Finalized returned 0
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.379959: I horovod/common/operations.cc:1115] [1]: MaxG Shutdown B
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.379992: I horovod/common/operations.cc:1134] [1]: MaxG Shutdown C
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.379999: I horovod/common/operations.cc:1141] [1]: MaxG Shutdown D
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380011: I horovod/common/operations.cc:1154] [4]: MaxG Shutdown F
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380030: I horovod/common/operations.cc:1160] [4]: MaxG Shutdown G
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380045: I horovod/common/operations.cc:1166] [4]: MaxG Shutdown H
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380064: I horovod/common/operations.cc:1172] [4]: MaxG Shutdown I
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380079: I horovod/common/operations.cc:1176] [4]: MaxG Shutdown J
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380088: I horovod/common/operations.cc:1185] [4]: ~~ using default MPI errhandler
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380093: I horovod/common/operations.cc:1148] [1]: MaxG Shutdown E
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380131: I horovod/common/operations.cc:1154] [1]: MaxG Shutdown F
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380146: I horovod/common/operations.cc:1160] [1]: MaxG Shutdown G
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380098: I horovod/common/operations.cc:1148] [5]: MaxG Shutdown E
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.380068: I horovod/common/operations.cc:1195] [7]: MaxG Shutdown J.b
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380096: I horovod/common/operations.cc:1189] [4]: MaxG Shutdown J.a
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380106: I horovod/common/operations.cc:1193] [4]: MaxG MPI_Finalized returned 0
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.380115: I horovod/common/operations.cc:1195] [4]: MaxG Shutdown J.b
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380156: I horovod/common/operations.cc:1166] [1]: MaxG Shutdown H
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380172: I horovod/common/operations.cc:1172] [1]: MaxG Shutdown I
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380184: I horovod/common/operations.cc:1176] [1]: MaxG Shutdown J
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380159: I horovod/common/operations.cc:1154] [5]: MaxG Shutdown F
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380180: I horovod/common/operations.cc:1160] [5]: MaxG Shutdown G
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380190: I horovod/common/operations.cc:1166] [5]: MaxG Shutdown H
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380191: I horovod/common/operations.cc:1185] [1]: ~~ using default MPI errhandler
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380199: I horovod/common/operations.cc:1189] [1]: MaxG Shutdown J.a
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380213: I horovod/common/operations.cc:1193] [1]: MaxG MPI_Finalized returned 0
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380209: I horovod/common/operations.cc:1172] [5]: MaxG Shutdown I
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.380219: I horovod/common/operations.cc:1195] [1]: MaxG Shutdown J.b
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380224: I horovod/common/operations.cc:1176] [5]: MaxG Shutdown J
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380232: I horovod/common/operations.cc:1185] [5]: ~~ using default MPI errhandler
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380240: I horovod/common/operations.cc:1189] [5]: MaxG Shutdown J.a
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380250: I horovod/common/operations.cc:1193] [5]: MaxG MPI_Finalized returned 0
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.380259: I horovod/common/operations.cc:1195] [5]: MaxG Shutdown J.b
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380403: I horovod/common/operations.cc:1115] [0]: MaxG Shutdown B
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380446: I horovod/common/operations.cc:1134] [0]: MaxG Shutdown C
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380458: I horovod/common/operations.cc:1141] [0]: MaxG Shutdown D
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380594: I horovod/common/operations.cc:1148] [0]: MaxG Shutdown E
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380645: I horovod/common/operations.cc:1154] [0]: MaxG Shutdown F
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380662: I horovod/common/operations.cc:1160] [0]: MaxG Shutdown G
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380673: I horovod/common/operations.cc:1166] [0]: MaxG Shutdown H
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380692: I horovod/common/operations.cc:1172] [0]: MaxG Shutdown I
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380706: I horovod/common/operations.cc:1176] [0]: MaxG Shutdown J
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380714: I horovod/common/operations.cc:1185] [0]: ~~ using default MPI errhandler
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380721: I horovod/common/operations.cc:1189] [0]: MaxG Shutdown J.a
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380736: I horovod/common/operations.cc:1193] [0]: MaxG MPI_Finalized returned 0
Tue May 28 15:56:20 2019[1,0]<stdout>:[2019-05-28 15:56:20.380744: I horovod/common/operations.cc:1195] [0]: MaxG Shutdown J.b
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.428706: I horovod/common/operations.cc:1200] [6]: MaxG MPI_Finalize returned 0
Tue May 28 15:56:20 2019[1,6]<stdout>:[2019-05-28 15:56:20.428757: I horovod/common/operations.cc:1205] [6]: MaxG Shutdown K
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.428702: I horovod/common/operations.cc:1200] [5]: MaxG MPI_Finalize returned 0
Tue May 28 15:56:20 2019[1,5]<stdout>:[2019-05-28 15:56:20.428752: I horovod/common/operations.cc:1205] [5]: MaxG Shutdown K
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.428703: I horovod/common/operations.cc:1200] [4]: MaxG MPI_Finalize returned 0
Tue May 28 15:56:20 2019[1,4]<stdout>:[2019-05-28 15:56:20.428754: I horovod/common/operations.cc:1205] [4]: MaxG Shutdown K
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.429920: I horovod/common/operations.cc:1200] [7]: MaxG MPI_Finalize returned 0
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.429914: I horovod/common/operations.cc:1200] [1]: MaxG MPI_Finalize returned 0
Tue May 28 15:56:20 2019[1,7]<stdout>:[2019-05-28 15:56:20.429959: I horovod/common/operations.cc:1205] [7]: MaxG Shutdown K
Tue May 28 15:56:20 2019[1,1]<stdout>:[2019-05-28 15:56:20.429956: I horovod/common/operations.cc:1205] [1]: MaxG Shutdown K
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.431074: I horovod/common/operations.cc:1200] [2]: MaxG MPI_Finalize returned 0
Tue May 28 15:56:20 2019[1,2]<stdout>:[2019-05-28 15:56:20.431114: I horovod/common/operations.cc:1205] [2]: MaxG Shutdown K
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.432082: I horovod/common/operations.cc:1200] [3]: MaxG MPI_Finalize returned 0
Tue May 28 15:56:20 2019[1,3]<stdout>:[2019-05-28 15:56:20.432126: I horovod/common/operations.cc:1205] [3]: MaxG Shutdown K
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment