Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save yqshao/103d55ea6fdc53043dfbfca3e08dd6a0 to your computer and use it in GitHub Desktop.
Save yqshao/103d55ea6fdc53043dfbfca3e08dd6a0 to your computer and use it in GitHub Desktop.
(partial) EasyBuild log for failed build of /dev/shm/eb-oob3zdsh/files_pr20358/t/TensorFlow/TensorFlow-2.15.1-foss-2023a-CUDA-12.1.1.eb (PR(s) #20358) (easyblock PR(s) #3303)
Expected: true
i = 8 Tx[i] = 0.79477077722549438 Ty[i] = 0.79414105415344238
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.67218607664108276 not close to 0.67186534404754639)
Expected: true
i = 9 Tx[i] = 0.67218607664108276 Ty[i] = 0.67186534404754639
tensorflow/core/framework/tensor_testutil.cc:187: Failure
Expected: (num_failures) < (max_failures), actual: 10 vs 10
Too many mismatches (atol = 1.0000000000000001e-05 rtol = -1), giving up.
2024-06-19 09:11:34.728535: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.738050: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (3.8846702575683594 not close to 3.8839976787567139)
Expected: true
i = 0 Tx[i] = 3.8846702575683594 Ty[i] = 3.8839976787567139
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (-2.4200799465179443 not close to -2.4202702045440674)
Expected: true
i = 1 Tx[i] = -2.4200799465179443 Ty[i] = -2.4202702045440674
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.8368873596191406 not close to 2.8364553451538086)
Expected: true
i = 2 Tx[i] = 2.8368873596191406 Ty[i] = 2.8364553451538086
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.7212533950805664 not close to 2.7214481830596924)
Expected: true
i = 3 Tx[i] = 2.7212533950805664 Ty[i] = 2.7214481830596924
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.94675666093826294 not close to 0.94643855094909668)
Expected: true
i = 4 Tx[i] = 0.94675666093826294 Ty[i] = 0.94643855094909668
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.0059888362884521484 not close to 0.0061817169189453125)
Expected: true
i = 5 Tx[i] = 0.0059888362884521484 Ty[i] = 0.0061817169189453125
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.040736198425292969 not close to 0.041174411773681641)
Expected: true
i = 6 Tx[i] = 0.040736198425292969 Ty[i] = 0.041174411773681641
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (-1.0978603363037109 not close to -1.0974183082580566)
Expected: true
i = 7 Tx[i] = -1.0978603363037109 Ty[i] = -1.0974183082580566
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.2201886177062988 not close to 1.2197984457015991)
Expected: true
i = 8 Tx[i] = 1.2201886177062988 Ty[i] = 1.2197984457015991
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (-1.0960085391998291 not close to -1.0955471992492676)
Expected: true
i = 9 Tx[i] = -1.0960085391998291 Ty[i] = -1.0955471992492676
tensorflow/core/framework/tensor_testutil.cc:187: Failure
Expected: (num_failures) < (max_failures), actual: 10 vs 10
Too many mismatches (atol = 1.0000000000000001e-05 rtol = -1), giving up.
[ FAILED ] Test/FusedMatMulWithBiasOpTest/0.MatMul256x128x64, where TypeParam = float (631 ms)
[ RUN ] Test/FusedMatMulWithBiasOpTest/0.MatMul1x256x256
2024-06-19 09:11:34.745520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.784358: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.802377: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.821462: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.64193946123123169 not close to 0.64122533798217773)
Expected: true
i = 0 Tx[i] = 0.64193946123123169 Ty[i] = 0.64122533798217773
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (-1.7487689256668091 not close to -1.748792290687561)
Expected: true
i = 1 Tx[i] = -1.7487689256668091 Ty[i] = -1.748792290687561
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (-1.1690042018890381 not close to -1.1691427230834961)
Expected: true
i = 2 Tx[i] = -1.1690042018890381 Ty[i] = -1.1691427230834961
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.5510716438293457 not close to 2.5504193305969238)
Expected: true
i = 3 Tx[i] = 2.5510716438293457 Ty[i] = 2.5504193305969238
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (-0.98713958263397217 not close to -0.98724865913391113)
Expected: true
i = 4 Tx[i] = -0.98713958263397217 Ty[i] = -0.98724865913391113
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (-0.67567801475524902 not close to -0.67523503303527832)
Expected: true
i = 5 Tx[i] = -0.67567801475524902 Ty[i] = -0.67523503303527832
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.84158563613891602 not close to 0.84170770645141602)
Expected: true
i = 6 Tx[i] = 0.84158563613891602 Ty[i] = 0.84170770645141602
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (-1.590267539024353 not close to -1.5905402898788452)
Expected: true
i = 7 Tx[i] = -1.590267539024353 Ty[i] = -1.5905402898788452
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.18795979022979736 not close to 0.18810474872589111)
Expected: true
i = 8 Tx[i] = 0.18795979022979736 Ty[i] = 0.18810474872589111
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.2186254262924194 not close to 1.2183554172515869)
Expected: true
i = 9 Tx[i] = 1.2186254262924194 Ty[i] = 1.2183554172515869
tensorflow/core/framework/tensor_testutil.cc:187: Failure
Expected: (num_failures) < (max_failures), actual: 10 vs 10
Too many mismatches (atol = 1.0000000000000001e-05 rtol = -1), giving up.
2024-06-19 09:11:34.829383: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.837024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.845106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.852300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.861741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.870768: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
[ FAILED ] Test/FusedMatMulWithBiasOpTest/0.MatMul1x256x256, where TypeParam = float (132 ms)
[ RUN ] Test/FusedMatMulWithBiasOpTest/0.MatMul256x256x1
2024-06-19 09:11:34.878029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.885002: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.891811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.897361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (-0.47275495529174805 not close to -0.4722825288772583)
Expected: true
i = 0 Tx[i] = -0.47275495529174805 Ty[i] = -0.4722825288772583
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (-1.9528877735137939 not close to -1.9522664546966553)
Expected: true
i = 1 Tx[i] = -1.9528877735137939 Ty[i] = -1.9522664546966553
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.2612571716308594 not close to 2.2610714435577393)
Expected: true
i = 2 Tx[i] = 2.2612571716308594 Ty[i] = 2.2610714435577393
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.0227344036102295 not close to 1.0224490165710449)
Expected: true
i = 3 Tx[i] = 1.0227344036102295 Ty[i] = 1.0224490165710449
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.6487056016921997 not close to 1.6485648155212402)
Expected: true
i = 4 Tx[i] = 1.6487056016921997 Ty[i] = 1.6485648155212402
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.07120215892791748 not close to 0.071267485618591309)
Expected: true
i = 5 Tx[i] = 0.07120215892791748 Ty[i] = 0.071267485618591309
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.4944894313812256 not close to 1.4949222803115845)
Expected: true
i = 6 Tx[i] = 1.4944894313812256 Ty[i] = 1.4949222803115845
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.0336471796035767 not close to 1.0338342189788818)
Expected: true
i = 7 Tx[i] = 1.0336471796035767 Ty[i] = 1.0338342189788818
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.3226463794708252 not close to 1.3229568004608154)
Expected: true
i = 8 Tx[i] = 1.3226463794708252 Ty[i] = 1.3229568004608154
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (-1.6501600742340088 not close to -1.6500155925750732)
Expected: true
i = 9 Tx[i] = -1.6501600742340088 Ty[i] = -1.6500155925750732
tensorflow/core/framework/tensor_testutil.cc:187: Failure
Expected: (num_failures) < (max_failures), actual: 10 vs 10
Too many mismatches (atol = 1.0000000000000001e-05 rtol = -1), giving up.
2024-06-19 09:11:34.903124: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.910368: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.916944: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.923376: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.929985: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.936353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
[ FAILED ] Test/FusedMatMulWithBiasOpTest/0.MatMul256x256x1, where TypeParam = float (65 ms)
[ RUN ] Test/FusedMatMulWithBiasOpTest/0.MatMul1x256x1
2024-06-19 09:11:34.942926: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:34.947983: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
[ OK ] Test/FusedMatMulWithBiasOpTest/0.MatMul1x256x1 (10 ms)
[ RUN ] Test/FusedMatMulWithBiasOpTest/0.MatMul256x128x64WithActivation
2024-06-19 09:11:34.955156: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:35.107543: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.8359682559967041 not close to 0.83576458692550659)
Expected: true
i = 1 Tx[i] = 0.8359682559967041 Ty[i] = 0.83576458692550659
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.6040606498718262 not close to 2.6036930084228516)
Expected: true
i = 2 Tx[i] = 2.6040606498718262 Ty[i] = 2.6036930084228516
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.047066211700439453 not close to 0.046637892723083496)
Expected: true
i = 3 Tx[i] = 0.047066211700439453 Ty[i] = 0.046637892723083496
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (3.2687506675720215 not close to 3.2683916091918945)
Expected: true
i = 4 Tx[i] = 3.2687506675720215 Ty[i] = 3.2683916091918945
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.4333869218826294 not close to 1.4331045150756836)
Expected: true
i = 5 Tx[i] = 1.4333869218826294 Ty[i] = 1.4331045150756836
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.43990206718444824 not close to 0.44011807441711426)
Expected: true
i = 8 Tx[i] = 0.43990206718444824 Ty[i] = 0.44011807441711426
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.46132540702819824 not close to 0.46143496036529541)
Expected: true
i = 10 Tx[i] = 0.46132540702819824 Ty[i] = 0.46143496036529541
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.6572375297546387 not close to 1.6569837331771851)
Expected: true
i = 12 Tx[i] = 1.6572375297546387 Ty[i] = 1.6569837331771851
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.42939138412475586 not close to 0.43009209632873535)
Expected: true
i = 13 Tx[i] = 0.42939138412475586 Ty[i] = 0.43009209632873535
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.2736339569091797 not close to 2.2731962203979492)
Expected: true
i = 14 Tx[i] = 2.2736339569091797 Ty[i] = 2.2731962203979492
tensorflow/core/framework/tensor_testutil.cc:187: Failure
Expected: (num_failures) < (max_failures), actual: 10 vs 10
Too many mismatches (atol = 1.0000000000000001e-05 rtol = -1), giving up.
2024-06-19 09:11:35.126103: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:35.287105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.28547006845474243 not close to 0.28565779328346252)
Expected: true
i = 0 Tx[i] = 0.28547006845474243 Ty[i] = 0.28565779328346252
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.40718865394592285 not close to 0.40797686576843262)
Expected: true
i = 1 Tx[i] = 0.40718865394592285 Ty[i] = 0.40797686576843262
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.3653874397277832 not close to 0.3657681941986084)
Expected: true
i = 2 Tx[i] = 0.3653874397277832 Ty[i] = 0.3657681941986084
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.8045408725738525 not close to 1.8033087253570557)
Expected: true
i = 3 Tx[i] = 1.8045408725738525 Ty[i] = 1.8033087253570557
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.8206048011779785 not close to 1.8206955194473267)
Expected: true
i = 4 Tx[i] = 1.8206048011779785 Ty[i] = 1.8206955194473267
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.6212331056594849 not close to 1.6211655139923096)
Expected: true
i = 5 Tx[i] = 1.6212331056594849 Ty[i] = 1.6211655139923096
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.602130651473999 not close to 2.6023159027099609)
Expected: true
i = 6 Tx[i] = 2.602130651473999 Ty[i] = 2.6023159027099609
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.6459248065948486 not close to 2.646120548248291)
Expected: true
i = 8 Tx[i] = 2.6459248065948486 Ty[i] = 2.646120548248291
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.7717893123626709 not close to 1.7720063924789429)
Expected: true
i = 10 Tx[i] = 1.7717893123626709 Ty[i] = 1.7720063924789429
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.0171277523040771 not close to 2.0171072483062744)
Expected: true
i = 11 Tx[i] = 2.0171277523040771 Ty[i] = 2.0171072483062744
tensorflow/core/framework/tensor_testutil.cc:187: Failure
Expected: (num_failures) < (max_failures), actual: 10 vs 10
Too many mismatches (atol = 1.0000000000000001e-05 rtol = -1), giving up.
2024-06-19 09:11:35.304556: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:35.499047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (3.6455307006835938 not close to 3.6458392143249512)
Expected: true
i = 2 Tx[i] = 3.6455307006835938 Ty[i] = 3.6458392143249512
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.1020441055297852 not close to 2.1014847755432129)
Expected: true
i = 3 Tx[i] = 2.1020441055297852 Ty[i] = 2.1014847755432129
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.7777352333068848 not close to 2.7777528762817383)
Expected: true
i = 5 Tx[i] = 2.7777352333068848 Ty[i] = 2.7777528762817383
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.7984299659729004 not close to 2.7984819412231445)
Expected: true
i = 6 Tx[i] = 2.7984299659729004 Ty[i] = 2.7984819412231445
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.33777934312820435 not close to 0.33790296316146851)
Expected: true
i = 8 Tx[i] = 0.33777934312820435 Ty[i] = 0.33790296316146851
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (4.5897302627563477 not close to 4.5904135704040527)
Expected: true
i = 9 Tx[i] = 4.5897302627563477 Ty[i] = 4.5904135704040527
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.7685508728027344 not close to 1.7686034440994263)
Expected: true
i = 13 Tx[i] = 1.7685508728027344 Ty[i] = 1.7686034440994263
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.9485669136047363 not close to 2.9485468864440918)
Expected: true
i = 15 Tx[i] = 2.9485669136047363 Ty[i] = 2.9485468864440918
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (5.2114028930664062 not close to 5.211359977722168)
Expected: true
i = 16 Tx[i] = 5.2114028930664062 Ty[i] = 5.211359977722168
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (1.0042421817779541 not close to 1.0044519901275635)
Expected: true
i = 17 Tx[i] = 1.0042421817779541 Ty[i] = 1.0044519901275635
tensorflow/core/framework/tensor_testutil.cc:187: Failure
Expected: (num_failures) < (max_failures), actual: 10 vs 10
Too many mismatches (atol = 1.0000000000000001e-05 rtol = -1), giving up.
2024-06-19 09:11:35.527119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:35.763850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.46155077219009399 not close to 0.46171647310256958)
Expected: true
i = 1 Tx[i] = 0.46155077219009399 Ty[i] = 0.46171647310256958
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (3.7600879669189453 not close to 3.760930061340332)
Expected: true
i = 3 Tx[i] = 3.7600879669189453 Ty[i] = 3.760930061340332
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.6236052513122559 not close to 2.6236631870269775)
Expected: true
i = 5 Tx[i] = 2.6236052513122559 Ty[i] = 2.6236631870269775
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.68924880027771 not close to 2.6888184547424316)
Expected: true
i = 6 Tx[i] = 2.68924880027771 Ty[i] = 2.6888184547424316
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (0.51600128412246704 not close to 0.51613467931747437)
Expected: true
i = 7 Tx[i] = 0.51600128412246704 Ty[i] = 0.51613467931747437
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (3.2079949378967285 not close to 3.2077460289001465)
Expected: true
i = 8 Tx[i] = 3.2079949378967285 Ty[i] = 3.2077460289001465
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (3.4697041511535645 not close to 3.4698390960693359)
Expected: true
i = 10 Tx[i] = 3.4697041511535645 Ty[i] = 3.4698390960693359
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (6.4911842346191406 not close to 6.4912204742431641)
Expected: true
i = 11 Tx[i] = 6.4911842346191406 Ty[i] = 6.4912204742431641
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (3.5709230899810791 not close to 3.5705535411834717)
Expected: true
i = 12 Tx[i] = 3.5709230899810791 Ty[i] = 3.5705535411834717
tensorflow/core/framework/tensor_testutil.cc:184: Failure
Value of: IsClose(Tx[i], Ty[i], typed_atol, typed_rtol)
Actual: false (2.9294028282165527 not close to 2.9302127361297607)
Expected: true
i = 13 Tx[i] = 2.9294028282165527 Ty[i] = 2.9302127361297607
tensorflow/core/framework/tensor_testutil.cc:187: Failure
Expected: (num_failures) < (max_failures), actual: 10 vs 10
Too many mismatches (atol = 1.0000000000000001e-05 rtol = -1), giving up.
2024-06-19 09:11:35.824307: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:35.916619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:35.934123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:35.950548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:35.966813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:35.994523: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.006483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.015719: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.026998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.035493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.041981: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.060563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.066703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.085538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.091789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.110539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.116692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.137542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.157106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.199745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.206021: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.211878: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.240786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.247069: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
[ FAILED ] Test/FusedMatMulWithBiasOpTest/0.MatMul256x128x64WithActivation, where TypeParam = float (1299 ms)
[ RUN ] Test/FusedMatMulWithBiasOpTest/0.MatMul1x256x256WithActivation
2024-06-19 09:11:36.278029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.429245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.437262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.444024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.451126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.457683: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.464568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.471310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
[ OK ] Test/FusedMatMulWithBiasOpTest/0.MatMul1x256x256WithActivation (225 ms)
[ RUN ] Test/FusedMatMulWithBiasOpTest/0.MatMul256x256x1WithActivation
2024-06-19 09:11:36.478374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.666154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.674372: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.692128: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.699028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.705677: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.712289: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.718767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
[ OK ] Test/FusedMatMulWithBiasOpTest/0.MatMul256x256x1WithActivation (259 ms)
[ RUN ] Test/FusedMatMulWithBiasOpTest/0.MatMul1x256x1WithActivation
2024-06-19 09:11:36.737952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.922673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.939157: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.956506: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.963416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.969339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.986216: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
2024-06-19 09:11:36.992268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38553 MB memory: -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
[ OK ] Test/FusedMatMulWithBiasOpTest/0.MatMul1x256x1WithActivation (271 ms)
[----------] 8 tests from Test/FusedMatMulWithBiasOpTest/0 (2895 ms total)
[----------] Global test environment tear-down
[==========] 8 tests from 1 test suite ran. (2895 ms total)
[ PASSED ] 4 tests.
[ FAILED ] 4 tests, listed below:
[ FAILED ] Test/FusedMatMulWithBiasOpTest/0.MatMul256x128x64, where TypeParam = float
[ FAILED ] Test/FusedMatMulWithBiasOpTest/0.MatMul1x256x256, where TypeParam = float
[ FAILED ] Test/FusedMatMulWithBiasOpTest/0.MatMul256x256x1, where TypeParam = float
[ FAILED ] Test/FusedMatMulWithBiasOpTest/0.MatMul256x128x64WithActivation, where TypeParam = float
4 FAILED TESTS
== 2024-06-19 11:16:18,096 build_log.py:171 ERROR EasyBuild crashed with an error (at easybuild/base/exceptions.py:126 in __init__): At least 1 gpu tests failed:
//tensorflow/core/kernels:matmul_op_test_gpu (at easybuild/framework/easyblock.py:2292 in report_test_failure)
== 2024-06-19 11:16:18,097 build_log.py:267 INFO ... (took 1 hour 0 mins 59 secs)
== 2024-06-19 11:16:18,100 build_log.py:267 INFO ... (took 1 hour 2 mins 0 secs)
== 2024-06-19 11:16:18,102 filetools.py:2013 INFO Removing lock /local/tmp.2444677/software/.locks/_local_tmp.2444677_software_TensorFlow_2.15.1-foss-2023a-CUDA-12.1.1.lock...
== 2024-06-19 11:16:18,102 filetools.py:383 INFO Path /local/tmp.2444677/software/.locks/_local_tmp.2444677_software_TensorFlow_2.15.1-foss-2023a-CUDA-12.1.1.lock successfully removed.
== 2024-06-19 11:16:18,102 filetools.py:2017 INFO Lock removed: /local/tmp.2444677/software/.locks/_local_tmp.2444677_software_TensorFlow_2.15.1-foss-2023a-CUDA-12.1.1.lock
== 2024-06-19 11:16:18,103 easyblock.py:4285 WARNING build failed (first 300 chars): At least 1 gpu tests failed:
//tensorflow/core/kernels:matmul_op_test_gpu
== 2024-06-19 11:16:18,103 easyblock.py:328 INFO Closing log for application name TensorFlow version 2.15.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment