Skip to content

Instantly share code, notes, and snippets.

@sanjoy

sanjoy/gist:ba102ae122d2eaaa2073e5afe0ee0de2 Secret

Created February 1, 2020 06:22
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save sanjoy/ba102ae122d2eaaa2073e5afe0ee0de2 to your computer and use it in GitHub Desktop.
Save sanjoy/ba102ae122d2eaaa2073e5afe0ee0de2 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //tensorflow/compiler/tf2tensorrt:trt_engine_op_test
-----------------------------------------------------------------------------
2020-02-01 06:22:08.514493: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
Running main() from test_main.cc
[==========] Running 4 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 2 tests from TRTEngineOpTestBase
[ RUN ] TRTEngineOpTestBase.DynamicShapes
2020-02-01 06:22:08.529819: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
2020-02-01 06:22:08.536561: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-02-01 06:22:09.064201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 0 with properties:
pciBusID: 0000:84:00.0 name: TITAN V computeCapability: 7.0
coreClock: 1.455GHz coreCount: 80 deviceMemorySize: 11.78GiB deviceMemoryBandwidth: 607.97GiB/s
2020-02-01 06:22:09.065218: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 1 with properties:
pciBusID: 0000:03:00.0 name: Quadro GP100 computeCapability: 6.0
coreClock: 1.4425GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
2020-02-01 06:22:09.070643: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 2 with properties:
pciBusID: 0000:04:00.0 name: Quadro M2000 computeCapability: 5.2
coreClock: 1.1625GHz coreCount: 6 deviceMemorySize: 3.93GiB deviceMemoryBandwidth: 98.44GiB/s
2020-02-01 06:22:09.070671: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-02-01 06:22:09.089741: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-02-01 06:22:09.099666: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-02-01 06:22:09.104590: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-02-01 06:22:09.118771: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-02-01 06:22:09.123557: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-02-01 06:22:09.129602: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-02-01 06:22:09.133177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1685] Ignoring visible gpu device (device: 2, name: Quadro M2000, pci bus id: 0000:04:00.0, compute capability: 5.2) with core count: 6. The minimum required count is 8. You can adjust this requirement with the env var TF_MIN_GPU_MULTIPROCESSOR_COUNT.
2020-02-01 06:22:09.133188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1700] Adding visible gpu devices: 0, 1
2020-02-01 06:22:09.865926: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1099] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-01 06:22:09.865959: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] 0 1 2
2020-02-01 06:22:09.865965: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 0: N N N
2020-02-01 06:22:09.865968: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 1: N N N
2020-02-01 06:22:09.865971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 2: N N N
2020-02-01 06:22:09.867674: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1244] Created TensorFlow device (/job:worker/replica:0/task:0/device:GPU:0 with 180 MB memory) -> physical GPU (device: 0, name: TITAN V, pci bus id: 0000:84:00.0, compute capability: 7.0)
2020-02-01 06:22:09.887996: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:791] Building a new TensorRT engine for myop with input shapes: [[2,2]]
2020-02-01 06:22:09.888029: I tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1135] Linked TensorRT version: 6.0.1
2020-02-01 06:22:09.921772: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-02-01 06:22:09.921797: I tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1136] Loaded TensorRT version: 6.0.1
2020-02-01 06:22:09.938154: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
2020-02-01 06:22:10.293047: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-02-01 06:22:10.737479: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2020-02-01 06:22:10.737944: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2020-02-01 06:22:10.738019: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:806] Engine creation for myop failed. The native segment will be used instead. Reason: Internal: Failed to build TensorRT engine
2020-02-01 06:22:10.741568: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:791] Building a new TensorRT engine for myop with input shapes: [[3,2]]
2020-02-01 06:22:10.756872: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2020-02-01 06:22:10.756924: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2020-02-01 06:22:10.756950: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:806] Engine creation for myop failed. The native segment will be used instead. Reason: Internal: Failed to build TensorRT engine
2020-02-01 06:22:10.758351: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:791] Building a new TensorRT engine for myop with input shapes: [[10,10]]
2020-02-01 06:22:10.760297: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2020-02-01 06:22:10.760329: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2020-02-01 06:22:10.760343: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:806] Engine creation for myop failed. The native segment will be used instead. Reason: Internal: Failed to build TensorRT engine
[ OK ] TRTEngineOpTestBase.DynamicShapes (2233 ms)
[ RUN ] TRTEngineOpTestBase.AllowBuildAtRuntime
2020-02-01 06:22:10.763157: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 0 with properties:
pciBusID: 0000:84:00.0 name: TITAN V computeCapability: 7.0
coreClock: 1.455GHz coreCount: 80 deviceMemorySize: 11.78GiB deviceMemoryBandwidth: 607.97GiB/s
2020-02-01 06:22:10.764056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 1 with properties:
pciBusID: 0000:03:00.0 name: Quadro GP100 computeCapability: 6.0
coreClock: 1.4425GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
2020-02-01 06:22:10.764576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 2 with properties:
pciBusID: 0000:04:00.0 name: Quadro M2000 computeCapability: 5.2
coreClock: 1.1625GHz coreCount: 6 deviceMemorySize: 3.93GiB deviceMemoryBandwidth: 98.44GiB/s
2020-02-01 06:22:10.764602: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-02-01 06:22:10.764632: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-02-01 06:22:10.764646: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-02-01 06:22:10.764660: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-02-01 06:22:10.764690: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-02-01 06:22:10.764704: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-02-01 06:22:10.764733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-02-01 06:22:10.768351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1685] Ignoring visible gpu device (device: 2, name: Quadro M2000, pci bus id: 0000:04:00.0, compute capability: 5.2) with core count: 6. The minimum required count is 8. You can adjust this requirement with the env var TF_MIN_GPU_MULTIPROCESSOR_COUNT.
2020-02-01 06:22:10.768360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1700] Adding visible gpu devices: 0, 1
2020-02-01 06:22:11.057931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1099] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-01 06:22:11.057990: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] 0 1 2
2020-02-01 06:22:11.057996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 0: N N N
2020-02-01 06:22:11.057999: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 1: N N N
2020-02-01 06:22:11.058002: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 2: N N N
2020-02-01 06:22:11.059050: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1244] Created TensorFlow device (/job:worker/replica:0/task:0/device:GPU:0 with 180 MB memory) -> physical GPU (device: 0, name: TITAN V, pci bus id: 0000:84:00.0, compute capability: 7.0)
2020-02-01 06:22:11.060594: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:780] Found no engine in cache matching input shapes. Not building a new engine because allow_build_at_runtime=False. The native segment will be used instead.
tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op_test.cc:190: Failure
Expected equality of these values:
0
cache->size()
Which is: 1
[ FAILED ] TRTEngineOpTestBase.AllowBuildAtRuntime (299 ms)
[----------] 2 tests from TRTEngineOpTestBase (2532 ms total)
[----------] 1 test from TRTEngineOpTest/0, where TypeParam = float
[ RUN ] TRTEngineOpTest/0.Basic
2020-02-01 06:22:11.062203: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 0 with properties:
pciBusID: 0000:84:00.0 name: TITAN V computeCapability: 7.0
coreClock: 1.455GHz coreCount: 80 deviceMemorySize: 11.78GiB deviceMemoryBandwidth: 607.97GiB/s
2020-02-01 06:22:11.063089: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 1 with properties:
pciBusID: 0000:03:00.0 name: Quadro GP100 computeCapability: 6.0
coreClock: 1.4425GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
2020-02-01 06:22:11.063570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 2 with properties:
pciBusID: 0000:04:00.0 name: Quadro M2000 computeCapability: 5.2
coreClock: 1.1625GHz coreCount: 6 deviceMemorySize: 3.93GiB deviceMemoryBandwidth: 98.44GiB/s
2020-02-01 06:22:11.063600: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-02-01 06:22:11.063614: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-02-01 06:22:11.063628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-02-01 06:22:11.063659: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-02-01 06:22:11.063673: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-02-01 06:22:11.063686: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-02-01 06:22:11.063699: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-02-01 06:22:11.067212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1685] Ignoring visible gpu device (device: 2, name: Quadro M2000, pci bus id: 0000:04:00.0, compute capability: 5.2) with core count: 6. The minimum required count is 8. You can adjust this requirement with the env var TF_MIN_GPU_MULTIPROCESSOR_COUNT.
2020-02-01 06:22:11.067222: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1700] Adding visible gpu devices: 0, 1
2020-02-01 06:22:11.067241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1099] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-01 06:22:11.067246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] 0 1 2
2020-02-01 06:22:11.067249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 0: N N N
2020-02-01 06:22:11.067252: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 1: N N N
2020-02-01 06:22:11.067255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 2: N N N
2020-02-01 06:22:11.068134: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1244] Created TensorFlow device (/job:worker/replica:0/task:0/device:GPU:0 with 180 MB memory) -> physical GPU (device: 0, name: TITAN V, pci bus id: 0000:84:00.0, compute capability: 7.0)
2020-02-01 06:22:11.068951: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:791] Building a new TensorRT engine for myop with input shapes: [[1,2]]
2020-02-01 06:22:11.071213: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2020-02-01 06:22:11.071261: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2020-02-01 06:22:11.071278: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:806] Engine creation for myop failed. The native segment will be used instead. Reason: Internal: Failed to build TensorRT engine
[ OK ] TRTEngineOpTest/0.Basic (11 ms)
[----------] 1 test from TRTEngineOpTest/0 (11 ms total)
[----------] 1 test from TRTEngineOpTest/1, where TypeParam = Eigen::half
[ RUN ] TRTEngineOpTest/1.Basic
2020-02-01 06:22:11.072670: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 0 with properties:
pciBusID: 0000:84:00.0 name: TITAN V computeCapability: 7.0
coreClock: 1.455GHz coreCount: 80 deviceMemorySize: 11.78GiB deviceMemoryBandwidth: 607.97GiB/s
2020-02-01 06:22:11.073820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 1 with properties:
pciBusID: 0000:03:00.0 name: Quadro GP100 computeCapability: 6.0
coreClock: 1.4425GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
2020-02-01 06:22:11.074485: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 2 with properties:
pciBusID: 0000:04:00.0 name: Quadro M2000 computeCapability: 5.2
coreClock: 1.1625GHz coreCount: 6 deviceMemorySize: 3.93GiB deviceMemoryBandwidth: 98.44GiB/s
2020-02-01 06:22:11.074505: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-02-01 06:22:11.074537: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-02-01 06:22:11.074568: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-02-01 06:22:11.074591: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-02-01 06:22:11.074604: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-02-01 06:22:11.074616: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-02-01 06:22:11.074629: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-02-01 06:22:11.078077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1685] Ignoring visible gpu device (device: 2, name: Quadro M2000, pci bus id: 0000:04:00.0, compute capability: 5.2) with core count: 6. The minimum required count is 8. You can adjust this requirement with the env var TF_MIN_GPU_MULTIPROCESSOR_COUNT.
2020-02-01 06:22:11.078087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1700] Adding visible gpu devices: 0, 1
2020-02-01 06:22:11.078105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1099] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-01 06:22:11.078110: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] 0 1 2
2020-02-01 06:22:11.078113: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 0: N N N
2020-02-01 06:22:11.078116: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 1: N N N
2020-02-01 06:22:11.078119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 2: N N N
2020-02-01 06:22:11.078914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1244] Created TensorFlow device (/job:worker/replica:0/task:0/device:GPU:0 with 180 MB memory) -> physical GPU (device: 0, name: TITAN V, pci bus id: 0000:84:00.0, compute capability: 7.0)
2020-02-01 06:22:11.079708: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:791] Building a new TensorRT engine for myop with input shapes: [[1,2]]
2020-02-01 06:22:11.081542: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2020-02-01 06:22:11.081575: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2020-02-01 06:22:11.081588: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:806] Engine creation for myop failed. The native segment will be used instead. Reason: Internal: Failed to build TensorRT engine
[ OK ] TRTEngineOpTest/1.Basic (10 ms)
[----------] 1 test from TRTEngineOpTest/1 (10 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 3 test suites ran. (2553 ms total)
[ PASSED ] 3 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] TRTEngineOpTestBase.AllowBuildAtRuntime
1 FAILED TEST
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment