TF_Gather_ONNX

trtexec_output.log

&&&& RUNNING TensorRT.trtexec # trtexec --onnx=/home/soccer_ball_tracker_poc/tests/debug/simple_tf_gather_cnn.onnx --verbose
[10/03/2021-15:57:44] [I] === Model Options ===
[10/03/2021-15:57:44] [I] Format: ONNX
[10/03/2021-15:57:44] [I] Model: /home/soccer_ball_tracker_poc/tests/debug/simple_tf_gather_cnn.onnx
[10/03/2021-15:57:44] [I] Output:
[10/03/2021-15:57:44] [I] === Build Options ===
[10/03/2021-15:57:44] [I] Max batch: explicit
[10/03/2021-15:57:44] [I] Workspace: 16 MiB
[10/03/2021-15:57:44] [I] minTiming: 1
[10/03/2021-15:57:44] [I] avgTiming: 8
[10/03/2021-15:57:44] [I] Precision: FP32
[10/03/2021-15:57:44] [I] Calibration: 
[10/03/2021-15:57:44] [I] Refit: Disabled
[10/03/2021-15:57:44] [I] Safe mode: Disabled
[10/03/2021-15:57:44] [I] Save engine: 
[10/03/2021-15:57:44] [I] Load engine: 
[10/03/2021-15:57:44] [I] Builder Cache: Enabled
[10/03/2021-15:57:44] [I] NVTX verbosity: 0
[10/03/2021-15:57:44] [I] Tactic sources: Using default tactic sources
[10/03/2021-15:57:44] [I] Input(s)s format: fp32:CHW
[10/03/2021-15:57:44] [I] Output(s)s format: fp32:CHW
[10/03/2021-15:57:44] [I] Input build shapes: model
[10/03/2021-15:57:44] [I] Input calibration shapes: model
[10/03/2021-15:57:44] [I] === System Options ===
[10/03/2021-15:57:44] [I] Device: 0
[10/03/2021-15:57:44] [I] DLACore: 
[10/03/2021-15:57:44] [I] Plugins:
[10/03/2021-15:57:44] [I] === Inference Options ===
[10/03/2021-15:57:44] [I] Batch: Explicit
[10/03/2021-15:57:44] [I] Input inference shapes: model
[10/03/2021-15:57:44] [I] Iterations: 10
[10/03/2021-15:57:44] [I] Duration: 3s (+ 200ms warm up)
[10/03/2021-15:57:44] [I] Sleep time: 0ms
[10/03/2021-15:57:44] [I] Streams: 1
[10/03/2021-15:57:44] [I] ExposeDMA: Disabled
[10/03/2021-15:57:44] [I] Data transfers: Enabled
[10/03/2021-15:57:44] [I] Spin-wait: Disabled
[10/03/2021-15:57:44] [I] Multithreading: Disabled
[10/03/2021-15:57:44] [I] CUDA Graph: Disabled
[10/03/2021-15:57:44] [I] Separate profiling: Disabled
[10/03/2021-15:57:44] [I] Skip inference: Disabled
[10/03/2021-15:57:44] [I] Inputs:
[10/03/2021-15:57:44] [I] === Reporting Options ===
[10/03/2021-15:57:44] [I] Verbose: Enabled
[10/03/2021-15:57:44] [I] Averages: 10 inferences
[10/03/2021-15:57:44] [I] Percentile: 99
[10/03/2021-15:57:44] [I] Dump refittable layers:Disabled
[10/03/2021-15:57:44] [I] Dump output: Disabled
[10/03/2021-15:57:44] [I] Profile: Disabled
[10/03/2021-15:57:44] [I] Export timing to JSON file: 
[10/03/2021-15:57:44] [I] Export output to JSON file: 
[10/03/2021-15:57:44] [I] Export profile to JSON file: 
[10/03/2021-15:57:44] [I] 
[10/03/2021-15:57:45] [I] === Device Information ===
[10/03/2021-15:57:45] [I] Selected Device: NVIDIA GeForce GTX 1050
[10/03/2021-15:57:45] [I] Compute Capability: 6.1
[10/03/2021-15:57:45] [I] SMs: 5
[10/03/2021-15:57:45] [I] Compute Clock Rate: 1.493 GHz
[10/03/2021-15:57:45] [I] Device Global Memory: 4040 MiB
[10/03/2021-15:57:45] [I] Shared Memory per SM: 96 KiB
[10/03/2021-15:57:45] [I] Memory Bus Width: 128 bits (ECC disabled)
[10/03/2021-15:57:45] [I] Memory Clock Rate: 3.504 GHz
[10/03/2021-15:57:45] [I] 
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::BatchTilePlugin_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::CoordConvAC version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::GenerateDetection_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::Proposal version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[10/03/2021-15:57:45] [V] [TRT] Registered plugin creator - ::Split version 1
----------------------------------------------------------------
Input filename:   /home/soccer_ball_tracker_poc/tests/debug/simple_tf_gather_cnn.onnx
ONNX IR version:  0.0.7
Opset version:    12
Producer name:    tf2onnx
Producer version: 1.9.2
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::BatchTilePlugin_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::BatchedNMS_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::BatchedNMSDynamic_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::CoordConvAC version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::CropAndResize version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::DetectionLayer_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::FlattenConcat_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::GenerateDetection_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::GridAnchor_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::GridAnchorRect_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::InstanceNormalization_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::LReLU_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::MultilevelCropAndResize_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::MultilevelProposeROI_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::NMS_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::Normalize_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::PriorBox_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::ProposalLayer_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::Proposal version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::PyramidROIAlign_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::Region_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::Reorg_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::ResizeNearest_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::RPROI_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::SpecialSlice_TRT version 1
[10/03/2021-15:57:52] [V] [TRT] Plugin creator already registered - ::Split version 1
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:206: Adding network input: input_9 with dtype: float32, dimensions: (-1, 64, 64, 3)
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: input_9 for ONNX tensor: input_9
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:90: Importing initializer: new_shape__41
[10/03/2021-15:57:52] [10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:90: Importing initializer: conv2d_25_2/kernel/read__10
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:90: Importing initializer: conv2d_24_2/kernel/read__8
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:90: Importing initializer: conv2d_24_2/bias/read__9
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:90: Importing initializer: conv2d_23_2/kernel/read__6
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:90: Importing initializer: conv2d_23_2/bias/read__7
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:90: Importing initializer: const_fold_opt__43
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: conv2d_23_2/BiasAdd__13 [Transpose]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: input_9
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: conv2d_23_2/BiasAdd__13 [Transpose] inputs: [input_9 -> (-1, 64, 64, 3)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:154: Registering layer: conv2d_23_2/BiasAdd__13 for ONNX node: conv2d_23_2/BiasAdd__13
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: conv2d_23_2/BiasAdd__13:0 for ONNX tensor: conv2d_23_2/BiasAdd__13:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:183: conv2d_23_2/BiasAdd__13 [Transpose] outputs: [conv2d_23_2/BiasAdd__13:0 -> (-1, 3, 64, 64)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: conv2d_23_2/BiasAdd [Conv]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: conv2d_23_2/BiasAdd__13:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: conv2d_23_2/kernel/read__6
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: conv2d_23_2/bias/read__7
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: conv2d_23_2/BiasAdd [Conv] inputs: [conv2d_23_2/BiasAdd__13:0 -> (-1, 3, 64, 64)], [conv2d_23_2/kernel/read__6 -> (32, 3, 3, 3)], [conv2d_23_2/bias/read__7 -> (32)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/builtin_op_importers.cpp:450: Convolution input dimensions: (-1, 3, 64, 64)
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:154: Registering layer: conv2d_23_2/BiasAdd for ONNX node: conv2d_23_2/BiasAdd
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/builtin_op_importers.cpp:539: Using kernel: (3, 3), strides: (1, 1), prepadding: (0, 0), postpadding: (0, 0), dilations: (1, 1), numOutputs: 32
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/builtin_op_importers.cpp:540: Convolution output dimensions: (-1, 32, 62, 62)
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: conv2d_23_2/BiasAdd:0 for ONNX tensor: conv2d_23_2/BiasAdd:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:183: conv2d_23_2/BiasAdd [Conv] outputs: [conv2d_23_2/BiasAdd:0 -> (-1, 32, 62, 62)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: conv2d_23_2/Relu [Relu]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: conv2d_23_2/BiasAdd:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: conv2d_23_2/Relu [Relu] inputs: [conv2d_23_2/BiasAdd:0 -> (-1, 32, 62, 62)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:154: Registering layer: conv2d_23_2/Relu for ONNX node: conv2d_23_2/Relu
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: conv2d_23_2/Relu:0 for ONNX tensor: conv2d_23_2/Relu:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:183: conv2d_23_2/Relu [Relu] outputs: [conv2d_23_2/Relu:0 -> (-1, 32, 62, 62)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: max_pooling2d_17_2/MaxPool [MaxPool]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: conv2d_23_2/Relu:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: max_pooling2d_17_2/MaxPool [MaxPool] inputs: [conv2d_23_2/Relu:0 -> (-1, 32, 62, 62)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:154: Registering layer: max_pooling2d_17_2/MaxPool for ONNX node: max_pooling2d_17_2/MaxPool
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: max_pooling2d_17_2/MaxPool:0 for ONNX tensor: max_pooling2d_17_2/MaxPool:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:183: max_pooling2d_17_2/MaxPool [MaxPool] outputs: [max_pooling2d_17_2/MaxPool:0 -> (-1, 32, 31, 31)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: conv2d_24_2/BiasAdd [Conv]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: max_pooling2d_17_2/MaxPool:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: conv2d_24_2/kernel/read__8
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: conv2d_24_2/bias/read__9
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: conv2d_24_2/BiasAdd [Conv] inputs: [max_pooling2d_17_2/MaxPool:0 -> (-1, 32, 31, 31)], [conv2d_24_2/kernel/read__8 -> (16, 32, 3, 3)], [conv2d_24_2/bias/read__9 -> (16)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/builtin_op_importers.cpp:450: Convolution input dimensions: (-1, 32, 31, 31)
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:154: Registering layer: conv2d_24_2/BiasAdd for ONNX node: conv2d_24_2/BiasAdd
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/builtin_op_importers.cpp:539: Using kernel: (3, 3), strides: (1, 1), prepadding: (0, 0), postpadding: (0, 0), dilations: (1, 1), numOutputs: 16
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/builtin_op_importers.cpp:540: Convolution output dimensions: (-1, 16, 29, 29)
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: conv2d_24_2/BiasAdd:0 for ONNX tensor: conv2d_24_2/BiasAdd:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:183: conv2d_24_2/BiasAdd [Conv] outputs: [conv2d_24_2/BiasAdd:0 -> (-1, 16, 29, 29)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: conv2d_24_2/Relu [Relu]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: conv2d_24_2/BiasAdd:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: conv2d_24_2/Relu [Relu] inputs: [conv2d_24_2/BiasAdd:0 -> (-1, 16, 29, 29)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:154: Registering layer: conv2d_24_2/Relu for ONNX node: conv2d_24_2/Relu
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: conv2d_24_2/Relu:0 for ONNX tensor: conv2d_24_2/Relu:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:183: conv2d_24_2/Relu [Relu] outputs: [conv2d_24_2/Relu:0 -> (-1, 16, 29, 29)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: max_pooling2d_18_2/MaxPool [MaxPool]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: conv2d_24_2/Relu:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: max_pooling2d_18_2/MaxPool [MaxPool] inputs: [conv2d_24_2/Relu:0 -> (-1, 16, 29, 29)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:154: Registering layer: max_pooling2d_18_2/MaxPool for ONNX node: max_pooling2d_18_2/MaxPool
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: max_pooling2d_18_2/MaxPool:0 for ONNX tensor: max_pooling2d_18_2/MaxPool:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:183: max_pooling2d_18_2/MaxPool [MaxPool] outputs: [max_pooling2d_18_2/MaxPool:0 -> (-1, 16, 14, 14)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: conv2d_25_2/convolution [Conv]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: max_pooling2d_18_2/MaxPool:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: conv2d_25_2/kernel/read__10
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: conv2d_25_2/convolution [Conv] inputs: [max_pooling2d_18_2/MaxPool:0 -> (-1, 16, 14, 14)], [conv2d_25_2/kernel/read__10 -> (1, 16, 3, 3)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/builtin_op_importers.cpp:450: Convolution input dimensions: (-1, 16, 14, 14)
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:154: Registering layer: conv2d_25_2/convolution for ONNX node: conv2d_25_2/convolution
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/builtin_op_importers.cpp:539: Using kernel: (3, 3), strides: (1, 1), prepadding: (1, 1), postpadding: (1, 1), dilations: (1, 1), numOutputs: 1
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/builtin_op_importers.cpp:540: Convolution output dimensions: (-1, 1, 14, 14)
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: conv2d_25_2/convolution:0 for ONNX tensor: conv2d_25_2/convolution:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:183: conv2d_25_2/convolution [Conv] outputs: [conv2d_25_2/convolution:0 -> (-1, 1, 14, 14)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: conv2d_25_2/convolution__31 [Reshape]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: conv2d_25_2/convolution:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: new_shape__41
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: conv2d_25_2/convolution__31 [Reshape] inputs: [conv2d_25_2/convolution:0 -> (-1, 1, 14, 14)], [new_shape__41 -> (4)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:154: Registering layer: conv2d_25_2/convolution__31 for ONNX node: conv2d_25_2/convolution__31
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: conv2d_25_2/convolution__31:0 for ONNX tensor: conv2d_25_2/convolution__31:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:183: conv2d_25_2/convolution__31 [Reshape] outputs: [conv2d_25_2/convolution__31:0 -> (-1, 14, 14, 1)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: Flatten__11 [Flatten]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: conv2d_25_2/convolution__31:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: Flatten__11 [Flatten] inputs: [conv2d_25_2/convolution__31:0 -> (-1, 14, 14, 1)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:154: Registering layer: Flatten__11 for ONNX node: Flatten__11
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: Flatten__11:0 for ONNX tensor: Flatten__11:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:183: Flatten__11 [Flatten] outputs: [Flatten__11:0 -> (-1, -1)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: decode_layer_7_2/TopKV2 [TopK]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: Flatten__11:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: const_fold_opt__43
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: decode_layer_7_2/TopKV2 [TopK] inputs: [Flatten__11:0 -> (-1, -1)], [const_fold_opt__43 -> (1)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:154: Registering layer: decode_layer_7_2/TopKV2 for ONNX node: decode_layer_7_2/TopKV2
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: decode_layer_7_2/TopKV2:0 for ONNX tensor: decode_layer_7_2/TopKV2:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: decode_layer_7_2/TopKV2:1 for ONNX tensor: decode_layer_7_2/TopKV2:1
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:183: decode_layer_7_2/TopKV2 [TopK] outputs: [decode_layer_7_2/TopKV2:0 -> (-1, 10)], [decode_layer_7_2/TopKV2:1 -> (-1, 10)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: decode_layer_7_2/TopKV2__35 [Cast]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: decode_layer_7_2/TopKV2:1
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: decode_layer_7_2/TopKV2__35 [Cast] inputs: [decode_layer_7_2/TopKV2:1 -> (-1, 10)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/builtin_op_importers.cpp:320: Casting to type: int32
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:154: Registering layer: decode_layer_7_2/TopKV2__35 for ONNX node: decode_layer_7_2/TopKV2__35
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: decode_layer_7_2/TopKV2__35:0 for ONNX tensor: decode_layer_7_2/TopKV2__35:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:183: decode_layer_7_2/TopKV2__35 [Cast] outputs: [decode_layer_7_2/TopKV2__35:0 -> (-1, 10)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: Cast__37 [Cast]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: decode_layer_7_2/TopKV2__35:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: Cast__37 [Cast] inputs: [decode_layer_7_2/TopKV2__35:0 -> (-1, 10)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/builtin_op_importers.cpp:320: Casting to type: int32
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:154: Registering layer: Cast__37 for ONNX node: Cast__37
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: Cast__37:0 for ONNX tensor: Cast__37:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:183: Cast__37 [Cast] outputs: [Cast__37:0 -> (-1, 10)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: Unsqueeze__39 [Unsqueeze]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: Cast__37:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: Unsqueeze__39 [Unsqueeze] inputs: [Cast__37:0 -> (-1, 10)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/onnx2trt_utils.cpp:1815: Original shape: (_, 10), unsqueezing to: (_, _, _)
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:154: Registering layer: Unsqueeze__39 for ONNX node: Unsqueeze__39
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ImporterContext.hpp:120: Registering tensor: Unsqueeze__39:0 for ONNX tensor: Unsqueeze__39:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:183: Unsqueeze__39 [Unsqueeze] outputs: [Unsqueeze__39:0 -> (-1, -1, -1)], 
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:107: Parsing node: decode_layer_7_2/GatherV2 [GatherND]
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: Flatten__11:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:123: Searching for input: Unsqueeze__39:0
[10/03/2021-15:57:52] [V] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:129: decode_layer_7_2/GatherV2 [GatherND] inputs: [Flatten__11:0 -> (-1, -1)], [Unsqueeze__39:0 -> (-1, -1, -1)], 
[10/03/2021-15:57:52] [I] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/ModelImporter.cpp:139: No importer registered for op: GatherND. Attempting to import as plugin.
[10/03/2021-15:57:52] [I] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/builtin_op_importers.cpp:3775: Searching for plugin: GatherND, plugin_version: 1, plugin_namespace: 
[10/03/2021-15:57:52] --- Begin node ---
input: "Flatten__11:0"
input: "Unsqueeze__39:0"
output: "decode_layer_7"
name: "decode_layer_7_2/GatherV2"
op_type: "GatherND"
attribute {
  name: "batch_dims"
  i: 1
  type: INT
}
domain: ""

--- End node ---
[10/03/2021-15:57:52] [10/03/2021-15:57:52] [10/03/2021-15:57:52] [10/03/2021-15:57:52] &&&& FAILED TensorRT.trtexec # trtexec --onnx=/home/soccer_ball_tracker_poc/tests/debug/simple_tf_gather_cnn.onnx --verbose


[W] [TRT] /opt/tensorrt/TensorRT/parsers/onnx/onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[E] [TRT] INVALID_ARGUMENT: getPluginCreator could not find plugin GatherND version 1
While parsing node number 14 [GatherND -> "decode_layer_7"]:
ERROR: /opt/tensorrt/TensorRT/parsers/onnx/builtin_op_importers.cpp:3777 In function importFallbackPluginImporter:
[8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
[E] Failed to parse onnx file
[E] Parsing model failed
[E] Engine creation failed
[E] Engine set up failed

vs_code_notebook.py

#%%
import keras
from keras.layers import *
from keras.models import Model
import cv2
import tf2onnx
import sys
import onnx

import numpy as np
import tensorflow as tf
from pathlib import Path
import trt_inference
import matplotlib.pyplot as plt

#%%
DEBUG_DIR = "tests/debug"

#%%
class DecodeLayer(keras.layers.Layer):

    def call(self, inputs):
        b = tf.shape(inputs)[0]
        inputs_reshaped = tf.reshape(inputs, (b, -1))
        _, indices = tf.nn.top_k(inputs_reshaped, k=10)
        return tf.gather(inputs_reshaped, indices, batch_dims=1)

def get_model():
    inputs = Input(shape=(64,64,3))
    x = Conv2D(32, kernel_size=3, activation='relu')(inputs)
    x = MaxPooling2D(pool_size=(2, 2))(x)
    x = Conv2D(16, kernel_size=3, activation='relu')(x)
    x = MaxPooling2D(pool_size=(2, 2))(x)
    heatmap = Conv2D(1, kernel_size=3, padding='same', use_bias=False)(x)

    output = DecodeLayer()(heatmap)

    return Model(inputs=inputs, outputs=output)

#%%
model = get_model()
model.summary()

#%%
res = model.predict(np.zeros((1, 64, 64, 3), dtype=np.uint8))
res.shape

#%%
saved_model_path = Path(DEBUG_DIR) / "simple_tf_gather_cnn.h5"
saved_model_path.parent.mkdir(exist_ok=True, parents=True)
saved_model_path = str(saved_model_path)
model.save(saved_model_path)
print("saved to: ", saved_model_path)

#%%
model = keras.models.load_model(saved_model_path, 
                                custom_objects={"DecodeLayer": DecodeLayer})

#%%
input_layer = model.layers[0].input
input_name = input_layer.name.replace(":0", "")
input_shape = tuple(input_layer.shape.as_list())
# input_shape = (1, 288, 640, 5)


tf_spec_input_layer = (tf.TensorSpec(input_shape, input_layer.dtype, input_name),)
model_name = Path(saved_model_path).stem
output_dir = Path(saved_model_path).parent
onnx_model_path = output_dir / (model_name + ".onnx")

model_proto, _ = tf2onnx.convert.from_keras(model, 
                                            input_signature=tf_spec_input_layer, 
                                            opset=12, 
                                            output_path=onnx_model_path)
output_names = [n.name for n in model_proto.graph.output]
print(output_names)
#%%

model_onnx = onnx.load(onnx_model_path)
onnx.checker.check_model(model_onnx)