get_cached_outputs issue

gistfile1.txt

	2021-08-02 18:53:06.794504: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.0
INFO:absl:Running driver for BigQueryExampleGen
INFO:absl:MetadataStore with gRPC connection initialized
INFO:absl:Adding KFP pod name weeklyrunnerdemo274lzs-820-3701849744-2645169506 to execution
Traceback (most recent call last):
  File "/opt/ee/python/3.7/lib/python3.7/site-packages/ml_metadata/metadata_store/metadata_store.py", line 202, in _call_method
    response.CopyFrom(grpc_method(request, timeout=self._grpc_timeout_sec))
  File "/opt/ee/python/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/opt/ee/python/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
  status = StatusCode.RESOURCE_EXHAUSTED
  details = "Received message larger than max (5112237 vs. 4194304)"
  debug_error_string = "{"created":"@1627930404.485900067","description":"Received message larger than max (5112237 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":203,"grpc_status":8}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/ee/python/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/ee/python/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/ee/python/3.7/lib/python3.7/site-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 360, in <module>
    main()
  File "/opt/ee/python/3.7/lib/python3.7/site-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 353, in main
    execution_info = launcher.launch()
  File "/opt/ee/python/3.7/lib/python3.7/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 198, in launch
    self._exec_properties)
  File "/opt/ee/python/3.7/lib/python3.7/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 167, in _run_driver
    component_info=self._component_info)
  File "/opt/ee/python/3.7/lib/python3.7/site-packages/tfx/dsl/components/base/base_driver.py", line 300, in pre_execution
    component_info=component_info)
  File "/opt/ee/python/3.7/lib/python3.7/site-packages/tfx/orchestration/metadata.py", line 824, in get_cached_outputs
    (e.id, e) for e in self._store.get_executions_by_context(context.id))
  File "/opt/ee/python/3.7/lib/python3.7/site-packages/ml_metadata/metadata_store/metadata_store.py", line 1263, in get_executions_by_context
    self._call('GetExecutionsByContext', request, response)
  File "/opt/ee/python/3.7/lib/python3.7/site-packages/ml_metadata/metadata_store/metadata_store.py", line 177, in _call
    return self._call_method(method_name, request, response)
  File "/opt/ee/python/3.7/lib/python3.7/site-packages/ml_metadata/metadata_store/metadata_store.py", line 207, in _call_method
    raise _make_exception(e.details(), e.code().value[0])  # pytype: disable=attribute-error
ml_metadata.errors.ResourceExhaustedError: Received message larger than max (5112237 vs. 4194304)