Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Beam portable word count fails on Spark with `pre_optimize=all` set
ERROR:root:java.lang.IllegalArgumentException: PCollectionNodes [PCollectionNode{id=ref_PCollection_PCollection_4, PCollection=unique_name: "57read/Read/Reshuffle/ReshufflePerKey/Map(reify_timestamps).None"
coder_id: "ref_Coder_TupleCoder_5_length_prefixed"
is_bounded: BOUNDED
windowing_strategy_id: "ref_Windowing_Windowing_2"
}, PCollectionNode{id=ref_PCollection_PCollection_24, PCollection=unique_name: "46write/Write/WriteImpl/WindowInto(WindowIntoFn).None"
coder_id: "ref_Coder_TupleCoder_4_length_prefixed"
is_bounded: BOUNDED
windowing_strategy_id: "ref_Windowing_Windowing_1"
}, PCollectionNode{id=ref_PCollection_PCollection_12, PCollection=unique_name: "13pair_with_one.None"
coder_id: "ref_Coder_TupleCoder_11"
is_bounded: BOUNDED
windowing_strategy_id: "ref_Windowing_Windowing_1"
}] were consumed but never produced
INFO:root:Job state changed to FAILED
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/google/home/kcweaver/go/src/github.com/apache/beam/sdks/python/apache_beam/examples/wordcount.py", line 135, in <module>
run()
File "/usr/local/google/home/kcweaver/go/src/github.com/apache/beam/sdks/python/apache_beam/examples/wordcount.py", line 115, in run
result.wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 436, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline BeamApp-kcweaver-0513181909-eb76aba3_3e8af5c5-f9db-4a67-8a35-f18a59bc5299 failed in state FAILED: java.lang.IllegalArgumentException: PCollectionNodes [PCollectionNode{id=ref_PCollection_PCollection_4, PCollection=unique_name: "57read/Read/Reshuffle/ReshufflePerKey/Map(reify_timestamps).None"
coder_id: "ref_Coder_TupleCoder_5_length_prefixed"
is_bounded: BOUNDED
windowing_strategy_id: "ref_Windowing_Windowing_2"
}, PCollectionNode{id=ref_PCollection_PCollection_24, PCollection=unique_name: "46write/Write/WriteImpl/WindowInto(WindowIntoFn).None"
coder_id: "ref_Coder_TupleCoder_4_length_prefixed"
is_bounded: BOUNDED
windowing_strategy_id: "ref_Windowing_Windowing_1"
}, PCollectionNode{id=ref_PCollection_PCollection_12, PCollection=unique_name: "13pair_with_one.None"
coder_id: "ref_Coder_TupleCoder_11"
is_bounded: BOUNDED
windowing_strategy_id: "ref_Windowing_Windowing_1"
}] were consumed but never produced
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.