Skip to content

Instantly share code, notes, and snippets.

@takezoux2
Created April 10, 2020 05:01
Show Gist options
  • Save takezoux2/e7be1471710f719a76ddfe63293da0e3 to your computer and use it in GitHub Desktop.
Save takezoux2/e7be1471710f719a76ddfe63293da0e3 to your computer and use it in GitHub Desktop.
Dataflow Log for sample program
2020-04-10 13:40:22.901 JSTExecuting operation saveAsTextFile@{WordCount.scala:25}:1/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create
2020-04-10 13:40:22.939 JSTExecuting operation saveAsTextFile@{WordCount.scala:25}:1/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
2020-04-10 13:40:22.941 JSTStarting worker pool setup.
2020-04-10 13:40:22.975 JSTStarting 1 workers in us-central1-c...
2020-04-10 13:40:22.975 JSTExecuting operation countByValue@{WordCount.scala:23}:1/pApply:1/Combine.perKey(Count)/GroupByKey/Create
2020-04-10 13:40:23.025 JSTFinished operation saveAsTextFile@{WordCount.scala:25}:1/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create
2020-04-10 13:40:23.041 JSTFinished operation countByValue@{WordCount.scala:23}:1/pApply:1/Combine.perKey(Count)/GroupByKey/Create
2020-04-10 13:40:23.041 JSTFinished operation saveAsTextFile@{WordCount.scala:25}:1/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
2020-04-10 13:40:23.105 JSTValue "saveAsTextFile@{WordCount.scala:25}:1/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Session" materialized.
2020-04-10 13:40:23.141 JSTValue "countByValue@{WordCount.scala:23}:1/pApply:1/Combine.perKey(Count)/GroupByKey/Session" materialized.
2020-04-10 13:40:23.172 JSTValue "saveAsTextFile@{WordCount.scala:25}:1/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Session" materialized.
2020-04-10 13:40:23.206 JSTExecuting operation textFile@{WordCount.scala:20}:1/Read+map@{WordCount.scala:21}:2+flatMap@{WordCount.scala:22}:2+countByValue@{WordCount.scala:23}:1/pApply:1/Init/Map+countByValue@{WordCount.scala:23}:1/pApply:1/Combine.perKey(Count)/GroupByKey+countByValue@{WordCount.scala:23}:1/pApply:1/Combine.perKey(Count)/Combine.GroupedValues/Partial+countByValue@{WordCount.scala:23}:1/pApply:1/Combine.perKey(Count)/GroupByKey/Reify+countByValue@{WordCount.scala:23}:1/pApply:1/Combine.perKey(Count)/GroupByKey/Write
2020-04-10 13:40:49.902 JSTAutoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s).
2020-04-10 13:41:18.249 JSTWorkers have started successfully.
2020-04-10 13:41:18.282 JSTWorkers have started successfully.
2020-04-10 13:46:22.805 JSTChecking permissions granted to controller Service Account.
2020-04-10 13:52:22.818 JSTChecking permissions granted to controller Service Account.
2020-04-10 13:58:22.814 JSTChecking permissions granted to controller Service Account.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment