Skip to content

Instantly share code, notes, and snippets.

@rtyler
Created May 30, 2019 14:37
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rtyler/aa37404b0beb399b70c90804e7674ff9 to your computer and use it in GitHub Desktop.
Save rtyler/aa37404b0beb399b70c90804e7674ff9 to your computer and use it in GitHub Desktop.
Caused by: [22/802]
java.io.NotSerializableException: org.jruby.javasupport.Java$InterfaceProxyHandler
Serialization stack:
- object not serializable (class: org.jruby.javasupport.Java$InterfaceProxyHandler, value: org.jruby.javasupport.Java$InterfaceProxyHandler@36d7e8d2)
- field (class: java.lang.reflect.Proxy, name: h, type: interface java.lang.reflect.InvocationHandler)
- object (class com.sun.proxy.$Proxy44, com.sun.proxy.$Proxy44{org.jruby.javasupport.Java$InterfaceProxyHandler@36d7e8d2})
- field (class: org.apache.spark.sql.Dataset$$anonfun$foreach$2, name: func$5, type: interface org.apache.spark.api.java.function.ForeachFunction)
- object (class org.apache.spark.sql.Dataset$$anonfun$foreach$2, <function1>)
improveException at org/apache/spark/serializer/SerializationDebugger.scala:40
writeObject at org/apache/spark/serializer/JavaSerializer.scala:46
serialize at org/apache/spark/serializer/JavaSerializer.scala:100
ensureSerializable at org/apache/spark/util/ClosureCleaner.scala:400
org$apache$spark$util$ClosureCleaner$$clean at org/apache/spark/util/ClosureCleaner.scala:393
clean at org/apache/spark/util/ClosureCleaner.scala:162
clean at org/apache/spark/SparkContext.scala:2326
apply at org/apache/spark/rdd/RDD.scala:926
apply at org/apache/spark/rdd/RDD.scala:925
withScope at org/apache/spark/rdd/RDDOperationScope.scala:151
withScope at org/apache/spark/rdd/RDDOperationScope.scala:112
withScope at org/apache/spark/rdd/RDD.scala:363
foreach at org/apache/spark/rdd/RDD.scala:925
apply$mcV$sp at org/apache/spark/sql/Dataset.scala:2716
apply at org/apache/spark/sql/Dataset.scala:2716
apply at org/apache/spark/sql/Dataset.scala:2716
apply at org/apache/spark/sql/Dataset.scala:3349
apply at org/apache/spark/sql/execution/SQLExecution.scala:78
withSQLConfPropagated at org/apache/spark/sql/execution/SQLExecution.scala:125
withNewExecutionId at org/apache/spark/sql/execution/SQLExecution.scala:73
withNewRDDExecutionId at org/apache/spark/sql/Dataset.scala:3345
foreach at org/apache/spark/sql/Dataset.scala:2715
foreach at org/apache/spark/sql/Dataset.scala:2726
invoke0 at sun/reflect/NativeMethodAccessorImpl.java:-2
invoke at sun/reflect/NativeMethodAccessorImpl.java:62
invoke at sun/reflect/DelegatingMethodAccessorImpl.java:43
invoke at java/lang/reflect/Method.java:498
invokeDirectWithExceptionHandling at org/jruby/javasupport/JavaMethod.java:455
invokeDirect at org/jruby/javasupport/JavaMethod.java:316
call at org/jruby/java/invokers/InstanceMethodInvoker.java:46
cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:339
call at org/jruby/runtime/callsite/CachingCallSite.java:170
invokeOther49:foreach at simple.rb:42
<main> at simple.rb:42
invokeWithArguments at java/lang/invoke/MethodHandle.java:627
load at org/jruby/ir/Compiler.java:94
runScript at org/jruby/Ruby.java:856
runNormally at org/jruby/Ruby.java:779
runNormally at org/jruby/Ruby.java:797
runFromMain at org/jruby/Ruby.java:609
doRunFromMain at org/jruby/Main.java:415
internalRun at org/jruby/Main.java:307
run at org/jruby/Main.java:234
main at org/jruby/Main.java:206
invoke0 at sun/reflect/NativeMethodAccessorImpl.java:-2
invoke at sun/reflect/NativeMethodAccessorImpl.java:62
invoke at sun/reflect/DelegatingMethodAccessorImpl.java:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment