Skip to content

Instantly share code, notes, and snippets.

@erasmas

erasmas/api_test.clj

Created August 25, 2015 07:40
Show Gist options
  • Save erasmas/b4bda894115c5a0d6eac to your computer and use it in GitHub Desktop.
Save erasmas/b4bda894115c5a0d6eac to your computer and use it in GitHub Desktop.
Download ZIP
Raw
api_test.clj
(testing
"take-ordered returns the first N elements of an RDD using the natural ordering"
(is (= (-> (s/parallelize c [[1 -1] [2 -2] [3 -3] [4 -4]])
(s/take-ordered 1))
[[1 -1]])))
(testing
"take-ordered returns the first N elements of an RDD as defined by the specified comparator"
(is (= (-> (s/parallelize c [[1 -1] [2 -2] [3 -3] [4 -4]])
(s/take-ordered 1 (comparator (fn [[_ y1] [_ y2]] (< y1 y2)))))
[[4 -4]])))
Raw
test.log
lein test sparkling.accumulator-test
2015-08-25 10:40:21,488 WARN util.NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
lein test sparkling.api-test
2015-08-25 10:40:32,311 ERROR executor.Executor:96 - Exception in task 0.0 in stage 19.0 (TID 67)
com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.spark-project.guava.collect.NaturalOrdering
Note: To register this class use: kryo.register(org.spark-project.guava.collect.NaturalOrdering.class);
Serialization trace:
cmp$2 (scala.math.LowPriorityOrderingImplicits$$anon$7)
$outer (scala.math.Ordering$$anon$4)
ord (org.apache.spark.util.BoundedPriorityQueue)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:250)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:236)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Class is not registered: org.spark-project.guava.collect.NaturalOrdering
Note: To register this class use: kryo.register(org.spark-project.guava.collect.NaturalOrdering.class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:561)
... 16 more
2015-08-25 10:40:32,312 ERROR executor.Executor:96 - Exception in task 2.0 in stage 19.0 (TID 69)
com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.spark-project.guava.collect.NaturalOrdering
Note: To register this class use: kryo.register(org.spark-project.guava.collect.NaturalOrdering.class);
Serialization trace:
cmp$2 (scala.math.LowPriorityOrderingImplicits$$anon$7)
$outer (scala.math.Ordering$$anon$4)
ord (org.apache.spark.util.BoundedPriorityQueue)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:250)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:236)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Class is not registered: org.spark-project.guava.collect.NaturalOrdering
Note: To register this class use: kryo.register(org.spark-project.guava.collect.NaturalOrdering.class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:561)
... 16 more
2015-08-25 10:40:32,312 ERROR executor.Executor:96 - Exception in task 3.0 in stage 19.0 (TID 70)
com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.spark-project.guava.collect.NaturalOrdering
Note: To register this class use: kryo.register(org.spark-project.guava.collect.NaturalOrdering.class);
Serialization trace:
cmp$2 (scala.math.LowPriorityOrderingImplicits$$anon$7)
$outer (scala.math.Ordering$$anon$4)
ord (org.apache.spark.util.BoundedPriorityQueue)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:250)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:236)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Class is not registered: org.spark-project.guava.collect.NaturalOrdering
Note: To register this class use: kryo.register(org.spark-project.guava.collect.NaturalOrdering.class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:561)
... 16 more
2015-08-25 10:40:32,312 ERROR executor.Executor:96 - Exception in task 1.0 in stage 19.0 (TID 68)
com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.spark-project.guava.collect.NaturalOrdering
Note: To register this class use: kryo.register(org.spark-project.guava.collect.NaturalOrdering.class);
Serialization trace:
cmp$2 (scala.math.LowPriorityOrderingImplicits$$anon$7)
$outer (scala.math.Ordering$$anon$4)
ord (org.apache.spark.util.BoundedPriorityQueue)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:250)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:236)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Class is not registered: org.spark-project.guava.collect.NaturalOrdering
Note: To register this class use: kryo.register(org.spark-project.guava.collect.NaturalOrdering.class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:561)
... 16 more
2015-08-25 10:40:32,324 WARN scheduler.TaskSetManager:71 - Lost task 2.0 in stage 19.0 (TID 69, localhost): com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.spark-project.guava.collect.NaturalOrdering
Note: To register this class use: kryo.register(org.spark-project.guava.collect.NaturalOrdering.class);
Serialization trace:
cmp$2 (scala.math.LowPriorityOrderingImplicits$$anon$7)
$outer (scala.math.Ordering$$anon$4)
ord (org.apache.spark.util.BoundedPriorityQueue)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:250)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:236)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Class is not registered: org.spark-project.guava.collect.NaturalOrdering
Note: To register this class use: kryo.register(org.spark-project.guava.collect.NaturalOrdering.class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:561)
... 16 more
2015-08-25 10:40:32,326 ERROR scheduler.TaskSetManager:75 - Task 2 in stage 19.0 failed 1 times; aborting job
lein test :only sparkling.api-test/actions
ERROR in (actions) (DAGScheduler.scala:1266)
take-ordered returns the first N elements of an RDD using the natural ordering
expected: (= (-> (s/parallelize c [[1 -1] [2 -2] [3 -3] [4 -4]]) (s/take-ordered 1)) [[1 -1]])
2015-08-25 10:40:32,429 ERROR executor.Executor:96 - Exception in task 3.0 in stage 20.0 (TID 74)
com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: clojure.core$comparator$fn__4661
Note: To register this class use: kryo.register(clojure.core$comparator$fn__4661.class);
Serialization trace:
cmp$2 (scala.math.LowPriorityOrderingImplicits$$anon$7)
$outer (scala.math.Ordering$$anon$4)
ord (org.apache.spark.util.BoundedPriorityQueue)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:250)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:236)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Class is not registered: clojure.core$comparator$fn__4661
Note: To register this class use: kryo.register(clojure.core$comparator$fn__4661.class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:561)
... 16 more
2015-08-25 10:40:32,432 WARN scheduler.TaskSetManager:71 - Lost task 3.0 in stage 20.0 (TID 74, localhost): com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: clojure.core$comparator$fn__4661
Note: To register this class use: kryo.register(clojure.core$comparator$fn__4661.class);
Serialization trace:
cmp$2 (scala.math.LowPriorityOrderingImplicits$$anon$7)
$outer (scala.math.Ordering$$anon$4)
ord (org.apache.spark.util.BoundedPriorityQueue)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:250)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:236)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Class is not registered: clojure.core$comparator$fn__4661
Note: To register this class use: kryo.register(clojure.core$comparator$fn__4661.class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:561)
... 16 more
2015-08-25 10:40:32,432 ERROR scheduler.TaskSetManager:75 - Task 3 in stage 20.0 failed 1 times; aborting job
2015-08-25 10:40:32,433 ERROR executor.Executor:96 - Exception in task 0.0 in stage 20.0 (TID 71)
com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: clojure.core$comparator$fn__4661
Note: To register this class use: kryo.register(clojure.core$comparator$fn__4661.class);
Serialization trace:
cmp$2 (scala.math.LowPriorityOrderingImplicits$$anon$7)
$outer (scala.math.Ordering$$anon$4)
ord (org.apache.spark.util.BoundedPriorityQueue)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:250)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:236)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Class is not registered: clojure.core$comparator$fn__4661
Note: To register this class use: kryo.register(clojure.core$comparator$fn__4661.class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:561)
... 16 more
2015-08-25 10:40:32,435 ERROR executor.Executor:96 - Exception in task 2.0 in stage 20.0 (TID 73)
com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: clojure.core$comparator$fn__4661
Note: To register this class use: kryo.register(clojure.core$comparator$fn__4661.class);
Serialization trace:
cmp$2 (scala.math.LowPriorityOrderingImplicits$$anon$7)
$outer (scala.math.Ordering$$anon$4)
ord (org.apache.spark.util.BoundedPriorityQueue)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:250)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:236)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Class is not registered: clojure.core$comparator$fn__4661
Note: To register this class use: kryo.register(clojure.core$comparator$fn__4661.class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:561)
... 16 more
2015-08-25 10:40:32,436 ERROR executor.Executor:96 - Exception in task 1.0 in stage 20.0 (TID 72)
com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: clojure.core$comparator$fn__4661
Note: To register this class use: kryo.register(clojure.core$comparator$fn__4661.class);
Serialization trace:
cmp$2 (scala.math.LowPriorityOrderingImplicits$$anon$7)
$outer (scala.math.Ordering$$anon$4)
ord (org.apache.spark.util.BoundedPriorityQueue)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:250)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:236)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Class is not registered: clojure.core$comparator$fn__4661
Note: To register this class use: kryo.register(clojure.core$comparator$fn__4661.class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:561)
... 16 more
actual: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 19.0 failed 1 times, most recent failure: Lost task 2.0 in stage 19.0 (TID 69, localhost): com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.spark-project.guava.collect.NaturalOrdering
Note: To register this class use: kryo.register(org.spark-project.guava.collect.NaturalOrdering.class);
Serialization trace:
cmp$2 (scala.math.LowPriorityOrderingImplicits$$anon$7)
$outer (scala.math.Ordering$$anon$4)
ord (org.apache.spark.util.BoundedPriorityQueue)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:250)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:236)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Class is not registered: org.spark-project.guava.collect.NaturalOrdering
Note: To register this class use: kryo.register(org.spark-project.guava.collect.NaturalOrdering.class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:561)
... 16 more
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages (DAGScheduler.scala:1266)
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply (DAGScheduler.scala:1257)
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply (DAGScheduler.scala:1256)
scala.collection.mutable.ResizableArray$class.foreach (ResizableArray.scala:59)
scala.collection.mutable.ArrayBuffer.foreach (ArrayBuffer.scala:47)
org.apache.spark.scheduler.DAGScheduler.abortStage (DAGScheduler.scala:1256)
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply (DAGScheduler.scala:730)
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply (DAGScheduler.scala:730)
scala.Option.foreach (Option.scala:236)
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed (DAGScheduler.scala:730)
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive (DAGScheduler.scala:1450)
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive (DAGScheduler.scala:1411)
org.apache.spark.util.EventLoop$$anon$1.run (EventLoop.scala:48)
lein test :only sparkling.api-test/actions
ERROR in (actions) (DAGScheduler.scala:1266)
take-ordered returns the first N elements of an RDD as defined by the specified comparator
expected: (= (-> (s/parallelize c [[1 -1] [2 -2] [3 -3] [4 -4]]) (s/take-ordered 1 (comparator (fn [[_ y1] [_ y2]] (< y1 y2))))) [[4 -4]])
actual: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 20.0 failed 1 times, most recent failure: Lost task 3.0 in stage 20.0 (TID 74, localhost): com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: clojure.core$comparator$fn__4661
Note: To register this class use: kryo.register(clojure.core$comparator$fn__4661.class);
Serialization trace:
cmp$2 (scala.math.LowPriorityOrderingImplicits$$anon$7)
$outer (scala.math.Ordering$$anon$4)
ord (org.apache.spark.util.BoundedPriorityQueue)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21)
at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:250)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:236)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Class is not registered: clojure.core$comparator$fn__4661
Note: To register this class use: kryo.register(clojure.core$comparator$fn__4661.class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:561)
... 16 more
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages (DAGScheduler.scala:1266)
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply (DAGScheduler.scala:1257)
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply (DAGScheduler.scala:1256)
scala.collection.mutable.ResizableArray$class.foreach (ResizableArray.scala:59)
scala.collection.mutable.ArrayBuffer.foreach (ArrayBuffer.scala:47)
org.apache.spark.scheduler.DAGScheduler.abortStage (DAGScheduler.scala:1256)
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply (DAGScheduler.scala:730)
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply (DAGScheduler.scala:730)
scala.Option.foreach (Option.scala:236)
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed (DAGScheduler.scala:730)
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive (DAGScheduler.scala:1450)
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive (DAGScheduler.scala:1411)
org.apache.spark.util.EventLoop$$anon$1.run (EventLoop.scala:48)
lein test sparkling.broadcast-test
lein test sparkling.conf-test
lein test sparkling.core-test
lein test sparkling.function-test
lein test sparkling.kryoserializer
lein test sparkling.protocol-test
lein test sparkling.rdd.hadoopAvro-test
lein test sparkling.rdd.jdbc-test
lein test sparkling.serialization-test
#object[com.esotericsoftware.kryo.Registration 0x32c6d164 [105, sparkling.scalaInterop.ScalaFunction0]]
Writing #object[sparkling.scalaInterop.ScalaFunction0 0x645c9f0f <function0>]
Reading clazz
lein test sparkling.test-registrator
lein test sparkling.testutils
lein test sparkling.testutils.records.domain
lein test sparkling.testutils.records.registrator
Ran 19 tests containing 86 assertions.
0 failures, 2 errors.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment