Last active
December 17, 2015 15:41
-
-
Save PanAeon/c7d7bbd4dfd7e1c7afb9 to your computer and use it in GitHub Desktop.
Serialization Error
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
val rdd = sc.parallelize(Stream.from(1).take(1000).toList,4) | |
class NotSerializable() | |
val bar = new NotSerializable() | |
val simpleVariable = Set(1, 2, 3, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 19) | |
// here's the problem, this code runs forever: | |
rdd | |
.filter(x => simpleVariable(x)) // translates to this.simpleVariable | |
.collect() | |
// btw, and this works, fine because of parameter capture: | |
rdd | |
.filter(simpleVariable) | |
.collect() | |
/* =========== here's a workaround: ============================= | |
object Foo1 { | |
val simpleVariable = Set(1, 2, 3, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 19) | |
} | |
import Foo1._ | |
rdd | |
.filter(x => simpleVariable(x)) | |
.collect() | |
*/ | |
// But for a newbie scala user it would be quite difficult to figure out this solution!!!! | |
// so my question is - why not we wrap each line in an object automatically, as in a workaround? | |
// If the answer is positive I will create a new ticket and will try to fix this and submit PR. | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rdd | |
.filter(x => simpleVariable(x)) // translates to $line11.$read$$iw.......$iw.simpleVariable, works fine | |
.collect() |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
yeah, I already see a problem with defining companion objects for example, and out of order definitions, maybe we'll need a special cell type for them, which will complicate things, but I hope that we will avoid a greater evil