Skip to content

Instantly share code, notes, and snippets.

View stephenh's full-sized avatar

Stephen Haberman stephenh

270 followers · 37 following
View GitHub Profile
@stephenh
stephenh / hadoop1.0.3.java
Created April 18, 2013 18:31
retries
public synchronized int read() throws IOException {
int result = in.read();
if (result != -1) {
pos++;
}
return result;
}
@stephenh
stephenh / stderr
Created March 26, 2013 21:20
registrar not found
This file has been truncated, but you can view the full file.
SLF4J: This version of SLF4J requires log4j version 1.2.12 or later. See also http://www.slf4j.org/codes.html#log4j_version
13/03/26 20:19:54 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
13/03/26 20:19:54 INFO actor.ActorSystemImpl: RemoteServerStarted@akka://spark@10.224.6.227:52400
13/03/26 20:19:54 INFO spark.SparkEnv: Registering BlockManagerMaster
13/03/26 20:19:54 INFO storage.MemoryStore: MemoryStore started with capacity 323.9 MB.
13/03/26 20:19:54 INFO storage.DiskStore: Created local directory at /mnt/spark/local/spark-local-20130326201954-696e
13/03/26 20:19:55 INFO network.ConnectionManager: Bound socket to port 41101 with id = ConnectionManagerId(ip-10-224-6-227.ec2.internal,41101)
13/03/26 20:19:55 INFO storage.BlockManagerMaster: Trying to register BlockManager
13/03/26 20:19:55 INFO storage.BlockManagerMaster: Registered BlockManager
13/03/26 20:19:55 INFO server.Server: jetty-7.5.3.v20111011
@stephenh
stephenh / gist:5156672
Created March 13, 2013 21:50
object foo {
var comparisons = 0 //> comparisons : Int = 0
class Foo {
override def equals(other: Any) = { comparisons += 1; super.equals(other) }
}
class Bar extends Foo
class Baz extends Foo
val s1: Set[Foo] = Seq.fill(10)(new Bar).toSet //> s1 : Set[foo.Foo] = Set(foo$$anonfun$main$1$Bar$1@1529d183, foo$$anonfun$ma
//| in$1$Bar$1@7d1f0c98, foo$$anonfun$main$1$Bar$1@1740f55, foo$$anonfun$main$1$
//| Bar$1@c8a3c71, foo$$anonfun$main$1$Bar$1@6a9a9631, foo$$anonfun$main$1$Bar$1
@stephenh
stephenh / gist:5108259
Created March 7, 2013 14:04
kryo snapshot stack trace
[info] BlockManagerSuite:
[info] - master + 2 managers interaction *** FAILED ***
[info] java.lang.IllegalArgumentException: Unable to create serializer "com.esotericsoftware.kryo.serializers.FieldSerializer" for class: scala.Tuple2$mcII$sp
[info] at com.esotericsoftware.kryo.Kryo.newSerializer(Kryo.java:347)
[info] at com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:326)
[info] at com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:319)
[info] at de.javakaffee.kryoserializers.KryoReflectionFactorySupport.getDefaultSerializer(KryoReflectionFactorySupport.java:45)
[info] at com.esotericsoftware.kryo.Kryo.register(Kryo.java:363)
[info] at spark.KryoSerializer$$anonfun$createKryo$1.apply(KryoSerializer.scala:140)
[info] at spark.KryoSerializer$$anonfun$createKryo$1.apply(KryoSerializer.scala:139)
@stephenh
stephenh / gist:5066221
Created March 1, 2013 17:20
Implementing class error when compiled with openjdk7
Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
@stephenh
stephenh / Testing.scala
Created February 27, 2013 18:43
Reproduce EOF Exception
import scala.collection.JavaConversions._
import java.util.Collections
import spark.KryoSerializer
import de.javakaffee.kryoserializers.KryoReflectionFactorySupport
import com.esotericsoftware.kryo.io.Output
import com.esotericsoftware.kryo.io.Input
object Testing {
def main(args: Array[String]): Unit = {
val list = new java.util.ArrayList[String]()
@stephenh
stephenh / gist:5045042
Created February 27, 2013 04:19
EOFException in broadcast
13/02/27 03:33:18 INFO cluster.TaskSetManager: Loss was due to java.io.EOFException
at spark.KryoDeserializationStream.readObject(KryoSerializer.scala:44)
at spark.broadcast.HttpBroadcast$.read(HttpBroadcast.scala:129)
at spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala:40)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
@stephenh
stephenh / command.sh
Created February 2, 2013 16:18
chatty akka
./sbt/sbt "test-only spark.DistributedSuite" > distributed.txt 2>&1
@stephenh
stephenh / output
Created February 1, 2013 22:32
distributed
Listening for transport dt_socket at address: 5005
[info] Loading project definition from /home/stephen/other/spark/project/project
[info] Loading project definition from /home/stephen/other/spark/project
[info] Set current project to root (in build file:/home/stephen/other/spark/)
[INFO] [02/01/2013 16:23:07.149] [spray-io-worker-2] [IoWorker] IoWorker thread 'spray-io-worker-2' stopped
[INFO] [02/01/2013 16:23:07.152] [spray-io-worker-1] [IoWorker] IoWorker thread 'spray-io-worker-1' stopped
[INFO] [02/01/2013 16:23:07.156] [spray-io-worker-0] [IoWorker] IoWorker thread 'spray-io-worker-0' stopped
[INFO] [02/01/2013 16:23:07.158] [spark-akka.actor.default-dispatcher-5] [ActorSystem(spark)] RemoteClientShutdown@akka://spark@192.168.99.93:53127
[INFO] [02/01/2013 16:23:07.158] [spark-akka.actor.default-dispatcher-1] [ActorSystem(spark)] RemoteClientShutdown@akka://spark@192.168.99.93:53127
[INFO] [02/01/2013 16:23:07.161] [spark-3] [ActorSystem(spark
@stephenh
stephenh / sbt sucks
Created February 1, 2013 22:11
> ./sbt/sbt test-only spark.DistributedSuite
Listening for transport dt_socket at address: 5005
[info] Loading project definition from /home/stephen/other/spark/project/project
[info] Loading project definition from /home/stephen/other/spark/project
[info] Set current project to root (in build file:/home/stephen/other/spark/)
[INFO] [02/01/2013 16:09:52.354] [spark-akka.actor.default-dispatcher-2] [ActorSystem(spark)] RemoteServerShutdown@akka://spark@192.168.99.93:57697
[INFO] [02/01/2013 16:09:52.617] [spark-akka.actor.default-dispatcher-4] [ActorSystem(spark)] RemoteServerShutdown@akka://spark@192.168.99.93:52063
[INFO] [02/01/2013 16:09:52.853] [spark-akka.actor.default-dispatcher-1] [ActorSystem(spark)] RemoteServerShutdown@akka://spark@192.168.99.93:50396
[INFO] [02/01/2013 16:09:53.72] [spark-akka.actor.default-dispatcher-3] [ActorSystem(spark)] RemoteServerShutdown@akka://spark@192.168.99.93:56448
[INFO] [02/01/2013 16:09:53.284] [spark-akka.actor.default-dispatcher-1] [ActorSystem(spark)] RemoteServe