Skip to content

Instantly share code, notes, and snippets.

View JoshRosen's full-sized avatar

Josh Rosen JoshRosen

577 followers · 36 following
View GitHub Profile
@JoshRosen
JoshRosen / .gitignore
Created March 8, 2014 20:53
Importing RxJs in Typescript
Test.js
@JoshRosen
JoshRosen / failing.log
Created August 6, 2014 20:14
Logs for bug in https://github.com/apache/spark/pull/1758
14/08/06 13:07:00.529 INFO SecurityManager: Changing view acls to: joshrosen
14/08/06 13:07:00.530 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(joshrosen)
14/08/06 13:07:00.981 INFO ConnectionManager: Bound socket to port 58887 with id = ConnectionManagerId(joshs-mbp,58887)
14/08/06 13:07:01.080 DEBUG ConnectionManager: Before Sending [BufferMessage(id = 2, size = 10485760)] to [ConnectionManagerId(joshs-mbp,58887)] connectionid: joshs-mbp_58887_1
14/08/06 13:07:01.081 DEBUG ConnectionManager: Sending [BufferMessage(id = 2, size = 10485760)] to [ConnectionManagerId(joshs-mbp,58887)]
14/08/06 13:07:01.081 DEBUG SendingConnection: Added [BufferMessage(id = 2, size = 10485760)] to outbox for sending to [ConnectionManagerId(joshs-mbp,58887)]
14/08/06 13:07:01.082 DEBUG ConnectionManager: Selector selected 0 of 1 keys
14/08/06 13:07:01.084 INFO SendingConnection: Initiating connection to [joshs-mbp/192.168.1.245:58887]
14/08/06 13:07:01.087 INFO
@JoshRosen
JoshRosen / failing.log
Last active August 29, 2015 14:04
Logs for bug in https://github.com/apache/spark/pull/1758 while only running the single failing test.
14/08/06 13:38:22.898 INFO SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(joshrosen)
14/08/06 13:38:22.935 INFO ConnectionManager: Bound socket to port 59507 with id = ConnectionManagerId(joshs-mbp,59507)
14/08/06 13:38:22.945 INFO ConnectionManager: Bound socket to port 59508 with id = ConnectionManagerId(joshs-mbp,59508)
14/08/06 13:38:23.066 DEBUG SecurityMessage: message total size is : 42
14/08/06 13:38:23.067 DEBUG SendingConnection: Added [BufferMessage(id = 3, size = 42)] to outbox for sending to [ConnectionManagerId(joshs-mbp,59507)]
14/08/06 13:38:23.068 DEBUG ConnectionManager: adding connectionsAwaitingSasl id: joshs-mbp_59507_1
14/08/06 13:38:23.068 DEBUG ConnectionManager: Selector selected 0 of 1 keys
14/08/06 13:38:23.068 DEBUG ConnectionManager: Before Sending [BufferMessage(id = 2, size = 10485760)] to [ConnectionManagerId(joshs-mbp,59507)] connectionid: joshs-mbp_59507_1
14/08/06 13:38:23.069 DEBUG ConnectionManager: getAuthentic
@JoshRosen
JoshRosen / .gitignore
Last active August 29, 2015 14:05
Spark 1.1.0 release candidate benchmarking
results/
@JoshRosen
JoshRosen / install-mvn.sh
Last active August 29, 2015 14:05
Script for installing Maven on Amazon Linux
#!/usr/bin/env bash
# Script for installing Maven on Amazon Linux
MVN_VERSION=3.2.3
MVN_MD5=2fcfdb327eb94b8595d97ee4181ef0a6
# wget http://www.apache.org/dyn/closer.cgi/maven/maven-3/$MVN_VERSION/binaries/apache-maven-$MVN_VERSION-bin.tar.gz
wget http://apache.claz.org/maven/maven-3/$MVN_VERSION/binaries/apache-maven-$MVN_VERSION-bin.tar.gz
md5=`md5sum apache-maven-$MVN_VERSION-bin.tar.gz | awk '{ print $1 }'`
@JoshRosen
JoshRosen / gist:5d8df835516dc367389d
Created September 8, 2014 01:08
[error] * method getApplicationList()scala.collection.Seq in class org.apache.spark.deploy.history.HistoryServer has now a different result type; was: scala.collection.Seq, is now: scala.collection.Iterable
[error] filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.history.HistoryServer.getApplicationList")
[error] * method foreachAsync(org.apache.spark.api.java.function.VoidFunction)org.apache.spark.FutureAction in trait org.apache.spark.api.java.JavaRDDLike does not have a correspondent in old version
[error] filter with: ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.foreachAsync")
[error] * the type hierarchy of object org.apache.spark.scheduler.SparkListenerApplicationStart has changed in new version. Missing types {scala.runtime.AbstractFunction3}
[error] filter with: ProblemFilters.exclude[MissingTypesProblem]("org.apache.spark.scheduler.SparkListenerApplicationStart$")
[error] * method apply(java.lang.Object,jav
@JoshRosen
JoshRosen / HadoopThreadSafetyTest.scala
Last active August 29, 2015 14:07
import java.util.concurrent.{TimeUnit, ExecutorService, Executors}
import org.apache.hadoop.conf.Configuration
import scala.collection.JavaConversions._
import scala.util.Random
object HadoopThreadSafetyTest {
val operations: Seq[(Random, Configuration) => Any] = Seq(
@JoshRosen
JoshRosen / RDD.scala
Created October 13, 2014 19:38
import java.util.Comparator
trait RDDLike[T] {
def max(comp: Comparator[T]): T = {
(1.0).asInstanceOf[T]
}
}
class DoubleRDD extends RDDLike[java.lang.Double] {
// This space intentionally left blank.
@JoshRosen
JoshRosen / BitmapBenchmark.scala
Last active August 29, 2015 14:08
import java.util
import java.util.Random
import org.roaringbitmap.RoaringBitmap
object BitmapBenchmark extends scala.testing.Benchmark {
val rand = new Random(42)
def generateRandomData(numItems: Int, probEmpty: Double): Seq[Int] = {
val nums = for (
@JoshRosen
JoshRosen / README.md
Last active August 29, 2015 14:08
Scala 2.11.1 SerialVersionUID behavior change

Confusing Scala serialization puzzle

This writeup describes a tricky Scala serialization issue that we ran into while porting Spark to Scala 2.11.

First, let's create a factory that builds anonymous functions:

class FunctionFactory extends Serializable {
  // This method returns a function whose $outer scope is this class.
  // Note that the function does not reference any variables in this scope,