Let's say you have a Scala function which takes type parameter:
def myFunc[K]: T
Let's say I have several functions like that. Right now, if K could be one of several different values, I'd need some code like the following:
kType match {
package spark.jobserver.api | |
import com.typesafe.config.Config | |
import org.scalactic._ | |
import spark.jobserver.api._ | |
trait ContextProvider[C] { | |
val ctx: C with ContextLike = null | |
def context: C with ContextLike = ctx | |
} |
:: problems summary :: | |
:::: WARNINGS | |
[FAILED ] org.scala-sbt#main;0.13.11!main.jar: The HTTP response code for https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/main/0.13.11/jars/main.jar did not indicate a success. See log for more detail. (323ms) | |
[FAILED ] org.scala-sbt#main;0.13.11!main.jar: The HTTP response code for https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/main/0.13.11/jars/main.jar did not indicate a success. See log for more detail. (323ms) | |
==== typesafe-ivy-releases: tried | |
https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/main/0.13.11/jars/main.jar | |
[FAILED ] org.scala-sbt#compiler-interface;0.13.11!compiler-interface.jar: The HTTP response code for https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/compiler-interface/0.13.11/jars/compiler-interface.jar did not indicate a success. See log for more detail. (308ms) | |
[FAILED ] org.scala-sbt#compiler-interface;0.13.11!compiler-interface.jar: The HTTP response code for https: |
* See all current contexts | |
* For each context see all the jobs that are running | |
* History of past jobs for a given context. Actually this is not technically available via API; but history of past jobs | |
* Be able to see configuration for each job present or past (this is available via a tiny "C" link in current UI) | |
* Job results | |
* Link to Spark UI for detailed analysis |
Let's say you have a Scala function which takes type parameter:
def myFunc[K]: T
Let's say I have several functions like that. Right now, if K could be one of several different values, I'd need some code like the following:
kType match {
// Primitive.scala - only change is asCql method | |
implicit object BlobIsPrimitive extends Primitive[ByteBuffer] { | |
override type PrimitiveType = java.nio.ByteBuffer | |
val cassandraType = CQLSyntax.Types.Blob | |
override def fromRow(column: String, row: Row): Try[ByteBuffer] = nullCheck(column, row) { | |
r => r.getBytes(column) |
...to be turned into a blog post later. These are notes with references to commits, the blog post will have snippets of code so folks don't have to look things up.
Filo is an extreme serialization library for vector data. Think of it as the good parts of Parquet without the HDFS and file format garbage -- just the serdes and fast columnar storage.
I recently added a JMH benchmark for reading a Filo binary buffer containing 10,000 Ints using the simplest apply() method to sum up all the Ints.
Oh, and before we get started - avoid throwing exceptions in inner loops, especially Try(....).getOrElse(...)
patterns. Even if they occur only occasionally they can be extremely expensive.
import com.vividsolutions.jts.geom._ | |
import com.vividsolutions.jts.geom.impl.PackedCoordinateSequence | |
import com.vividsolutions.jts.geom.util.GeometryTransformer | |
/** | |
* A custom CoordSequence based on byte arrays for compactness and speed | |
* Just 2 dimensions for now. | |
* | |
* It's an example of creating a custom CoordinateSequence. | |
* NOTE: This is much more memory efficient, but slower because of deserialization cost. |
case StopContext(name) => | |
if (contexts contains name) { | |
logger.info("Shutting down context {}", name) | |
context.watch(contexts(name)) // watch for termination event | |
contexts(name) ! PoisonPill | |
contexts.remove(name) | |
resultActors.remove(name) | |
sender ! ContextStopped | |
} else { | |
sender ! NoSuchContext |
// We want a generic way to chunk large arrays into segments of byte arrays | |
// and to do so in a way that doesn't blow up memory for large objects, ie ability to chunk / page / stream | |
trait ChunkingArraySerDe[T] { | |
def apply(data: Array[T], chunkSize: Int): Iterator[Array[Byte]] | |
def unapply(serialized: Iterator[Array[Byte]]): Array[T] | |
} | |
// For arrays with fixed-size elements | |
trait PrimitiveChunkingSerDe[@specialized(Int, Long, Boolean) T] extends ChunkingArraySerDe[T] { |
So, there have been some discussions and angry tweets recently about irritating Scala "features" (like value discarding and auto-tupling) that can actually be turned off by selecting the right compiler flag in conjunction with -Xfatal-warnings
. I highly recommend a set of options something like those below.
scalacOptions ++= Seq(
"-deprecation",
"-encoding", "UTF-8", // yes, this is 2 args
"-feature",
"-language:existentials",