Skip to content

Instantly share code, notes, and snippets.

{
"id": "/spark-streaming-labels-dev",
"cmd": "/usr/local/spark-2.1.1/bin/spark-submit --class com.koddi.etl.spark.streaming.LabelStream --master mesos://leader.mesos:5050 --conf spark.ssl.noCertVerification=true --conf spark.mesos.executor.docker.image=koddidev/spark:mesos-1.0.1 --conf spark.executor.home=/usr/local/spark-2.1.1 --executor-memory 2g /mnt/mesos/sandbox/spark-streaming.jar dev",
"cpus": 1,
"mem": 1024,
"disk": 1024,
"instances": 0,
"container": {
"type": "DOCKER",
"volumes": [],
@kyrozetera
kyrozetera / spark-jobserver-create-context-log
Created January 31, 2017 17:07
Log of Spark JobServer spinning up a context on marathon
[2017-01-31 16:59:42,330] INFO ocalContextSupervisorActor [] [akka://JobServer/user/context-supervisor] - Creating a SparkContext named test-context
[2017-01-31 16:59:42,336] INFO .jobserver.JobManagerActor [] [akka://JobServer/user/context-supervisor/test-context] - Starting actor spark.jobserver.JobManagerActor
[2017-01-31 16:59:42,382] INFO k.jobserver.JobStatusActor [] [akka://JobServer/user/context-supervisor/test-context/$a] - Starting actor spark.jobserver.JobStatusActor
[2017-01-31 16:59:42,382] INFO k.jobserver.JobResultActor [] [akka://JobServer/user/context-supervisor/test-context/$b] - Starting actor spark.jobserver.JobResultActor
[2017-01-31 16:59:42,441] WARN org.apache.spark.SparkConf [] [akka://JobServer/user/context-supervisor/test-context] - The configuration key spark.akka.threads is not supported any more because Spark doesn't use Akka since 2.0
[2017-01-31 16:59:42,483] INFO parkContextFactory$$anon$1 [] [akka://JobServer/user/context-supervisor/test-context] - Running Spark version
@kyrozetera
kyrozetera / spark-jobserver-marathon.json
Last active January 30, 2017 23:14
Spark JobServer Marathon config
{
"id": "/spark-jobserver",
"cmd": null,
"cpus": 0.5,
"mem": 1024,
"disk": 100,
"instances": 1,
"container": {
"type": "DOCKER",
"volumes": [],
@kyrozetera
kyrozetera / gist:c163a9b83ef840bf692de26ab5967d9e
Created November 4, 2016 15:27
ChannelException with Alluxio on Lambda using EPOLL channel
io.netty.channel.ChannelException: Unable to create Channel from class class io.netty.channel.epoll.EpollSocketChannel: java.io.IOException
java.io.IOException: io.netty.channel.ChannelException: Unable to create Channel from class class io.netty.channel.epoll.EpollSocketChannel
at alluxio.client.netty.NettyUnderFileSystemFileWriter.write(NettyUnderFileSystemFileWriter.java:103)
at alluxio.client.file.UnderFileSystemFileOutStream.writeToWorker(UnderFileSystemFileOutStream.java:161)
at alluxio.client.file.UnderFileSystemFileOutStream.flush(UnderFileSystemFileOutStream.java:85)
at alluxio.client.file.UnderFileSystemFileOutStream.close(UnderFileSystemFileOutStream.java:75)
at alluxio.client.file.FileOutStream.close(FileOutStream.java:162)
at com.koddi.etl.lambda.SparkLauncher.writeFileToAlluxio(SparkLauncher.scala:127)
at com.koddi.etl.lambda.SparkLauncher$$anonfun$handleRequest$1.apply(SparkLauncher.scala:55)
at com.koddi.etl.lambda.SparkLauncher$$anonfun$handleRequest$1.apply(SparkLauncher.scala:46)
@kyrozetera
kyrozetera / fun-arithmetic.php
Created July 6, 2016 21:47
Floating Point Fun
$thing = 50 * (1.1);
var_dump($thing);
var_dump(55 == $thing);
/* Output:
* float(55)
* bool(false)
*/
$thing = 50 + (50 * (.1));