IgorBerman

package com.example;

import java.util.concurrent.CompletionStage;
import java.util.concurrent.TimeUnit;

import scala.runtime.BoxedUnit;
import scala.util.Try;
import akka.Done;
import akka.NotUsed;
import akka.actor.ActorSystem;

IgorBerman

1. General Background and Overview

• Probabilistic Data Structures for Web Analytics and Data Mining : A great overview of the space of probabilistic data structures and how they are used in approximation algorithm implementation.
• Models and Issues in Data Stream Systems
• Philippe Flajolet’s contribution to streaming algorithms : A presentation by Jérémie Lumbroso that visits some of the hostorical perspectives and how it all began with Flajolet
• Approximate Frequency Counts over Data Streams by Gurmeet Singh Manku & Rajeev Motwani : One of the early papers on the subject.
• [Methods for Finding Frequent Items in Data Streams](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.9800&rep

IgorBerman

//without backpressure:
ActorRef myActor = system.actorOf(
				new akka.routing.ConsistentHashingPool(maxParallelism)
				.withHashMapper(hashMapper)
				.props(MyActor.props(blLogic).withDispatcher("my-dispatcher"))
				, "my-ordered-processor-pool");
		
final Timeout timeout = Timeout.apply(1000L, TimeUnit.MILLISECONDS);
		
Flow.of(Event.class)

IgorBerman

1. Setup your process java properties
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1098 -Dcom.sun.management.jmxremote.rmi.port=1098 -Djava.rmi.server.hostname=127.0.0.1

2. ssh to your server on ec2 with a) port forwarding of 1098(jmx port) and b) with socks server on 10003
ssh -L 1098:127.0.0.1:1098 -D 10003 my.ec2.instance.com

3. Open visualvm with socks proxy on on same proxy port
jvisualvm -J-DsocksProxyHost=localhost -J-DsocksProxyPort=10003

4. create local jmx connection on localhost:1098 in visualvm

IgorBerman

Supposing you've installed spark with brew install spark@1.6
it will place it into /usr/local/opt/apache-spark@1.6

to use spark locally from intellij from your script define following:

1. environment variables:
PYSPARK_PYTHON=/Users/igorberm/.runtimes/Python34/bin/python3
PYTHONPATH=/usr/local/opt/apache-spark@1.6/libexec/python/lib/pyspark.zip:/usr/local/opt/apache-spark@1.6/libexec/python/lib/py4j-0.9-src.zip
PYTHONUNBUFFERED=1
SPARK_HOME=/usr/local/opt/apache-spark@1.6/libexec

IgorBerman

package igorprivate;

import shaded.parquet.org.slf4j.LoggerFactory;
import static org.apache.spark.sql.functions.concat_ws;
import static org.apache.spark.sql.functions.date_format;
import static org.apache.spark.sql.functions.from_json;
import static org.apache.spark.sql.functions.lit;
import static org.apache.spark.sql.functions.struct;
import static org.apache.spark.sql.functions.sum;
import static org.apache.spark.sql.functions.to_json;