apg

Streaming Systems:

• Differential Dataflow: https://github.com/frankmcsherry/timely-dataflow - The code is ugly Rust, but the logic and linked papers are quite interesting.
• Spinning Fast Iterative Dataflows: http://vldb.org/pvldb/vol5/p1268_stephanewen_vldb2012.pdf - Flink's execution model. See also http://us9.campaign-archive2.com/?u=4188b6afbe9e5d43111fef4d4&id=46ab8c2adf&e=04a33a53a5
• Discretized Streams: http://www.cs.berkeley.edu/~matei/papers/2013/sosp_spark_streaming.pdf - Spark Streaming's model of operation.
• Google's Dataflow Model: http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43864.pdf - This is now also available as Apache (Incubating) Beam https://wiki.apache.org/incubator/BeamProposal

Streaming More Generally:

• Algorithms + Data Structures: https://gist.github.com/debasishg/8172796

apg

package main

import (
	"fmt"
	"sync"
)

type Event map[string]interface{}

type asyncStreamFunc func(done <-chan struct{}, in <-chan Event, out chan<- Event)

apg

package main

type Plugin struct {
	Match(Msg) bool // continue the Reporter Chain.
	Run()
	Shutdown()
}

type LoggingConfig struct {
	ReporterPlugins []Plugin

apg

1. General Background and Overview

• Probabilistic Data Structures for Web Analytics and Data Mining : A great overview of the space of probabilistic data structures and how they are used in approximation algorithm implementation.
• Models and Issues in Data Stream Systems
• Philippe Flajolet’s contribution to streaming algorithms : A presentation by Jérémie Lumbroso that visits some of the hostorical perspectives and how it all began with Flajolet
• Approximate Frequency Counts over Data Streams by Gurmeet Singh Manku & Rajeev Motwani : One of the early papers on the subject.
• [Methods for Finding Frequent Items in Data Streams](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.9800&rep

apg

[apg@amend soa]$ go test -bench=. 
testing: warning: no tests to run
PASS
BenchmarkAOS10	20000000	        77.2 ns/op
Benchmark1AOS10	20000000	        78.1 ns/op
BenchmarkSOA10	20000000	        65.2 ns/op
BenchmarkAOS100	 2000000	       748 ns/op
Benchmark1AOS100	 2000000	       749 ns/op
BenchmarkSOA100	 2000000	       649 ns/op
BenchmarkAOS1000	  200000	      6864 ns/op

apg

#lang racket/base

(require racket/draw
         racket/gui)

(define files (for/list [(i (in-range 7))]
                (format "/tmp/banana/banana~a.png" i)))

(define bitmaps (map (lambda (x) (read-bitmap x)) files))
(define dcs (map (lambda (x) (new bitmap-dc% [bitmap x])) bitmaps))

apg

Welcome to Racket v6.1.
racket@> (require racket/draw)
racket@> (define files (for/list
                           [(i (in-range 7))]
                         (format "/tmp/banana/banana~a.png" i)))
racket@> files
'("/tmp/banana/banana0.png"
  "/tmp/banana/banana1.png"
  "/tmp/banana/banana2.png"
  "/tmp/banana/banana3.png"

apg

$ go run score.go 
Test: [1 2 3]
	score=0
Test: [2 1 3]
	score=1
Test: [3 2 1]
	score=2
Test: [1 2 4 3 5]
	score=1
Test: [1 2 4 3 5 6 8 9 7]

apg

apg

Internet Scale Services Checklist

A checklist for designing and developing internet scale services, inspired by James Hamilton's 2007 paper "On Desgining and Deploying Internet-Scale Services."

• http://mvdirona.com/jrh/talksandpapers/jamesrh_lisa.pdf

Basic tenets

• Does the design expect failures to happen regularly and handle them gracefully?
• Have we kept things as simple as possible?