Skip to content

Instantly share code, notes, and snippets.

View top-k-threshold.md

This relates to work I'm doing on a system for distributed outlier detection, as described here.

In this system, the count-min sketch is used as a distributed filter. Over some interval, the CMS is populated on all the edge nodes, sent to a server for merging, and the merged CMS is broadcast back out to the edges. On the following interval, each new key is run through the "global" CMS, and if it's over some threshold, it is added to the set of potential outliers, which are precisely counted.

This is, as described so far, an adaptation of this 2003 paper from single-node to multi-node. However, that paper assumes that the threshold is known ahead of time.

In most cases, though, what constitutes an "outlier" is a moving target. We'd like to adapt as the distribution changes. Previously, I was using a consistent uniform sample of the keys to identify the threshold, but we'd have to take a very

anonymous
anonymous / playground.rs
Created Mar 29, 2016
Shared via Rust Playground
View playground.rs
use std::sync::{Arc, Mutex};
trait Set {
fn set(&mut self, t: Str);
}
struct Str<'a>(&'a str);
pub struct Port(Arc<Mutex<Set>>);
anonymous
anonymous / playground.rs
Created Mar 29, 2016
Shared via Rust Playground
View playground.rs
use std::sync::{Arc, Mutex};
trait Get<T> {
fn get(&mut self) -> T;
}
trait Set {
fn set(&mut self, t: Str);
}
View elasticsearchcreator.py
from elasticsearch import Elasticsearch
# Important thing, because by default it tries localhost:9200 which is not responsible by some reason
es = Elasticsearch(hosts=({"host": "127.0.0.1", "port": 9200},))
# Wait for elasticSearch to initialize
while True:
try:
es.ping()
except Exception:
@oza
oza / SparkOnYARN.md
Last active Sep 22, 2021
How to run Spark on YARN with dynamic resource allocation
View SparkOnYARN.md

YARN

  1. General resource management layer on HDFS
  2. A part of Hadoop

Spark

  1. In memory processing framework

Spark on YARN