Storm scheduler notes

RAS_vs_Isolation

1. Problems with isolation scheduler
- Hand out dedicated machines to topology, it's a problem in heterogeneous cluster
- Low overall cluster resource utilization, users are not utilizing their isolated resource very well. 
- Unbalanced resource usage: some machines are not used when others are over used. 
- Per topology scheduling strategy

2. Resource aware scheduling 
Refer to http://storm.apache.org/releases/current/Resource_Aware_Scheduler_overview.html

This scheduler takes into account resource avalibility on machines and resource requirement of workloads when scheduling the topology
- fine grain resource control
- RAS implemention includes many nice multi-tenant features
- Constraint based scheduling?? 

3. RAS features
- Pluggable per topology scheduling strategies (multi-tenant strategy vs. default resource aware strategy vs. constraint based scheduling strategy)
conf.setTopologyStrategy(org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class);
Users can implement the IStrategy interface and define new strategies to schedule specific topologies. 

- Resource isolation vs. cgroups
CGroups are used by Storm to limit the resource usage of workers to guarantee fairness and QOS. CGroups can be used in conjunction with the Resource Aware Scheduler. CGroups will then enforce the resource usage of workers as allocated by the Resource Aware Scheduler. To use cgroups with the Resource Aware Scheduler, simply enable cgroups and be sure NOT to set storm.worker.cgroup.memory.mb.limit and storm.worker.cgroup.cpu.limit configs.
https://storm.apache.org/releases/2.0.0-SNAPSHOT/cgroups_in_storm.html

- Per user resource gyarantees
Specify # of CPU and amount of memory on user basis

- Topology priority
Users can set the priority of a topology to indicate its importance

- Pluggable topology priority
which topology should be scheduled first? take into account resource guarantees and topology priority

- Pluggable topology eviction strategies
When there is not enough resource, which topology from which user to evict?