These are a couple real examples of heka lua filters that do anomaly detection and alerting, which a both very new experimental features. Eventually we should have some nicer abstractions for detecting and alerting common scenarios that can be easily configured instead of having to write all the logic yourself. There's currently a lot of flexabilty but at the price of more code.
Since heka filters can process and generate messages, you can make pretty complex meta-dataflows to control alerts, and can wire them up to any output (I think), irc, email, sms, whatevs.
The module sources are here:
https://github.com/mozilla-services/heka/tree/dev/sandbox/lua/modules
Our current dashboard displays annotations, but the UX still needs a lot of work; there's too much clicking and navigating required. I'd ❤︎ to have some time to contribute.