I think we can all agree that the addition of Stellar has given us significant capabilities in Metron that we would not otherwise have. The ability to program in real-time on security telemetry is very powerful.
There are some ways that we can improve it though. Primarily, the way that a user programs with Stellar in Metron has some friction points.
-
It is difficult to understand, especially for beginners.
- Which JSON element do I need to add this Stellar snippet to?
- Do I need to escape this quote?
- Why do I need to fit all of my logic on this single, long line?
-
The code doesn't all live in one place which makes it difficult to understand and debug holistically.
- Threat triage code is added here, Enrichment code goes over there.
- Oops, I changed the field name to 'hostname', but forgot to change it in my Threat Triage rule.
-
Maintaining, migrating, and sharing Stellar code is difficult when so deeply embedded in the JSON-y guts of Metron.
I am proposing an event-driven model for programming in Metron with Stellar. I argue that we can do this with minimal changes to the underlying implementation of Stellar and the topologies.
The user simply creates one (or more) scripts specifying all of their Stellar code (code that may execute across parsing, enrichment, threat triage, etc). In this script, the user registers functions that get called when events occur. For example, I register a function that performs a Geo-IP lookup when a YAF flow record goes through Enrichment.
The event-driven model allows a user to define all of their code in one place, even though under-the-hood it may be executed across separate topologies in Metron.
This would mimic the type of abstraction that is used in Bro [2]. That abstraction has proven useful in the cybersecurity domain. I think we need the same sort of abstraction in Metron.
For example, consider the 7-part blog series that describes operating on Squid telemetry [1]. All of the logic described in that series could be boiled down to the following. I would argue that this is far easier to grok than all of the JSON I see in that series.
# executed when a squid record has been successfully parsed
def whenSquidParsed(msg) {
msg['full_hostname'] = URL_TO_HOST(msg['url'])
msg['primary_domain'] = DOMAIN_REMOVE_SUBDOMAINS(msg['full_hostname'])
}
# executed when a squid record goes through enrichment
def whenSquidEnriched(msg) {
msg['whois'] = ENRICHMENT_GET('whois', msg['full_hostname'], 'enrichments', 'whois')
msg['zeus'] = ENRICHMENT_GET('zeus', msg['full_hostname'], 'enrichments', 'zeus')
msg['user'] = ENRICHMENT_GET('user', msg['ip_src_addr'], 'enrichments', 'users')
}
# executed when a squid record goes through threat triage
def whenSquidTriaged(msg) {
threat_score = 0
if exists(msg['zeus']) then threat_score += 5
if not ENDS_WITH(msg['primary_domain'], '.com') then threat_score += 10
msg['threat_score'] = threat_score
}
# register my Stellar functions to execute when events fire
register(PARSE_EVENT, "squid", whenSquidParsed)
register(ENRICHMENT_EVENT, "squid", whenSquidEnriched)
register(TRIAGE_EVENT, "squid", whenSquidTriaged)To implement this we would need some enhancements to the core Stellar language. Enhancements like variable assignment, named function definitions, and simpler key/value operations on Maps.
For the topologies, we just need a Stellar execution engine that accepts Stellar scripts and allows events to be injected by the topology itself.
// load the user's code from somewhere - Zk?
String stellarCode = ...
// initialize the execution environment
StellarExecutor executor = new StellarExecutor();
executor.load(stellarCode);When the Enrichment topology gets a message, all it needs to do is create an Event and inject that into the execution environment. This would execute the user's code and allow it to read/alter the message in-flight.
executor.createEvent(new EnrichmentEvent(message, ...));The Threat Triage bolt would not be much different.
executor.createEvent(new TriageEvent(message, ...));I'm sure the details will get a good bit more complicated once I dig into the code, but I can't think of any major complications right now.