- Allow the R&R team to find and effectively prioritize app instances for analysis
- Allow each R&R analyst to be significantly more productive and reduce manual drudgery
- Create a system that is flexible enough to adapt with a changing malware landscape
A matcher is an ElasticSearch query, that can be executed on any of our existing indices.
It can be created/updated by any analyst at any time.
An action is a predifined set of business logic to execute.
Each action may have arguments required for it's execution.
For example an autoclose action will require a comment.
Adding a new action WILL REQUIRE a new deploy.
A rule joins a matcher to N actions.
On top of that, it will have some other information related to notifications (email etc.) and a name.
We will create a RULES API endpoint under /api/v1/rules
Rule JSON payload Subject to change
{
"id": 1,
"name": "autoclose_ActSpat.A",
"notifications": {
"type": daily,
"emails": [
"christophe@lookout.com"
]
}
"matcher": {
{
"query": {
"range" : {
"version_code" : {
"gte" : 1,
"lte" : 2
}
}
}
}
}
"actions": [
{
"name": "autoclose",
"args": {
"comment": "This is bad so we are closing it."
}
}
]
}
Upon creation of rule, we will store it's matcher in the PERCOLATOR API of Elasticsearch. this will give us an easy access to all rules matched with an document.
In order to be executed, most actions will require arguments, (comments, assessment_id etc.) so when each action should define all it's required arguments. this is already done for mass actions, so we can leverage the batch action implementation.
We only execute rules at two different points.
-
When a new document is (re)indexed.
We call to the percolator API for this document, it will give us in one query all the matchers that it matched against and we can process them in order.
-
When we create/update a new rule.
We execute the query against the index using the scroll API, so that we don't conflict with incoming ingestion/changes.
Of course, all of that happens in background jobs.
On the frontend, we will need a new UI to create rules. It should be pretty close to the saved queries UI but should support adding actions and their required parameters.
We could try to do something fancy that will automagicly generate forms from the definitions of Actions but I think it will be better to create one view per action so that we don't have to do some meta programming magic.
In order to allow the analyst to do all the matching they need, we might need to add data to our current index as well as adding filters to support them. This might require some extra time to reindex and migrate ES schemas.