Skip to content

Instantly share code, notes, and snippets.

@djpate
Last active August 29, 2015 14:11
Show Gist options
  • Save djpate/f48e2009847ed5a05e3a to your computer and use it in GitHub Desktop.
Save djpate/f48e2009847ed5a05e3a to your computer and use it in GitHub Desktop.
Response Policy Design

Response Policy

Goals

  • Allow the R&R team to find and effectively prioritize app instances for analysis
  • Allow each R&R analyst to be significantly more productive and reduce manual drudgery
  • Create a system that is flexible enough to adapt with a changing malware landscape

Term definition

Matcher

A matcher is an ElasticSearch query, that can be executed on any of our existing indices.

It can be created/updated by any analyst at any time.

Action

An action is a predifined set of business logic to execute.

Each action may have arguments required for it's execution.

For example an autoclose action will require a comment.

Adding a new action WILL REQUIRE a new deploy.

Rule

A rule joins a matcher to N actions.

On top of that, it will some other informations related to notifications (email etc.) and a name.

Design

Backend (Estimation 2 weeks)

Rules API

We will create a RULES API endpoint under /api/v1/rules

Rule JSON payload Subject to change

{
  "id": 1,
  "name": "autoclose_ActSpat.A",
  "notifications": {
    "type": daily,
    "emails": [
      "christophe@lookout.com"
    ]
  }
  "matcher": {
    {
      "query": {
        "range" : {
          "version_code" : {
              "gte" : 1,
              "lte" : 2
          }
        }
      }
    }
  }
  "actions": [
    {
      "name": "autoclose",
      "args": {
        "comment": "This is bad so we are closing it."
      }
    }
  ]
}

Upon creation of rule, we will store it's matcher in the PERCOLATOR API of Elasticsearch. this will give us an easy access to all rules matched with an document.

Actions

In order to be executed, most actions will require arguments, (comments, assessment_id etc.) so when each actions should define all it's required arguments. this is already done for mass actions, so we can leverage the bacth action implementation.

Executing Rules

We onlu execute rules at two different points.

  • When a new document is (re)indexed.

    We call to the percolator API for this document, it will give us in one query all the matchers that it matched against and we can process them in order.

  • When we create/update a new rule.

    We execute the query against the index using the scroll API, so that we don't conflict with incoming ingestion/changes.

Of course, all of that happens in background jobs.

Frontend (3 Weeks)

On the frontend, we will need a new UI to create rules, It should be pretty close to the saved queries UI but should support adding actions and their required parameters.

We could try to do something fancy that will automagicly generate forms from the definitions of Actions but I think it will be better to create one view per action so that we don't have to do some meta programming magic.

Additional Considerations

In order to allow the analyst to do all the matching they need, we might need to add data to our current index as well as adding filters to suppor them. This might require some extra time to reindex and migrate ES schemas.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment