nickwallen

2016-06-29 09:20:33,928 p=12803 u=nallen |  TASK [deployment-report : Sanity check Metron web] *****************************
2016-06-29 09:20:34,007 p=12803 u=nallen |  fatal: [node1 -> localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "sudo: a password is required\n", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false}
2016-06-29 09:20:34,009 p=12803 u=nallen |  PLAY RECAP *********************************************************************
2016-06-29 09:20:34,009 p=12803 u=nallen |  node1                      : ok=97   changed=0    unreachable=0    failed=1   
2016-06-29 09:21:27,465 p=13129 u=nallen |  PLAY ***************************************************************************
2016-06-29 09:21:27,465 p=13129 u=nallen |  skipping: no hosts matched
2016-06-29 09:21:27,466 p=13129 u=nallen |  PLAY ***************************************************************************
2016-06-29 09:21:27,466 p=13129 u=nallen |  skipping: no hosts match

nickwallen

Name		Description
profile	Required	A unique name identifying the profile. The field is treated as a string.
foreach	Required	A separate profile is maintained for each of these. This is effectively the entity that the profile is describing. The field is expected to contain a Stellar expression whose result is the entity name. For example, if ip_src_addr then a separate profile would be maintained for each unique IP source address in the data; 10.0.0.1, 10.0.0.2, etc.
onlyif	Optional	An expression that determines if a message should be applied to the profile. A Stellar expression is expected that when executed returns a boolean. A message is only applied to a profile if this condition is true. This allows a profile to filter the messages that it receives.
init	Optional	A set of expressions that is executed at the start of a window period. A map is expected where the key is the variable name a

nickwallen

Metron Profiler

The Profiler is a feature extraction mechanism that can generate a profile describing the behavior of an entity on a network. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.

This is achieved by summarizing the streaming telemetry data consumed by Metron over sliding windows. A summary statistic is applied to the data received within a given window. Collecting this summary across many windows results in a time series that is useful for analysis.

Usage

Any field contained within a message can be used to generate a profile. A profile can even be produced from combining fields that originate in different data sources. A user has considerable power to transform the data used in a profile by leveraging the Stellar language. A user only need configure the desired profiles in Zookeeper and ensure that the Profiler topology is running.

nickwallen

import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Iterator;

/*
 * This source code has been placed into the public domain.
 */

/**

nickwallen

| Name 	    |            	| Description 	|
|---	    |---	        |---	        |
| profile  	| Required   	| A unique name identifying the profile.  The field is treated as a string. |
| foreach  	| Required  	| A separate profile is maintained *for each* of these.  This is effectively the entity that the profile is describing.  The field is expected to contain a Stellar expression whose result is the entity name.  For example, if `ip_src_addr` then a separate profile would be maintained for each unique IP source address in the data; 10.0.0.1, 10.0.0.2, etc. | 
| onlyif  	| Optional  	| An expression that determines if a message should be applied to the profile.  A Stellar expression is expected that when executed returns a boolean.  A message is only applied to a profile if this condition is true. This allows a profile to filter the messages that it receives. |
| groupBy   | Optional      | A list of Stellar expressions used to group the profile measurements when persisted. This is intended to sort the Profil

nickwallen

Metron Profiler

The Profiler is a feature extraction mechanism that can generate a profile describing the behavior of an entity on a network. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.

This is achieved by summarizing the streaming telemetry data consumed by Metron over sliding windows. A summary statistic is applied to the data received within a given window. Collecting this summary across many windows results in a time series that is useful for analysis.

Usage

Any field contained within a message can be used to generate a profile. A profile can even be produced from combining fields that originate in different data sources. A user has considerable power to transform the data used in a profile by leveraging the Stellar language. A user only need configure the desired profiles in Zookeeper and ensure that the Profiler topology is running.

nickwallen

{
  "template": "sensor1_index*",
  "mappings": {
    "sensor1_doc": {
      "_timestamp": {
        "enabled": true
      },
      "properties": {
        "timestamp": {
          "type": "date",

nickwallen

1

Launch "Quick Dev" then stop all sensors and topologies.

service monit stop
service elasticsearch stop
storm kill bro
storm kill snort

nickwallen

1

Launch "Quick Dev" then stop all sensors and topologies.

service monit stop
service elasticsearch stop
storm kill bro
storm kill snort

nickwallen

Generate PCAP

We need a source of PCAP data to ingest. In a production environment there is likely to be host(s) configured with one or more span port(s) that receives raw packet data from a packet aggregator device. To simulate this, we will use Metron's Pcap Replay service.

service pcap-replay start

Validate that that packet data is being replayed correctly.