Name | Description | |
---|---|---|
profile | Required | A unique name identifying the profile. The field is treated as a string. |
foreach | Required | A separate profile is maintained for each of these. This is effectively the entity that the profile is describing. The field is expected to contain a Stellar expression whose result is the entity name. For example, if ip_src_addr then a separate profile would be maintained for each unique IP source address in the data; 10.0.0.1, 10.0.0.2, etc. |
onlyif | Optional | An expression that determines if a message should be applied to the profile. A Stellar expression is expected that when executed returns a boolean. A message is only applied to a profile if this condition is true. This allows a profile to filter the messages that it receives. |
init | Optional | A set of expressions that is executed at the start of a window period. A map is expected where the key is the variable name a |
2016-06-29 09:20:33,928 p=12803 u=nallen | TASK [deployment-report : Sanity check Metron web] ***************************** | |
2016-06-29 09:20:34,007 p=12803 u=nallen | [0;31mfatal: [node1 -> localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "sudo: a password is required\n", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false}[0m | |
2016-06-29 09:20:34,009 p=12803 u=nallen | PLAY RECAP ********************************************************************* | |
2016-06-29 09:20:34,009 p=12803 u=nallen | node1 : ok=97 changed=0 unreachable=0 failed=1 | |
2016-06-29 09:21:27,465 p=13129 u=nallen | PLAY *************************************************************************** | |
2016-06-29 09:21:27,465 p=13129 u=nallen | [0;36mskipping: no hosts matched[0m | |
2016-06-29 09:21:27,466 p=13129 u=nallen | PLAY *************************************************************************** | |
2016-06-29 09:21:27,466 p=13129 u=nallen | [0;36mskipping: no hosts match |
The Profiler is a feature extraction mechanism that can generate a profile describing the behavior of an entity on a network. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.
This is achieved by summarizing the streaming telemetry data consumed by Metron over sliding windows. A summary statistic is applied to the data received within a given window. Collecting this summary across many windows results in a time series that is useful for analysis.
Any field contained within a message can be used to generate a profile. A profile can even be produced from combining fields that originate in different data sources. A user has considerable power to transform the data used in a profile by leveraging the Stellar language. A user only need configure the desired profiles in Zookeeper and ensure that the Profiler topology is running.
import java.util.Arrays; | |
import java.util.Collections; | |
import java.util.HashMap; | |
import java.util.Iterator; | |
/* | |
* This source code has been placed into the public domain. | |
*/ | |
/** |
| Name | | Description | | |
|--- |--- |--- | | |
| profile | Required | A unique name identifying the profile. The field is treated as a string. | | |
| foreach | Required | A separate profile is maintained *for each* of these. This is effectively the entity that the profile is describing. The field is expected to contain a Stellar expression whose result is the entity name. For example, if `ip_src_addr` then a separate profile would be maintained for each unique IP source address in the data; 10.0.0.1, 10.0.0.2, etc. | | |
| onlyif | Optional | An expression that determines if a message should be applied to the profile. A Stellar expression is expected that when executed returns a boolean. A message is only applied to a profile if this condition is true. This allows a profile to filter the messages that it receives. | | |
| groupBy | Optional | A list of Stellar expressions used to group the profile measurements when persisted. This is intended to sort the Profil |
The Profiler is a feature extraction mechanism that can generate a profile describing the behavior of an entity on a network. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.
This is achieved by summarizing the streaming telemetry data consumed by Metron over sliding windows. A summary statistic is applied to the data received within a given window. Collecting this summary across many windows results in a time series that is useful for analysis.
Any field contained within a message can be used to generate a profile. A profile can even be produced from combining fields that originate in different data sources. A user has considerable power to transform the data used in a profile by leveraging the Stellar language. A user only need configure the desired profiles in Zookeeper and ensure that the Profiler topology is running.
{ | |
"template": "sensor1_index*", | |
"mappings": { | |
"sensor1_doc": { | |
"_timestamp": { | |
"enabled": true | |
}, | |
"properties": { | |
"timestamp": { | |
"type": "date", |
We need a source of PCAP data to ingest. In a production environment there is likely to be host(s) configured with one or more span port(s) that receives raw packet data from a packet aggregator device. To simulate this, we will use Metron's Pcap Replay service.
service pcap-replay start
Validate that that packet data is being replayed correctly.
Followed the instructions in the blog series to add Squid.
We need an index template for Squid so that it interprets the timestamp field as a timestamp. I did not see this in the blog series, but I might have missed.
PUT _template/values
{