A service has been created to simulate the behavior of a sensor by sending canned telemetry data to a Kafka topic. These "Sensor Stubs" consume fewer resources than the actual sensor that they replace.
The stubs are installed with a set of canned data for each sensor type. A subset of this canned data is randomly selected and sent to the Kafka topic in batches. The timestamp of each message is updated to match current system time.
===Private IPv4 address spaces
| RFC1918 name | IP address range | number of addresses |
|---|---|---|
| 24-bit block | 10.0.0.0 - 10.255.255.255 | 16,777,216 |
| 20-bit block | 172.16.0.0 - 172.31.255.255 | 1,048,576 |
| 16-bit block | 192.168.0.0 - 192.168.255.255 | 65,536 |
Follow these steps to create a vertical bar chart in Kibana.
Followed the instructions in the blog series to add Squid.
We need an index template for Squid so that it interprets the timestamp field as a timestamp. I did not see this in the blog series, but I might have missed.
PUT _template/values
{
We need a source of PCAP data to ingest. In a production environment there is likely to be host(s) configured with one or more span port(s) that receives raw packet data from a packet aggregator device. To simulate this, we will use Metron's Pcap Replay service.
service pcap-replay start
Validate that that packet data is being replayed correctly.
| { | |
| "template": "sensor1_index*", | |
| "mappings": { | |
| "sensor1_doc": { | |
| "_timestamp": { | |
| "enabled": true | |
| }, | |
| "properties": { | |
| "timestamp": { | |
| "type": "date", |
The Profiler is a feature extraction mechanism that can generate a profile describing the behavior of an entity on a network. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.
This is achieved by summarizing the streaming telemetry data consumed by Metron over sliding windows. A summary statistic is applied to the data received within a given window. Collecting this summary across many windows results in a time series that is useful for analysis.
Any field contained within a message can be used to generate a profile. A profile can even be produced from combining fields that originate in different data sources. A user has considerable power to transform the data used in a profile by leveraging the Stellar language. A user only need configure the desired profiles in Zookeeper and ensure that the Profiler topology is running.
| | Name | | Description | | |
| |--- |--- |--- | | |
| | profile | Required | A unique name identifying the profile. The field is treated as a string. | | |
| | foreach | Required | A separate profile is maintained *for each* of these. This is effectively the entity that the profile is describing. The field is expected to contain a Stellar expression whose result is the entity name. For example, if `ip_src_addr` then a separate profile would be maintained for each unique IP source address in the data; 10.0.0.1, 10.0.0.2, etc. | | |
| | onlyif | Optional | An expression that determines if a message should be applied to the profile. A Stellar expression is expected that when executed returns a boolean. A message is only applied to a profile if this condition is true. This allows a profile to filter the messages that it receives. | | |
| | groupBy | Optional | A list of Stellar expressions used to group the profile measurements when persisted. This is intended to sort the Profil |