Skip to content

Instantly share code, notes, and snippets.

Stellar REPL Management Utilities

In order to augment the functionality of the Stellar REPL, a few management functions surrounding the management of the configurations and the management of Stellar transformations in the following areas have been added:

  • Stellar field transformations in the Parsers
  • Stellar enrichments in the Enrichment topology
  • Stellar threat triage rules

Troubleshooting issues when programming against a live stream of data is difficult. It would be useful to have a means to run the entire threat triage process within the REPL before deploying your rule set on a Metron cluster. This creates a set of functions to allow simulation of Threat Triage inside of the Stellar REPL. This is useful for creating new triage rules, debugging existing triage rules, and to iterate quickly when testing rule sets.

Changes

Created the following new functions.

  • THREAT_TRIAGE_INIT
  • THREAT_TRIAGE_SCORE
  • THREAT_TRIAGE_CONFIG

Here is an example using the Profiler's "Group By" functionality. It is often the case that patterns and behaviors will differ based on calendar effects like day of week. For example, activity on a weekday can be very different from a weekend. The Profiler's "Group By" functionality is one way to account for calendar effects.

This profile definition operates over any incoming telemetry that has an ip_src_addr and a timestamp field. It produces a profile that segments the data by the day of week. It does by using a 'groupBy' expression to extract the day of week from the telemetry's timestamp field.

{
  "profiles": [
    {
 "profile": "calender-effects",
[HDP-2.5]
name=HDP-2.5
baseurl=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0
path=/
enabled=1
gpgcheck=0

Setup Vagrant

vagrant init suse/sles12-sp1
vagrant up --provider virtualbox
vagrant ssh

Install repository

Setup Vagrant

vagrant init suse/sles11sp3
vagrant up --provider virtualbox
vagrant ssh

Install repository

  • Could also try isolcpus

Average number of messages per batch

cat fastcapa.err | grep "produce messageset with" | awk '{print $7}' | awk '{ s+=$1 } END { print s/NR }'

Average size of each batch

cat fastcapa.err | grep "produce messageset with" | awk '{print $9}' | sed -s "s/[(]//g" | awk '{ s+=$1 } END { printf "%.2f MB \n", (s/NR)/1000000 }'

Calculate the rate of dating hitting HDFS. Should roughly match the incoming packet rate.

hadoop fs -ls /apps/metron/pcap | awk '{sum += $5} END { print sum}' && \
  sleep 60 && \
  hadoop fs -ls /apps/metron/pcap | awk '{sum += $5} END { print sum}'

Are the pcap files rolling correctly?