nickwallen

```
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at

nickwallen

FROM centos:7

# install development tools
RUN yum -y groupinstall "Development Tools"
RUN yum -y install cmake make gcc gcc-c++ flex bison libpcap libpcap-devel openssl-devel python-devel swig zlib-devel perl cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi git

# install bro
RUN curl -L https://www.bro.org/downloads/bro-2.5.5.tar.gz | tar xvz

nickwallen

for k in $(git branch --sort=committerdate | sed /\*/d); do 
  if [ -z "$(git log -1 --since='2 months ago' -s $k)" ]; then
    echo ""
    echo ""
    echo "branch: $k"
    git show $k --stat --oneline
    git show $k --pretty="format:  Last commit: %cd" | head -n 1
    
    read -p "  delete '$k'? [yN] " -n 1 -r
    echo

nickwallen

Stellar Interpreter for Apache Zeppelin

Apache Zeppelin is a web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more. This project provides a means to run the Stellar REPL directly within a Zeppelin Notebook.

Installation

Currently, you need to manually install the Stellar Interpreter in Zeppelin. In the future this step could be automated by the Metron Mpack.

To install the Stellar Interpreter in your Apache Zeppelin installation, follow these instructions. This is paraphrased from the Zeppelin docs.

nickwallen

Changes needed.

• gateway_recover_after_data_nodes: 1
• index_number_of_replicas: 0
• index_number_of_shards: 1
• masters_also_are_datanodes: "true"
• zen_discovery_ping_unicast_hosts: [ centos1 ]

nickwallen

export PATH=$PATH:/var/lib/ambari-server/

python service_check.py \
  SERVICE_CHECK \
  /var/lib/ambari-agent/data/command-312.json \
  /var/lib//ambari-agent/cache/common-services/METRON//0.4.1/package/ \
  output.txt \
  DEBUG \
 /var/lib/ambari-agent/tmp

nickwallen

One liner to stop / remove all of Docker containers:

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

nickwallen

Metron Profiler

The Profiler is a feature extraction mechanism that can generate a profile describing the behavior of an entity. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.

This is achieved by summarizing the streaming telemetry data consumed by Metron over sliding windows. A summary statistic is applied to the data received within a given window. Collecting this summary across many windows results in a time series that is useful for analysis.

Any field contained within a message can be used to generate a profile. A profile can even be produced by combining fields that originate in different data sources. A user has considerable power to transform the data used in a profile by leveraging the Stellar language. A user only need configure the desired profiles and ensure that the Profiler topology is running.

• Installation
• [Creating Profiles]

nickwallen

find ./ -name "metron-api-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-common-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-data-management-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-elasticsearch-0.4.1-uber.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-enrichment-0.4.1-uber.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-management-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-parsers-0.4.1-uber.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-pcap-backend-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-profiler-0.4.1-uber.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-rest-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;

nickwallen

Stellar Language

For a variety of components (threat intelligence triage and field transformations) we have the need to do simple computation and transformation using the data from messages as variables. For those purposes, there exists a simple, scaled down DSL created to do simple computation and transformation.

• Introduction
• Stellar Core Functions
• Stellar Benchmarks
• Stellar Shell