FROM centos:7
# install development tools
RUN yum -y groupinstall "Development Tools"
RUN yum -y install cmake make gcc gcc-c++ flex bison libpcap libpcap-devel openssl-devel python-devel swig zlib-devel perl cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi git
# install bro
RUN curl -L https://www.bro.org/downloads/bro-2.5.5.tar.gz | tar xvz
``` | |
#!/bin/bash | |
# | |
# Licensed to the Apache Software Foundation (ASF) under one | |
# or more contributor license agreements. See the NOTICE file | |
# distributed with this work for additional information | |
# regarding copyright ownership. The ASF licenses this file | |
# to you under the Apache License, Version 2.0 (the | |
# "License"); you may not use this file except in compliance | |
# with the License. You may obtain a copy of the License at |
for k in $(git branch --sort=committerdate | sed /\*/d); do | |
if [ -z "$(git log -1 --since='2 months ago' -s $k)" ]; then | |
echo "" | |
echo "" | |
echo "branch: $k" | |
git show $k --stat --oneline | |
git show $k --pretty="format: Last commit: %cd" | head -n 1 | |
read -p " delete '$k'? [yN] " -n 1 -r | |
echo |
Apache Zeppelin is a web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more. This project provides a means to run the Stellar REPL directly within a Zeppelin Notebook.
Currently, you need to manually install the Stellar Interpreter in Zeppelin. In the future this step could be automated by the Metron Mpack.
To install the Stellar Interpreter in your Apache Zeppelin installation, follow these instructions. This is paraphrased from the Zeppelin docs.
Changes needed.
- gateway_recover_after_data_nodes: 1
- index_number_of_replicas: 0
- index_number_of_shards: 1
- masters_also_are_datanodes: "true"
- zen_discovery_ping_unicast_hosts: [ centos1 ]
export PATH=$PATH:/var/lib/ambari-server/
python service_check.py \
SERVICE_CHECK \
/var/lib/ambari-agent/data/command-312.json \
/var/lib//ambari-agent/cache/common-services/METRON//0.4.1/package/ \
output.txt \
DEBUG \
/var/lib/ambari-agent/tmp
One liner to stop / remove all of Docker containers:
docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)
The Profiler is a feature extraction mechanism that can generate a profile describing the behavior of an entity. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.
This is achieved by summarizing the streaming telemetry data consumed by Metron over sliding windows. A summary statistic is applied to the data received within a given window. Collecting this summary across many windows results in a time series that is useful for analysis.
Any field contained within a message can be used to generate a profile. A profile can even be produced by combining fields that originate in different data sources. A user has considerable power to transform the data used in a profile by leveraging the Stellar language. A user only need configure the desired profiles and ensure that the Profiler topology is running.
- Installation
- [Creating Profiles]
find ./ -name "metron-api-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \; | |
find ./ -name "metron-common-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \; | |
find ./ -name "metron-data-management-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \; | |
find ./ -name "metron-elasticsearch-0.4.1-uber.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \; | |
find ./ -name "metron-enrichment-0.4.1-uber.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \; | |
find ./ -name "metron-management-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \; | |
find ./ -name "metron-parsers-0.4.1-uber.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \; | |
find ./ -name "metron-pcap-backend-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \; | |
find ./ -name "metron-profiler-0.4.1-uber.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \; | |
find ./ -name "metron-rest-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \; |
For a variety of components (threat intelligence triage and field transformations) we have the need to do simple computation and transformation using the data from messages as variables. For those purposes, there exists a simple, scaled down DSL created to do simple computation and transformation.