Skip to content

Instantly share code, notes, and snippets.

View delete_collection.sh
```
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
View Dockerfile for Bro Plugin Testing.md
FROM centos:7

# install development tools
RUN yum -y groupinstall "Development Tools"
RUN yum -y install cmake make gcc gcc-c++ flex bison libpcap libpcap-devel openssl-devel python-devel swig zlib-devel perl cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi git

# install bro
RUN curl -L https://www.bro.org/downloads/bro-2.5.5.tar.gz | tar xvz
View cleanup-branches.sh
for k in $(git branch --sort=committerdate | sed /\*/d); do
if [ -z "$(git log -1 --since='2 months ago' -s $k)" ]; then
echo ""
echo ""
echo "branch: $k"
git show $k --stat --oneline
git show $k --pretty="format: Last commit: %cd" | head -n 1
read -p " delete '$k'? [yN] " -n 1 -r
echo
View Stellar in a Notebook.md

Stellar Interpreter for Apache Zeppelin

Apache Zeppelin is a web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more. This project provides a means to run the Stellar REPL directly within a Zeppelin Notebook.

Installation

Currently, you need to manually install the Stellar Interpreter in Zeppelin. In the future this step could be automated by the Metron Mpack.

To install the Stellar Interpreter in your Apache Zeppelin installation, follow these instructions. This is paraphrased from the Zeppelin docs.

View Elasticsearch Single Node.md

Changes needed.

  • gateway_recover_after_data_nodes: 1
  • index_number_of_replicas: 0
  • index_number_of_shards: 1
  • masters_also_are_datanodes: "true"
  • zen_discovery_ping_unicast_hosts: [ centos1 ]
View Mpack Tests.md
export PATH=$PATH:/var/lib/ambari-server/

python service_check.py \
  SERVICE_CHECK \
  /var/lib/ambari-agent/data/command-312.json \
  /var/lib//ambari-agent/cache/common-services/METRON//0.4.1/package/ \
  output.txt \
  DEBUG \
  /var/lib/ambari-agent/tmp
View Remove All Docker Containers.md

One liner to stop / remove all of Docker containers:

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)
View Profiler README.md

Metron Profiler

The Profiler is a feature extraction mechanism that can generate a profile describing the behavior of an entity. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.

This is achieved by summarizing the streaming telemetry data consumed by Metron over sliding windows. A summary statistic is applied to the data received within a given window. Collecting this summary across many windows results in a time series that is useful for analysis.

Any field contained within a message can be used to generate a profile. A profile can even be produced by combining fields that originate in different data sources. A user has considerable power to transform the data used in a profile by leveraging the Stellar language. A user only need configure the desired profiles and ensure that the Profiler topology is running.

View Update Metron Libs.txt
find ./ -name "metron-api-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-common-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-data-management-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-elasticsearch-0.4.1-uber.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-enrichment-0.4.1-uber.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-management-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-parsers-0.4.1-uber.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-pcap-backend-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-profiler-0.4.1-uber.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
find ./ -name "metron-rest-0.4.1.jar" -exec scp {} root@node1:/usr/metron/0.4.1/lib/ \;
View README.md

Stellar Language

For a variety of components (threat intelligence triage and field transformations) we have the need to do simple computation and transformation using the data from messages as variables. For those purposes, there exists a simple, scaled down DSL created to do simple computation and transformation.

You can’t perform that action at this time.