inthecloud247

# Create using kubectl:
#   $ kubectl create -f splunk-daemonset.yaml
# 
# You should also add config on your indexer to deal with the json formatted files:
#   https://answers.splunk.com/answers/148307/how-to-parse-and-extract-json-log-files-in-splunk.html
#
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: splunk-forwarder

inthecloud247

curl -s 10.0.0.145:8500/v1/kv/my_key/my_otehr_key/this_is_the_key?dc=dc1 | jq -r '.[0].Value' | base64 --decode

inthecloud247

{
  "title": "Cisco ASA",
  "services": {
    "query": {
      "list": {
        "0": {
          "query": "*",
          "alias": "",
          "color": "#7EB26D",
          "id": 0,

inthecloud247

nginx/
!nginx/.gitkeep
!nginx/logs/.gitkeep

src/

tmp/

inthecloud247

Boostrapping Salt on Linux EC2 with Cloud-Init

Salt is a great tool for remote execution and configuration management, however you will still need to bootstrap the daemon when spinning up a new node. One option is to create and save a custom AMI, but this creates another resource to maintain and document.

A better method for Linux machines uses Canonical's CloudInit to run a bootstrap script during an EC2 Instance initialization. Cloud-init takes the user_data string passed into a new AWS instance and runs it in a manner similar to rc.local. The bootstrap script needs to:

1. Install Salt with dependencies
2. Point the minion to the master

inthecloud247

# salt-minion.conf
description "salt-minion upstart daemon"
author  "Jeff Bauer <jbauer@rubic.com>"

# copy this file to /etc/init

start on (net-device-up and local-filesystems)
stop on shutdown

expect fork

inthecloud247

require "cjson"
local util = require("util") 
-- https://github.com/mozilla-services/lua_sandbox/blob/master/modules/util.lua


local message = {
   Type = "IGNORE",
   Fields = {}
}

inthecloud247

// define these shard spaces
shardSpaces = [
  {name: "raw", retention: "4h"},
  {name: "week", retention: "7d"},
  {name: "month", retention: "30d"},
  {name: "year", retention: "365d"},
  {name: "keep", retention: "inf"}
]

// apply these rules for mapping data to shard spaces

inthecloud247

[hekad]
maxprocs = 2
base_dir = "/tmp/cache"
pid_file = "/tmp/hekad.pid"
share_dir = "/usr/share/heka"

[log1]
type = "LogstreamerInput"
log_directory = "/var/log/"
file_match = 'yum\.log'

inthecloud247

Go here for the latest + some code.

numbat

An alerting engine for a metrics & monitoring system.

This is the same approach I wanted in my initial spike, only instead of writing a custom collector & using an existing alerting engine (riemann), I'm proposing using an existing collector (hekad) and writing the alerting engine.

The system