Skip to content

Instantly share code, notes, and snippets.

@jeffgeiger
Last active December 22, 2015 14:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jeffgeiger/6487459 to your computer and use it in GitHub Desktop.
Save jeffgeiger/6487459 to your computer and use it in GitHub Desktop.
Generate named grok filters and a stub config from bro log headers on the fly. `cd` to your Bro logs directory and run this.
#!/bin/bash
TSTAMP=$(date +%s)
CONF="/tmp/logstash-$TSTAMP.conf"
PATTERNS="/tmp/bro_patterns_$TSTAMP"
for j in *.log; do
LOG=$(echo $j | sed 's/.log//g')
FILTNAME=$(echo $LOG | tr '[:lower:]' '[:upper:]')
echo -e "BRO$FILTNAME \c" >> $PATTERNS
for i in $(grep -E "^#fields" $LOG.log | sed 's/^#fields\t//g'); do echo -e "%{DATA:$i}\\\0134\\\0164\c"; done | sed 's/\\\0134\\\0164$//g' >> $PATTERNS
# done
echo "" >> $PATTERNS
echo "
input {
file {
path => [ \"$PWD/$j\" ] # array (required)
type => \"bro_$LOG\" # string (optional)
}
}
" >> $CONF
echo "
filter{
grok {
type => \"bro_$LOG\"
patterns_dir => \"/opt/logstash/patterns\"
break_on_match => false
pattern => \"%{BRO$FILTNAME}\"
add_tag => \"bro$LOG\"
}
}
" >> $CONF
done
echo -e "================= PATTERNS =================\n$(cat "$PATTERNS")\n================= CONFIG =================\n$(cat $CONF)\n================= END ================="
rm $CONF
rm $PATTERNS
exit 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment