Skip to content

Instantly share code, notes, and snippets.

View NiceGuyIT's full-sized avatar

David Randall NiceGuyIT

View GitHub Profile
@NiceGuyIT
NiceGuyIT / README.md
Last active March 4, 2024 18:10
nginx JSON to Filebeat to Logstash to Elasticsearch

Intro

This is an example configuration to have nginx output JSON logs to make it easier for Logstash processing. I was trying to get nginx > Filebeat > Logstash > ES working and it wasn't until I connected Filebeat directly to Elasticsearch that I saw the expected data. Google led me to ingest-convert.sh and I realized filebeat setup works for Filebeat > ES but not Filebeat > Logstash > ES. This is because Logstash does not use ingest pipelines by default. You have to enable them in the elasticsearch output block.

Having nginx log JSON in the format required for Elasticsearch means there's very little processing (i.e. grok) to be done in Logstash. nginx can only output JSON for access logs; the error_log format cannot be changed.

Extra fields are output and not used by the Kibana dashboards. I included them in case they might be useful. Since they are not declared in the filebeat setup, their default is "string" when yo

@NiceGuyIT
NiceGuyIT / auto.nfs4
Last active August 29, 2015 14:12 — forked from thwarted/auto.nfs4
#!/bin/bash
# name this script /etc/auto.nfs4 and make it executable
# /bin is too restrictive
#PATH=/bin
tryconnect() {
local host="$1"