Skip to content

Instantly share code, notes, and snippets.

@brablc
Last active September 27, 2020 20:10
Show Gist options
  • Save brablc/6139440 to your computer and use it in GitHub Desktop.
Save brablc/6139440 to your computer and use it in GitHub Desktop.
Logstash example configuration for parsing Postfix mail log files
# Postfix stuff based on https://gist.github.com/jbrownsc/4694374:
QUEUEID (?:[A-F0-9]+|NOQUEUE)
EMAILADDRESSPART [a-zA-Z0-9_.+-=:]+
EMAILADDRESS %{EMAILADDRESSPART:local}@%{EMAILADDRESSPART:remote}
RELAY (?:%{HOSTNAME:relayhost}(?:\[%{IP:relayip}\](?::[0-9]+(.[0-9]+)?)?)?)
POSREAL [0-9]+(.[0-9]+)?
DELAYS (%{POSREAL}[/]*)+
DSN %{NONNEGINT}.%{NONNEGINT}.%{NONNEGINT}
STATUS sent|deferred|bounced|expired
PERMERROR 5[0-9]{2}
MESSAGELEVEL reject|warning|error|fatal|panic
POSTFIXACTION discard|dunno|filter|hold|ignore|info|prepend|redirect|replace|reject|warn
# postfix/smtp and postfix/lmtp and postfix/local
POSTFIXSMTPRELAY %{QUEUEID:qid}: to=<%{EMAILADDRESS:to}>,(?:\sorig_to=<%{EMAILADDRESS:orig_to}>,)? relay=%{RELAY}, delay=%{POSREAL:delay}, delays=%{DELAYS:delays}, dsn=%{DSN:dsn}, status=%{STATUS:result} \(%{DATA:reason}\)
POSTFIXSMTPCONNECT connect to %{RELAY}: %{GREEDYDATA:reason}
POSTFIXSMTP4XX %{QUEUEID:qid}: host %{RELAY} said: %{GREEDYDATA:reason}
POSTFIXSMTP5XX %{QUEUEID:qid}: to=<%{EMAILADDRESS:to}>,(?:\sorig_to=<%{EMAILADDRESS:orig_to}>,)? relay=%{RELAY}, delay=%{POSREAL:delay}, delays=%{DELAYS:delays}, dsn=%{DSN:dsn}, status=%{STATUS:result} \(host %{HOSTNAME}\[%{IP}\] said: %{PERMERROR:responsecode} %{DATA:smtp_response} \(in reply to %{DATA:command} command\)\)
POSTFIXSMTPREFUSAL %{QUEUEID:qid}: host %{RELAY} refused to talk to me: %{GREEDYDATA:reason}
POSTFIXSMTPLOSTCONNECTION %{QUEUEID:qid}: lost connection with %{RELAY} while %{GREEDYDATA:reason}
POSTFIXSMTPTIMEOUT %{QUEUEID:qid}: conversation with %{RELAY} timed out while %{GREEDYDATA:reason}
# postfix/smtpd
POSTFIXSMTPDCONNECTS (?:dis)?connect from %{RELAY}
POSTFIXSMTPDACTIONS %{QUEUEID:qid}: %{POSTFIXACTION}: %{DATA:command} from %{RELAY}: %{DATA:smtp_response}: %{DATA:reason}; from=<%{EMAILADDRESS:from}> to=<%{EMAILADDRESS:to}> proto=%{DATA:proto} helo=<%{IPORHOST}>
POSTFIXSMTPDTIMEOUTS timeout after %{DATA:command} from %{RELAY}
POSTFIXSMTPDLOGIN %{QUEUEID:qid}: client=%{DATA:client}, sasl_method=%{DATA:saslmethod}, sasl_username=%{EMAILADDRESS:saslusername}
POSTFIXSMTPDCLIENT %{QUEUEID:qid}: client=%{DATA:client}
# postfix/cleanup
POSTFIXCLEANUP %{QUEUEID:qid}: message-id=<%{EMAILADDRESS:messageid}>
# postfix/bounce
POSTFIXBOUNCE %{QUEUEID:qid}: sender non-delivery notification: %{QUEUEID:bouncequeueid}
# postfix/qmgr and postfix/pickup
POSTFIXQMGR %{QUEUEID:qid}: (?:removed|from=<(?:%{EMAILADDRESS:from})?>(?:, size=%{POSINT:size}, nrcpt=%{POSINT:nrcpt} \(%{GREEDYDATA:queuestatus}\))?)
input {
file {
type => "postfix"
path => "/var/log/mail.info"
}
}
filter {
grok {
type => "postfix"
patterns_dir => [ "/etc/logstash/grok-patterns" ]
pattern => [
"%{SYSLOGBASE} %{POSTFIXSMTPDCONNECTS}",
"%{SYSLOGBASE} %{POSTFIXSMTPDACTIONS}",
"%{SYSLOGBASE} %{POSTFIXSMTPDTIMEOUTS}",
"%{SYSLOGBASE} %{POSTFIXSMTPDLOGIN}",
"%{SYSLOGBASE} %{POSTFIXSMTPDCLIENT}",
"%{SYSLOGBASE} %{POSTFIXSMTPRELAY}",
"%{SYSLOGBASE} %{POSTFIXSMTPCONNECT}",
"%{SYSLOGBASE} %{POSTFIXSMTP4XX}",
"%{SYSLOGBASE} %{POSTFIXSMTP5XX}",
"%{SYSLOGBASE} %{POSTFIXSMTPREFUSAL}",
"%{SYSLOGBASE} %{POSTFIXSMTPLOSTCONNECTION}",
"%{SYSLOGBASE} %{POSTFIXSMTPTIMEOUT}",
"%{SYSLOGBASE} %{POSTFIXBOUNCE}",
"%{SYSLOGBASE} %{POSTFIXQMGR}",
"%{SYSLOGBASE} %{POSTFIXCLEANUP}"
]
named_captures_only => true
}
}
@somoza
Copy link

somoza commented Sep 2, 2015

You are using a deprecated config setting "pattern" set in grok. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. You should use this instead: match => { "message" => "your pattern here" } If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"pattern", :plugin=><LogStash::Filters::Grok patterns_dir=>["/etc/logstash/patterns.d"], pattern=>["%{SYSLOGBASE} %{POSTFIXSMTPDCONNECTS}", "%{SYSLOGBASE} %{POSTFIXSMTPDACTIONS}", "%{SYSLOGBASE} %{POSTFIXSMTPDTIMEOUTS}", "%{SYSLOGBASE} %{POSTFIXSMTPDLOGIN}", "%{SYSLOGBASE} %{POSTFIXSMTPDCLIENT}", "%{SYSLOGBASE} %{POSTFIXSMTPRELAY}", "%{SYSLOGBASE} %{POSTFIXSMTPCONNECT}", "%{SYSLOGBASE} %{POSTFIXSMTP4XX}", "%{SYSLOGBASE} %{POSTFIXSMTP5XX}", "%{SYSLOGBASE} %{POSTFIXSMTPREFUSAL}", "%{SYSLOGBASE} %{POSTFIXSMTPLOSTCONNECTION}", "%{SYSLOGBASE} %{POSTFIXSMTPTIMEOUT}", "%{SYSLOGBASE} %{POSTFIXBOUNCE}", "%{SYSLOGBASE} %{POSTFIXQMGR}", "%{SYSLOGBASE} %{POSTFIXCLEANUP}"], named_captures_only=>"true">, :level=>:warn}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment