Grok ESXi 5.x Pattern (for Logstash) (including puppet format with special escaping!)

gistfile1.txt

filter {
 grok {
  pattern => ['(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGHOST:logsource}) (?:%{SYSLOGPROG}): (?<messagebody>(?:\[(?<esxi_thread_id>[0-9A-Z]{8,8}) %{DATA:esxi_loglevel} \'%{DATA:esxi_service}\'\] %{GREEDYDATA:esxi_message}|%{GREEDYDATA}))']
  type => "esxi"
 }
}

# Puppet format with escaping

pattern => [ "(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:.* (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}|(?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}): (?:(?:\[[0-9A-Z]{8,8}) (?:%{GREEDYDATA:esxi_loglevel}) \\\'(?:%{GREEDYDATA:esxi_service})\\\'] (?:%{GREEDYDATA:message})|(?:%{GREEDYDATA:message}))" ],

If anyone knows what the 8-digit Hex-Digits mean, please tell me, so i can update the Pattern - The Digit is now parsed as trash only

Thanks for the grok!

I was working with good tech person from VMware and asked about the hex code and he said this:

"The hex number you refer to is a thread ID within the host agent for one or more tasks its executing, therefore it would not be possible to map that value to anything meaningful except when used for debugging purposes."

Updated the ESXi Grok Pattern for a more accurate matching and integrated the new esxi_thread_id field, thanks to @gungazoo

If you use the standard syslog parser in Logstash it will parse most of this automatically

Hi Guys,

I have a problem with this not matching all messages from my vmware logs:

The below matches:

Jan 15 18:38:57 HOSTNAME Hostd: [6BBC3B90 info 'DvsTracker'] FetchUplinkDVPortgroups: added 24 items

The below does not match:

Jan 15 18:40:18 HOSTNAME Vpxa: [FFE3EB90 verbose 'VpxaHalCnxHostagent' opID=WFU-ac4ab664] [WaitForUpdatesDone] Received callback

Can you spot the problem ?

Sure. 😄

Ok, let's not keep the suspense too long.

The difference why the Hostd: output matches and Vpxa does not is the difference in the content between the first square brackets.

You can try something like this:

(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) %{SYSLOGHOST:esxi_hostname} %{SYSLOGPROG:esxi_program}(\[%{INT:esxi_pid}\])?: (?<messagebody>(?:\[(?<esxi_thread_id>[0-9A-Z]{8,8}) %{DATA:esxi_loglevel} \'%{DATA:esxi_service}\'(\s.*)?\] %{GREEDYDATA:esxi_message}|%{GREEDYDATA}))

This ESXi logging is quite a mess indeed. Every service seems to have an entirely different format.
That's why this attempt at capturing the output contains the "OR %GREEDYDATA" at the end in case the quite detailed filter starting at "messagebody" does not match, which it will not for Vpxa.

In any case I believe you should get acquainted with grok (http://www.logstash.net/docs/1.4.2/filters/grok) and also regular expressions.