Skip to content

Instantly share code, notes, and snippets.

@vincenewell
Created December 20, 2022 18:49
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save vincenewell/8e53114263c7868fc2c6631340f99b2a to your computer and use it in GitHub Desktop.
Save vincenewell/8e53114263c7868fc2c6631340f99b2a to your computer and use it in GitHub Desktop.
[filebeat 8.5.3] extracting log_level from message with dissect processor, failing Right padding modifier
# filebeat version
filebeat version 8.5.3 (amd64), libbeat 8.5.3 [6d03209df870c63ef9d59d609268c11dfdc835dd built 2022-12-04 04:51:48 +0000 UTC
filebeat.inputs:
- type: filestream
id: app-services
enabled: true
paths:
- /opt/tomcat/logs/app-services.log
tags: ["tomcat", "app"]
parsers:
- multiline:
# Any line that starts with these contents should be considered the start of a new log entry
type: pattern
pattern: '^I:'
negate: true
match: after
processors:
- dissect:
target_prefix: ""
tokenizer: "I:%{log_level->} [%{thread}] %{class} %{log_time}: "
field: "message"
ignore_failure: true
- timestamp:
# Processing to set the @timestamp field based on the log.
field: log_time
ignore_missing: true
ignore_failure: true
layouts:
# Define the layout using reference time = 1136239445
# https://pkg.go.dev/time
- '02 Jan 2006 15:04:05.000'
test:
# This section just allows us to copy paste timestamps
# from the log and validate one of the layouts can parse
# the timestamp from the test timestamp.
- '20 Dec 2022 00:00:40.584'
timezone: Local
- drop_fields:
fields:
- log_time
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
output.logstash:
hosts: ["localhost:5044"]
> db.filebeat.findOne({"log.file.path": "/opt/tomcat/logs/app-services.log", "log_level": "DEBUG " })
{
"_id" : ObjectId("63a1fe694244c574f0000005"),
"@timestamp" : ISODate("2022-12-20T18:25:00.929Z"),
"log" : {
"file" : {
"path" : "/opt/tomcat/logs/app-services.log"
},
"offset" : 11123820
},
"message" : "I:DEBUG [appTaskScheduler-8] CustomHeaderClientFilter 20 Dec 2022 13:25:00.929: Added header X-Requested-With:XMLHttpRequest to request context org.apache.cxf.jaxrs.client.spec.ClientRequestContextImpl@53578c30",
"class" : "CustomHeaderClientFilter",
"log_level" : "DEBUG ",
"@version" : "1",
"thread" : "appTaskScheduler-8",
"tags" : [
"tomcat",
"app"
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment