-
-
Save vincenewell/8e53114263c7868fc2c6631340f99b2a to your computer and use it in GitHub Desktop.
[filebeat 8.5.3] extracting log_level from message with dissect processor, failing Right padding modifier
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# filebeat version | |
filebeat version 8.5.3 (amd64), libbeat 8.5.3 [6d03209df870c63ef9d59d609268c11dfdc835dd built 2022-12-04 04:51:48 +0000 UTC |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
filebeat.inputs: | |
- type: filestream | |
id: app-services | |
enabled: true | |
paths: | |
- /opt/tomcat/logs/app-services.log | |
tags: ["tomcat", "app"] | |
parsers: | |
- multiline: | |
# Any line that starts with these contents should be considered the start of a new log entry | |
type: pattern | |
pattern: '^I:' | |
negate: true | |
match: after | |
processors: | |
- dissect: | |
target_prefix: "" | |
tokenizer: "I:%{log_level->} [%{thread}] %{class} %{log_time}: " | |
field: "message" | |
ignore_failure: true | |
- timestamp: | |
# Processing to set the @timestamp field based on the log. | |
field: log_time | |
ignore_missing: true | |
ignore_failure: true | |
layouts: | |
# Define the layout using reference time = 1136239445 | |
# https://pkg.go.dev/time | |
- '02 Jan 2006 15:04:05.000' | |
test: | |
# This section just allows us to copy paste timestamps | |
# from the log and validate one of the layouts can parse | |
# the timestamp from the test timestamp. | |
- '20 Dec 2022 00:00:40.584' | |
timezone: Local | |
- drop_fields: | |
fields: | |
- log_time | |
filebeat.config.modules: | |
path: ${path.config}/modules.d/*.yml | |
reload.enabled: false | |
output.logstash: | |
hosts: ["localhost:5044"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
> db.filebeat.findOne({"log.file.path": "/opt/tomcat/logs/app-services.log", "log_level": "DEBUG " }) | |
{ | |
"_id" : ObjectId("63a1fe694244c574f0000005"), | |
"@timestamp" : ISODate("2022-12-20T18:25:00.929Z"), | |
"log" : { | |
"file" : { | |
"path" : "/opt/tomcat/logs/app-services.log" | |
}, | |
"offset" : 11123820 | |
}, | |
"message" : "I:DEBUG [appTaskScheduler-8] CustomHeaderClientFilter 20 Dec 2022 13:25:00.929: Added header X-Requested-With:XMLHttpRequest to request context org.apache.cxf.jaxrs.client.spec.ClientRequestContextImpl@53578c30", | |
"class" : "CustomHeaderClientFilter", | |
"log_level" : "DEBUG ", | |
"@version" : "1", | |
"thread" : "appTaskScheduler-8", | |
"tags" : [ | |
"tomcat", | |
"app" | |
] | |
} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment