Last active
April 2, 2019 08:46
-
-
Save EricVS/9ddb58840913c765acac6114c0ed81dd to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
log message: | |
2019-03-28 06:25:29.203 [http-nio-8080-exec-48] INFO LSL_LOG {"timestamp":"2019-03-28T06:25:29.199Z","gametime":"00:00:00.610","ipaddress":"128.199.97.19","uri":"/ps/game/GameContainer.action","params":"[brandId=1,gameId=star,platform=STAR,playForReal=false,isMobile=true]","playerid":"","partyid":""} | |
parser: | |
(?<time>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{1,})\s\[(?<threadid>.{1,})\]\sINFO\s.*LSL_LOG\s.*\{"timestamp":"(?<timestamp>.{1,})","gametime":"(?<gametime>.{1,})","ipaddress":"(?<ipaddress>.{1,})","uri":"(?<uri>.{1,})","params":"\[brandId=(?<brandid>\d{1,}),gameId=(?<gameid>.{1,}),platform=(?<platform>.{1,}),playForReal=(?<playforreal>.{1,}),isMobile=(?<ismobile>.{1,})\]","playerid":"(?<playerid>.{0,})","partyid":"(?<partyid>.{0,})"\} | |
works in Rubular and regex101 but in Elasticsearch after fluent-bit pushes it the entire message is container in 'log'. | |
{ | |
"_index": "flbit-fg-2019.04.02", | |
"_type": "flb_type", | |
"_id": "N6XF22kBr9CA8ewdU10K", | |
"_version": 1, | |
"_score": null, | |
"_source": { | |
"@timestamp": "2019-04-02T01:58:55.092Z", | |
"log": "2019-03-28 08:34:08.584 [http-nio-8080-exec-23] INFO LSL_LOG {\"timestamp\":\"2019-03-28T08:34:08.582Z\",\"gametime\":\"00:00:00.053\",\"ipaddress\":\"150.95.175.3\",\"uri\":\"/ps/game/GameContainer.action\",\"params\":\"[brandId=1,gameId=star,platform=STAR,playForReal=false,isMobile=true]\",\"playerid\":\"\",\"partyid\":\"\"}" | |
}, | |
"fields": { | |
"@timestamp": [ | |
"2019-04-02T01:58:55.092Z" | |
] | |
}, | |
"sort": [ | |
1554170335092 | |
] | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[INPUT] | |
Name tail | |
Tag mytag.* | |
Path /home/centos/omg.log | |
Parser myparser | |
Mem_Buf_Limit 50M | |
Buffer_Chunk_Size 50M | |
Buffer_Max_Size 100M | |
Skip_Long_Lines On | |
Refresh_Interval 10 | |
DB /tmp/fluent-bit.db | |
[FILTER] | |
Name grep | |
Match mytag.* | |
Regex log /.{1,}LSL_LOG.{1,}/ | |
[OUTPUT] | |
Name es | |
Match mytag.* | |
Host 127.0.0.1 | |
Port 9200 | |
Logstash_Format on | |
Logstash_Prefix flbit-fg |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[PARSER] | |
Name myparser | |
Format regex | |
Regex (?<date>[^ ]*)\s(?<time>[^ ]*)\s\[(?<threadid>.{1,})\]\sINFO\s.*LSL_LOG\s.*\{"timestamp":"(?<timestamp>.{1,})","gametime":"(?<gametime>.{1,})","ipaddress":"(?<ipaddress>.{1,})","uri":"(?<uri>.{1,})","params":"\[brandId=(?<brandid>\d{1,}),gameId=(?<gameid>.{1,}),platform=(?<platform>.{1,}),playForReal=(?<playforreal>.{1,}),isMobile=(?<ismobile>.{1,})\]","playerid":"(?<playerid>.{0,})","partyid":"(?<partyid>.{0,})"\} | |
Time_Key time | |
Time_Format %Y-%m-%d %H:%M:%S | |
Time_Keep Off |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[SERVICE] | |
Flush 1 | |
Log_Level debug | |
Daemon off | |
Parsers_file parsers.conf | |
HTTP_Server On | |
HTTP_Listen 0.0.0.0 | |
HTTP_Port 2020 | |
@INCLUDE input.conf |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment