Skip to content

Instantly share code, notes, and snippets.

@EricVS
Last active April 2, 2019 08:46
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save EricVS/9ddb58840913c765acac6114c0ed81dd to your computer and use it in GitHub Desktop.
Save EricVS/9ddb58840913c765acac6114c0ed81dd to your computer and use it in GitHub Desktop.
log message:
2019-03-28 06:25:29.203 [http-nio-8080-exec-48] INFO LSL_LOG {"timestamp":"2019-03-28T06:25:29.199Z","gametime":"00:00:00.610","ipaddress":"128.199.97.19","uri":"/ps/game/GameContainer.action","params":"[brandId=1,gameId=star,platform=STAR,playForReal=false,isMobile=true]","playerid":"","partyid":""}
parser:
(?<time>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{1,})\s\[(?<threadid>.{1,})\]\sINFO\s.*LSL_LOG\s.*\{"timestamp":"(?<timestamp>.{1,})","gametime":"(?<gametime>.{1,})","ipaddress":"(?<ipaddress>.{1,})","uri":"(?<uri>.{1,})","params":"\[brandId=(?<brandid>\d{1,}),gameId=(?<gameid>.{1,}),platform=(?<platform>.{1,}),playForReal=(?<playforreal>.{1,}),isMobile=(?<ismobile>.{1,})\]","playerid":"(?<playerid>.{0,})","partyid":"(?<partyid>.{0,})"\}
works in Rubular and regex101 but in Elasticsearch after fluent-bit pushes it the entire message is container in 'log'.
{
"_index": "flbit-fg-2019.04.02",
"_type": "flb_type",
"_id": "N6XF22kBr9CA8ewdU10K",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-04-02T01:58:55.092Z",
"log": "2019-03-28 08:34:08.584 [http-nio-8080-exec-23] INFO LSL_LOG {\"timestamp\":\"2019-03-28T08:34:08.582Z\",\"gametime\":\"00:00:00.053\",\"ipaddress\":\"150.95.175.3\",\"uri\":\"/ps/game/GameContainer.action\",\"params\":\"[brandId=1,gameId=star,platform=STAR,playForReal=false,isMobile=true]\",\"playerid\":\"\",\"partyid\":\"\"}"
},
"fields": {
"@timestamp": [
"2019-04-02T01:58:55.092Z"
]
},
"sort": [
1554170335092
]
}
[INPUT]
Name tail
Tag mytag.*
Path /home/centos/omg.log
Parser myparser
Mem_Buf_Limit 50M
Buffer_Chunk_Size 50M
Buffer_Max_Size 100M
Skip_Long_Lines On
Refresh_Interval 10
DB /tmp/fluent-bit.db
[FILTER]
Name grep
Match mytag.*
Regex log /.{1,}LSL_LOG.{1,}/
[OUTPUT]
Name es
Match mytag.*
Host 127.0.0.1
Port 9200
Logstash_Format on
Logstash_Prefix flbit-fg
[PARSER]
Name myparser
Format regex
Regex (?<date>[^ ]*)\s(?<time>[^ ]*)\s\[(?<threadid>.{1,})\]\sINFO\s.*LSL_LOG\s.*\{"timestamp":"(?<timestamp>.{1,})","gametime":"(?<gametime>.{1,})","ipaddress":"(?<ipaddress>.{1,})","uri":"(?<uri>.{1,})","params":"\[brandId=(?<brandid>\d{1,}),gameId=(?<gameid>.{1,}),platform=(?<platform>.{1,}),playForReal=(?<playforreal>.{1,}),isMobile=(?<ismobile>.{1,})\]","playerid":"(?<playerid>.{0,})","partyid":"(?<partyid>.{0,})"\}
Time_Key time
Time_Format %Y-%m-%d %H:%M:%S
Time_Keep Off
[SERVICE]
Flush 1
Log_Level debug
Daemon off
Parsers_file parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
@INCLUDE input.conf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment