Skip to content

Instantly share code, notes, and snippets.

@vatshat
Last active September 18, 2019 09:24
Show Gist options
  • Save vatshat/503a2f42e0bde192d80a41050b587350 to your computer and use it in GitHub Desktop.
Save vatshat/503a2f42e0bde192d80a41050b587350 to your computer and use it in GitHub Desktop.
Example of how to upload 10 multiple JSON log events once every second
#!/bin/bash
a=0
while [ $a -lt 20 ]
do
now=$(TZ='UTC' date +%s%3N)
message=$(cat << EndOfMessage
{
"timeMillis": $now,
"thread": "ReqProc-StageThread[3]/bcuNSY6xbNk7ADiQiG0KrA",
"level": "INFO",
"loggerName": "engineaudit",
"message": "PA Audit with data ",
"endOfBatch": true,
"loggerFqcn": "org.apache.logging.slf4j.Log4jLogger",
"contextMap": {
"AUDIT.duration": "1552467944801",
"AUDIT.host": "ip-10-211-10-56.eu-west-1.compute.internal",
"AUDIT.method": "POST",
"AUDIT.pathPrefix": "/*",
"AUDIT.pathPrefixType": "Any",
"AUDIT.proxyRoundTripMS": "130",
"AUDIT.reqReceivedMillisec": "1552467944643",
"AUDIT.reqSentMillisec": "1552467944671",
"AUDIT.requestUri": "/nwb/as/token.oauth2",
"AUDIT.resource": "PF-RUNTIME",
"AUDIT.respReceivedMillisec": "1552467944801",
"AUDIT.respSentMillisec": "1552467944802",
"AUDIT.responseCode": "200",
"AUDIT.roundTripMS": "159",
"AUDIT.targetHost": "dplsp2test.web.rbsgrp.ppe:6020",
"AUDIT.userInfoReqSentMillisec": "0",
"AUDIT.userInfoRespReceivedMillisec": "0",
"AUDIT.userInfoRoundTripMS": "0",
"exchangeId": "bcuNSY6xbNk7ADiQiG0KrA"
},
"threadId": 11,
"threadPriority": 5,
"brand": "nwb",
"nodeType": "Runtime"
}
EndOfMessage
)
message=`echo $message | jq -c . | jq --raw-input .`
json=$(cat << EndOfMessage
[
{
"timestamp": $now,
"message": $message
},
{
"timestamp": $now,
"message": "Member public test InternalPartitionService"
}
]
EndOfMessage
)
echo $json
a=`expr $a + 1`
sequenceToken=$(aws logs describe-log-streams --log-group-name thabile-support --log-stream-name-prefix thabile-support | jq --raw-output ".logStreams[0].uploadSequenceToken")
if [[ $sequenceToken == "null" ]]
then
aws logs put-log-events --log-group-name thabile-support --log-stream-name thabile-support --log-events "$json"
else
aws logs put-log-events --log-group-name thabile-support --log-stream-name thabile-support --log-events "$json" --sequence-token $sequenceToken
fi
sleep 1s
done
@vatshat
Copy link
Author

vatshat commented Sep 18, 2019

It's important to note that this method uses DescribeLogStream which is not best practice, as it has very low limit thresholds. For correct way of doing it refer to this https://gist.github.com/vatshat/63cad6f15082ec06851bb40eb0394931

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment