I wanted to send the metrics data of an application to the prometheus and grafana.
The application is generating logs in plain text. To send meaningful data to prometheus and grafana it is important to parse them.
Below is the log content of the application and saved them into opcron.txt file
2020-03-30 06:20:43,012 [INFO ] [test.py:136] Number of Files identified in the directory = 3
2020-03-30 06:20:43,012 [INFO ] [test.py:337] Number of Files from the directory present in object storage = 3
2020-03-30 06:20:43,012 [INFO ] [test.py:438] Number of Files successfully uploaded = 0
2020-03-30 06:20:43,013 [INFO ] [test.py:909] Number of Files upload failed for = 0
Create a shell script to parse these logs and convert them into JSON format.\
cat > generatejson.sh
awk '{print "{\"cron_stats\": { \"completion_time\": \""$1" " $2"\", \"Task\": "}' opcron.txt > file1.txt
cat opcron.txt | cut -d']' -f3 | cut -d' ' -f2-15 | cut -d'=' -f1 > file2.txt
cat opcron.txt | cut -d']' -f3 | cut -d' ' -f2-15 | cut -d'=' -f2 > file3.txt
sed -i -e 's/[[:blank:]]*$//g' -e 's/\ /_/g' -e 's/^/"/g' -e 's/$/"/g' file2.txt
sed -i -e 's/\ / "/g' -e 's/$/"/g' -e 's/^/, "value"\:/g' -e 's/$/} }/' file3.txt
i=1
END=$(wc -l file1.txt | cut -d' ' -f1)
truncate -s 0 cronstat.txt
for i in $(seq 1 $END); do var1=$(sed -n ""$i"p" file1.txt); var2=$(sed -n ""$i"p" file2.txt); var3=$(sed -n ""$i"p" file3.txt); var4="$var1$var2$var3"; i=$(expr $i + 1); echo $var4 | tee -a cronstat.txt; done
# Remove all tmp file
rm -rf file1.txt file2.txt file3.txt