Get CloudWatch logs from AWS ECS instance and format to be easier to read the timestamps

README.md

Get CloudWatch logs from AWS

With easy to read format

CLI Package to fetch CloudWatch logs and format them with human readable datetimes

Setup

• Download all files in this gist
• Run $ yarn
• Run $ chmod +x ./get-logs.sh
• Install AWS CLI
• Log in to AWS CLI by runnin aws configure

Usage

The bash script was written to be run from terminal with the following parameters:

./get-logs.sh <log-group-name> <log-stream-name> <epoch-time-integer> <output-file>

Example:

./get-logs.sh /ecs/group dev-group-ecs-instance/dev-group-ecs-instance/a77f726e-938a-4c5b-894a-fb3df7e5d98b 1565100257000 my-output-log.txt

Getting epoch time integer

You can get the start time string from CloudWatch url, for example 2019-08-06T14:04:17Z, then simply go to momentjs.com and run this from chrome dev tools:

moment('2019-08-06T14:04:17Z').valueOf()

fix-log-format.js

const fs = require('fs');
const moment = require('moment');

var contents = fs.readFileSync('/tmp/ecs.log');

fs.writeFileSync(
  process.argv[3], 
  contents.toString()
    .replace(/b\/\d+?	f\/\d+?\n/, '')
    .replace(new RegExp(`	${process.argv[2]}\\d{9}\\n`, 'g'),'')
    .replace(new RegExp(`${process.argv[2]}\\d{9}`, 'g'), (i) => `[${moment(parseInt(i)).format('YYYY-MM-DD HH:mm:ss')}]`).replace(/EVENTS\s+?/g,'')
);

get-log.sh

# RUN:
#
# ./get-logs.sh /ecs/group dev-group-ecs-instance/dev-group-ecs-instance/a77f726e-938a-4c5b-894a-fb3df7e5d98b 1565100257000 my-output-log.txt
#
aws logs get-log-events \
   --start-time $3 \
   --log-group-name "$1" \
   --log-stream-name "$2" \
   --output text > /tmp/ecs.log

node fix-log-format.js `echo $3 | awk '{print substr($0,0,4)}'` $4

package.json

{
  "name": "log-getter",
  "version": "1.0.0",
  "main": "get-logs.sh",
  "license": "MIT",
  "dependencies": {
    "moment": "^2.24.0"
  }
}