Using AWS CLI to query CloudWatch Logs with Insights. Looking -30 mins to now.

query-aws-logs-insights.bash

# this script queries aws logs with insights filtering on ERROR

# explanation of start and end times
#--start-time = unix timestamp 30 mins in the past
#--end-time = unix timestamp now

QUERY_ID=$(aws logs start-query \
 --profile $profile \
 --log-group-name /aws/lambda/aap-event-consumer-dev \
 --start-time `date -v-30M "+%s"` \
 --end-time `date "+%s"` \
 --query-string 'fields @message filter @message like /ERROR/' \
 | jq -r '.queryId')
 
echo "Query started (query id: $QUERY_ID), please hold ..." && sleep 5 # give it some time to query

aws --profile $profile logs get-query-results --query-id $QUERY_ID

This will return a query id. Use that to pull the actually logs like so:

 aws --profile clientProfile logs get-query-results --query-id <query-id>

Example of querying error level:

aws logs start-query \
 --profile clientProfile \
 --log-group-name MY-LOG_GROUP \
 --start-time `date -v-30M "+%s"` \
 --end-time `date "+%s"` \
 --query-string 'fields @message | filter @message like /\[error\]/'

@dsmrt

aws logs start-query \
 --profile clientProfile \
 --log-group-name MY-LOG_GROUP \
 --start-time `date -d -30minutes +%s` \
 --end-time `date +%s` \
 --query-string 'fields @message | filter @message like /\[error\]/'

Got this error when try the script

aws: error: argument --start-time: invalid int value: 'date -v-30M "+%s"'

I'm using this awscli version aws-cli/2.0.26 Python/3.7.3 Linux/4.14.181-140.257.amzn2.x86_64 botocore/2.0.0dev30

👋 @HieronyM This works my Mac: date -d -30minutes +%s . I believe it's bsd version of date. You may want to verify that works. If not, use another way to convert the last 30 mins into a unix timestamp.

I updated the gist to start and get query.

Thanks @dsmrt ,

it works now, I think I have some typos previously.
Btw I'm wondering, did you ever try to export the query result to S3?

jq can be replaced with jmespath query

--query queryId

QUERY_ID=$(aws logs start-query \
 --profile $profile \
 --log-group-name /aws/lambda/aap-event-consumer-dev \
 --start-time `date -v-30M "+%s"` \
 --end-time `date "+%s"` \
 --query-string 'fields @message filter @message like /ERROR/' --query queryId

Also, I had to change query-string

from

'fields @message filter @message like /ERROR/'

to

'fields @message | filter level like "error"'