Export logs from CloudWatch Logs to S3 with AWS Lambda

aws_lambda_cloudwatch_logs_exporter.md

event example

TASK_NAME=${1:-'export_logs'}
FROM=${2:-'2017-12-24 12:00'}
TO=${3:-'2017-12-24 12:05'}
cat <<-EOF > event.json
{
    "taskName": "$TASK_NAME",
    "fromTime": `gdate --date="$FROM" +%s%3N`,
    "toTime": `gdate --date="$TO" +%s%3N`
}
EOF

Check the result

$ aws logs describe-export-tasks --status-code "COMPLETED"
$ aws s3 ls [S3_BUCKET_NAME]/[S3_DESTINATION_NAME]/[TASK_ID]/ --recursive

aws_lambda_cloudwatch_logs_exporter.py

import json
import os

import boto3

LOG_GROUP_NAME = os.environ['LOG_GROUP_NAME']
S3_BUCKET_NAME = os.environ['S3_BUCKET_NAME']
S3_DESTINATION_NAME = os.environ['S3_DESTINATION_NAME']

def lambda_handler(event, context):
    print('Received event: {json.dumps(event, indent=4)}')
    task_name = event['taskName']
    fromTime = event['fromTime']
    toTime = event['toTime']

    client = boto3.client('logs')
    response = client.create_export_task(
        taskName=task_name,
        logGroupName=LOG_GROUP_NAME,
        fromTime=fromTime,
        to=toTime,
        destination=S3_BUCKET_NAME,
        destinationPrefix=S3_DESTINATION_NAME)

    return response