Script to dump/download logs from AWS CloudWatch Logs

aws_cwlogs_get_event_logs_full.py

#!/usr/bin/env python3

# TODO - Possible improvements
#
# - Optionally add timestamp (events.timestamp) in a human readable format
#   to the output. Useful if the date/time is not in the message itself.
# - Use filter_log_events instead of get_log_events, accepts a log stream
#   pattern, and add log stream name to the output. Useful when we want to
#   download logs from multiple log streams.
#
# FIXME
#
# - Bugs in `date_str_to_timestamp` logic. Just use timestamps in UTC ending with `Z` to avoid them.

import argparse
from datetime import datetime

import boto3


def date_str_to_timestamp(date_str):
    date_str = date_str.replace("Z", ".000+00:00")
    return int(datetime.fromisoformat(date_str).timestamp()) * 1000


def main(log_group_name, log_stream_name, start_time, end_time, region):
    logs_client = boto3.client("logs", region_name=region)
    previous_token = None
    while True:
        params = {
            "logGroupName": log_group_name,
            "logStreamName": log_stream_name,
            "startFromHead": True
        }
        if start_time:
            params["startTime"] = date_str_to_timestamp(start_time)
        if end_time:
            params["endTime"] = date_str_to_timestamp(end_time)
        if previous_token:
            params["nextToken"] = previous_token
        resp = logs_client.get_log_events(**params)
        next_token = resp['nextForwardToken']

        if next_token == previous_token:
            return

        previous_token = next_token
        for e in resp['events']:
            print(e['message'])


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--log-group-name", required=True)
    parser.add_argument("--log-stream-name", required=True)
    parser.add_argument("--start-time")
    parser.add_argument("--end-time")
    parser.add_argument("--region")
    args = parser.parse_args()
    main(args.log_group_name, args.log_stream_name, args.start_time,
            args.end_time, args.region)

Dump/download logs from AWS CloudWatch Logs

AWS CloudWatch Logs has APIs and CLI, like the GetLogEvents API and the correspondent aws logs get-log-events command, to allow download the logs.

However, they have the limitation of only being able to get up to 1 MB of size or 10K messages. If you need more than that, you must implement a logic following the nextForwardToken until it repeats itself.

The goal of this script is to implement this logic via Python/Boto3, so it'll be easier if someone needs to download them.

Alternative approaches:

• Exporting logs to S3 in CSV is more recommended in cases of huge log volume.
• The project awslogs works very good as may solve multiple use cases (without the need of dumping/downloading the logs)
  • Be aware that I faced a few issues with its --tail parameter (when new log streams are created over time)
• CloudWatch Log Insights is another good feature

Details

• This script accepts ISO 8601 date format, easier then timestamps
  • Be aware that there are lot's of limitations and TZ logic problems in my logic. Just put your date in UTC ending in 'Z' to avoid problems.
  • I may fix this in future, but can't guarantee
• Logs are not downloaded in JSON, but in plaintext lines separated by newlines

Install

https://gist.githubusercontent.com/rarylson/f5037653f425fcbb8a9ecdaa7f488e2f/raw/d681d691348ec6d9f5e558d9e619c79b6945ea18/aws_cwlogs_get_event_logs_full.py
chmod +x aws_cwlogs_get_event_logs_full.py

Usage

Basic usage:

./aws_cwlogs_get_event_logs_full.py --log-group-name MY-LOG-GROUP --log-stream-name MY-LOG-STREAM

More complex usage:

./aws_cwlogs_get_event_logs_full.py --log-group-name MY-LOG-GROUP --log-stream-name MY-LOG-STREAM \
    --start-time '2021-12-01T03:00:00Z'  --end-time '2021-12-10T16:00:00Z' > MY-OUTPUT.log