-
-
Save omardulaimi/2e5f8468c063e584c072336df32c74e2 to your computer and use it in GitHub Desktop.
import boto3 | |
import os | |
import datetime | |
""" | |
This portion will obtain the Environment variables from AWS Lambda. | |
""" | |
GROUP_NAME = os.environ['GROUP_NAME'] | |
DESTINATION_BUCKET = os.environ['DESTINATION_BUCKET'] | |
PREFIX = os.environ['PREFIX'] | |
NDAYS = os.environ['NDAYS'] | |
nDays = int(NDAYS) | |
""" | |
This portion will receive the nDays value (the date/day of the log you want | |
want to export) and calculate the start and end date of logs you want to | |
export to S3. Today = 0; yesterday = 1; so on and so forth... | |
Ex: If today is April 13th and NDAYS = 0, April 13th logs will be exported. | |
Ex: If today is April 13th and NDAYS = 1, April 12th logs will be exported. | |
Ex: If today is April 13th and NDAYS = 2, April 11th logs will be exported. | |
""" | |
currentTime = datetime.datetime.now() | |
StartDate = currentTime - datetime.timedelta(days=nDays) | |
EndDate = currentTime - datetime.timedelta(days=nDays - 1) | |
""" | |
Convert the from & to Dates to milliseconds | |
""" | |
fromDate = int(StartDate.timestamp() * 1000) | |
toDate = int(EndDate.timestamp() * 1000) | |
""" | |
The following will create the subfolders' structure based on year, month, day | |
Ex: BucketNAME/LogGroupName/Year/Month/Day | |
""" | |
BUCKET_PREFIX = os.path.join(PREFIX, StartDate.strftime('%Y{0}%m{0}%d').format(os.path.sep)) | |
""" | |
Based on the AWS boto3 documentation | |
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/logs.html#CloudWatchLogs.Client.create_export_task | |
""" | |
def lambda_handler(event, context): | |
client = boto3.client('logs') | |
client.create_export_task( | |
logGroupName=GROUP_NAME, | |
fromTime=fromDate, | |
to=toDate, | |
destination=DESTINATION_BUCKET, | |
destinationPrefix=BUCKET_PREFIX | |
) |
Thanks for sharing this. I'm curious do we define variables at the beginning? Or, at the end of the file? Also, where do we get identified variable from? GROUP_NAME (Is this arn for log group name?) PREFIX (I assume this is any string?)
Hi PiExpr,
So these will be defined in the Environment Variable Section for the Lambda Function. This article will walk you through
Part 1: https://omardulaimi.medium.com/export-ec2-logs-to-cloudwatch-and-s3-89285029a345
Part 2: https://omardulaimi.medium.com/export-cloudwatch-logs-to-s3-with-lambda-dd45cf246766
Now if you are not using Lambda, then you can define the variables in this section of the code:
GROUP_NAME = "Log_goup1"
DESTINATION_BUCKET = "BucketName"
PREFIX = "Random String, can be anything"
NDAYS = "30"
The prefix if I recall correctly is the name of the bucket, and the Group Name is the log group name in CloudWatch.
Note: this articles might be a bit outdated due to AWS changing their layout/things from time to time.
Thanks for sharing this. I'm curious do we define variables at the beginning? Or, at the end of the file?
Also, where do we get identified variable from?
GROUP_NAME (Is this arn for log group name?)
PREFIX (I assume this is any string?)