Skip to content

Instantly share code, notes, and snippets.

@vatshat
Last active January 29, 2020 16:04
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save vatshat/63cad6f15082ec06851bb40eb0394931 to your computer and use it in GitHub Desktop.
Save vatshat/63cad6f15082ec06851bb40eb0394931 to your computer and use it in GitHub Desktop.
Example of how to upload logs to CloudWatch using best practices to avoid throttling
import json
import sys
import boto3
import time
## boto3 client
logs = boto3.client('logs', region_name='eu-west-1')
## Define log group and log stream. Default log group for glue logs and logstream defined dynamically upon job run.
log_group = "my_custom_log_group"
log_stream = "my_custom_log_stream"
## Grab current time in milliseconds since Jan 1, 1970 00:00:00 UTC for use in cloudwatch put_log_events
current_milli_time = lambda: int(round(time.time() * 1000))
## Functions
# Grabs next sequence token
def get_sequence_token():
response = logs.describe_log_streams(
logGroupName = log_group,
logStreamNamePrefix = log_stream
)
print(response)
next_token = response['logStreams'][0]["uploadSequenceToken"]
return next_token
# Actually puts the event
def put_event(msg):
# NOTE: This should only be called from send_msg()
# next_token = get_sequence_token()
response_put = logs.put_log_events(
logGroupName = log_group,
logStreamName = log_stream,
logEvents = [
{
'timestamp' : current_milli_time(),
'message' : msg
}
]
)
print(response_put)
# Provides interface for put_events, change default loglevel here
def send_msg(msg, level="INFO"):
# Generate timestamp for the header
timestamp = time.strftime("%y/%m/%d %H:%M:%S", time.gmtime())
# Customize header here
put_event(timestamp + " " + level + " " + msg)
# example log string/object provided
json_object = {
"message": "My custom message as a serializable Python object"
}
send_msg(json.dumps(json_object))
@vatshat
Copy link
Author

vatshat commented Sep 18, 2019

Instead of using the DescribeLogStream (which has a TPS limit of 5 per account/region) to get sequence token, perform the 1st PutLogEvents (which has a TPS limit of 5 per log stream) without a sequence token in a try except block because, it'll return an error with the expected sequence token in the response error. Then simply use the returned "expected sequence token" when performing the PutLogEvents API call again in your except block

@joshjiang
Copy link

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment