Skip to content

Instantly share code, notes, and snippets.

@emmanuelnk

emmanuelnk/mwaa_cloudwatch_get_worker_count.md

Created April 21, 2023 17:00
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save emmanuelnk/9a9bb7dcb4748de7ff6994a129674a53 to your computer and use it in GitHub Desktop.
Save emmanuelnk/9a9bb7dcb4748de7ff6994a129674a53 to your computer and use it in GitHub Desktop.
Download ZIP
Managed Worklows for Apache Airflow get worker Count from Cloudwatch Insights
Raw
mwaa_cloudwatch_get_worker_count.md

Cloudwatch does not provide metrics on how many workers you are running in a MWAA Environment. Fortunately we can get this information from Cloudwatch Insights using the MWAA_ENVIRONMENT_NAME-Worker log group. Every log stream has the ip address of the worker in the @logStream field

  1. Open the relevant MWAA_ENVIRONMENT_NAME-Workerlog group
  2. Open in CloudWatch Insights and paste and run the following query for your desired time range.
  3. you can then add the visualization to your desired Cloudwatch dashboard
fields @timestamp
| parse @logStream "*.*.*.*.*" as 
workerIpAddress, 
workerService, 
workerLogCreationTimestampSeconds, 
workerLogCreationTimestampMicroSeconds, 
WorkerLogFileExtension 
| stats count_distinct(workerIpAddress) as workerCount by bin(1h)

Sample screenshot

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment