guide_setting_up cloudwatch_agents_inside_no_internet_environments.md

A guide to setting up cloudwatch agents in AWS if you do not have internet access

1. Adds the needed endpoints:

The folowing endpoints must be created inside you AWS VPC

ssm.region.amazonaws.com
ssmmessages.region.amazonaws.com
ec2messages.region.amazonaws.com
logs.region.amazonaws.com

https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-create-vpc.html

If you use the terraform-aws-modules/vpc/aws module for vpcs you can enable these endpoints with the folowing Note: enable_ssmmessages_endpoint not strictly required (it allows remote ssh acess via the gui)

 //allows ssm to manadge ec2 instances
  enable_ssm_endpoint              = true
  ssm_endpoint_private_dns_enabled = true
  ssm_endpoint_security_group_ids  = [aws_security_group.ssm.id] # should allow port 443 https ingress
  //allows ssh sesshon from ssm
  enable_ssmmessages_endpoint              = true
  ssmmessages_endpoint_private_dns_enabled = true
  ssmmessages_endpoint_security_group_ids  = [aws_security_group.ssm.id]
  //allows run command from ssm
  enable_ec2messages_endpoint              = true
  ec2messages_endpoint_private_dns_enabled = true
  ec2messages_endpoint_security_group_ids  = [aws_security_group.ssm.id]
  //allows acess to cloudwatch
  enable_logs_endpoint = true
  logs_endpoint_private_dns_enabled = true
  logs_endpoint_security_group_ids = [aws_security_group.ssm.id]

2. Create a config for cloud watch

Here is an example of config to monitor a test log file this should be saved into parameter store

{
	"agent": {
		"metrics_collection_interval": 60,
		"run_as_user": "cwagent"
	},
	"logs": {
		"logs_collected": {
			"files": {
				"collect_list": [
					{
						"file_path": "/tmp/testlog.txt",
						"log_group_name": "testlog.txt",
						"log_stream_name": "{instance_id}"
					}
				]
			}
		}
	},
	"metrics": {
		"append_dimensions": {
			"AutoScalingGroupName": "${aws:AutoScalingGroupName}",
			"ImageId": "${aws:ImageId}",
			"InstanceId": "${aws:InstanceId}",
			"InstanceType": "${aws:InstanceType}"
		},
		"metrics_collected": {
			"disk": {
				"measurement": [
					"used_percent"
				],
				"metrics_collection_interval": 60,
				"resources": [
					"*"
				]
			},
			"mem": {
				"measurement": [
					"mem_used_percent"
				],
				"metrics_collection_interval": 60
			}
		}
	}
}

A wisard on the ec2 instance can also generate these configs wiht user input: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file-wizard.html

3. Permissions

The ec2 instances need to be attached to a role with the following AWS managed policies:

• CloudWatchAgentServerPolicy
• AmazonSSMManagedInstanceCore
• AmazonSSMDirectoryServiceAccess

4. Start cloudwatch agent

once all the flowing has been done there are ssm cmds to run to install and configure the cloud watch agent on the instance.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-EC2-Instance-fleet.html

A better solution may be to install the agent during the build of amis and the configure step as a user_data bootstrap step

A recurring configure step could be secdualed so that changes to config are pushed to ec2 instances automatically (else the change will have to be run manually)