In this example, we are using Centos7, journalctl and systemctl so that we can monitor logs from DC/OS instances (masters, agents and public agents). It is useful for anyone using systemd, journald in an AWS EC2 enviroment that wants logging. The nice thing about Amazon CloudWatch is that it integrates well with Amazon EMR and Amazon Elasticsearch. (For more background on this subject see this article which covers using CloudFormation, Packr, etc. for Immutable Infrastructure to build DC/OS and deploy it to Amazon Web Services.)
We will install journald-cloudwatch-logs. We are going to setup a daemon into systemd that forwards logs to Amazon CloudWatch log streams.
This utility journald-cloudwatch-logs monitors the systemd journal, managed by journald, and writes journal entries into AWS Cloudwatch Logs.
This program is an alternative to the AWS-provided logs agent which works only with sending text log files into AWS Cloudwatch. Conversely the journald-cloudwatch-logs utility reads directly from the systemd journal.
Step 1) Create an IAM role to start your DC/OS instances.
Step 2) Create an IAM policy as follows and associate it with your the DC/OS IAM role.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams"
],
"Resource": [
"arn:aws:logs:*:*:log-group:*",
"arn:aws:logs:*:*:log-group:*:log-stream:*"
]
}
]
}Step 3) Download and Install the agent software and setup config files
mkdir /tmp/logagent
cd /tmp/logagent
curl -OL https://github.com/saymedia/journald-cloudwatch-logs/releases/download/v0.0.1/journald-cloudwatch-logs-linux.zip
unzip journald-cloudwatch-logs-linux.zip
sudo mv journald-cloudwatch-logs/journald-cloudwatch-logs /usr/bin
sudo mkdir -p /var/lib/journald-cloudwatch-logs/
sudo mv /home/centos/install/journald-cloudwatch.conf /etc/
sudo mv /home/centos/install/journald.unit /etc/systemd/system/journald-cloudwatch.service
sudo chmod 664 /etc/systemd/system/journald-cloudwatch.service
sudo chown -R centos /var/lib/journald-cloudwatch-logs/
sudo systemctl enable journald-cloudwatch.service[Unit]
Description=journald-cloudwatch-logs
Wants=basic.target
After=basic.target network.target
[Service]
User=centos
Group=centos
ExecStart=/usr/bin/journald-cloudwatch-logs /etc/journald-cloudwatch.conf
KillMode=process
Restart=on-failure
RestartSec=42s
[Install]
WantedBy=getty.target
log_group = "dcos-logstream"
state_file = "/var/lib/journald-cloudwatch-logs/state"
log_priority = "WARNING"
We created packer file to automate the creation of our base AMI image for DC/OS.
{
"variables": {
"aws_access_key": "",
"aws_secret_key": "",
"aws_region": "us-east-1",
"aws_ami_image": "ami-6d1c2007",
"aws_instance_type": "m4.large"
},
"builders": [
{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "{{user `aws_region`}}",
"source_ami": "{{user `aws_ami_image`}}",
"instance_type": "{{user `aws_instance_type`}}",
"ssh_username": "centos",
"ami_name": "base-centos-7-dcos-{{timestamp}}",
"tags": {
"Name": "ami-centos7-dcos-v1",
"OS_Version": "LinuxCentOs7",
"Release": "7",
"Description": "Base CentOs7 image with prerequisites for DC/OS"
},
"user_data_file": "files/user-data.txt"
}
],
"provisioners": [
{
"type": "file",
"source": "files/overlay.conf",
"destination": "/home/centos/install/overlay.conf"
},
{
"type": "shell",
"inline": [
"sudo mkdir -p /etc/modules-load.d",
"sudo mv /home/centos/install/overlay.conf /etc/modules-load.d/overlay.conf",
"sudo reboot"
]
},
{
"type": "file",
"source": "files/docker.repo",
"destination": "/home/centos/install/docker.repo"
},
{
"type": "file",
"source": "files/override.conf",
"destination": "/home/centos/install/override.conf"
},
{
"type": "shell",
"inline": [
"echo installing Docker -----------------------------------",
"sudo mkdir -p /etc/systemd/system/docker.service.d",
"sudo mv /home/centos/install/override.conf /etc/systemd/system/docker.service.d/override.conf",
"sudo mv /home/centos/install/docker.repo /etc/yum.repos.d/docker.repo",
"sudo yum install -y docker-engine-1.11.2",
"sudo systemctl start docker",
"sudo systemctl enable docker",
"sudo docker ps",
"echo DONE installing Docker -----------------------------",
"sudo yum install -y tar xz unzip curl ipset nano tree",
"sudo sed -i s/SELINUX=enforcing/SELINUX=permissive/g /etc/selinux/config",
"sudo groupadd nogroup",
"sudo reboot"
]
},
{
"type": "shell",
"inline": [
"echo installing aws cli -------------------------------",
"mkdir /tmp/awscli",
"cd /tmp/awscli",
"curl https://s3.amazonaws.com/aws-cli/awscli-bundle.zip -o awscli-bundle.zip",
"unzip awscli-bundle.zip",
"sudo ./awscli-bundle/install -i /usr/lib/aws -b /usr/bin/aws",
"echo DONE installing aws cli -------------------------------"
]
},
{
"type": "file",
"source": "files/journald-cloudwatch.conf",
"destination": "/home/centos/install/journald-cloudwatch.conf"
},
{
"type": "file",
"source": "files/journald.unit",
"destination": "/home/centos/install/journald.unit"
},
{
"type": "shell",
"inline": [
"echo install log agent -------------------------------",
"mkdir /tmp/logagent",
"cd /tmp/logagent",
"curl -OL https://github.com/saymedia/journald-cloudwatch-logs/releases/download/v0.0.1/journald-cloudwatch-logs-linux.zip",
"unzip journald-cloudwatch-logs-linux.zip",
"sudo mv journald-cloudwatch-logs/journald-cloudwatch-logs /usr/bin",
"sudo mkdir -p /var/lib/journald-cloudwatch-logs/",
"sudo mv /home/centos/install/journald-cloudwatch.conf /etc/",
"sudo mv /home/centos/install/journald.unit /etc/systemd/system/journald-cloudwatch.service",
"sudo chmod 664 /etc/systemd/system/journald-cloudwatch.service",
"sudo chown -R centos /var/lib/journald-cloudwatch-logs/",
"sudo systemctl enable journald-cloudwatch.service",
"echo DONE installing log agent -------------------------------"
]
},
{
"type": "shell",
"inline": [
"echo DONE installing packages for CentOS7 DC/OS"
]
}
]
}