Kinesis Agent on Ubuntu 16.04

README.md

Setting Up Kinesis Agent on Ubuntu 16.04

To ease the process, I created a script that automatizes the installation of the Agent. The steps are:

1. Create the configuration file for the agent following these readings:

   • Writing to a Stream
   • Writing to Firehose

   There is a agent.json file that exemplifies the structure.

2. Next, open a shell and execute the following:

   chmod +x setup
   ./setup
   

   By the end of the installation, the shell will show the tailing of the agente logs. It is possible to exist it without exiting the agente (Ctrl+C).

Notes

• It is possible that the following error shows up:

  error: package lombok does not exist
  

  In this case, try to rexecute the agent script again.

agent.json

{
    "checkpointFile": "/tmp/aws-kinesis-agent-checkpoints/main.log",
    "cloudwatch.emitMetrics": true,
    "cloudwatch.endpoint": "https://monitoring.us-west-2.amazonaws.com",
    "kinesis.endpoint": "https://kinesis.us-west-2.amazonaws.com",
    "awsAccessKeyId": "ACCESSKEY",
    "awsSecretAccessKey": "SECRETKEY",
    "flows": [
        {
            "filePattern": "/tmp/aws-kinesis-agent-test1.log*",
            "initialPosition": "END_OF_FILE",
            "kinesisStream": "aws-kinesis-agent-test1",
            "dataProcessingOptions": [
                {
                    "optionName": "CSVTOJSON",
                    "customFieldNames": [
                        "field1",
                        "field2",
                        "field3"
                    ],
                    "delimiter": "\\t"
                }
            ]
        },
        {
            "filePattern": "/tmp/aws-kinesis-agent-test2.log*",
            "initialPosition": "START_OF_FILE",
            "kinesisStream": "aws-kinesis-agent-test2",
            "dataProcessingOptions": [
                {
                    "optionName": "LOGTOJSON",
                    "logFormat": "COMMONAPACHELOG"
                }
            ]
        }
    ]
}

setup

#!/bin/bash

AGENT_VERSION=1.1.4
JAVA_START_HEAP=256m
JAVA_MAX_HEAP=512m
LOG_LEVEL=INFO
WORK_DIR=$PWD

# Installation
sudo apt update
sudo apt-get install -y default-jre default-jdk
curl -LO https://github.com/awslabs/amazon-kinesis-agent/archive/${AGENT_VERSION}.tar.gz
tar xvzf ${AGENT_VERSION}.tar.gz
rm ${AGENT_VERSION}.tar.gz 
mv amazon-kinesis-agent-${AGENT_VERSION} amazon-kinesis-agent
cd amazon-kinesis-agent
sudo ./setup --install

# Some setup
mkdir -p /tmp/kinesis
sudo cp $WORK_DIR/agent.json /etc/aws-kinesis/agent.json

# Starting service and tailing log
sudo service aws-kinesis-agent start
sleep 10 
tail -f /var/log/aws-kinesis-agent/aws-kinesis-agent.log

It should be tar -xvzf ${AGENT_VERSION}.tar.gz