Logstash logging with lambdas

lambda_logging.md

• Its a pain to log events with a lambda, given that there is no server to run the agents (splunk, filebeat, etc.) on. Here is a quick and easy way to set up ELK logging by writing directly to logstash via TCP.

• Make sure the lambda is running in the right Subnet and has the right Security Group to be able to talk to Logstash server:port.

pom.xml

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>4.11</version>
</dependency>
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.2.3</version>
</dependency>

logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
  <!-- Replace the destination with the logstash server: logstash port -->
        <destination>localhost:5000</destination>
  <!-- There can be multiple destinations -->
        <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
    </appender>

    <root level="DEBUG">
        <appender-ref ref="stash" />
    </root>
</configuration>

finally block

Make sure the code closes the logging context before the JVM dies.

private final Logger logger = LoggerFactory.getLogger(this.getClass());
try {
    logger.info("This is an info");
    logger.debug("This is a debug");
    logger.trace("This is a trace");
    logger.error("This is an error", new RuntimeException("Runtime Exception!"));
    logger.warn("This is a warning");
}finally{
    LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();
    loggerContext.stop();
}

For instance, for WARN, it will produce output -

@timestamp:October 11th 2017, 21:09:34.941 port:33,292 @version:1 host:172.19.0.1 message:{"@timestamp":"2017-10-
11T21:09:20.444-07:00","@version":1,"message":"This is a warning", 
"logger_name":"com.foo.serverless.logging.Test","thread_name":"main","level":"WARN","level_value":30000} 
_id:AV8OxfPrVuC4n7F7pVhY _type:logs _index:logstash-2017.10.12 _score:

You will notice the message within a message. The solution to this is discussed here.

To fix this, locate $LOGSTASH_HOME/pipeline/logstash.conf and change the input to include the json_lines codec.

input {
        tcp {
                port => 5000
                codec => json_lines
        }
}

Restart logstash, and this time the output will be -

@timestamp:October 11th 2017, 23:04:16.845 level:WARN port:33,448 thread_name:main level_value:30000 @version:1 
host:172.19.0.1 logger_name:com.foo.serverless.logging.Test message:This is a warning _id:AV8PL05oe2YvJOO3IrZ1 
_type:logs _index:logstash-2017.10.12 _score: -

Testing with ELK

Install the ELK stack on Docker locally via https://github.com/deviantony/docker-elk and create the default index. It will listen on port 5000 for LogStash.