Logstash Logback setup for Mesosphere for debugging distributed microservices

setup_for_logback_logstash.md

Logstash and Logback setup for running in Mesosphere.

.../salt/logstash/files/conf.d/input-json.conf

input {
  udp {
    port => 5000
    codec => "json"
  }
  tcp {
    port => 5000
    codec => "json"
  }
}

Logging setup for Logstash, Logback, when services are running in Mesosphere.

logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <appender name="STASH-UDP" class="net.logstash.logback.appender.LogstashSocketAppender">
        <host>logstash.service.ops-prod-us-west-2.consul</host>
        <port>5000</port>
        <customFields>
            {
                  "serviceName":"employeeService",
                  "serviceHost":"${HOST}",
                  "servicePort":"${PORT0}",
                  "serviceAdminPort":"${PORT1}"

            }
        </customFields>
    </appender>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>
                %magenta(%d{yyyy-MM-dd HH:mm:ss.SSS}) %yellow([%thread]) %highlight(%-5level) %cyan(%logger{15}) - %msg%n
            </pattern>
            <!-- use this pattern to include the caller of the message.  It works like a stacktrace.  Good for debugging, but very slow, so don't use it normally -->
            <!--<pattern>%magenta(%d{yyyy-MM-dd HH:mm:ss.SSS}) %yellow([%thread]) %highlight(%-5level) %cyan(%caller{1}) - %msg %n</pattern>-->
        </encoder>
    </appender>

    <root level="ALL">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="STASH-UDP"/>
    </root>


    <logger name="com" level="INFO"/>
    <logger name="org" level="INFO"/>
    <logger name="com.datastax" level="INFO"/>
    <logger name="io.advantageous" level="INFO"/>
    <logger name="io.vertx" level="INFO"/>

</configuration>

Notice that we configured custom fields to send to Logstash.

        <customFields>
            {
                  "serviceName":"employeeService",
                  "serviceHost":"${HOST}",
                  "servicePort":"${PORT0}",
                  "serviceAdminPort":"${PORT1}"

            }
        </customFields>

The serviceName is important when searching for one service or another. This is important if you have services that use a lot of common base classes. This is so you can decide which message goes with which service. Kibana allows you to filter on fields easily.

The ${HOST} pulls the field out of the environment variable $HOST. See logback config for more details.

The serviceHost allows you to filter which messages go to which location. Again, this can be helpful if debugging and you have many instance of a particular serviceName.

The servicePort not seem obvious at first. You basically use this because a serviceName could be deployed to the same mesos slave, i.e., the same server/HOST. This helps you identify which service is which.

You could add serviceId as in "serviceId":"employeeService-${HOST}-${PORT0}", if you wanted to filter a serviceId quickly.

The QBit microservices lib has support for logging MDC.

You could log per URI and/or requestId or headers, etc.