Openshift EFK Multiline concat for Java / Springboot apps

ocp-efk-multiline.md

Introduction

Openshift 3.11 EFK comes with fluentd-concat plugin and therefore multiline support for stacktrace-merging can be configured as shown below.

Modify config-map logging-fluentd

Open the config-map logging-fluentd in openshift-logging project. In the middle you'll see the label @INGRESS, modify/split it into two labels.

original label @INGRESS:

....
<label @INGRESS>
## filters
  @include configs.d/openshift/filter-pre-*.conf
  @include configs.d/openshift/filter-retag-journal.conf
  @include configs.d/openshift/filter-k8s-meta.conf
  @include configs.d/openshift/filter-kibana-transform.conf
  @include configs.d/openshift/filter-k8s-flatten-hash.conf
  @include configs.d/openshift/filter-k8s-record-transform.conf
  @include configs.d/openshift/filter-syslog-record-transform.conf
  @include configs.d/openshift/filter-viaq-data-model.conf
  @include configs.d/openshift/filter-post-*.conf
##
</label>
...

New modified and splitted label:

...
<label @INGRESS>
## filters
  @include configs.d/openshift/filter-pre-*.conf
  @include configs.d/openshift/filter-retag-journal.conf
  @include configs.d/openshift/filter-k8s-meta.conf
  <filter **>
    @type concat
    key log
    multiline_start_regexp /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}/
    timeout_label @INGRESSCONTINUE
    flush_interval 3s
  </filter>
  <match **>
    @type relabel
    @label @INGRESSCONTINUE
  </match>  
</label>
  
<label @INGRESSCONTINUE>
  @include configs.d/openshift/filter-kibana-transform.conf
  @include configs.d/openshift/filter-k8s-flatten-hash.conf
  @include configs.d/openshift/filter-k8s-record-transform.conf
  @include configs.d/openshift/filter-syslog-record-transform.conf
  @include configs.d/openshift/filter-viaq-data-model.conf
  @include configs.d/openshift/filter-post-*.conf
##
</label>
....

This change introduces the concat filter to look for loglines matching "YYYY-MM-DD HH:MM:SS.sss" and looking for following lines not starting with this pattern and therefor merging it together into one single message for ES/Kibana.

Complete new config-map key fluent.conf:

# This file is the fluentd configuration entrypoint. Edit with care.

@include configs.d/openshift/system.conf
#<system>
#  log_level trace
#</system>

# In each section below, pre- and post- includes don't include anything initially;
# they exist to enable future additions to openshift conf as needed.

## sources
## ordered so that syslog always runs last...
@include configs.d/openshift/input-pre-*.conf
@include configs.d/dynamic/input-docker-*.conf
@include configs.d/dynamic/input-syslog-*.conf
@include configs.d/openshift/input-post-*.conf
##

<label @INGRESS>
## filters
  @include configs.d/openshift/filter-pre-*.conf
  @include configs.d/openshift/filter-retag-journal.conf
  @include configs.d/openshift/filter-k8s-meta.conf
  <filter **>
    @type concat
    key log
    multiline_start_regexp /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}/
    timeout_label @INGRESSCONTINUE
    flush_interval 3s
  </filter>
  <match **>
    @type relabel
    @label @INGRESSCONTINUE
  </match>  
</label>
  
<label @INGRESSCONTINUE>
  @include configs.d/openshift/filter-kibana-transform.conf
  @include configs.d/openshift/filter-k8s-flatten-hash.conf
  @include configs.d/openshift/filter-k8s-record-transform.conf
  @include configs.d/openshift/filter-syslog-record-transform.conf
  @include configs.d/openshift/filter-viaq-data-model.conf
  @include configs.d/openshift/filter-post-*.conf
##
</label>

<label @OUTPUT>
## matches
  @include configs.d/openshift/output-pre-*.conf
  @include configs.d/openshift/output-operations.conf
  @include configs.d/openshift/output-applications.conf
  # no post - applications.conf matches everything left
##
</label>

Have fun with stacktraces merged together into one message for EFK! :)