Parse OpenShift logs in Graylog

parse-openshift-logs-in-graylog.md

Problems

• Application logs are mixed with kubernetes logs. They can be searched using a query similar to this one, but it's a bit unwieldy as it's not using Graylog fields (for example, the quotes can't be left out): "container_name=mycontainer"

• The actual log message is prefixed with a bunch of extra information that makes the logs difficult to read in the Graylog UI. For example, the message in Graylog shows up as:

  app1.openshift.example.org output_tag: namespace_name=mynamespace, container_name=myapplication, pod_name=myapplication-7dd79c7bb7-qkfzk, message=Web server listening at: http://0.0.0.0:3000
  

  But the actual log message is:

  Web server listening at: http://0.0.0.0:3000
  

Solution

This rule solves most of this:

• It extracts the fields in the OpenShift log into Graylog fields
• It removes all of the extra information in front of the message
• It also sets application_name based on container_name for more consistent behaviour across application logs

rule "Parse OpenShift logs"
when
    // Replace this with a regex to match your OpenShift servers
    regex(".*app.*\\.openshift\\.example\\.org", to_string($message.source)).matches == true &&
    // Only parse OpenShift logs that contain "namespace_name"
    contains(to_string($message.message), "namespace_name")
then
    let message_regex = regex(".*app.*\\.openshift\\.example\\.org\\soutput_tag:(.*)\\snamespace_name=(.*),\\scontainer_name=(.*),\\spod_name=(.*),\\smessage=(.*)", to_string($message.message));
    set_field("output_tag", message_regex["0"]);
    set_field("namespace_name", message_regex["1"]);
    set_field("container_name", message_regex["2"]);
    set_field("application_name", message_regex["2"]);
    set_field("pod_name", message_regex["3"]);
    // If the message is set to an empty string it will revert to its previous contents, so this is a hack to leave it empty since it can't be deleted without a separate rule
    set_field("message", concat(to_string(message_regex["4"]), "\u00a0"));
end