Using Fluentd to convert Kubernetes Jenkins dynamic builder label to save ElasticSearch indexes

gistfile1.txt

I started with the fluentd-elasticsearch docker image at:
  https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

I added to the Gemfile:
  gem 'fluent-plugin-record-modifier', '~>1.1.0'
and built my own docker image.

Then I made a helm chart from the official (but now deprecated) fluentd-elasticsearch helm chart at:
  https://github.com/helm/charts/tree/master/stable/fluentd-elasticsearch

I used the docker image I built above.
I added a values file for my deployment target and overrode these:

---
env:
  OUTPUT_NAME: fluentd
  OUTPUT_TYPE: forward

configMaps:
  system.conf: |-
    <system>
      root_dir /tmp/fluentd-buffers/
    </system>
  #containers.input.conf: ""
  #system.input.conf: ""
  forward.input.conf: ""
  #monitoring.conf: ""
  output.conf: |
    # Enriches records with Kubernetes metadata
    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>

    <filter kubernetes.**>
      @type grep
      <exclude>
        key $.kubernetes.container_name
        pattern ^(calico-node|fluentd-fluentd-elasticsearch|sysdig)$
      </exclude>
    </filter>

    # Jenkins does this dynamic key name which wreaks havoc on ES's auto field indexing.
    # Capture the key name, add it back as the value to a static key, and then delete it.
    <filter kubernetes.var.log.containers.jenkins-slave**>
      @type record_modifier
      remove_keys _dummy_
      <record>
        _dummy_ ${value = ''; record['kubernetes']['labels'].keys.each { |label| if label =~ /^jenkins\/buildpod.*/; value = label; end }; if value != ''; record['jenkins_build'] = value; record['kubernetes']['labels'].delete(value); end; nil}
      </record>
    </filter>

    <match **>
      @id "#{ENV['OUTPUT_NAME']}"
      @type "#{ENV['OUTPUT_TYPE']}"
      @log_level info
      time_as_integer true
      include_tag_key true
      type_name _doc
      <server>
        host "#{ENV['OUTPUT_HOST']}"
        port "#{ENV['OUTPUT_PORT']}"
        weight 100
      </server>
      <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.system.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size "#{ENV['OUTPUT_BUFFER_CHUNK_LIMIT']}"
        queue_limit_length "#{ENV['OUTPUT_BUFFER_QUEUE_LIMIT']}"
        overflow_action block
      </buffer>
    </match>