Creates a Kubernetes DaemonSet that will monitor container logs and forward them to a Splunk Indexer

splunk-daemonset.yaml

# Create using kubectl:
#   $ kubectl create -f splunk-daemonset.yaml
# 
# You should also add config on your indexer to deal with the json formatted files:
#   https://answers.splunk.com/answers/148307/how-to-parse-and-extract-json-log-files-in-splunk.html
#
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: splunk-forwarder
spec:
  template:
    metadata:
      labels:
        name: splunk-forwarder
    spec:
      hostNetwork: true
      containers:
      - name: splunk-forwarder
        image: splunk/universalforwarder:latest
        env:
          - name: SPLUNK_START_ARGS
            value: "--accept-license"
          - name: SPLUNK_FORWARD_SERVER
            value: your_splunk_indexer:9997
          - name: SPLUNK_USER
            value: root
          - name: SPLUNK_ADD_1
            value: 'monitor /var/log/containers -sourcetype docker_json'
        volumeMounts:
          - mountPath: /var/log
            name: varlog
            readOnly: true
          - mountPath: /var/lib/docker/containers
            name: varlibdockercontainers
            readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
        - hostPath:
            path: /var/log
          name: varlog
        - hostPath:
            path: /var/lib/docker/containers
          name: varlibdockercontainers

I was able to add custom configuration by adding command and args as follows:

command: [ "/bin/bash", "-c" ]
args: [
  "timeout 10 /sbin/entrypoint.sh start-service; \
  echo 'sslCertPath = /opt/splunk/etc/auth/server.pem' >> /opt/splunk/etc/system/local/outputs.conf && \
  echo 'sslRootCAPath = /opt/splunk/etc/auth/cacert.pem' >> /opt/splunk/etc/system/local/outputs.conf && \
  echo 'sslVerifyServerCert = false' >> /opt/splunk/etc/system/local/outputs.conf && \
  grep sslPassword /opt/splunk/etc/system/local/server.conf >> /opt/splunk/etc/system/local/outputs.conf && \
  /sbin/entrypoint.sh start-service"
]

It's a bit hacky but allows you to avoid building a custom Docker image.