Skip to content

Instantly share code, notes, and snippets.

@rushi47
Last active March 6, 2024 21:58
Show Gist options
  • Save rushi47/823e97a4793027ee9a05b9c16a58b05b to your computer and use it in GitHub Desktop.
Save rushi47/823e97a4793027ee9a05b9c16a58b05b to your computer and use it in GitHub Desktop.
Dynamic topic for output kafka in fluentbit
Place below config in fluent bit.config. And annotate your pods with `k8s_annotations_fluentbit.topic : <destination-topic>`.
For the pods which have this annotation defined, kafka will output : respective `<destination-topic>`
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[FILTER]
Name nest
Match *
Operation lift
Nested_under kubernetes
Add_prefix kubernetes_
[FILTER]
Name nest
Match *
Operation lift
Nested_under kubernetes_annotations
Add_prefix k8s_annotations_
[FILTER]
Name nest
Match *
Operation nest
Wildcard kubernetes_*
Nest_under kubernetes
[OUTPUT]
Name kafka
Match kube.*
Brokers ${KAFKA_BROKER}
Topics default_topic_here
Topic_key k8s_annotations_fluentbit.topic
Dynamic_topic On
...remaining config
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment