Skip to content

Instantly share code, notes, and snippets.

@mbrownnycnyc

mbrownnycnyc/splk_backpressure.md

Last active August 28, 2023 23:56
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mbrownnycnyc/2d0c2437a3d74468bae5d87a848e68e6 to your computer and use it in GitHub Desktop.
Save mbrownnycnyc/2d0c2437a3d74468bae5d87a848e68e6 to your computer and use it in GitHub Desktop.
Download ZIP
info on backpressure mechanism for splunk forwarders. You should _NOT_ be changing these settings usually, but may consider it when dealing with extremely high volume data sources (such as a UF on a syslog server).
Raw
splk_backpressure.md

https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Protectagainstlossofin-flightdata

How can data loss be avoided?

The architecture is such that the UDP data sources must be converted to TCP backed by reliable delivery. Additionally, the forwarders and indexers may be configured to send application level ACKs back to sending forwarders.

splunkd delivery of packets is as follows:

  • data is sent in chunks of 64KB.
  • By default the forwarder is not looking for, nor is the indexer signaled to send, ACKs upon block receipt.
  • If you wish the forwarder to be configured to ask for ACKs by the indexer upon block receipt, the following items can be configured:
    • outputs.conf: [tcpoutput] stanza, useACK value: default is false. This enables the indexer app level ACKing mechanism. The forwarder holds data blocks until the Indexer sends an ACK for this block back to the forwarder. The forwarder will continue to send data blocks while waiting, but if none are ACKed, the forwarder will wait until the readTimeout value is hit, then pick another indexer.
    • outputs.conf: [tcpoutput] stanza, readTimeout value: default is 300 seconds.
    • outputs.conf: [tcpoutput] stanza, writeTimeout value: default is 300 seconds.
    • outputs.conf: [tcpoutput] stanza, maxQueueSize value: default is auto, which means 500KB with useACK = false, or 7MB with useACK = true. This setting directly describes/configures the forwarder’s “output queue,” however it also indirectly adjust the forwarder’s “wait queue,” where wait queue size is equal to (maxQueueSize*3).

If their maxQueueSize is reached, then pressure will be placed back upon the processing pipeline. This probably means that your inputs will start needing to deal with a cache/queue of their own. Inputs queue components by order of backpressure:

  1. in-memory input queue (inputs.conf: [tcp] stanza, queueSize value: default is 500KB): queue in RAM resident
  2. persistent queue: (inputs.conf: [tcp] stanza, persistentQueueSize value: default is disabled): queue is disk resident

Consider reviewing autolb to understand ingsetion pipeline delivery.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment