Skip to content

Instantly share code, notes, and snippets.

@automine
Created December 22, 2022 18:03
Show Gist options
  • Save automine/095ceb8348aec83d027a512db4b48277 to your computer and use it in GitHub Desktop.
Save automine/095ceb8348aec83d027a512db4b48277 to your computer and use it in GitHub Desktop.
Explanation of various HTTP(s) timeouts in Splunk

Original version is here, I just wanted something easier to read.

Config Default Max Recommended Purpose When to use
statusTimeout 10 sec 120 sec Connect/read/write timeout to get search peer's info ( from SH main splunkd to peer main splunkd).  One thread handles many peers. Check DistributedPeer* component WARN/ERROR for timeouts. Impacts bundle replication as peers are marked down when peer is up and running. This can happen  to due to busy mgmt port on peer(check peer  metrics.log for dutycycle)
sendTimeout 30 sec 120 sec send/write timeout for SH search process to  peer's main splunkd. One thread handles many peers. Check search.log for send/write failure. This can happen  to due to busy mgmt port on peer(check peer  metrics.log for dutycycle)
receiveTimeout 600 sec 1200 sec read/receive timeout for SH search process to  peer's main splunkd. One thread handles many peers. Check search.log for read failure.
connectionTimeout 10 sec 120 sec connect timeout for SH search process to  peer's main splunkd. One thread handles many peers. Check search.log for connect failure. This can happen  to due to busy mgmt port on peer(check peer  metrics.log for dutycycle)
authTokenConnectionTimeout 5 sec 120 sec connect timeout to get search peer's auth token ( from SH main splunkd to peer main splunkd). One thread handles many peers. Check DistributedPeer* component WARN/ERROR for timeouts. Impacts bundle replication as peers are marked down when peer is up and running.  This can happen  to due to busy mgmt port on peer(check peer  metrics.log for dutycycle)
authTokenSendTimeout 10 sec 120 sec send/write timeout get search peer's auth token ( from SH main splunkd to peer main splunkd). One thread handles many peers. Check DistributedPeer* component WARN/ERROR for timeouts. Impacts bundle replication as peers are marked down when peer is up and running.  This ca happen  to due to busy mgmt port on peer(check peer  metrics.log for dutycycle)
authTokenReceiveTimeout 10 sec 120 sec read/receive timeout get search peer's auth token ( from SH main splunkd to peer main splunkd). One thread handles many peers. Check DistributedPeer* component WARN/ERROR for timeouts. Impacts bundle replication as peers are marked down when peer is up and running.
connectionTimeout 60 sec 120 sec connect timeout for actual bundle replication from SH main splunkd to peer main splunkd. One thread handles one peer. Impacts actual bundle replication. Normally this never fails.
sendRcvTimeout 60 sec 120 sec read/write timeout for actual bundle replication from SH main splunkd to peer main splunkd. One thread handles one peer. Impacts actual bundle replication. Normally this never fails.
remote_timeline_connection_timeout 5 sec 30 sec connect timeout for SH search process to  peer's main splunkd for timeliner. Multi-threaded. Normally this never fails.
remote_timeline_send_timeout 10 sec 30 sec send timeout for SH search process to  peer's main splunkd for timeliner. Multi-threaded. Normally this never fails.
cxn_timeout_raft 2 sec 30 sec SH to SH raft communication connect timeout. One thread handles one member.
send_timeout_raft 5 sec 30 sec SH to SH raft communication send timeout. One thread handles one member.
rcv_timeout_raft 5 sec 30 sec SH to SH raft communication read timeout. One thread handles one member.
sendRcvTimeout 60  sec 120 sec Read/Write timeout by search peer for bundle replication ( from peer main splunkd to SH main splunkd) Impacts actual bundle replication.
busyKeepAliveIdleTimeout 12 sec 120 sec SH Peer/Member HTTP server keep-alive connection idle timeout. When number of HTTP connections > 60, it disconnects any idle connection. Impacts bundle replication and search.
streamInWriteTimeout 5 sec 30 sec Read/Write timeout when receiving http request by http server Normally this never fails.
dedicatedIoThreads 0 5 Http Listener thread will let additional I/O threads do read/write from socket. Very useful for knowledge replication/config replication/raft/search artifact replication  and searches.
You will see in metrics.log that  mgmt_httpd  is consistently > 0.8
Metrics - group=dutycycle, name=misc, mgmt_httpd=
useClientSSLCompression TRUE FALSE SSL compression used by Http client for SH→SH / SH→IDX / SH→CM/ IDX->CM Change this settings  if increase in network bandwidth usage is not an issue.
connect_timeout 60 120 DC->DS If DCs are timing out while connecting("Connect timeout").
send_timeout 60 120 DC->DS If there is a send timeout while sending request to DS
recv_timeout 60 120 DC->DS If DC read timed out while waiting for a response from DS.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment