Kafka Rollout Test.md

Bet Settlement Worker (x1)

• 40-50k items per minute (3000 cursor size, 1000 concurrency counter, 500 batch size, 250 ms timeout)

Account Operation Workers (x6)

• 25-45k items per minute (with 3ms SQL Delay)
• 30-50k items per minute (with 0ms SQL Delay)

Combo Bonus Workers (x6) (Slowest)

• 18-40k items per minute (with 3ms SQL Delay)
• 25-45k items per minute (with 0ms SQL Delay
• Huge spike (100k items per minute with 0ms SQL Delay, while refreshing the worker details page), no idea why.

DWH Workers (x6)

• 25-45k items per minute (with 3ms SQL Delay)
• 36-50k items per minute (with 0ms SQL Delay)

Bottleneck

• 11-17ms per single item publish
• With this restriction, workers that use the SingleToSingle pattern almost never go above 66 items per second
• Which means roughly 4000 items per minute, per worker.
• This is if we consider only the Kafka I/O operations, when we include Mongo and SQL I/O's, the things get even slower.

Possible actions for optimization

• Tune Windows TCP/IP stack for performance
• Change SingleToSingle workers to use different flow