InKolev

1. Get authorization token POST
   http://craftcluster/api/FacebookLogin/GetAuthorizationToken?facebookAccessToken=...&facebookProfileId=...

InKolev

Building Kafka from the Hardware - up

• Higher Message Retention ? - Increase disk size
• Higher Message Throughput ? - Increase network capacity
• Higher Producer Performance ? - Increase Disk I/O speed
• Higher Consumer Performance ? - Increase Memory

Critical Configurations (Consumer)

• queued.min.messages
• fetch.wait.max.ms
• socket.blocking.max.ms

InKolev

Bet Settlement Worker (x1)

• 40-50k items per minute (3000 cursor size, 1000 concurrency counter, 500 batch size, 250 ms timeout)

Account Operation Workers (x6)

• 25-45k items per minute (with 3ms SQL Delay)
• 30-50k items per minute (with 0ms SQL Delay)

Combo Bonus Workers (x6) (Slowest)

• 18-40k items per minute (with 3ms SQL Delay)
• 25-45k items per minute (with 0ms SQL Delay

InKolev

Conway's law

"Any organization that designs a system, will produce a design
whose structure is a copy of the organization's communication structure."

Jimmy's law

"A broken, dysfunctional organization, driven by meeting unhealthy goals and metrics
will produce broken, dysfunctional systems, even if the developers in that organization are good."

Kosh (Babylon 5)

"The avalanche has already started. It is too late for the pebbles to vote."

InKolev

Find largest files on disk (where size is bigger than 100MB)

sudo find / -xdev -type f -size +100M -exec du -sh {} ';' | sort -rh | head -n50

PM2 node process manager

pm2 start ecosystem.config.js

Sample ecosystem file

InKolev

Multi-tenant systems

Check if the observed behavior is the same for other tenants, not just for a single one. The wider the dataset, the easier it becomes to isolate the probem.

Check for data bleed accross tenants

• Configuration mix
• HTTP requests routed to the wrong tenant
• Message bus listeners consuming and processing data for the wrong tenant or for more than 1 tenants.