kafka-hang-debug.md

TLDR

If a program is writing to stdout, there is a big risk that if the other end of the pipe is frozen, the program itself will hang.

from a previous instance, not shutdown cleanly

root@bigfoot-kafka3:/etc/service/fk-3p-kafka/log/main# fuser *
current:             32391
lock:                32391

root@bigfoot-kafka3:/etc/service/fk-3p-kafka/log/main# ls -l /proc/32390
lrwxrwxrwx 1 root root 0 Apr 11 16:53 cwd ->  (deleted)/etc/sv/fk-3p-kafka/log
lrwxrwxrwx 1 root root 0 Apr 11 16:53 exe -> /usr/bin/supervise

from the current instance

root@bigfoot-kafka3:/etc/service/fk-3p-kafka/log/main# ls -l /proc/4174 
lrwxrwxrwx 1 root root 0 Apr 11 16:53 cwd -> /etc/sv/fk-3p-kafka/log
lrwxrwxrwx 1 root root 0 Apr 11 16:53 exe -> /usr/bin/supervise

Learnings

1. Log4j default synchronous appenders use a synchronized block to write to output stream. This can lock up your process.
2. Pipes and fifos freeze the left hand side when buffer limit is reached. TODO find out how and what the limits are.
3. Supervise + Supervise + Multilog needs to be treated carefully and idiomatically.