Reaper gotcha

README.md

Signal handling gotcha

A process running as PID 1 that calls proc.StartReaper() must be careful when explicitly or implicitly issuing wait syscalls to forked processes the application code explicitly manages. In this example, using the stdlib Cmd.CombinedOutput() function is shown to race with proc.StartReaper().

1. Cmd.CombinedOutput() is called
2. Internally, the stdib calls Cmd.Run() and the /bin/true process is forked
3. The /bin/true process exits
4. Race begins between:
   1. Cmd.Wait() (called from Cmd.Run() after the process exits)
   2. Reaper goroutine (triggered because the current process, as PID 1, receives SIGCHLD when the process exits)

When the Reaper wins, the Cmd.Wait() call will return an error because the child was reaped. The caller must then (on unix systems) interpret an os.SyscallError and inspect the wait syscall name to understand the benign nature of the failure.

docker build -t reaper-test . && docker run -it reaper-test

Sending build context to Docker daemon  4.608kB
Step 1/4 : FROM golang:1.13.5-buster
 ---> ed081345a3da
Step 2/4 : COPY main.go /go/reaper.go
 ---> 94c4fd412ce9
Step 3/4 : RUN go build -o /bin/reaper /go/reaper.go
 ---> Running in 47e95ab18e79
Removing intermediate container 47e95ab18e79
 ---> 13675ad858c8
Step 4/4 : ENTRYPOINT ["/bin/reaper"]
 ---> Running in 5518da86191c
Removing intermediate container 5518da86191c
 ---> b5d0ed6c822c
Successfully built b5d0ed6c822c
Successfully tagged reaper-test:latest
started reaper!
Signal received: child exited
Signal received: child exited
Signal received: child exited
Signal received: child exited
Signal received: child exited
Reaped process with pid 19
panic: waitid: no child processes

goroutine 1 [running]:
main.main()
        /go/reaper.go:17 +0x7d

Dockerfile

FROM golang:1.13.5-buster

COPY main.go /go/reaper.go

RUN go build -o /bin/reaper /go/reaper.go

ENTRYPOINT ["/bin/reaper"]

reaper.go

package main

import (
	"fmt"
	"os"
	"os/exec"
	"os/signal"
	"syscall"
)

func main() {
	StartReaper()

	for {
		cmd := exec.Command("/bin/true")
		if _, err := cmd.CombinedOutput(); err != nil {
			panic(err)
		}
	}
}

// StartReaper starts a goroutine to reap processes if called from a process
// that has pid 1.
func StartReaper() {
	if os.Getpid() == 1 {
		go func() {
			sigs := make(chan os.Signal, 1)
			signal.Notify(sigs, syscall.SIGCHLD)
			for {
				// Wait for a child to terminate
				sig := <-sigs
				fmt.Printf("Signal received: %v\n", sig)
				for {
					// Reap processes
					cpid, _ := syscall.Wait4(-1, nil, syscall.WNOHANG, nil)
					if cpid < 1 {
						break
					}

					fmt.Printf("Reaped process with pid %d\n", cpid)
				}
			}
		}()
	}
}