When we run a rolling restart on our orchestrator statefulset, the node that is the previous master will get stuck in a crash loop.
Findings so far:
- The pod that gets stuck in a crash loop seems to be the node that used to be the master.
- When you delete the pod, it somehow gets out of the crash loop.
- Using pprof, Leo tracked the crash to within the martini web framework (used by orchestrator - Profile map, problem region), while writing the response.
- More specifically, the
X-Forwarded-Forheader in the response (which is supposed to contain the IP addresses of each proxy the server goes through) seems to [accumulate](https://github.com/golang/go/blob/b5be877ba4318422547068b85c673639cd843b7d/src/net/http/httputil/