This demonstrates that d3-zoom doesn't compute the zoom point as expected if you position a zoom-catching rect with "x" and "y" attributes. If you position the rect with "transform", it works as expected.
Point your mouse at a certain point, e.g. (100, 100). Try to zoom in on it. See how the point (100, 100) starts to scroll out of sight?
Now click "Fix it" and try it again. It works.