When you build a community you're creating a culture. That culture will be about more than the code, the modules, or the language. The people you draw in will have their own biases and behaviors that impact the kinds of people you continue to draw as you grow.
Cultures will naturally fight behavior that is divisive. That is, behavior that is divisive to the established members of that community. As a community grows larger it is harder and harder to change what the culture finds acceptable because changing it, even if it is inclusive in nature, is disturbing and divisive to existing membership. Fighting for change in established cultures means dealing with a lot of dismissive language and attacks for the "tone" of your argument.
That is why it is so important that a culture becomes comfortable with aggressively fighting exclusionary behavior. While it is certainly more beneficial to make pro-active steps to increase diversity we cannot be dismissive of the effect that passionate reactions to poor behavior