I want to start off by saying that I strongly respect John De Goes and the decision he made. If you haven't already read his extensive post about what he did and why he did it, I highly recommend you take the time. His post is thorough and generally persuaded me to his way of thinking, which is not that the works of Moldbug should be accepted or ignored, but that we need to be inclusive of differing beliefs in a community, so long as the holder of those beliefs is willing to "leave them at the door". This is a strong, intellectually defensible and consistent position.
However, as the day has gone on, and I've discussed this issue at length with many members of the community. I have come to the conclusion that this is idealistic, and ultimately missing the point. And as strange as it seems, the point is not that there is a literal Neo Nazi who will be speaking at a well-regarded conference! The point is somehow bigger than even that, and it has to do with community.
Community is everything. I'm fond of saying that programmers are not automata, and that's true, but it radically undersells how important people are. Every time you "do the hallway track" at a conference, you're explicitly making the claim that the people at the conference are more important than the content. People often talk about how you should try to choose your workplace for the coworkers, not the problem. And when we punch our pillows in frustration after a difficult day, we're rarely thinking about a thorny technical problem or an annoying technology: we're thinking of a person (or persons). Community is everything.
And community is being disrupted. Not even by Moldbug himself, but by the idea of Moldbug being accepted in any form. I called this the "meta-controversy" on twitter, and the more I think about it, the more I've realized that it's the only controversy that matters in this case. Community is disrupted by Moldbug. It is now being disrupted by the announcement that he will be speaking at LambdaConf, and it will (presumably) be even further disrupted by the actual event. People are divided, some of whom believe that we can (as a community) divorce ourselves from these ulterior associations and judge solely on technical merit, and others who believe that allowing Moldbug a platform on any topic is benefiting and supporting his philosophy.
What I'm realizing is that it doesn't matter which of these opinions holds sway. The division and disruption of the community itself is the problem, and it needs to be rectified. This would be true whether Moldbug were the second coming of Hitler or of Ghandi. Social equillibrium is tremendously important, and disruption to that equillibrium is a cancer which the community must stamp out for its own preservation. Moldbug must be the recipient of this correction, not because of his views (which are reprehensible), but because of the effect he has. As horrifying as it seems, the fact that he advocates for literal slavery (and more) is secondary to the impact he has on the community as a whole.
The community must always act for the preservation of its own harmony. And to that end, I believe that Moldbug should be removed from his speaking slot at LamdaConf. All arguments about inclusiveness or the technical nature of a conference are an aside, because the community itself has decided that they are an aside by their reaction.
If you step back a few more steps and look not just at the LambdaConf community and its social equilibrium, but of society as a whole, I am not sure the position you state above is easily defensible.
Whenever you have a situation that boils down to, "we will eject this person because we want to eject this person" it is not simply a matter of the community being everything. That can be and often has been the source of terrible injustice. You need to advocate that the will of the group is a just decision. I am quite glad that various groups that promoted all sorts of inhumane acts and attitudes are no more.
I don't have statistics, presently, but it seems to me that it's overwhelmingly perpetrated by charismatic demagogues. It is the retreat from engagement of unpalatable ideas, not the failure to boycott their proponents, that fosters demagoguery. Boycotts do not train one to use reason to persuade, to demand reason and evidence, to deconstruct bad ideas to expose precisely why they are bad. (People don't want to believe ideas that are just plain bad. They might want to believe an idea which is unpopular but which they can convince themselves is true.)
So although I agree that social cohesion writ large is a good thing, cohesive subgroups within society can promote qualities that are dangerous to society as a whole. And because of that, you can't just write a blank check that "community is everything". It's not everything; it exists in a larger social context.
There are arguments that I think you could make to give a strong defense of the wisdom of sanctioning Yarvin (starting with an endorsement of the principles of affirmative action). This isn't it. Not only does it abandon any pretense of moral standing, it also does more to promote Yarvin's ideas save for actually advocating for them. By requiring only a community, "Noooooo!", toxic ideas remain unchallenged, unexamined, still tempting to those who can be swayed by appeal to evidence and reason.