TLDR: The cascade={"remove"} is like a "software" onDelete="CASCADE", and will remove objects from the database only when an explicit call to $em->remove() occurs. Thus, it could result in more than one object being deleted. orphanRemoval can remove objects from the database even if there was no explicit call to ->remove().
I answered this question a few times to different people so I will try to sum things up in this Gist.
Let's take two entities A and B as an example. I will use a OneToOne relationship in this example but it works exactly the same with OneToMany relationships.
class A