Referencing this discussion on how to deny loopback access, create the opposite which is to only allow access from localhost. The goal is to only allow access after the client has connected with SSH using the port tunneling option.
Add new filter rules to block VNC (port 5900) access when not connected via SSH with port forwarding. Plus add blocks to file sharing (SMB and AFP). Just in case I am running a web server block that too, or any other ports listed in this link.
# pfctl is the packet filter firewall for modern OS X.
# For more info -> man pfctl
# flush to the default