The line that reads `if(in_array($term, $terms_of_the_post))` is what's causing the issue. The object stored in $term will never match any of the objects in the array $terms_of_the_post. So I changed that code to match the following:
So instead of comparing the object $term to the array of objects in $terms_of_the_post, I created an array of the term_ids and checked if the value $term->term_id is in the array. Now when I update a post that is in a manual ordering, the post isn't removed from the end result.