filter the countries where the VAT number field will show up for

wc-eu-vat-countries.php

add_filter( 'woocommerce_eu_vat_number_country_codes', 'woo_custom_eu_vat_number_country_codes' );
function woo_custom_eu_vat_number_country_codes( $vat_countries ) {
// only show field for users in BE
return array( 'BE' );
}

If you need to remove a couple of countries use array_diff passing an array of the country codes you wish to remove as the second param, no need to loop and unset values.
As @jakecausier mentioned pass the modified array through array_values to reindex the values.

add_filter( 'woocommerce_eu_vat_number_country_codes', 'woo_custom_eu_vat_number_country_codes' );
function woo_custom_eu_vat_number_country_codes( $vat_countries ) {
    $display_vat = array_diff($vat_countries, ['SE', 'GB']); // remove countries in second array
    return array_values($display_vat); // reindex array
}