Intel Sandy-Bridge EVC patch for BOSH

readme.md

In order to vMotion VMs from a cluster with a newer CPU architecture to a cluster with an older CPU architecture (downgrade) requires that we set an EVC mode that is compatible with the older target cluster CPU architecture. Downgrading a VM's supported CPU feature set requires that we set an EVC mode on each VM so as to not impact the VMs that are staying on the source cluster. To set a per VM EVC mode on a BOSH deployment requires that we use a modified BOSH CPI and upgrade each VM's hardware version to 14 or higher.

In a vSphere environment, the default VM hardware version for any VM deployed by BOSH is version 9 (ESXi 5.5). This article describes how to force the BOSH Director to upgrade the VM hardware version to the highest level compatible with the ESXi host and set a per VM EVC mode. This is a required prerequisite to vMotion VMs to an older vSphere cluster using the vMotion Migration for BOSH Deployments tool.

You will need to modify the BOSH director configuration via the Operations Manager API and the BOSH CPI directly on the BOSH director VM via SSH. These instructions will modify the BOSH director CPI code directly on the BOSH VM, so if you upgrade Operations Manager or redeploy the BOSH director the BOSH CPI patch will be removed. Do not remove the CPI patch until your foundation has been migrated to the new vSphere cluster and apply changes has run successfully.

Instructions

Upgrade VM Hardware

1. Install the OM CLI tool on your workstation or jumpbox and set the following environment variables to match your TAS foundation:

export OM_TARGET="opsman.example.com"
export OM_USERNAME=admin
export OM_PASSWORD="$ecret"
export OM_SKIP_SSL_VALIDATION=true
export DIRECTOR_IP='192.168.1.5'

2. Backup the latest director configuration properties:

om curl -p /api/v0/staged/director/properties > director-props.json

3. Set the upgrade hardware flag to true:

om curl -p /api/v0/staged/director/properties \
  -x PUT \
  -d '{"iaas_configuration":{"additional_cloud_properties":{"upgrade_hw_version":true}}}'

Patch BOSH CPI

1. From your workstation or jumpbox grab the SSH key from the Operations Manager Director tile.

om credentials -p p-bosh -c .director.bbr_ssh_credentials -f private_key_pem > ~/.ssh/bbr.key && chmod 0400 ~/.ssh/bbr.key

2. From your workstation or jumpbox that has internet access download the BOSH CPI patch to the BOSH director.

wget -O - --no-check-certificate \
  https://gist.githubusercontent.com/sneal/6f77d2278c1234eca9fa1d6e49795c10/raw/3d4f8ab30b80e6910473d1e247d1b0bcfd19e168/evc.patch | \
ssh -i ~/.ssh/bbr.key "bbr@$DIRECTOR_IP" 'cat > /home/bbr/evc.patch'

3. From your workstation or jumpbox apply the patch to the BOSH Director's CPI:

ssh -t -i ~/.ssh/bbr.key "bbr@$DIRECTOR_IP" "sudo sh -c 'cd /var/vcap/jobs/vsphere_cpi/packages/vsphere_cpi; patch -p1 -i /home/bbr/evc.patch'"

Recreate VMs with Apply Changes

1. From your workstation or jumpbox enable the recreate all VMs option while also disabling recreate the BOSH director VM:

om curl -x PUT -p '/api/v0/staged/director/properties' \
  -d '{ "director_configuration": { "bosh_director_recreate_on_next_deploy": false, "bosh_recreate_on_next_deploy": true }}'

2. For any on-demand service tiles you have installed make sure you enable any upgrade errands.

3. Execute apply changes ensuring to recreate all the VMs in the foundation:

om apply-changes

Operations Manager VM & BOSH Director VM EVC Mode

Login to the vSphere GUI and find the TAS installation's Operations Manager VM and BOSH Director VM. Apply the below process to both VMs.

1. Right click on the VM and select Compatibility | Schedule VM Upgrade Compatibility. Select Yes. Select ESXi 6.7 U2 and later. Click OK.

2. Reboot the VM. Right click on the VM and select Power | Restart Guest OS. Click Yes.

3. After the VM finishes rebooting, ensure the VM's compatibility now says "ESXi 6.7 U2 and later (VM version 15)"

4. Power down the VM. Right click on the VM and select Power | Shutdown Guest OS. Click Yes.

5. Select the VM's Configure tab | VMware EVC. If it doesn't show up click the Refresh button in the GUI. Click Edit | Enable EVC for Intel® hosts. Change CPU mode to Intel Sandy Bridge Generation. Click OK.

6. Power up the VM. Right click on the VM and select Power | Power On.

7. For the Operations Manager VM don't forget to login to the GUI and decrypt the installation.

References

https://community.pivotal.io/s/article/upgrade-vm-hardware-version-for-Bosh-VM-managed?language=en_US

evc.patch

diff --git vsphere_cpi/lib/cloud/vsphere/resources/vm.rb vsphere_cpi/lib/cloud/vsphere/resources/vm.rb
index 69cb1a56..8e944a75 100644
--- vsphere_cpi/lib/cloud/vsphere/resources/vm.rb
+++ vsphere_cpi/lib/cloud/vsphere/resources/vm.rb
@@ -232,6 +232,10 @@ module VSphereCloud
         @client.upgrade_vm_virtual_hardware(@mob)
       end
 
+      def set_evc_mode
+        @client.apply_evc_mode(@mob)
+      end
+
       def delete
         @client.delete_vm(@mob)
       end
diff --git vsphere_cpi/lib/cloud/vsphere/vcenter_client.rb vsphere_cpi/lib/cloud/vsphere/vcenter_client.rb
index 12356ea4..1fbb1b0c 100644
--- vsphere_cpi/lib/cloud/vsphere/vcenter_client.rb
+++ vsphere_cpi/lib/cloud/vsphere/vcenter_client.rb
@@ -85,6 +85,51 @@ module VSphereCloud
       end
     end
 
+    def apply_evc_mode(vm)
+      logger.info("Setting VM EVC mode")
+      mask = []
+      # intel-sandybridge
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.AES", :feature_name => "cpuid.AES", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.AVX", :feature_name => "cpuid.AVX", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.CMPXCHG16B", :feature_name => "cpuid.CMPXCHG16B", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.DS", :feature_name => "cpuid.DS", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.FAMILY", :feature_name => "cpuid.FAMILY", :value => "Val:6"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.FCMD", :feature_name => "cpuid.FCMD", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.IBPB", :feature_name => "cpuid.IBPB", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.IBRS", :feature_name => "cpuid.IBRS", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.Intel", :feature_name => "cpuid.Intel", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.LAHF64", :feature_name => "cpuid.LAHF64", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.LM", :feature_name => "cpuid.LM", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.MDCLEAR", :feature_name => "cpuid.MDCLEAR", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.MODEL", :feature_name => "cpuid.MODEL", :value => "Val:0x2d"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.MWAIT", :feature_name => "cpuid.MWAIT", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.NUMLEVELS", :feature_name => "cpuid.NUMLEVELS", :value => "Val:0xd"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.NUM_EXT_LEVELS", :feature_name => "cpuid.NUM_EXT_LEVELS", :value => "Val:0x80000008"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.NX", :feature_name => "cpuid.NX", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.PCID", :feature_name => "cpuid.PCID", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.PCLMULQDQ", :feature_name => "cpuid.PCLMULQDQ", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.POPCNT", :feature_name => "cpuid.POPCNT", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.RDTSCP", :feature_name => "cpuid.RDTSCP", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.SS", :feature_name => "cpuid.SS", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.SSBD", :feature_name => "cpuid.SSBD", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.SSE3", :feature_name => "cpuid.SSE3", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.SSE41", :feature_name => "cpuid.SSE41", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.SSE42", :feature_name => "cpuid.SSE42", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.SSSE3", :feature_name => "cpuid.SSSE3", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.STEPPING", :feature_name => "cpuid.STEPPING", :value => "Val:2"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.STIBP", :feature_name => "cpuid.STIBP", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.VMX", :feature_name => "cpuid.VMX", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.XCR0_MASTER_SSE", :feature_name => "cpuid.XCR0_MASTER_SSE", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.XCR0_MASTER_YMM_H", :feature_name => "cpuid.XCR0_MASTER_YMM_H", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "cpuid.XSAVE", :feature_name => "cpuid.XSAVE", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "hv.capable", :feature_name => "hv.capable", :value => "Val:1"})
+      mask << VimSdk::Vim::Host::FeatureMask.new({:key => "vt.realmode", :feature_name => "vt.realmode", :value => "Val:1"})
+      logger.info(mask)
+      wait_for_task do
+        vm.apply_evc_mode(mask)
+      end
+    end
+
     def power_on_vm(datacenter, vm)
       wait_for_task do
         result = wait_for_task do
diff --git vsphere_cpi/lib/cloud/vsphere/vm_creator.rb vsphere_cpi/lib/cloud/vsphere/vm_creator.rb
index c2c76994..5d9aa6bd 100644
--- vsphere_cpi/lib/cloud/vsphere/vm_creator.rb
+++ vsphere_cpi/lib/cloud/vsphere/vm_creator.rb
@@ -217,6 +217,7 @@ module VSphereCloud
             # 2. Global upgrade hardware flag @upgrade_hw_version
             if vm_config.upgrade_hw_version?(vm_config.vm_type.upgrade_hw_version, @upgrade_hw_version)
               created_vm.upgrade_vm_virtual_hardware
+              created_vm.set_evc_mode
             end
           rescue VSphereCloud::VCenterClient::AlreadyUpgraded
           end