VDP generates a self-signed certificate without a proper common name (CN), there are two things that need to be done to make the VDP SSL connection green. First, the certificate needs the proper FQDN and second, the certificate must be signed by a trusted CA.
Most of these commands were pulled out of the VDP SHA256 hotfix script and documentation on signing tomcat keys.
If anything goes wrong, you can delete the tomcat key, run the self-signed certificate command without modification and restart the webapp to get the system back to a default state.
Delete the current tomcat key:
/usr/java/latest/bin/keytool -delete -alias tomcat -storepass changeit
Then we're going to use VMWare's command for generating the self-signed certificate with a modification, change the CN from localhost.localmachine to the FQDN of the machine you're generating the certificate for:
/usr/java/latest/bin/keytool -genkeypair -v -alias tomcat -keyalg RSA -sigalg SHA256withRSA -keystore /root/.keystore -storepass changeit -keypass changeit -validity 3650 -dname "CN=localhost.localmachine, OU=Avamar, O=EMC, L=Irvine, S=California, C=US"
Should be changed to something like this:
/usr/java/latest/bin/keytool -genkeypair -v -alias tomcat -keyalg RSA -sigalg SHA256withRSA -keystore /root/.keystore -storepass changeit -keypass changeit -validity 3650 -dname "CN=vdp.domain.com, OU=Avamar, O=EMC, L=Irvine, S=California, C=US"
Now we can generate a signing request for that certificate:
/usr/java/latest/bin/keytool -certreq -keystore /root/.keystore -alias tomcat -file /root/signing-request -storepass changeit
Once you sign the certificate on the CA server, download the base64 certificate chain file from the CA and save it to a flat file in the root folder of the VDP server. Then import the signed certificate chain with this command:
/usr/java/latest/bin/keytool -importcert -keystore /root/.keystore -alias tomcat -file /root/chain-file.pem -trustcacerts -storepass changeit
Your new certificate should now be in the keystore, restart the web app so tomcat sees the changes:
emwebapp.sh --stop; sleep 15; emwebapp.sh --start
It may take several minutes for tomcat to fully start, give it about 5 minutes and then attempt to connect to the vdp-configure page for this server.