Finally realised I was copying over the wrong cert you need to copy over the castore.pem cert, not the rui.cert one. Anyways, here's a short procedure which might help someone:
- go to your ESXi host, open Configure Management Network, open DNS Configuration, set hostname to whatever your desired name is (server3 in this case). Esc out of the settings back to the main screen saving the network configuration when prompted
- Alt+F1 to go into console (Alt+F2 to get out of it), log in as root and run the following to regenerate certs for the new host name:
cd /etc/vmware/ssl
/sbin/generate-certificates
reboot
- when the host is up and running, use WinSCP (or some other method) to copy the castore.pem from the /etc/vmware/ssl folder on the host to your local workstation. Hit start menu button on the workstation, start typing certif... and pick Manage computer ceritifactes, then in the left hand side list pick Trusted Root Ceritifaction Authorities / Ceritifactes, then on the list on the right, right click and pick All Tasks->Import. Select the downloaded castore.pem (select to view All files rather than crt only) and run through the import process
- open your hosts file (c:\Windows\System32\Drivers\etc\hosts) and insert a new entry with your IP address and hostname
- open up your browser and type in http://[hostname] which should open up the vsphere client login webpage with a valid cert and no issues