Hi again!
I've done a quick sweep and I've got something to report after the analysis of the images you provided:
- esx01:Read Latency is way too high on Trudy-B-VM (81.12/cmd) and vTracer (48.57/cmd) - generally where you would like to have latencies under 10 ms (it could go at about 30 ms critical, but this is VDI so users are going to be super-sensitive on this) - this is a response of guest operating system. Also Device Average/cmd is not too fancy on naa.60...d9.
- esx02: %ready is exceeding it's rule-of-the-thumb value (>5 for 1 vCPU assigned, in your case it's ~30%) on te VMs Nick O, view6-CS and BC-Bill, although not by a big margin and that's nothing compared to some of the VMs below:
- esx03: A larger amount over of %RDY is on VMs ams-acct-3 (15%), artisan-6 (28%), artisan-16 (29%) & artisan-8 (17%) although the idle CPU time is pretty much the majority of ticks
- esx09: Everything is more or less fine
- esx10: High ready time on artisan-12 (27%), artisan-2 (66%), life-case-1 (34%), IT-Jim (47%).
- esx11: System has a huge ready value (~44%)
The thing that I'd suggest here is to observe the trend of VMs that are not mentioned above and cutting their vCPUs so that they can run in a context of one NUMA node (vCPUs less or equal than 4). Also please watch out for over-provisioning the Virtual CPUs. I see that you have quite some VMs that use both CPUs at the same time, which would read to resource contention or outright "choking" both the physical CPUs, making the resource contention a real problem. This is why you have to check your performance graphs for how much CPU is used at a peak time (let's say for the last week), and if your 8-vCPU VM is only 40% used, you'd make a calculation of 8 x 0,4 which would be 3,2 cores, so you'd be safe with cutting down this VM to 4 cores. I think this is really essential because I see a heavy potential resource oversubscription on all your hosts. while running VMware, less is more because the ESXi hosts would have trouble with co-scheduling your VMs. Remember that all the vCPUs have to be serviced in one CPU's clock cycle, so if one of your 8-vCPU VMs has to tick (even for an idle cycle), it effectively affect every other VM from ticking as well.
I strongly recommend that you do an individual, per-VM tuning and determining a proper vCPU count depending on the recent usage trend.
Good luck!