Please note that your ESXi Hypervisor needs to run on Apple hardware to install an OSX guest system.
So installing ESXi on MAC mini server and OSX 10.9. as guest there would work.
OSX guest under ESXi running on non-apple x86 server hardware would be a violation of the apple EULA.
Tim