I was able to download the command line tool (esxcli) and that allowed me to set promiscuous mode. It was not trivial to figure this out, but at least I got around it. For anyone else running into this problem, you can do something like this:
To list interfaces /ports
esxcli --server IPADDRESS --username USER --password PASSWORD network ip interface list
My switch was vSwitch0 after running this.
To check the policy:
esxcli --server IPADDRESS --username USER --password PASSWORD network vswitch standard policy security get -v vSwitch0
To set the policy:
esxcli --server IPADDRESS --username USER --password PASSWORD network vswitch standard policy security set -f true -m false -p true -v vSwitch0