Virtual Infrastructure clients are unable to open consoles on VMs managed by my VCS 5.1 appliance. I see lots of people with this same issue, but they don't mention if they're running vcenter server in windows or with the supplied linux-based OVA appliance. I also haven't seen any fixes or anyone mention that they've successfully fixed it. I would try to reinstall, but i have a large inventory that I'd rather not have to spend two hours rebuilding all of their resource groups and HA configs.
It started happening when I moved the VCS appliance from one vlan to another in our datacenter, changed its IP address, and rebooted it. The new vlan is where its associated ESXi5 hosts live (10.10.11.0/24). Moving the VCS appliance to the old vlan did not fix the problem.
VI clients on same subnet (10.10.11.x) are able to connect to the VCS server for management, and the machine where the VI client is running on can ping it via both its hostname and IP address. All services are up, when accessing the VCS server via the web admin interface (https://vcs:5480)
When I try to open a console on a VM managed via the VCS, the console window comes up, stays black for a few seconds, then shows the error:
Unable to connect to the MKS: Failed to connet to server 10.10.11.165:902 [X]
When the VI client is connected directly to an ESXi host (bypassing the VCS), console windows open fine. There are no DNS errors on the clients, VCS, or ESXi hosts. I can log into ALL devices (client host, ESXi hosts and the VCS appliance) and run ping/traceroute/nslookup for eachother, by both hostname and IP address.
Running netstat -an and lsof -nP on the VCS appliance shows that port 902/tcp is NOT in a listening state. Running tcpdump on the vcs (tcpdump -tpnv tcp and port 902) while trying to open a console shows that the TCP packets from the client are getting to the VCS server, but being refused:
IP 10.10.11.102.55490 > 10.10.11.165.902: S 1324346965:1324346965(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>
IP 10.10.11.165.902 > 10.20.10.102.55490: R 0:0(0) ack 1324346966 win 0
IP 10.10.11.102.55490 > 10.10.11.165.902: S 1324346965:1324346965(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>
IP 10.10.11.165.902 > 10.20.10.102.55490: R 0:0(0) ack 1 win 0
IP 10.10.11.102.55490 > 10.10.11.165.902: S 1324346965:1324346965(0) win 8192 <mss 1460,nop,nop,sackOK>
IP 10.10.11.165.902 > 10.20.10.102.55490: R 0:0(0) ack 1 win 0
What daemon/process is responsible for 902/TCP? Is it vpxd (which currently has 902/UDP, 443/TCP, 80/TCP, 8085/TCP, 8089/TCP open)
Is it supposed to be a proxy that forwards it to the same port on the VM's associated ESXI host?