We have a 4 node vSphere 5.0 cluster, and I've just been finishing off replacing two of the older nodes with brand new (identical) servers. The first host came across fine, and joined the cluster without a problem. The second one will join, but HA will not initialize correctly. After a period of time, I just get 'vSphere HA Agent Unreachable' next to the new host. I've dug around, and have found quite a few references to this problem, but nothing seems to fix it. I'm reasonably sure that the very first time I added the new host, everything looked ok in vCenter. I then proceeded to up the EVC level (part of the plan), and although I can't say it happened exectly when I did this, on or around the same time, I started getting the error.
What I've tried.
1. Full reboot of the new host (no change)
2. Checking all management IP's on the hosts (all ok)
3. ICMP check between all cluster hosts and vCenter (all ok)
3. Checked DNS (all ok)
4. Checked: 'vCenter requires verfied SSL host certificates' (all ok)
5. Reverting EVC level back to the original setting (did not fix it)
I've had a look at vpxa.log, and there doesn't seem to be any problems I can see there, but fdm.log is reporting the following;
2014-11-14T14:51:08.306Z [FFD64B90 verbose 'Cluster' opID=SWI-e6ab007a] [ClusterManagerImpl::IsBadIP] 192.168.xxx.122 is bad ip
2014-11-14T14:51:08.306Z [FFD64B90 verbose 'Cluster' opID=SWI-e6ab007a] [ClusterManagerImpl::IsBadIP] 192.168.xxx.113 is bad ip
2014-11-14T14:51:08.359Z [FFD64B90 verbose 'Cluster' opID=SWI-e6ab007a] [ClusterManagerImpl::IsBadIP] 192.168.xxx.116 is bad ip
In this case, xxx.113 is the management IP of another host in the cluster. xxx.116 is the management IP of the cluster master and xxx.122 is another IP address bound to one of the other hosts, just used for mounting NFS volumes (not management traffic). It just repeats these thee addresses over and over as 'is bad ip' in fdm.log. There's nothing wrong with these addresses. They're not duplicates, and are all in DNS correctly.
So, while I can pretty much see the errors causing this, I don't know how to fix this. Could really do with some advice.