I have a previously working vcenter appliance 6.5 that was in a bad state after an unclean shutdown from a thunderstorm. Initially it was not fully booting, but I ran fsck on the volumes that needed it, and it gets all the way to the running info screen. After that vpxd fails to fully start. I can't find any more volumes that are in a bad state, but it appears that I might have some SQL corruption. Note the SQL error below.
Error message when trying to open web client is: 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007faab404a230] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
Output of service-control --status --all
Running:
applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-analytics vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
Stopped:
vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-vcha vmware-vpxd vsan-dps
Grep of all lines with error in them from vpxd log file:
2020-05-23T15:13:12.693Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Unable to read from '/etc/motd':N7Vmacore23FileIONotFoundExceptionE(Could not find file : /etc/motd)
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: AgentUpgrade.autoUpgradeAgents
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: AgentUpgrade.checkPeriodSeconds
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: alarms.upgraded
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Purge1
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Purge2
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Purge3
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Rollup1
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Rollup2
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Rollup3
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Calc1
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Calc2
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Calc3
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Calc4
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Purge1
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Purge2
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Purge3
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Purge4
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_LOG_BUFFER
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_LOGING_MODE
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_STATS_DELAY_MINS_1
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_STATS_DELAY_MINS_2
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_STATS_DELAY_MINS_3
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_STATS_DELAY_MINS_4
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: VirtualCenter.CacheSize
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: VirtualCenter.LDAPAdminPrincipal
2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: VirtualCenter.VimWebServicesUrl
2020-05-23T15:13:12.704Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Skipping bad entry config.vpxd.enableDebugBrowse from DB. Resetting to default.Exception: Fault cause: vmodl.fault.InvalidArgument
2020-05-23T15:13:15.388Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.cluster.VsanClusterHclInfo already exists.
2020-05-23T15:13:15.388Z error vpxd[04354] [Originator@6876 sub=Vsan] Managed type vim.cluster.VsanClusterHealthSystem already exists.
2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VsanHclControllerInfo already exists.
2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Managed type vim.host.VsanHealthSystem already exists.
2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VsanHostHclInfo already exists.
2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Managed type vim.host.VsanDiskManagementSystem already exists.
2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.vsan.host.DiskMapInfoEx already exists.
2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.vsan.host.VsanDiskManagementSystemCapability already exists.
2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANCmmdsFaultDomainInfo already exists.
2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANCmmdsNodeInfo already exists.
2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANCmmdsPreferredFaultDomainInfo already exists.
2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANStretchedClusterHostCapability already exists.
2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANStretchedClusterHostInfo already exists.
2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Managed type vim.host.VsanStretchedClusterSystem already exists.
2020-05-23T15:13:19.242Z error vpxd[04354] [Originator@6876 sub=profileUtil] [DeserializeFromFile] reading failed: Could not find file : /var/lib/vmware/hpMetadataCache.xml
2020-05-23T15:13:22.233Z error vpxd[04569] [Originator@6876 sub=Main opID=CheckCertificateExpiry-35702e2f] Unable to get certificate count for APPLMGMT_PASSWORD from VECS localhost, error: 0
2020-05-23T15:13:23.413Z error vpxd[04435] [Originator@6876 sub=vmomi.soapStub[12]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'
2020-05-23T15:13:23.729Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\Administrator"
2020-05-23T15:13:23.729Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for group "CAROLINA\Domain Admins"
2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\Administrator"
2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for group "VSPHERE.LOCAL\Administrators"
2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vsphere-webclient-80cfe84e-9df1-47e4-93e3-d2dfc3e72d72"
2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-80cfe84e-9df1-47e4-93e3-d2dfc3e72d72"
2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-extension-80cfe84e-9df1-47e4-93e3-d2dfc3e72d72"
2020-05-23T15:13:24.346Z error vpxd[04668] [Originator@6876 sub=vmomi.soapStub[13]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'
2020-05-23T15:13:25.152Z error vpxd[04354] [Originator@6876 sub=OsLayer_linux] [VpxOsLayer] Failed to write to config: Permission denied for file : /etc/vmware-vpx/vpxd.cfg.tmp
2020-05-23T15:14:05.925Z error vpxd[04454] [Originator@6876 sub=[SSO] opID=26e74229] [UserDirectorySso] GetUserInfo exception: N7Vmacore9Authorize25AuthUserNotFoundExceptionE(User localos\com.vmware.vim.eam)
2020-05-23T15:14:05.927Z error vpxd[04454] [Originator@6876 sub=[SSO] opID=26e74229] [UserDirectorySso] NormalizeUserName(com.vmware.vim.eam, false) exception: N7Vmacore9Authorize25AuthUserNotFoundExceptionE(User localos\com.vmware.vim.eam)
2020-05-23T15:14:10.713Z error vpxd[04429] [Originator@6876 sub=HttpSvc.HTTPService] Failed to read request; stream: <io_obj p:0x00007f4f5039b388, h:-1, <UNIX '/var/run/vmware/vpxd-webserver-pipe'>, <UNIX ''> FD Closed>, error: N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x00007f4f5039b388, h:-1, <UNIX '/var/run/vmware/vpxd-webserver-pipe'>, <UNIX ''> FD Closed>, duration: 00:00:45.250489 (hh:mm:ss.us))
2020-05-23T15:16:17.602Z error vpxd[04458] [Originator@6876 sub=MoEnvBrowser opID=75353f56] Can not find option descriptor for key vmx-14
2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Execute result code: -1
2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] SQL execution failed: select rule_topn1_proc()
2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Execution elapsed time: 11 ms
2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Statement diagnostic data from driver is XX001:0:1:ERROR: could not read block 2389 in file "base/16395/32425": read only 0 of 8192 bytes;
2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Connection diagnostic data from driver is XX001:0:110:ERROR: could not read block 2389 in file "base/16395/32425": read only 0 of 8192 bytes
2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Bind parameters:
2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from XX001:1
2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from XX001:110
2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] SQLError was thrown: "ODBC error: (XX001) - ERROR: could not read block 2389 in file "base/16395/32425": read only 0 of 8192 bytes;