After migrating from Windows 5.5 vCenter to VCSA 6.5 Update1 (intermediate step to 6.7) everything was working well. After the migration the vcenter machine cert was soon to be expired so I replaced it, though I did notice https://vc.domain.com:7444 was still using the old cert. I didn't worry about this as the old cert was not expired. A month or so passes by with vcenter being very stable. Completed host upgrades to 6.5U1 using VUM over the weekend - In the mean time the cert bound to 7444 expired, which caused problems with the Migration wizard to 6.7. Also, VUM service had stopped and wouldn't start (perhaps it was ok until I rebooted the VCSA and now VUM wont start)
With the help of VMWare support we fixed this by replacing the cert in the STS_INTERNAL_SSL_CERT. This fixed the expired cert for 7444 but did not help with the VUM issues. I'm pretty well stuck, I am just waiting on VMWare support to help. In the mean time I thought I'd post here in case someone could possible help.
Regards
I get the following errors in the SSH session when executing service-control --start vmware-updatemgr:
Error executing start on service updatemgr. Details {
"resolution": null,
"detail": [
{
"args": [
"updatemgr"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'updatemgr'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Service-control failed. Error {
"resolution": null,
"detail": [
{
"args": [
"updatemgr"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'updatemgr'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
tail -f updatemgr-utility.log give this:
[2018-10-10 23:29:10,255 INFO] Install Key store for Jetty
[2018-10-10 23:29:11,579 INFO] Keystore installed successfully.
[2018-10-10 23:29:11,953 INFO] Updating VUM extension with VC
[2018-10-10 23:29:12,236 INFO] Updating CM service info
[2018-10-10 23:29:12,402 ERROR] CM ReRegisterService failure. Exception is (cis.cm.fault.ComponentManagerFault) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = '',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
errorCode = 0,
errorMessage = 'UNKNOWN'
}
[2018-10-10 23:29:12,402 ERROR] Unable to update CM service info
Tail on cm.log give me this while starting service:
2018-10-10T23:24:35.847Z [pool-2-thread-1 [] WARN com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore (f15657f3-ac72-40f0-8c90-3e948201000c)] Call to lookup service failed; uri:https://VCENTER.DoMain.int/lookupservice/sdk [(vmodl.fault.InvalidArgument) {
faultCause = null,
faultMessage = null,
invalidProperty = Invalid certificate
}]
2018-10-10T23:24:35.847Z [pool-2-thread-1 [] ERROR com.vmware.cis.services.cm.service.ServiceManagerImplTemplate (f15657f3-ac72-40f0-8c90-3e948201000c)] reRegisterService v1: Failed to re-register c39131ca-cda5-446d-a6ac-44b51348c107 (vpxd-extension-a6f84462-5ea2-11e6-ada6-0050569777b5@vsphere.local, com.vmware.vcIntegrity/vcIntegrity 6.5.0)
com.vmware.vim.binding.vmodl.fault.InvalidArgument: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_141]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_141]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_141]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_141]
at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_141]
at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:150) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:101) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:88) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:83) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:40) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:274) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:613) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:594) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:345) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:305) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:179) ~[vlsi-client.jar:?]
at com.sun.proxy.$Proxy101.set(Unknown Source) ~[?:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper$3.execute(LsVmomiSiteStore.java:229) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper$3.execute(LsVmomiSiteStore.java:226) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper.callLs(LsVmomiSiteStore.java:302) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper.set(LsVmomiSiteStore.java:224) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore.updateService(LsVmomiSiteStore.java:622) ~[service-cm.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.vmware.cis.services.common.perfmon.PerfmonInterceptor.invoke(PerfmonInterceptor.java:31) ~[service-common.jar:?]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.sun.proxy.$Proxy67.updateService(Unknown Source) ~[?:?]
at com.vmware.cis.services.cm.service.ServiceManagerImplTemplate.reRegisterService(ServiceManagerImplTemplate.java:306) [service-cm.jar:?]
at com.vmware.cis.services.cm.service.ServiceManagerImpl.reRegisterService(ServiceManagerImpl.java:291) [service-cm.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.vmware.cis.services.common.perfmon.PerfmonInterceptor.invoke(PerfmonInterceptor.java:31) [service-common.jar:?]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.sun.proxy.$Proxy68.reRegisterService(Unknown Source) [?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:65) [vlsi-server.jar:?]
at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47) [vlsi-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
tcpdump on lo gives:
Request
POST /lookupservice/sdk HTTP/1.1
Content-Type: text/xml; charset=utf-8
SOAPAction: urn:lookup/2.0
Content-Length: 21474
Host: https://VCENTER.DoMain.int/lookupservice/sdk
Connection: Keep-Alive
User-Agent: VMware vim-java 1.0
Cookie: vmware_soap_session=a7600162-79b1-4797-9e7f-4b885dde550b
Accept-Encoding: gzip,deflate
X-Forwarded-For: 127.0.0.1
X-Forwarded-Proto: https
....CERT HASH in XML stream.........
Request
HTTP/1.1 500
Set-Cookie: vmware_soap_session=a7600162-79b1-4797-9e7f-4b885dde550b; HttpOnly
Content-Type: text/xml;charset=utf-8
Content-Length: 558
Date: Wed, 10 Oct 2018 01:45:27 GMT
Connection: close
<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soapenv:Body><soapenv:Fault><faultcode>ServerFaultCode</faultcode><faultstring/><detail><RuntimeFaultFault xsi:type="InvalidArgument" xmlns="urn:lookup"><invalidProperty>Invalid certificate</invalidProperty></RuntimeFaultFault></detail></soapenv:Fault></soapenv:Body></soapenv:Envelope>
Clik here to view.