My situation is 6.5 upgrade to 6.7u3
Symptoms
Accessing the vSphere Client (HTML5) of vCenter server in browser is failed with error message:
503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::LocalServiceSpec:0x0000025c4df77820] _serverNamespace = /ui action = Allow _port = 5091)
Services status in vCenter server with command "service-control --status --all" shows the "vsphere-ui" service in STOPPED status.
the vsphere_client_virgo.log file, you see entries similar to:
[2020-01-30T01:30:37.230+08:00] [INFO ] cm-catalog-manager-pool-46 com.vmware.vise.vim.lookup.LsCatalogManager Time to load properties (ms)C:\ProgramData\VMware\vCenterServer\cfg\vsphere-ui\cmCatalog\com.vmware.cis.cs.eam.zip:US: 2
[2020-01-30T01:30:37.931+08:00] [ERROR] SimpleAsyncTaskExecutor-1 com.vmware.vise.extensionfw.ExtensionManager Error retrieving deployment data for the plugin package at C:\ProgramData\VMware\vCenterServer\runtime\vsphere-ui\plugin-packages\vsphere-client
[2020-01-30T01:30:37.942+08:00] [ERROR] SimpleAsyncTaskExecutor-1 com.vmware.vise.util.concurrent.WorkerThread SimpleAsyncTaskExecutor-1 terminated with exception: java.lang.NoClassDefFoundError: java/util/concurrent/ExecutorCompletionService
at com.vmware.vise.extensionfw.impl.PackagesDeployer.<init>(PackagesDeployer.java:160)
at com.vmware.vise.extensionfw.impl.PackagesDeployer.deploy(PackagesDeployer.java:143)
at com.vmware.vise.extensionfw.ExtensionManager.addPluginPackages(ExtensionManager.java:520)
at com.vmware.vise.extensionfw.impl.ExtensionServiceImpl.addPluginPackages(ExtensionServiceImpl.java:164)
at com.vmware.vise.extensionfw.impl.CorePluginsDeploymentActivator.deployPluginsFromDir(CorePluginsDeploymentActivator.java:156)
at com.vmware.vise.extensionfw.impl.CorePluginsDeploymentActivator.deployInternal(CorePluginsDeploymentActivator.java:102)
at com.vmware.vise.extensionfw.impl.CorePluginsDeploymentActivator.access$000(CorePluginsDeploymentActivator.java:27)
at com.vmware.vise.extensionfw.impl.CorePluginsDeploymentActivator$1.run(CorePluginsDeploymentActivator.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [java.util.concurrent.ExecutorCompletionService]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1370)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1220)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1182)
at org.eclipse.equinox.servletbridge.FrameworkLauncher$ChildFirstURLClassLoader.loadClass0(FrameworkLauncher.java:1051)
at org.eclipse.equinox.servletbridge.FrameworkLauncher$ChildFirstURLClassLoader.loadClass(FrameworkLauncher.java:1036)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:371)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:364)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:161)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 9 common frames omitted
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.concurrent.ExecutorCompletionService]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1380)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1368)
... 18 common frames omitted