I try to do the convergence from the web UI with 6.7.0.40000 Build 14367737
I have a couple of VCS/PSC on two sites and replication between each PSC
The process succed on first couple, the second failed on step below
{
"key": "firstbootpart1",
"value": {
"description": "Run vmafd firstboot",
"status": "FAILED"
}
},
I think i'm exactly in that case : VMware Knowledge Base
so I disable TSO and GSO as explained on both PSC (source and replicat embedded previously converged)
But know, how can I restart the process ? should I restore backup ?
Another question, since the first converge succeed I haven't yet decommission it, I expect to do that after full convergence, but doesn't know the best practice.
converge.log
2019-10-10T12:28:28.18Z INFO converge Validating Provided Configuration ...
2019-10-10T12:28:28.23Z INFO converge Running the ldapsearch
2019-10-10T12:28:28.23Z INFO converge Running command: ['/opt/likewise/bin/ldapsearch', '-h', 'llivmwpsc01.my.domain.lan', '-w', '******', '-x', '-D', 'cn=administrator,cn=users,dc=vsphere,dc=local', '-b', 'cn=Sites,cn=Configuration,dc=vsphere,dc=local']
2019-10-10T12:28:28.238Z INFO converge Running command ldapsearch successful
2019-10-10T12:28:28.282Z INFO converge Following solutions have been found registered to the Platform Services Controller. You may have to re-register these after convergence. Please refer to the documentation for details.
2019-10-10T12:28:29.134Z INFO converge Infra node url https://llivmwpsc01.my.domain.lan:443/sts/STSService/ is up
2019-10-10T12:28:29.205Z INFO converge Infra node url https://llivmwpsc01.my.domain.lan:443/idm/ is up
2019-10-10T12:28:29.327Z INFO converge Verify AD domain input info.
2019-10-10T12:28:29.327Z INFO converge Skipping AD Domain related checks.
2019-10-10T12:28:29.331Z INFO converge Validation completed successfully.
2019-10-10T12:28:31.384Z INFO converge Validating Provided Configuration ...
2019-10-10T12:28:31.391Z INFO converge Running the ldapsearch
2019-10-10T12:28:31.392Z INFO converge Running command: ['/opt/likewise/bin/ldapsearch', '-h', 'llivmwpsc01.my.domain.lan', '-w', '******', '-x', '-D', 'cn=administrator,cn=users,dc=vsphere,dc=local', '-b', 'cn=Sites,cn=Configuration,dc=vsphere,dc=local']
2019-10-10T12:28:31.639Z INFO converge Running command ldapsearch successful
2019-10-10T12:28:31.684Z INFO converge Following solutions have been found registered to the Platform Services Controller. You may have to re-register these after convergence. Please refer to the documentation for details.
2019-10-10T12:28:32.364Z INFO converge Collected start_time:1570710512.3636036 as a part of Telemetry
2019-10-10T12:28:32.366Z INFO converge Collected build_no:14367737 as a part of Telemetry
2019-10-10T12:28:32.366Z INFO converge Collected version:6.7.0.40000 as a part of Telemetry
2019-10-10T12:28:32.367Z INFO converge Collected client:cli as a part of Telemetry
2019-10-10T12:28:32.367Z INFO converge Collected replication_flag:False as a part of Telemetry
2019-10-10T12:28:32.383Z DEBUG converge Starting new HTTPS connection (1): localhost
2019-10-10T12:28:33.75Z DEBUG converge https://localhost:443 "POST /rest/com/vmware/cis/session HTTP/1.1" 200 None
2019-10-10T12:28:33.88Z DEBUG converge Starting new HTTPS connection (1): localhost
2019-10-10T12:28:34.92Z DEBUG converge https://localhost:443 "GET /rest/vcenter/topology/nodes HTTP/1.1" 200 None
2019-10-10T12:28:34.95Z INFO converge Collected noOfVCs:2 as a part of Telemetry
2019-10-10T12:28:34.96Z INFO converge Collected noOfPSCs:2 as a part of Telemetry
2019-10-10T12:28:34.96Z INFO converge Collected topology info as a part of Telemetry
2019-10-10T12:28:34.185Z INFO converge Infra node url https://llivmwpsc01.my.domain.lan:443/sts/STSService/ is up
2019-10-10T12:28:34.224Z INFO converge Infra node url https://llivmwpsc01.my.domain.lan:443/idm/ is up
2019-10-10T12:28:34.326Z INFO converge Verify AD domain input info.
2019-10-10T12:28:34.326Z INFO converge Skipping AD Domain related checks.
2019-10-10T12:28:34.330Z INFO converge Validation completed successfully.
2019-10-10T12:28:34.330Z INFO converge Setting the converge running flag
2019-10-10T12:28:34.347Z INFO converge Running converge on Management node.
2019-10-10T12:28:34.348Z INFO converge Executing reconfiguring steps. This will take few minutes to complete.
Please wait ...
2019-10-10T12:28:34.348Z INFO converge Downloading RPMs
2019-10-10T12:28:34.349Z INFO converge Collected client:rest as a part of Telemetry
2019-10-10T12:28:34.418Z INFO converge Custom URL repositry not configured.Use default VMware repository URL.
2019-10-10T12:28:34.421Z INFO converge currentURL = https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.5.0.14000.latest/
2019-10-10T12:28:34.421Z INFO converge Manifest file = https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.5.0.14000.latest/manifest/manifest-latest.xml
2019-10-10T12:28:34.422Z INFO converge Running command: ['/usr/bin/wget', '--no-check-certificate']
2019-10-10T12:28:34.545Z INFO converge Downloaded file manifest-latest.xml
2019-10-10T12:28:34.546Z INFO converge Running command: ['/usr/bin/wget', '--no-check-certificate']
2019-10-10T12:28:35.73Z INFO converge Exception in downloading file : manifest-latest.xml.sha256
2019-10-10T12:28:35.91Z INFO converge Unsetting the converge running flag
2019-10-10T12:28:38.199Z INFO converge Cleanup successful with partial flag = True.
2019-10-10T12:28:38.201Z INFO converge Collected end_time:1570710518.2011995 as a part of Telemetry
2019-10-10T12:28:38.201Z INFO converge Collected status:False as a part of Telemetry
2019-10-10T12:28:38.202Z INFO converge Collected @type:convergence_telemetry as a part of Telemetry
2019-10-10T12:28:38.213Z DEBUG converge Starting new HTTP connection (1): localhost
2019-10-10T12:28:38.227Z DEBUG converge http://localhost:15080 "POST /analytics/telemetry/ph/api/hyper/send?_c=vcebc.1_0 HTTP/1.1" 201 0
2019-10-10T12:28:38.232Z DEBUG converge Pushed the Collected Data to VMWare Analytics Service as a part of Telemetry
2019-10-10T14:16:53.773Z INFO converge Validating Provided Configuration ...
2019-10-10T14:16:53.778Z INFO converge Running the ldapsearch
2019-10-10T14:16:53.778Z INFO converge Running command: ['/opt/likewise/bin/ldapsearch', '-h', 'llivmwpsc01.my.domain.lan', '-w', '******', '-x', '-D', 'cn=administrator,cn=users,dc=vsphere,dc=local', '-b', 'cn=Sites,cn=Configuration,dc=vsphere,dc=local']
2019-10-10T14:16:53.990Z INFO converge Running command ldapsearch successful
2019-10-10T14:16:54.37Z INFO converge Following solutions have been found registered to the Platform Services Controller. You may have to re-register these after convergence. Please refer to the documentation for details.
2019-10-10T14:16:58.512Z WARNING converge Unable to query solution regiestered
2019-10-10T14:16:58.638Z INFO converge Infra node url https://llivmwpsc01.my.domain.lan:443/sts/STSService/ is up
2019-10-10T14:16:58.675Z INFO converge Infra node url https://llivmwpsc01.my.domain.lan:443/idm/ is up
2019-10-10T14:16:58.767Z ERROR converge Failed to validate sso credential. Error SoapException:
faultcode: ns0:FailedAuthentication
faultstring: Invalid credentials
faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>
2019-10-10T14:16:58.767Z ERROR converge SSO validation failed..
2019-10-10T14:16:58.773Z ERROR converge Pre-checks failed for converge.
2019-10-10T14:16:58.782Z INFO converge Unsetting the converge running flag
2019-10-10T14:17:02.816Z INFO converge Cleanup successful with partial flag = True.
2019-10-10T14:17:35.329Z INFO converge Validating Provided Configuration ...
2019-10-10T14:17:35.334Z INFO converge Running the ldapsearch
2019-10-10T14:17:35.334Z INFO converge Running command: ['/opt/likewise/bin/ldapsearch', '-h', 'llivmwpsc01.my.domain.lan', '-w', '******', '-x', '-D', 'cn=administrator,cn=users,dc=vsphere,dc=local', '-b', 'cn=Sites,cn=Configuration,dc=vsphere,dc=local']
2019-10-10T14:17:35.596Z INFO converge Running command ldapsearch successful
2019-10-10T14:17:35.689Z INFO converge Following solutions have been found registered to the Platform Services Controller. You may have to re-register these after convergence. Please refer to the documentation for details.
2019-10-10T14:17:36.639Z INFO converge Infra node url https://llivmwpsc01.my.domain.lan:443/sts/STSService/ is up
2019-10-10T14:17:36.675Z INFO converge Infra node url https://llivmwpsc01.my.domain.lan:443/idm/ is up
2019-10-10T14:17:36.763Z INFO converge Running command: ['/opt/likewise/bin/ldapsearch', '-h', 'llivmwpsc01.my.domain.lan', '-w', '******', '-x', '-D', 'cn=administrator,cn=users,dc=vsphere,dc=local', '-b', 'cn=LNIVMWVCT01.my.domain.lan,ou=Domain Controllers,dc=vsphere,dc=local']
2019-10-10T14:17:36.782Z INFO converge Replication partner provided is correct.
2019-10-10T14:17:36.783Z INFO converge Verify AD domain input info.
2019-10-10T14:17:36.784Z INFO converge Skipping AD Domain related checks.
2019-10-10T14:17:36.787Z INFO converge Validation completed successfully.
2019-10-10T14:17:38.940Z INFO converge Validating Provided Configuration ...
2019-10-10T14:17:38.949Z INFO converge Running the ldapsearch
2019-10-10T14:17:38.949Z INFO converge Running command: ['/opt/likewise/bin/ldapsearch', '-h', 'llivmwpsc01.my.domain.lan', '-w', '******', '-x', '-D', 'cn=administrator,cn=users,dc=vsphere,dc=local', '-b', 'cn=Sites,cn=Configuration,dc=vsphere,dc=local']
2019-10-10T14:17:39.195Z INFO converge Running command ldapsearch successful
2019-10-10T14:17:39.297Z INFO converge Following solutions have been found registered to the Platform Services Controller. You may have to re-register these after convergence. Please refer to the documentation for details.
2019-10-10T14:17:40.169Z INFO converge Collected start_time:1570717060.1686823 as a part of Telemetry
2019-10-10T14:17:40.171Z INFO converge Collected build_no:14367737 as a part of Telemetry
2019-10-10T14:17:40.172Z INFO converge Collected version:6.7.0.40000 as a part of Telemetry
2019-10-10T14:17:40.172Z INFO converge Collected client:cli as a part of Telemetry
2019-10-10T14:17:40.173Z INFO converge Collected replication_flag:False as a part of Telemetry
2019-10-10T14:17:40.188Z DEBUG converge Starting new HTTPS connection (1): localhost
2019-10-10T14:17:41.12Z DEBUG converge https://localhost:443 "POST /rest/com/vmware/cis/session HTTP/1.1" 200 None
2019-10-10T14:17:41.26Z DEBUG converge Starting new HTTPS connection (1): localhost
2019-10-10T14:17:42.864Z DEBUG converge https://localhost:443 "GET /rest/vcenter/topology/nodes HTTP/1.1" 200 None
2019-10-10T14:17:42.867Z INFO converge Collected noOfVCs:1 as a part of Telemetry
2019-10-10T14:17:42.868Z INFO converge Collected noOfPSCs:2 as a part of Telemetry
2019-10-10T14:17:42.868Z INFO converge Collected topology info as a part of Telemetry
2019-10-10T14:17:42.954Z INFO converge Infra node url https://llivmwpsc01.my.domain.lan:443/sts/STSService/ is up
2019-10-10T14:17:42.991Z INFO converge Infra node url https://llivmwpsc01.my.domain.lan:443/idm/ is up
2019-10-10T14:17:43.70Z INFO converge Running command: ['/opt/likewise/bin/ldapsearch', '-h', 'llivmwpsc01.my.domain.lan', '-w', '******', '-x', '-D', 'cn=administrator,cn=users,dc=vsphere,dc=local', '-b', 'cn=LNIVMWVCT01.my.domain.lan,ou=Domain Controllers,dc=vsphere,dc=local']
2019-10-10T14:17:43.92Z INFO converge Replication partner provided is correct.
2019-10-10T14:17:43.93Z INFO converge Verify AD domain input info.
2019-10-10T14:17:43.94Z INFO converge Skipping AD Domain related checks.
2019-10-10T14:17:43.99Z INFO converge Validation completed successfully.
2019-10-10T14:17:43.100Z INFO converge Setting the converge running flag
2019-10-10T14:17:43.119Z INFO converge Running converge on Management node.
2019-10-10T14:17:43.120Z INFO converge Executing reconfiguring steps. This will take few minutes to complete.
Please wait ...
2019-10-10T14:17:43.120Z INFO converge Downloading RPMs
2019-10-10T14:17:43.121Z INFO converge Collected client:rest as a part of Telemetry
2019-10-10T14:17:43.169Z INFO converge VCSA ISO is mounted. Copying RPMs from the ISO.
2019-10-10T14:17:43.180Z INFO converge ISO version : VMware-vCenter-Server-Appliance-6.7.0.40000-14367737
2019-10-10T14:17:43.180Z INFO converge VC version : VMware-vCenter-Server-Appliance-6.7.0.40000-14367737
2019-10-10T14:17:48.89Z INFO converge Collected site_guid:af685cec-89f9-4dca-872c-1b40c56be288 as a part of Telemetry
2019-10-10T14:17:48.90Z DEBUG converge e5dd6353-547d-4ecc-b908-1cc1a2156dab
2019-10-10T14:17:48.126Z DEBUG converge e4729eae-0fc8-451a-889f-20d1b60d49d7
2019-10-10T14:17:48.465Z INFO converge Store all vecs certificates.
2019-10-10T14:17:49.311Z INFO converge final json ---->
2019-10-10T14:17:49.312Z INFO converge [{'entries': [{'Entry type': 'Private Key', 'Alias': '__MACHINE_CERT', 'cert_path': '/root/velma/old_certs/__MACHINE_CERT-MACHINE_SSL_CERT.crt', 'key_path': '/root/velma/old_certs/__MACHINE_CERT-MACHINE_SSL_CERT.key'}], 'store_name': 'MACHINE_SSL_CERT', 'Number of entries in store': '1'}, {'entries': [{'Entry type': 'Private Key', 'Alias': 'machine', 'cert_path': '/root/velma/old_certs/machine-machine.crt', 'key_path': '/root/velma/old_certs/machine-machine.key'}], 'store_name': 'machine', 'Number of entries in store': '1'}, {'entries': [{'Entry type': 'Private Key', 'Alias': 'vsphere-webclient', 'cert_path': '/root/velma/old_certs/vsphere-webclient-vsphere-webclient.crt', 'key_path': '/root/velma/old_certs/vsphere-webclient-vsphere-webclient.key'}], 'store_name': 'vsphere-webclient', 'Number of entries in store': '1'}, {'entries': [{'Entry type': 'Private Key', 'Alias': 'vpxd', 'cert_path': '/root/velma/old_certs/vpxd-vpxd.crt', 'key_path': '/root/velma/old_certs/vpxd-vpxd.key'}], 'store_name': 'vpxd', 'Number of entries in store': '1'}, {'entries': [{'Entry type': 'Private Key', 'Alias': 'vpxd-extension', 'cert_path': '/root/velma/old_certs/vpxd-extension-vpxd-extension.crt', 'key_path': '/root/velma/old_certs/vpxd-extension-vpxd-extension.key'}], 'store_name': 'vpxd-extension', 'Number of entries in store': '1'}, {'entries': [{'Entry type': 'Private Key', 'Alias': 'sms_self_signed', 'cert_path': '/root/velma/old_certs/sms_self_signed-SMS.crt', 'key_path': '/root/velma/old_certs/sms_self_signed-SMS.key'}], 'store_name': 'SMS', 'Number of entries in store': '1'}, {'entries': [], 'store_name': 'APPLMGMT_PASSWORD', 'Number of entries in store': '0'}, {'entries': [{'Entry type': 'Private Key', 'Alias': 'data-encipherment', 'cert_path': '/root/velma/old_certs/data-encipherment-data-encipherment.crt', 'key_path': '/root/velma/old_certs/data-encipherment-data-encipherment.key'}], 'store_name': 'data-encipherment', 'Number of entries in store': '1'}]
2019-10-10T14:17:49.505Z INFO converge Removing computer account of vCenter on Management node.
2019-10-10T14:17:49.506Z INFO converge Running command: ['/usr/lib/vmware-vmafd/bin/domainjoin', 'leave', '--username', 'administrator', '--password', '******']
2019-10-10T14:17:49.745Z INFO converge Removed vCenter computer account from management node.
2019-10-10T14:17:49.757Z INFO converge Running Firstboots for : vmafd-firstboot
2019-10-10T14:17:59.437Z INFO converge Stopping the services ...
2019-10-10T14:17:59.445Z INFO converge <class 'list'>
2019-10-10T14:17:59.446Z INFO converge ['vmware-certificatemanagement', 'vmware-pod', 'vmware-topologysvc', 'vmware-updatemgr', 'vmware-vcha', 'vmcam', 'vmware-postgres-archiver', 'vsan-dps', 'vmware-perfcharts', 'vmware-vsan-health', 'vsphere-ui', 'vmware-eam', 'vmware-imagebuilder', 'vmware-mbcs', 'vmware-netdumper', 'vmware-rbd-watchdog', 'vmware-vsm', 'vsphere-client', 'vmware-content-library', 'vmware-sps', 'vmware-vpxd', 'vmware-vpxd-svcs', 'vmware-vpostgres']
2019-10-10T14:17:59.446Z INFO converge Stopping services
2019-10-10T14:20:25.24Z INFO converge Stopped all the services.
2019-10-10T14:20:25.54Z INFO converge Removed Old Afd db.
2019-10-10T14:20:25.68Z INFO converge Cleared old ssl certificates.
2019-10-10T14:20:25.252Z INFO converge
2019-10-10T14:20:25.253Z INFO converge Installing PSC Rpms.
2019-10-10T14:20:25.253Z INFO converge Installing rpm: /root/velma/rpm/vmware-directory-6.7.0.4837-14192624.x86_64.rpm
2019-10-10T14:20:26.240Z INFO converge Installing rpm: /root/velma/rpm/vmware-certificate-server-6.7.0.5602-14192633.x86_64.rpm
2019-10-10T14:20:27.128Z INFO converge Installing rpm: /root/velma/rpm/vmware-psc-health-6.7.0.1949-14192626.x86_64.rpm
2019-10-10T14:20:27.494Z INFO converge Installing rpm: /root/velma/rpm/VMware-cis-license-6.7.0-13714356.x86_64.rpm
2019-10-10T14:20:28.220Z INFO converge Installing rpm: /root/velma/rpm/vmware-identity-sts-6.7.0.6011-14192627.noarch.rpm
2019-10-10T14:20:35.690Z INFO converge Installing rpm: /root/velma/rpm/vmware-dns-server-1.0.0-14192631.x86_64.rpm
2019-10-10T14:20:36.509Z INFO converge PSC Rpms Installed Success.
2019-10-10T14:20:36.510Z INFO converge Running the ldapsearch
2019-10-10T14:20:36.511Z INFO converge Running command: ['/opt/likewise/bin/ldapsearch', '-h', 'llivmwpsc01.my.domain.lan', '-w', '******', '-x', '-D', 'cn=administrator,cn=users,dc=vsphere,dc=local', '-b', 'cn=ConsentConfigContainer,cn=PhoneHome,cn=Services,dc=vsphere,dc=local']
2019-10-10T14:20:36.532Z INFO converge Running command successful
2019-10-10T14:20:36.533Z INFO converge CEIP is disabled.
2019-10-10T14:20:36.533Z INFO converge Setting Install Parameters
2019-10-10T14:20:36.543Z INFO converge Install Parameters Set.
2019-10-10T14:20:37.664Z INFO converge Perform update startuptype operation in start order. startup_type=Automatic, svc_names=['vmdird', 'vmcad', 'vmdnsd', 'vmware-sts-idmd', 'vmware-certificatemanagement'], include_vmonsvcs=False include_coreossvcs=False, include_leafossvcs=False
2019-10-10T14:20:37.979Z INFO converge Successfully changed startuptype for service vmdird
2019-10-10T14:20:38.441Z INFO converge Successfully changed startuptype for service vmcad
2019-10-10T14:20:39.9Z INFO converge Successfully changed startuptype for service vmware-sts-idmd
2019-10-10T14:20:39.582Z INFO converge Successfully changed startuptype for service vmdnsd
2019-10-10T14:20:39.601Z INFO converge Successfully changed startuptype for service certificatemanagement
2019-10-10T14:20:39.605Z INFO converge Collected replication_flag:True as a part of Telemetry
2019-10-10T14:20:39.606Z INFO converge Running command: ['/opt/likewise/bin/ldapsearch', '-h', 'LNIVMWVCT01.my.domain.lan', '-w', '******', '-x', '-D', 'cn=administrator,cn=users,dc=vsphere,dc=local', '-b', 'cn=slivctapp.my.domain.lan,ou=Computers,dc=vsphere,dc=local']
2019-10-10T14:20:39.677Z WARNING converge Unable to run ldapsearch
2019-10-10T14:20:39.678Z INFO converge Collected replication_latency:0.0727686882019043 as a part of Telemetry
2019-10-10T14:20:39.678Z INFO converge Leave fed is complete
2019-10-10T14:20:39.679Z INFO converge Leave fed complete status: True
2019-10-10T14:20:39.679Z INFO converge Running Firstboots for : visl-support-firstboot,vmafd-firstboot
2019-10-10T14:26:37.832Z ERROR converge Failed to run white listed firstboot scripts. Error: {
"problemId": null,
"detail": [
{
"id": "install.ciscommon.command.errinvoke",
"args": [
"Command: ['run-firstboot-scripts', '--action', 'firstboot', '--subaction', 'firstboot', '--fbWhiteList', 'visl-support-firstboot,vmafd-firstboot']\nStderr: "
],
"translatable": "An error occurred while invoking external command : '%(0)s'",
"localized": "An error occurred while invoking external command : 'Command: ['run-firstboot-scripts', '--action', 'firstboot', '--subaction', 'firstboot', '--fbWhiteList', 'visl-support-firstboot,vmafd-firstboot']\nStderr: '"
}
],
"componentKey": null,
"resolution": null
}
2019-10-10T14:26:37.849Z INFO converge Unsetting the converge running flag
2019-10-10T14:26:42.103Z INFO converge Cleanup successful with partial flag = True.
2019-10-10T14:26:42.104Z INFO converge Collected end_time:1570717602.1046734 as a part of Telemetry
2019-10-10T14:26:42.105Z INFO converge Collected status:False as a part of Telemetry
2019-10-10T14:26:42.105Z INFO converge Collected @type:convergence_telemetry as a part of Telemetry
2019-10-10T14:26:42.118Z DEBUG converge Starting new HTTP connection (1): localhost
2019-10-10T14:26:42.130Z DEBUG converge http://localhost:15080 "POST /analytics/telemetry/ph/api/hyper/send?_c=vcebc.1_0 HTTP/1.1" 201 0
2019-10-10T14:26:42.133Z DEBUG converge Pushed the Collected Data to VMWare Analytics Service as a part of Telemetry
Thanks for your help