503 error on vCSA 6.5.0a running on ESXi 6.5.0a
So, I opened a case with VMware and it’s a known issue for vCenter 6.5.0a. The below fix directly solved the error I was seeing.
Information attached for reference only. Please be aware that this may NOT be applicable to your system and may bork it completely!
On the webclient, the web page shows the error: 503 Service Unavailable (Failed to connect to endpoint). . .
PowerCLI cannot connect to vCenter.
vpxd on vCenter Server 6.5 intermittently crashes with the following error stack in the vpxd log file:
|
1 2 |
2016-11-28T02:09:36.418Z error vpxd[7F27BD4A9700] [Originator@6876 sub=Default opID=HB-host-83@226512-71ec21d0] [VdbStatement] SQLError was thrown: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device"; --> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)" |
- The VMware vCenter Server 6.5 vpxd logs are located in the %ALLUSERSPROFILE%\VMWare\vCenterServer\logs\vmware-vpx folder.
- The VMware vCenter Server Appliance vpxd 6.5 logs are located in the /var/log/vmware/vmware-vpx folder.
Before making database modifications, ensure there is a working backup of the database or snapshot.
If the embedded postgres database is being used, locate the duplicate key in the postgresql logs:
postgresql.log
|
1 2 |
UTC 583a9c81.5814 127750 VCDB vc ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device" UTC 583a9c81.5814 127750 VCDB vc DETAIL: Key (id, device_key)=(361, 3002) already exists. |
The postgresql.log can be found in var/log/vmware/vpostgres
To connect to the embedded postgres database:
|
1 |
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres |
To remove the duplicate key run the below command:
(modify the id and device_key according to your vpxd log below!):
|
1 |
DELETE FROM vc.vpx_vm_virtual_device where id='361' and device_key='3002'; |