Problem description
VMware backups in Phoenix fail with internal error 65535. NFC errors are visible in VMware logs. Phoenix logs show the error 14009 - The server refused connection.
Cause
Connectivity issues between the VPXd (vCenter) and VPXa (ESXi) are the root cause of NFC errors. In VMware VDDK 6.7.x and later, the issue can occur if VDDK crashes due to an I/O retry after an NBD asynchronous I/O (AIO) error.
Traceback
Phoenix logs
Log location: PhoenixLogs-Job<jobid>\<backupset>\PhoenixJob<jobid>\Phoenix.<timestamp>
Errors:
[2020-01-27 21:28:47,882] [ERROR] Error in reading from the Disk. Error Code: 14009. Retrying in 30 Seconds. Retry attempt remaining 1
[2020-01-27 21:29:17,905] [ERROR] DataManager: Error in reading data from the Disk [6000C299-9c0b-fa25-9933-edebbb1fa634] at start :10218. vix_error ==== 14009[The server refused connection.]
VMware logs
Log location: Highlight VM > Browse Datastore > Browse the VM folder > Download all vmware.log files.
Errors:
2020-01-27T21:27:42.724+01:00| vthread-8| W115: SSL: Unknown SSL Error
2020-01-27T21:27:42.724+01:00| vthread-8| I125: SSL Error: error:1409F07F:SSL routines:ssl3_write_pending:bad write retry
2020-01-27T21:27:42.724+01:00| vthread-8| W115: [NFC ERROR] NfcNetTcpWrite: bWritten: -1
2020-01-27T21:27:42.724+01:00| vthread-8| W115: [NFC ERROR] NfcSendMessage: send failed: NFC_NETWORK_ERROR
2020-01-27T21:27:42.724+01:00| vthread-8| W115: [NFC ERROR] NfcFssrvr_Close: failed to send close message
2020-01-27T21:27:42.724+01:00| vthread-8| W115: SSL: Unknown SSL Error
2020-01-27T21:27:42.724+01:00| vthread-8| I125: SSL Error: error:1409F07F:SSL routines:ssl3_write_pending:bad write retry
2020-01-27T21:27:42.724+01:00| vthread-8| W115: [NFC ERROR] NfcNetTcpWrite: bWritten: -1
2020-01-27T21:27:42.724+01:00| vthread-8| W115: [NFC ERROR] NfcSendMessage: send failed: NFC_NETWORK_ERROR
2020-01-27T21:27:43.791+01:00| vthread-8| I125: VixDiskLib: VixDiskLib_Close: Close disk.
2020-01-27T21:27:43.920+01:00| vthread-4| I125: VixDiskLib: VixDiskLib_Disconnect: Disconnect.
2020-01-28T18:27:18.959+01:00| vthread-6| I125: SSL: syscall error 110: Connection timed out
2020-01-28T18:27:18.959+01:00| vthread-6| W115: [NFC ERROR] NfcNetTcpWrite: bWritten: -1
2020-01-28T18:27:18.959+01:00| vthread-6| W115:
2020-01-28T18:27:18.959+01:00| vthread-6| W115: [NFC ERROR] NfcFssrvr_IOEx: failed to send io message
2020-01-28T18:27:18.959+01:00| vthread-6| I125: DISKLIB-LIB : RWv failed ioId: #1 (2338) (34)
Resolution
Perform a VMotion of the virtual machine. If this resolves the problem, then it is probably a network issue with the original ESX host.
If the issue persists, perform a Storage VMotion of the virtual machine. If this resolves the problem, then it is probably a network issue between the ESX host and the original datastore.
If the issue persists despite a VMotion and an SVMotion, reboot the ESX host and the vCenter.
For more information on how to resolve NFC error 14009 in VMware VDDK versions 6.7.x and later, see the VMware article VDDK fails with error 14009 while backup application tries to read/write after session is disconnected (71046)