Problem description

VM backup fails with the VMWARE VDDK65535 error (memory allocation error)

Traceback

Phoenix.log :
[2022-03-13 22:15:18,232] [ERROR] DataManager: Error in reading data from the Disk [6000c292-8b6c-4ec5-a71d-fa4c002829ac] at start :0. vix_error ==== 2[Memory allocation failed. Out of memory.]
[2022-03-13 22:15:18,234] [ERROR] Error in reading from the Disk. Error Code: 2. Retrying in 30 Seconds. Retry attempt remaining 2
[2022-03-13 22:15:18,235] [ERROR] 140153062401792 Backup of VMDK Failed. Fault: VMware VDDK: Internal Error (#10005ffff : 2) (Error Code : VMWARE_VDDK65535)

VDDK logs :
2022-03-13T22:14:18.193Z| vthread-12| E110: VixDiskLib: Detected DiskLib error 802 (NBD_ERR_INSUFFICIENT_RESOURCES).
2022-03-13T22:14:18.193Z| vthread-12| E110: VixDiskLib: VixDiskLib_Read: Read 8192 sectors at 0 failed. Error 2 (Memory allocation failed. Out of memory.) (DiskLib error 802: NBD_ERR_INSUFFICIENT_RESOURCES) at 5240

Cause

This issue is related to a VMWare decision to limit the NBD protocol to a maximum 32MB read buffer by default on ESXi hosts.
There is a limitation with ESX/VC host on NFC session (NBD/NBDSSL Transport) NFC Session Limits and Timeouts
NFC session has a limitation of 32 MB buffer allocation. If it exceeds 32 MB, NFC session fails.
The NFS buffer is calculated by the following formula.
NFS Buffer Size = Number of VMDKS x Block Size

Example:

In a Phoenix Backup Proxy, if you have set a block size of 4 MB and VM has 11 VMDKs connected using NBDSSL, the NFS buffer size will be 44 MB, which exceeds the limit.

📝 Note
This issue will occur when a substantial number of backups use the NBD or NBDSSL protocol for backup.

Considerations

Since this limitation is not valid for HOTADD transport mode, ensure that the HOTADD options work to avoid this error.
Deploy a proxy in the same cluster that contains vCenter (target VM) to back up the VM.
If a backup is not possible via HOTADD, adjust the block size to fix the issue.

Resolution

For VMware backup proxy version 7.0.0 and later, perform the following steps:

Go to the configuration file at /etc/Druva/EnterpriseWorkloads/vmware/VMwareConfig.yaml, and add the following parameter under the drst section:
vm_backup_block_size: 1048576
Restart the service:
service Druva-EnterpriseWorkloads restart

For VMware backup proxy version prior to 7.0.0, perform the following steps:

Log in to the VMWARE Backup proxy with default username root and password.
Ensure there are no jobs running.
Stop the Phoenix Agent service on the backup proxy using the command:
```
/etc/init.d/Phoenix stop 
```
Verify if any Phoenix processes are still running.
```
ps -ef | grep -i Phoenix
```
If any process is still found running, kill the process with the following command:
```
Kill -9 <process id>
```
Go to the path below:
```
cd /etc/Phoenix/VMWARE
```

Edit Phoenix.cfg file using vi editor using the below command:

vi Phoenix.cfg 
    Type ‘ i ‘ to activate insert mode 
    Add below line
    _VM_BACKUP_BLOCK_SIZE = 1048576
    Type esc key followed by :wq to save and exit file.

*This sets the block size to 1 MB

Verify if the above lines have been added using the below command
```
cat Phoenix.cfg
```
Start the Phoenix Agent service on the backup proxy as follows:
```
/etc/init.d/Phoenix start
```
Trigger the backup or restore

What is a backup proxy for VMware

Encrypted VM backup shows the Invalid configuration for device error on VMWare backup proxy

Scenarios when the performance of an incremental VM backup is impacted and the job fails with the Backup Window Expired error.

VMware VM Restore fails with Error VMWARE VDDK1

VMware VM backups fails with Phoenix 1 Internal Error