Skip to main content
All CollectionsKnowledge BaseEnterprise WorkloadsTroubleshooting - Enterprise Workloads
VM backup fails with the VMWARE VDDK65535 error (memory allocation error)
VM backup fails with the VMWARE VDDK65535 error (memory allocation error)
Updated over a week ago

This article applies to:

  • OS: ESX 5.0 and later

  • Product edition: Phoenix

Problem description

VM backup fails with the VMWARE VDDK65535 error (memory allocation error)

Traceback

Phoenix.log :
[2022-03-13 22:15:18,232] [ERROR] DataManager: Error in reading data from the Disk [6000c292-8b6c-4ec5-a71d-fa4c002829ac] at start :0. vix_error ==== 2[Memory allocation failed. Out of memory.]
[2022-03-13 22:15:18,234] [ERROR] Error in reading from the Disk. Error Code: 2. Retrying in 30 Seconds. Retry attempt remaining 2
[2022-03-13 22:15:18,235] [ERROR] 140153062401792 Backup of VMDK Failed. Fault: VMware VDDK: Internal Error (#10005ffff : 2) (Error Code : VMWARE_VDDK65535)
VDDK logs :
2022-03-13T22:14:18.193Z| vthread-12| E110: VixDiskLib: Detected DiskLib error 802 (NBD_ERR_INSUFFICIENT_RESOURCES).
2022-03-13T22:14:18.193Z| vthread-12| E110: VixDiskLib: VixDiskLib_Read: Read 8192 sectors at 0 failed. Error 2 (Memory allocation failed. Out of memory.) (DiskLib error 802: NBD_ERR_INSUFFICIENT_RESOURCES) at 5240

Cause

  • This issue is related to a VMWare decision to limit the NBD protocol to a maximum 32MB read buffer by default on ESXi hosts.

  • There is a limitation with ESX/VC host on NFC session (NBD/NBDSSL Transport) https://pubs.vmware.com/vsphere-50/i...truct.5.5.html

  • NFC session has a limitation of 32 MB buffer allocation. If it exceeds 32 MB, NFC session fails.

  • The NFS buffer is calculated by the following formula.

  • NFS Buffer Size = Number of VMDKS x Block Size

Example:

In a Phoenix Backup Proxy, if you have set a block size of 4 MB and VM has 11 VMDKs connected using NBDSSL, the NFS buffer size will be 44 MB, which exceeds the limit.


πŸ“ Note
​This issue will occur when a substantial number of backups use the NBD or NBDSSL protocol for backup.


Considerations

  • Since this limitation is not valid for HOTADD transport mode, ensure that the HOTADD options work to avoid this error.

  • Deploy a proxy in the same cluster that contains vCenter (target VM) to back up the VM.

  • If a backup is not possible via HOTADD, adjust the block size to fix the issue.

Resolution

For VMware backup proxy version 7.0.0 and later, perform the following steps:

  1. Go to the configuration file at /etc/Druva/EnterpriseWorkloads/vmware/VMwareConfig.yaml, and add the following parameter under the drst section:

    vm_backup_block_size: 1048576

  2. Restart the service:

    service Druva-EnterpriseWorkloads restart

For VMware backup proxy version prior to 7.0.0, perform the following steps:

  1. Log in to the VMWARE Backup proxy with default username root and password.

  2. Ensure there are no jobs running.

  3. Stop the Phoenix Agent service on the backup proxy using the command:
    ​

    /etc/init.d/Phoenix stop 
  4. Verify if any Phoenix processes are still running.
    ​

    ps -ef | grep -i Phoenix
  5. If any process is still found running, kill the process with the following command:
    ​

    Kill -9 <process id>
  6. Go to the path below:
    ​

    cd /etc/Phoenix/VMWARE
  7. Edit Phoenix.cfg file using vi editor using the below command:
    ​

    vi Phoenix.cfg 
        Type β€˜ i β€˜ to activate insert mode 
        Add below line
        _VM_BACKUP_BLOCK_SIZE = 1048576
        Type esc key followed by :wq to save and exit file.


    *This sets the block size to 1 MB

  8. Verify if the above lines have been added using the below command
    ​

    cat Phoenix.cfg
  9. Start the Phoenix Agent service on the backup proxy as follows:
    ​

    /etc/init.d/Phoenix start
  10. Trigger the backup or restore

Did this answer your question?