Skip to main content

VMware Virtual Machine Backup Fails Due to Backup Window Expiration After CBT Failure

VMware Virtual Machine Backup Fails Due to Backup Window Expiration After CBT Failure

Problem description

A scheduled backup job for a VMware virtual machine (VM) fails to complete within the configured backup window. The backup job starts as an incremental backup, but the agent fails to query VMware Changed Block Tracking (CBT) and falls back to a full disk scan (HCache). For VMs with multi-terabyte virtual disks, the full disk scan cannot complete within the allotted backup window, resulting in a timeout and job termination.

Cause

This issue occurs when VMware CBT fails to return changed disk areas, forcing the Druva Phoenix agent to perform a full disk scan to protect data integrity.

Common causes include:

  • Stale, corrupted, or orphaned CBT tracking files (*-ctk.vmdk).

  • The VM has reached or approached the VMware limit of 32 snapshots.

  • Orphaned snapshots left behind after failed backup or consolidation tasks.

  • Unconsolidated disks requiring manual snapshot consolidation.

Traceback

The Phoenix agent logs (cbtmanager.go) capture the failure to query changed disk areas:

level=error ts=<YYYY-MM-DD>T<HH:MM:SS>Z filename=cbtmanager.go:xxx message="Failed to query change disk area" VMDK="[<Datastore_Name>] <VM_UUID>/<VM_Name>.vmdk" Error="Error caused by file /vmfs/volumes/<Datastore_UUID>/<VM_UUID>/<VM_Name>.vmdk"

The Druva Phoenix Management Console Job Progress log reflects the switch from an incremental scan to a full disk scan prior to the window timeout:

YYYY-MM-DD HH:MM:SS | Starting backup operation YYYY-MM-DD HH:MM:SS | Actual Disk Scan type used: Incremental disk scan YYYY-MM-DD HH:MM:SS | HCache decision: Full disk scan detected - using HCache YYYY-MM-DD HH:MM:SS | Initial attempt Backup window expired YYYY-MM-DD HH:MM:SS | Finished backup operation with status: Failed

Resolution

1. Consolidate and Clean Up VMware Snapshots

  1. Log in to the VMware vSphere Client and select the affected VM.

  2. Check the VM summary banner. If a Configuration Issues alert states that virtual machine disks require consolidation, right-click the VM and select Snapshots > Consolidate.

  3. Open Snapshot Manager and review existing snapshots. Ensure total snapshots do not exceed VMware limits (maximum 32; VMware recommends ≤ 2-3).

  4. Delete any orphaned, stale, or temporary snapshots created during prior backup attempts.

2. Reset Changed Block Tracking (CBT)

  1. Power off the virtual machine (recommended to release file locks on all *-ctk.vmdk files).

  2. Right-click the VM, select Edit Settings, then navigate to VM Options > Advanced > Configuration Parameters > Edit Configuration.

  3. Set the following keys to false:

    • ctkEnabled = false

    • scsi0:0.ctkEnabled = false (Repeat for each attached disk, e.g., scsi0:1, scsi1:0)

  4. Click OK to save the changes.

  5. Using the vSphere Datastore Browser or SSH, open the VM working directory and delete all leftover *-ctk.vmdk files.

  6. Power on the VM.

  7. Create a temporary snapshot in vSphere, then immediately select Delete All snapshots. This forces VMware to update vmware.log and flush the CBT table.

  8. Power off the VM (or perform in-place if live CBT enable is supported), return to Configuration Parameters, and reset the keys to true:

    • ctkEnabled = true

    • scsi0:0.ctkEnabled = true (Repeat for all attached SCSI disks)

  9. Power on the VM, create another temporary snapshot, and delete it to establish a fresh CBT baseline.

3. Adjust Backup Window for Baseline Scan Because the first backup following a CBT reset requires a full disk scan to re-index changed blocks:

  1. In the Druva Phoenix Management Console, open the backup policy assigned to the VM.

  2. Temporarily extend the Backup Window duration to allow sufficient time for a multi-terabyte disk scan.

  3. Save the policy.

4. Trigger and Verify Backup

  1. Manually trigger a backup job for the VM from the Druva Phoenix Management Console.

  2. Confirm the baseline job completes successfully.

  3. Restore the backup window to its original scheduled duration for subsequent incremental backups.

Verification

  1. Review the new backup job progress log in the Druva Phoenix Management Console.

  2. Ensure the log confirms:

    • Successful query of the CBT change area.

    • Actual Disk Scan type used: Incremental disk scan (on the second backup post-reset).

    • Job status indicates Successful without triggering the HCache full scan fallback.

See also

Did this answer your question?

Related Articles