Skip to main content

VMware Backup Fails with "Nonmatching Change UUID" Error

VMware Backup Fails with "Nonmatching Change UUID" Error

Problem description

VMware backup jobs fail for one or more configured virtual machines during scheduled or manual backup operations. The backup fails during block analysis when querying changed disk sectors, resulting in job failure or an unexpected fallback to full disk scanning.This issue typically occurs after manual snapshot manipulation, storage migration (Storage vMotion), disk geometry changes, or an unexpected host shutdown/crash while CBT tracking is active.

Cause

VMware Changed Block Tracking (CBT) utilizes .ctk files on the datastore to track blocks modified since the last backup baseline. The error is caused by:

  • Snapshot Chain Discrepancy: Unconsolidated or orphaned VMware snapshots disrupting the sequential change ID chain between the Druva backup index and VMware vSphere.

  • Metadata Mismatch Following Storage Migration: Storage vMotion, cloning, or attaching/detaching .vmdk disks modified the VM disk metadata without updating the CBT tracking state.

  • CTK File Corruption: An ungraceful host shutdown, sudden power loss, or storage path interruption corrupted the virtual machine's -ctk.vmdk files.

Traceback

[2023-05-22 12:05:12,410] [ERROR] Failed to fetch changed blocks for disk scsi0:0. Error: VMware error: 'Nonmatching change UUID'. [2023-05-22 12:05:12,415] [ERROR] CBT tracking sequence mismatch between Druva backup index and vSphere QueryChangedDiskAreas API. [2023-05-22 12:05:13,102] [ERROR] Backup job failed for VM '<VM_Name>'. Error: Nonmatching change UUID.

Resolution

To resolve this issue, consolidate any outstanding snapshots and reset VMware Changed Block Tracking (CBT) on the affected virtual machine.

Prerequisites:

  1. Open the vSphere Client and inspect the affected VM.

  2. If snapshots exist, right-click the VM, select Snapshots > Consolidate (or delete/merge manual snapshots via Manage Snapshots).

  3. Schedule a brief maintenance window to power off the virtual machine.

Step 1: Disable CBT on the Target VM

  1. Open the vSphere Web Client.

  2. Right-click the virtual machine and click Power Off.

  3. Right-click the virtual machine, click the Snapshot and navigate to Snapshot Manager. Ensure there are no active snapshots. If there are snapshots present, consolidate them to commit the changes. For more information, see Consolidating/Committing snapshots in ESXi.

  4. Right-click the virtual machine and click Edit Settings.

  5. Click the Options tab, select the Advanced section and then click Configuration Parameters.

  6. Disable CBT for the virtual machine by setting the ctkEnabled value to false.

  7. Disable CBT for the individual virtual disks attached to the virtual machine by setting the scsix:x.ctkEnabled value for each attached virtual disk to false.
    Where scsix:x is SCSI controller and SCSI device ID of your virtual disk.

Step 2: Remove Corrupted CTK Files from the Datastore

  1. Go to Storage, right-click the datastore hosting the virtual machine, and select Browse Files.

  2. Open the VM's working folder.

  3. Locate all files ending with -ctk.vmdk.

  4. Delete the -ctk.vmdk files. (Do not delete standard .vmdk base disks or descriptor files).

Step 3: Re-enable CBT and Run Baseline Backup

  1. Power on the virtual machine.

  2. If Auto-enable CBT is selected in your Druva VMware Backup Policy:

    • No manual edit in vSphere is needed; Druva automatically turns CBT on during the next backup.

  3. If CBT must be enabled manually:

    • Repeat Step 1 while the VM is powered off, changing ctkEnabled and disk-specific scsiX:X.ctkEnabled parameters to TRUE, then power the VM back on.

  4. Log in to the Druva Management Console.

  5. Navigate to Protect > VMware > Configured VMs.

  6. Select the virtual machine and click Backup Now.

Druva will run a full-block scan to establish a fresh, healthy CBT baseline. Subsequent backups will resume normal incremental tracking.

Verification

  1. Log in to the Druva Management Console and navigate to Jobs > VMware.

  2. Verify that the manual backup job status displays Successful.

  3. In the job details pane, verify that block-level analysis completes without Nonmatching change UUID or CBT-related errors.

  4. Check the datastore browser in vSphere to ensure new, healthy -ctk.vmdk files have been automatically regenerated in the VM directory.

Did this answer your question?

Related Articles