Problem description

Nutanix VM Backups fail with error AHV 100 - Failed to discover iSCSI targets.
Error Description: This error may occur when Phoenix fails to discover the iSCSI targets during backup or restore.

Cause

iSCSI Data Service IP is not configured or not reachable

Traceback

level=error ts=2023-03-31T02:44:54.306441158-07:00 filename=iscsi.go:53 message="Failed to discover iSCSI targets on remote machine"

level=error ts=2023-03-31T02:44:54.306558865-07:00 stack="Code: 4\\nOriginal: exit status 4\\niSCSI: connect problem caused operation to fail.\\ngoroutine 13 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\nphoenix/ahv/client/pkg/logutils.(*loggerImpl).Error(0xc0002dc900, {0xf7ada0, 0xc00046c690}, {0xea2391, 0xc00061e200}, {0x0, 0x2, 0x15})\n\t/bld/go/src/phoenix/ahv/client/pkg/logutils/logger.go:80 +0x247\nphoenix/ahv/client/agent/ahv.(*AHVState).loginTargets(0xc00014e140)\n\t/bld/go/src/phoenix/ahv/client/agent/ahv/iscsi.go:53 +0x32f\nphoenix/ahv/client/agent/ahv.(*AHVState).PrepareBackup(0xc00014e140)\n\t/bld/go/src/phoenix/ahv/client/agent/ahv/backup_manager.go:140 +0xfbf\nphoenix/ahv/client/agent/backup.(*Agent).Prepare(0xc000161ef0, {0xf9c5a8, 0xc0002de1e0})\n\t/bld/go/src/phoenix/ahv/client/agent/backup/backup.go:267 +0x3ee\ngit.druva.org/phoenixdev/hyperloop/backup.(*backup).prepare(0xc0002de1e0)\n\t/bld/go/src/git.druva.org/phoenixdev/hyperloop/backup/prepare.go:12 +0x62\ngit.druva.org/phoenixdev/hyperloop/backup.(*backup).doBackup(0xc0002de1e0)\n\t/bld/go/src/git.druva.org/phoenixdev/hyperloop/backup/job.go:133 +0xd9\ngit.druva.org/phoenixdev/hyperloop/backup.(*backup).Start.func1()\n\t/bld/go/src/git.druva.org/phoenixdev/hyperloop/backup/job.go:21 +0x10f\ncreated by git.druva.org/phoenixdev/hyperloop/backup.(*backup).Start\n\t/bld/go/src/git.druva.org/phoenixdev/hyperloop/backup/job.go:14 +0x5b\n"

level=error ts=2023-03-31T02:44:54.308986621-07:00 filename=backup_manager.go:142 message="Failed to login iSCSI targets"

level=error ts=2023-03-31T02:44:54.309098715-07:00 filename=backup.go:269 message="Failed to prepare AHV backup"

level=info ts=2023-03-31T02:44:54.398562949-07:00 filename=job.go:85 message="Abort set" method=Abort error="Failed to prepare backup sources" msg="Code: 4\\nOriginal: exit status 4\\niSCSI: connect problem caused operation to fail."

Resolution

Connect to the PRISM Element
Check the Cluster setting and ensure that iSCSI Data Services IP is configured (Ref: Configure Data Services IP )

The iSCSI Data Services IP address is used to provide iSCSI access to the cluster storage. It is primarily used by Nutanix Volumes to present storage to your Guest OS directly via ISCSI. This IP address acts as an iSCSI target discovery portal and initial connection point. For example, Nutanix Volumes utilizes an iSCSI data services IP address to clients for target discovery. This simplifies external iSCSI configuration on clients.
Try to telnet to the Data Services IP over port 3261 , 3260 & 3205and make sure that it connects successfully
telnet <data_Services_IP> 3260 #iSCSI Discovery
telnet <data_Services_IP> 3261 #ISCSI Data
telnet <data_Services_IP> 3205 # Nutanix internal/v3 API gateway (required in some AOS versions
Run tcpdump -i any host “Data_Service_IP” -nn // it will show the port it is trying to connect.
Ensure that the Data Source IP is not in the DHCP scope, it needs to be a static IP address.
If in DHCP scope, assign a new static IP to it and update the same in Nutanix Cluster settings.
Restart the Proxy Services
Trigger a manual backup of the affected server. This time it should complete successfully