Skip to main content

Oracle PBS Export Service Fails Due to ZFS Pool in DEGRADED State

Oracle PBS Export Service Fails Due to ZFS Pool in DEGRADED State

Problem Description

Oracle Phoenix Backup Store (PBS) backup jobs fail because the Export Service fails to initialize. In the Phoenix Backup Store server environment, all Oracle PBS datasets remain mounted in read-write (rw) mode, and data remains accessible. However, backup jobs fail immediately during startup retries because the underlying ZFS pool (Phoenix) is in a DEGRADED state.

Cause

One of the disks belonging to the Phoenix ZFS pool has encountered hardware or I/O errors, causing the pool to enter a DEGRADED state.

Although the pool is degraded:

  • All datasets remain mounted in read-write (rw) mode.

  • The storage pool continues to serve data using the remaining healthy disk(s).

  • A previous ZFS scrub completed successfully with 0B repaired, confirming there is no data corruption.

  • During Export Service initialization, a health check validates the ZFS pool status. Since the pool is not healthy, the Export Service fails to start, resulting in Oracle PBS backup failures.

Traceback

1. Oracle PBS Job Log (oracle-pbs.log):

level=error ts=2026-08-04T17:23:51.398289638Z filename=main.go:282 layer=main message="Failed to start Export Service or service not ready" Error="failed to wait for Export Service: export service failed to start after 20 retries"

2. Export Service Log (export_service.log):

[2026-08-04 16:48:49,107] [ERROR] Zpool is in unhealthy state. please bring it to healthy state

3. ZFS Pool Status (zpool status):

pool: Phoenix state: DEGRADED scan: scrub repaired 0B in <duration> with 0 errors errors: No known data errors

4. Dataset Mount Status (mount | grep Phoenix):

/Phoenix               on /Phoenix               rw
/Phoenix/PRFP          on /Phoenix/PRFP          rw
/Phoenix/ASMTD         on /Phoenix/ASMTD         rw
/Phoenix/PRFD          on /Phoenix/PRFD          rw
/Phoenix/ASMTP         on /Phoenix/ASMTP         rw
/Phoenix/PREPROD2022   on /Phoenix/PREPROD2022   rw

Resolution

Schedule a maintenance window before proceeding and ensure no active backup or restore operations are running.

  1. Stop the Druva Enterprise Workloads service:

    • For agent version 7.x.x and later:

      sudo systemctl stop Druva-EnterpriseWorkloads

    • For agent version 6.x.x and earlier:

      service PhoenixBackupStore stop

  2. Clear the pool error counters: Reset transient device errors on the pool:

    zpool clear Phoenix

  3. Verify pool status: Check that the pool transitions to healthy:

    zpool status

    Expected output: state: ONLINE

  4. Execute a ZFS scrub to verify integrity:

    zpool scrub Phoenix

    Monitor progress using zpool status until the scrub completes. Confirm that scan: scrub repaired 0B with 0 errors and no read/write/checksum errors exist.

  5. Start the Druva service:

    • For agent version 7.x.x and later:

      sudo systemctl start Druva-EnterpriseWorkloads

    • For agent version 6.x.x and earlier:

      service PhoenixBackupStore start

Verification

  1. Run zpool status and confirm the pool state displays ONLINE.

  2. Review /var/log/Phoenix/export_service.log (or corresponding agent path) and confirm the Export Service initializes without Zpool is in unhealthy state errors.

  3. Trigger an ad-hoc Oracle PBS backup job from the Druva Management Console and verify that the job completes successfully without failing at the Export Service retry stage.

See also

Did this answer your question?

Related Articles