This article applies to:
OS: Ubuntu 18.04
Product edition: Enterprise Workloads
Feature Category: NA
Problem description:
Oracle Phoenix Backup store(PBS) jobs stay in queue and do not run.
Cause
This is due to various reasons including, but not limited to, the following:
Hardware Failure
Network interruptions
Multiple cron/scheduled tasks
A symptom of this issue will be that the RMAN backup script is fails with the below error:
Validating backup request
Response from Phoenix Backup Store: {"error_code":0,"error_msg":"","result":{"is_reached_max_snapshots":true,"is_version_correct":true,"message":"","mount_name":"PBS"}}
Validation for phoenix backup failed... exiting
Resolution
Before proceeding any further please note that this may result in data loss the snapshot may be the only reference of the data remaining. Typically RMAN will retain all needed backup parts in the NFS mount, but depending on circumstances the data may not be present.
To ensure your ZFS storage pool is healthy, run the following command:
zpool status
Next, check for any active RMAN processes on your Oracle Database host systems. If necessary, pause any scheduled tasks.
Primary procedure:
Cancel any running jobs in the console for the PBS if possible.
Stop the PhoenixBackupStore service, and wait 5 minutes so all cloud treads despawn.
systemctl stop PhoenixBackupStore.service
Stop the nfs-server service to release any holds on the file system.
systemctl stop nfs-server.service
Once the time has elapsed start the nfs-server service followed by the PhoenixBackupStore service.
systemctl start nfs-server.servicesystemctl start PhoenixBackupStore.service
Check the status of the PhoenixBackupStore service, and verify it is successful. If there is a failure it will need to be addressed prior to proceeding further.
A new job should spawn to upload the snapshot, and will need to be canceled.
Now restart the PhoenixBackupStore service, and cancel the job for each remaining snapshot.
systemctl restart PhoenixBackupStore.service
If the primary procedure fails to clear the snapshots then this process can be followed.
Stop the PhoenixBackupStore service, and the nfs-server service
Run the following to see the snapshots that exist on the filesystem:
zfs list -t snapshot
Release the snapshot using the following command:
zfs release -r <zpool name> <SNAP_NAME>Example zfs release -r phoenix Phoenix/proddb@UID_5314
Destroy the snapshot using the following command:
zfs destroy -R <zpool name> <SNAP_NAME>Example zfs destroy -R phoenix Phoenix/proddb@UID_5314
If an error occurs in any of the steps please contact technical support for assistance.