RMAN backup fails with Oracle31 error
Issue
RMAN backup fails with Oracle31 error and you get the following error message:
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of delete command on ORA_DISK_1 channel at 04/04/2023 18:15:08
ORA-27056: could not delete file
Linux-x86_64 Error: 30: Read-only file system
Cause
This error occurs when you perform the RMAN restore on the database by mounting the restore location from PBS on the Oracle server. After cataloging, RMAN refers to the mounted location which is in read-only mode. In the next backup, RMAN attempts to crosscheck and delete the files from a Read-Only mount which results in an error.
Resolution
After RMAN restore from PBS restore location is successful, do the following:
1. Unmount the restore path by running the following command:
df -h
umount -l <restorepath>
2. Crosscheck and delete backups by running the following command:
crosscheck backup; crosscheck copy; delete force noprompt expired backup; delete force noprompt expired copy;
3. Trigger the backup.
4. If the issue persists, run the following RMAN commands:
crosscheck backup; crosscheck copy; delete force noprompt backup; delete force noprompt copy;
Unable to start the NFS server on Phoenix Backup Store
Cause
This error occurs if the NFS server is unable to find the backup mount path on the Phoenix Backup Store.
Resolution
Add the backup mount path in the /etc/exports file manually.
Unable to start the Phoenix Backup Store service
Cause
This issue can occur due to multiple reasons.
Resolution
If the NFS service is not running, start the NFS service.
For other issues, contact Support.
A disconnected Phoenix Backup Store remains in the waiting to disable state even after disabling it
Cause
This happens when the Druva Cloud waits for the Phoenix Backup Store to get connected so that it can disable it. However, if the Phoenix Backup Store is decommissioned or there is no way it can connect to the Druva Cloud, it stays in the waiting to disable state on the Management Console.
Resolution
Re-register the Phoenix Backup Store
To disable the Phoenix Backup Store on the Management Console, contact Support.
Unable to disable the Phoenix Backup Store after replacing the old Phoenix Backup Store
Cause
This happens when the Phoenix Backup Store service is still running on the old Phoenix Backup Store and you attempt to disable the new Phoenix Backup Store.
Resolution
Contact Support to disable the new Phoenix Backup Store and stop the Phoenix Backup Store service on the old Phoenix Backup Store. For more information, see prerequisites to re-registering a Phoenix Backup Store.
Unable to deploy a Phoenix Backup Store on a VMware setup
Cause
The Phoenix Backup Store deployment fails when the MD5 checksum of the downloaded OVA package does not match with the MD5 checksum of the package mentioned on the downloads page. This can occur if there was a problem with the download.
Resolution
Ensure that the MD5 checksum of the downloaded OVA package matches with the MD5 checksum on the downloads page before deploying the Phoenix Backup Store OVA package. The MD5 checksum is mentioned below the Phoenix Backup Store package.
RMAN script failure scenarios
Unable to validate backup request with Phoenix Backup Store
Issue
The validation check fails to reach Phoenix Backup Store (curl returns non-zero exit code on /backup validation check).
Cause
This issue occurs due to one of the following reasons:
Network connectivity issues between the agent and the Phoenix Backup Store.
Port 9090 is blocked by a firewall or security group.
The EnterpriseWorkloads agent service is not running on the backup store.
Resolution
To resolve this issue, verify your network connectivity and the status of the agent service.
Verify network connectivity
Check basic network connectivity and ensure port 9090 is open. Run the following commands:
ping <backup_store_IP>
telnet <backup_store_IP> 9090
Note: If the ping command succeeds but the telnet command fails, check your firewall and security group rules to ensure port 9090 is open for incoming traffic.
Verify the Enterprise Workloads agent service
Ensure that the Druva agent service is actively running on the backup store.
Check the status of the service:
service Druva-EnterpriseWorkloads status
If the service is not running, restart it:
service Druva-EnterpriseWorkloads restart
Mount target has reached maximum snapshot limit
Issue
The mount target has exceeded the maximum snapshot limit.
Cause
The backup target has reached its maximum allowed snapshot count. This occurs because snapshots accumulate while background jobs are actively processing uploads.
Resolution
To resolve this issue, perform one of the following actions:
Option 1: Wait for processing to complete. Allow time for the existing background snapshot uploads to finish processing. You can monitor the progress and status of the snapshots from the Enterprise Workloads Management Console.
Option 2: Increase the maximum snapshot limit. Update the plugin configuration file to allow a higher number of snapshots.
Open the following configuration file:
/etc/Druva/EnterpriseWorkloads/oracle-pbs/OraclePBSPlugin.ymlLocate the max_oracle_snapshots parameter and increase its value.
Note: The default value for the max_oracle_snapshots parameter is 5.
Another backup is currently running for this database
Issue
A backup job is already in progress for the specified database.
Cause
Another instance of the Recovery Manager (RMAN) backup process is currently running for this Oracle Database SID.
Resolution
Wait for the active backup job to finish before initiating a new one. Monitor the active backup job on the Jobs page.