Problem description
Oracle PBS backups fails with RMAN error 0RA-19870 AND 0RA-19625 and phoenix error ORACLE31.
Cause
If the customer creates a new PBS to migrate the database from the old PBS to the new PBS then deletes the old PBS or PBS mount without updating the RMAN catalog.
Then whenever the next backup triggers, backup pieces that are residing in an old PBS mount (Deleted PBS)which is physically not available but exists in the RMAN catalog.
Since the information is still available in the RMAN catalog when backup gets triggered, RMAN knows where the old backup pieces are residing and looks for the data in deleted PBS or PBS mount which is physically not available then the RMAN fails with the below trace back in RMAN logs and with Oracle31 error code on Phoenix UI.
Traceback
RMAN-03002: failure of recover command at 05/31/2022 23:37:26 ORA-19870: error while restoring backup piece \\192.16.10.18\PHOENIX\OLD_PBS_MOUNT_DB1\ORACLE_DATA\FULL\FULL_BK_DATAFILE_AS1_2_1.RBK ORA-19625: error identifying file \\192.16.10.18\1PHOENIX\NEW_PBS_MOUNT_DB\1ORACLE_DATA\FULL\FULL_BK_DATAFILE_DATA_D-DB1_I-139400_TS-USER_DATA_FNO-4_QSO07.RBK ORA-27041: unable to open file OSD-04002: unable to open file O/S-Error: (OS 3) The system cannot find the path specified.
Resolution
Login to the RMAN host.
Connect to the RMAN target
# rman target /
Run below appropriate RMAN crosscheck commands on the RMAN host (DB Server) and delete all the control file metadata that is being referenced from the old backups from the deleted PBS mount.
Crosscheck copy; Crosscheck copy of archivelog all; Crosscheck archivelog all; Crosscheck backup; Delete force noprompt expired copy; Delete force noprompt expired archivelog all; Delete force noprompt expired backup; Delete force noprompt obsolete;On the source, the database disables block change tracking
Remove the old file for BCT post checking with your Oracle team
Recreate the BCT file and enable BCT from the sqlplus command line:
Select name from v$dbfile;
Use the below query to get the path for your database to create a BCT file.
Select name from v$dbfile;
Example o/p:: u03/oradata/DEV19C3/
Now you can save your BCT file in the above output location
Examples from Unix you use windows below are just examples:
# unix# alter database enable block change tracking using file'/u03/oradata/DEV19C3/change_tracking.f';
#windows# alter database enable block change tracking using file'D:oracle\oradata\SUP10\change_tracking.f';Since we have re-enabled the CBT, the Next backup will be considered as level-0 backup(Full Backup)
We ran crosscheck commands that marked expired and obsolete backup pieces for a clean up then we have cleaned up all the old files then finally re-enabled BCT to kick off a level-0 backup (Full-Backup)