Skip to main content
All CollectionsKnowledge BaseEnterprise WorkloadsTroubleshooting - Enterprise Workloads
Playbook - How to Collect logs and troubleshoot Oracle PBS and Oracle DTC issues
Playbook - How to Collect logs and troubleshoot Oracle PBS and Oracle DTC issues
Updated over 10 months ago

Overview

This article will help you understand the step by step approach to reach towards the exact cause of the backup failure reported for Oracle workload. Kindly read the document carefully and ensure that you follow the steps appropriately and collect the relevant information from the environment.

Details

►1. Perform the Basic environmental checks before you identify the issue ?

  • What kind of Druva setup is there for Oracle backups, is this regarding Oracle PBS (Phoenix Backup Store) or Oracle DTC ?

  • What is the version of Oracle application and OS version of the DB server?

  • What is the Druva agent version for PBS /DTC ?

  • Check the Support matrix for the supported version of the Oracle and OS.

  • Check the System requirements for Oracle DTC.

  • Is single DB impacted or multiple DBs impacted ?

  • Any recent changes ? like upgrading of OS , upgrading of Druva agent or migration etc ?

►2. Oracle - PBS

Scenario

Check-list

Unable to mount the PBS shares on the DB server.

  • Check the last successful backup respective to the mount.

  • Log into the oracle DB server and check the available mount points.

    • # mount (It should show all mounts that are configured to mount to this DB host)

  • Verify the df -h output on the DB server, it should show all the mounts that were shown on the above step.

  • If it is not showing all the mounts on df -h then verify the fs tab entries

    • # cat fstab

  • Then Verify the fstab entries and if you see entries for the mount, but still not available on the root filesystem then check if it is a part of /etc/exports on PBS.

  • If the mounts are available on fstab entries on DB server and etc/exports of PBS server but still its not showing in df -h which means OS is unable to mount the PBS share.

  • Restart nfs services if you are still not seeing the mount on DB server.

    • #systemctl start nfs-server

  • Reboot the DB server, then OS mounts all the mounts that are available on the fstab entries which will resolve the issue.

PBS is not connected to the cloud.

( Click Here )

  • Check the connection between PBS and DB server through ping.

  • Telnet to port 9090 should work from DB server to PBS server.

    • Note: The Phoenix Backup Store takes incoming requests on port 9090. Since the RMAN communicates with the Phoenix Backup Store API to check the status, ensure that the port 9090 on the Phoenix Backup Store can take incoming requests over http.

  • Verify if the DB server is connected to druva cloud.

    • #telnet backup-phoenix.druva.com 443

  • Check the /etc/exports on PBS and it should show all the entries.

    • #cat /etc/exports

  • Verify if there are any duplicate entries , stale entries or extra characters on /etc/exports

  • Also check any stale entries by running this command

    • #exportfs -r

  • If it is there then remove the stale entire by editing the exports file using vi command

    • vi /etc/exports

    • press i to go into edit mode remove those stale entries

    • save the file by pressing escape + :wq

  • Verify df -h, it should show all the mounts. If it's not showing then:

    • Check the zpool status on PBS server

      • #zpool status

    • If it's not showing any zpool status as online, then try importing the zpool with the below command.

      • #zpool import -f Phoenix

    • Post importing the command then verify the status which should show it as online.

      • #zpool status

  • Restart the NFS services on PBS

    • #systemctl start nfs-server

  • Restart the PBS services

    • #service PhoenixBackupStore restart

  • Now verify the PBS status on the druva console now it should show connected.

►3. Checklist to troubleshoot Oracle PBS Issues ?

  1. Check the status of PhoenixBackupStore service

# systemctl status PhoenixBackupStore

  1. Identify any error by checking main service logs

# tail -100f /var/log/Phoenix/main_service.log
//here - 100 is to display last 100 lines and f is for displaying live result

  1. For PBS Check for the backup request id and backup set id // backup set id is the folder name which is created under the backup and it is in numeric

Example: Backup request received.
bkpset_id[1903],req_id : [356]
Here bkpset_id is the folder name and req_id is your JOBID location: /var/log/Phoenix/ORACLE/backup/1903/
​
​# tail -100f /var/log/Phoenix/ORACLE/backup/1903/Phoenix.log

►4. Checklist to troubleshoot Oracle DTC Issues ?

  1. Check the status of PhoenixOracle services

# systemctl status PhoenixOracle

  1. Identify any error by checking main service logs

# tail -100f /var/log/PhoenixOracle/main_oracle_service.log

  1. Identify any error by checking main service logs

Example: Backup request received.
bkpset_id[1903],req_id : [356]
Here bkpset_id is the folder name and req_id is your JOBID
For further investigation, go to this folder /var/log/Phoenix/ORACLE/backup/1903/
open the Phoenix.log to identify the issue.
​# tail -100f /var/log/Phoenix/ORACLE/backup/1903/Phoenix.log

Related Articles
User Guide - Preparing your environment for successful Druva Oracle PBS Backup
User Guide - Preparing your environment for successful Druva Oracle PBS Restore
Oracle PBS does not show the ZFS pool mounted
How to manually clear snapshots from the Oracle Phoenix Backup Store
Logging, configuration files, and services for Oracle PBS
Did this answer your question?