Problem description
Oracle backup or restore fails with Oracle_DTC 17: Failed to establish session with GRPC Server or the detailed logs will contain the error “Failed to find an available port to start IOServer”
Cause
The Oracle DTC agent is unable to start the IOServer process due to a port conflict on the Oracle database server. The IOServer process will use a dynamic port range of 20000-20050 by default.
The server hostname is not properly configured with the /etc/hosts file.
Traceback
level=info ts=2023-09-18T13:36:25.933427377-04:00 filename=backup.go:2042 message="Starting PhoenixIOServer" level=error ts=2023-09-18T13:36:25.941451785-04:00 stack="No port is available in the given range : from 20000 to 20050\\ngoroutine 67 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 \n\t/bld/src/phoenix/oraclesbt/client/agent/backup/backup.go:2538level=info ts=2023-09-18T13:36:25.941598151-04:00 filename=backup.go:1684 message="Finishing backup operation with error status" ErrorCode=4295819281 Message="Failed to find an available port to start IOServer"
Verification:
IOServer process port conflict
Verify what process or application is using the ports by running one of the following commands below. These are only examples, and please feel free to use any command familiar to you, or available on the Oracle database server.
netstat -lntup | grep ":200[0-9][0-9]"ss -lntu | grep ":200[0-9][0-9]"
Hostname Verification
Ping the server’s hostname - ping $(hostname)
Ping should return the IPv4 address, not IPv6
Check the /etc/hosts file
There should be an entry for the server’s correct hostname and IPv4 address
Resolution
IOServer process port conflict (Phoenix Agent below 7.0)
The port range can be moved following the process below
Stop the PhoenixOracle service:
service PhoenixOracle stopEdit the following file:
/etc/PhoenixOracle/Phoenix.ymlUpdate the following entries with an available port range of at least 50 ports:
IOSERVER_START_PORT: 20000IOSERVER_END_PORT: 20050Start the PhoenixOracle service:
service PhoenixOracle start
Hostname Resolution (Valid for all agents)
Update the
/etc/hostsfile with an entry for the hostname and IPv4 address