EBS 12.2 -- CPU patch failing due to wrong value in AD_APPS_JDBC_URL / adop : IO Error: The Network Adapter could not establish the connection

We love working on the weekend.. Patching the systems heavily :) Here is an issue that we dealt with in the last weekend while patching an EBS 12.2 system with a CPU patch.

I was woken up by APPS DBAS early in the Saturday morning. (a unique way to start the weekend)

The issue was urgent and it was on PROD. Yes, we have online patching but the tolerance for the duration of this patching work was not so high and the patching issue should have been fixed as soon as possible to make the work continue and to leave time for the tests.

-Adop apply- was failing with java exceptions , and it was super clear that was related with the database connections. We made our diagnosis, checked the adop logs, adworker logs, got some artistic moves using adctrl :), but I won't go into the detailes of the diagnostics work.. Anyways; at the end we concluded that the value set for the AD_APPS_JDBC_URL environment variable in Patch FS was the problem.

AD utilities like Adop uses the value of AD_APPS_JDBC_URL variable (from the patch env) for the database connections and in this case, during the excessive patcing(multiple adop cycles), that value was somehow (it is another story) modified with a wrong one. So, the value of AD_APPS_JDBC_URL environment variable in Patch environment was wrong.

Actually the hostname part of it was wrong.. The connection string given there was pointing the physical hostnames of the database nodes, but this was a RAC (on Exadata X8-2M) and it was using the virtual IP addresses (and hostnames) for its local listeners.. So no one was listening on the physical database hostnames :) and that was the cause of the problem.. 

As a quick workaround we added the physical DB hostnames along with the virtual ip addresses (mapped the physical hosts to the virtual Ip addresses)  into the/etc/hosts files of the apps nodes. That fixed the issue. But! the actual solution should have been correcting the value of AD_APPS_JDBC_URL in the patch fs.. So here I'm sharing this with you. I hope it will help.

Background info:

EBS Version: 12.2.4

DB Version : 19.16.0.0

Failing Security patch : ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR JUL 2022 (Patch 34127951)

Failing Adop Phase: ADOP APPLY

 The error stack:

Calling /home/app/ebsapp01/R12/fs1/EBSapps/comn/util/jdk32/jre/bin/java ...
Exception in thread "main" java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:490)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:678)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:235)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:34)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:567)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at oracle.apps.ad.worker.AdJavaWorker.getAppsConnection(AdJavaWorker.java:1050)
at oracle.apps.ad.worker.AdJavaWorker.main(AdJavaWorker.java:278)
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:398)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:438)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:732)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:259)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1230)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:327)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at oracle.net.nt.MetricsEnabledSocket.connect(TcpNTAdapter.java:631)
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:164)
at oracle.net.nt.ConnOption.connect(ConnOption.java:137)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:374)
... 13 more


Workaround :

Add the relevant host records to /etc/hosts file of the apps nodes. (In this case, add the physical hostname of the database node with the IP address of the virtual hostame(s) and retry the failing adop apply action. It will solve the problem, but this is a specific case.! *So don't do it in your case:) The real problem is in the value of AD_APPS_JDBC_URL environment variable of the patch filesystem. Actually it is the one that should be corrected.

Solution:

Change the AD_APPS_JDBC_URL on Run FS. Run Autoconfig. Run adop prepare, check and see if the value of AD_APPS_JDBC_URL in the Patch FS is corrected. (If it is still not corrected, a fs_clone may be needed). Basically, we need to recreate the patch fs env files with the correct value set for the AD_APPS_JDBC_URL..  Anyways, once we see it is corrected, we retry the failing action.