Erman Arslan's Oracle Blog

Saturday, April 29, 2023

Erman Arslan's Oracle Forum / MAR 15 - APR 29, 2023 - "Q & A Series"

Supporting the Oracle users around the world. Let's check what we have in the last 45 days.

Remember, you can ask questions and get remote support using my forum. Just click on the link named "Erman Arslan's Oracle Forum is available now". Click here to ask a question", which is available on the main page of Erman Arslan's Oracle Blog. A total of 2165 questions were asked, almost 10000 comments have been made and here is the links to the latest questions from the last period.

-- or just use the direct link:

https://ermanarslan.blogspot.com/p/forum.html

Upgrade EBS R12.1.3 to R12.2.10 by satish

ebs_patch or <INSTANCE>_ebs_patch services by Harin

survey exadata by Roshan

refresh physical disk from shanpshots by Roshan

OMERROR notifications by big

upgrade R12.1.3 to R12.2 by satish

XTTS XML tables export by Roshan

move temp file by big

Custom top - R12.2 upgrade by satish

R12.1.3 upgrade to R12.2 by satish

Purge End Dated OMERROR Workflow by big

diskgroup out of space by Roshan

Stage -upgrade from R12.1.3 to R12.2 by satish

Applying patch to Local Inventory after upgrade by satish

Charecter Set - upgrading 11gr2 database to 19c by satish

English menu in french interface by big

32 bit Oracle client by Roshan

In-Place OS Upgrade from from RHEL 7.9 to 8.7 by Karthik

mount acfs filesystem by Roshan

Add X9M cell servers to existing X8M rack by Aman

permission denied by Roshan

Many WorkFlow notifications by big

FS-CLONE failed by big

move datafile by big

ORA-01821: date format not recognized by big

Prepare phase hangs R12.2.5 by satish

sys file by big

sql server AG by Roshan

sql_id by big

fsclone failed R12.2.5 by satish

Scan connection issue - EBS by satish

dropping a restore point in production by raiq1

Bandwidth calculation for Data Guard log shipping. by raiq1

Tuesday, April 11, 2023

EBS 12.2 -- CPU patch failing due to wrong value in AD_APPS_JDBC_URL / adop : IO Error: The Network Adapter could not establish the connection

We love working on the weekend.. Patching the systems heavily :) Here is an issue that we dealt with in the last weekend while patching an EBS 12.2 system with a CPU patch.

I was woken up by APPS DBAS early in the Saturday morning. (a unique way to start the weekend)

The issue was urgent and it was on PROD. Yes, we have online patching but the tolerance for the duration of this patching work was not so high and the patching issue should have been fixed as soon as possible to make the work continue and to leave time for the tests.

-Adop apply- was failing with java exceptions , and it was super clear that was related with the database connections. We made our diagnosis, checked the adop logs, adworker logs, got some artistic moves using adctrl :), but I won't go into the detailes of the diagnostics work.. Anyways; at the end we concluded that the value set for the AD_APPS_JDBC_URL environment variable in Patch FS was the problem.

AD utilities like Adop uses the value of AD_APPS_JDBC_URL variable (from the patch env) for the database connections and in this case, during the excessive patcing(multiple adop cycles), that value was somehow (it is another story) modified with a wrong one. So, the value of AD_APPS_JDBC_URL environment variable in Patch environment was wrong.

Actually the hostname part of it was wrong.. The connection string given there was pointing the physical hostnames of the database nodes, but this was a RAC (on Exadata X8-2M) and it was using the virtual IP addresses (and hostnames) for its local listeners.. So no one was listening on the physical database hostnames :) and that was the cause of the problem..

As a quick workaround we added the physical DB hostnames along with the virtual ip addresses (mapped the physical hosts to the virtual Ip addresses) into the/etc/hosts files of the apps nodes. That fixed the issue. But! the actual solution should have been correcting the value of AD_APPS_JDBC_URL in the patch fs.. So here I'm sharing this with you. I hope it will help.

Background info:

EBS Version: 12.2.4

DB Version : 19.16.0.0

Failing Security patch : ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR JUL 2022 (Patch 34127951)

Failing Adop Phase: ADOP APPLY

The error stack:

Calling /home/app/ebsapp01/R12/fs1/EBSapps/comn/util/jdk32/jre/bin/java ...
Exception in thread "main" java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:490)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:678)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:235)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:34)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:567)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at oracle.apps.ad.worker.AdJavaWorker.getAppsConnection(AdJavaWorker.java:1050)
at oracle.apps.ad.worker.AdJavaWorker.main(AdJavaWorker.java:278)
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:398)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:438)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:732)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:259)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1230)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:327)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at oracle.net.nt.MetricsEnabledSocket.connect(TcpNTAdapter.java:631)
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:164)
at oracle.net.nt.ConnOption.connect(ConnOption.java:137)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:374)
... 13 more

Workaround :

Add the relevant host records to /etc/hosts file of the apps nodes. (In this case, add the physical hostname of the database node with the IP address of the virtual hostame(s) and retry the failing adop apply action. It will solve the problem, but this is a specific case.! *So don't do it in your case:) The real problem is in the value of AD_APPS_JDBC_URL environment variable of the patch filesystem. Actually it is the one that should be corrected.

Solution:

Change the AD_APPS_JDBC_URL on Run FS. Run Autoconfig. Run adop prepare, check and see if the value of AD_APPS_JDBC_URL in the Patch FS is corrected. (If it is still not corrected, a fs_clone may be needed). Basically, we need to recreate the patch fs env files with the correct value set for the AD_APPS_JDBC_URL.. Anyways, once we see it is corrected, we retry the failing action.

Wednesday, March 29, 2023

EBS 12.2 -- Failed to load webapp: /OA_HTML , java.lang.NullPointerException , WebAppModule.java:397 & ClassFinderMetaDataEnumaration.java:32

Today, I want to share an interesting issue that I came across in an EBS 12.2 environment. The issue started after applying an EBS CPU... When the issue was escalated to me, the limits of tolerance had already been crossed, so we had to save the day.

Issue:

After applying an EBS CPU (2022), oacore server couldn't be started.. It was just putting itself to the ADMIN state and normally didn't service anything.. This issue resulted service interruption, as the EBS application lost a large part of its functionality.(login page was throwing HTTP - Error 404 Not Found)

Running autoconfig and taking some general actions didn't solve it. (Regeneration tasks... In case we might have a corrupted/or irregular ebs3rdpartyManifest_xml.tmp).

Applying a more recent CPU came to mind, but it was an intense work, because all the tests were done for CPU 2022, and those tests should be repeated for 2023 as well.. Besides, we needed to save the day, cause EBS was not functioning.

Here is part of the actual call/error stack:

weblogic.application.ModuleException: Failed to load webapp: /OA_HTML because of DeploymentException: java.lang.NullPointerException

at weblogic.servlet.internal.WebAppModule.prepare(WebAppModule.java:397)

at weblogic.application.internal.flow.ScopedModuleDriver.prepare(ScopedModuleDriver.java:176)

at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:199)

at weblogic.application.internal.flow.DeploymentCallbackFlow$1.next(DeploymentCallbackFlow.java:517)

at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)

.....

..................

Caused By: java.lang.NullPointerException

at weblogic.application.utils.ClassFinderMetaDataEnumaration.hasMoreElements(ClassFinderMetaDataEnumaration.java:32)

at weblogic.servlet.internal.WebAnnotationProcessorImpl.getAnnotatedClasses(WebAnnotationProcessorImpl.java:139)

at weblogic.servlet.internal.War.getAnnotatedClasses(War.java:1038)

Cause:

Some nonEBS-related files (for example: backup files created manually before applying some apps patches) were present in the APPS filesystem. Weblogic (oacore managed server in this case) was trying to use them, and that was creating the misbehaviour.

Note that, in some rare cases; file permissions can also be a cause of this issue..

The solution:

*Remove any nonEBS-related files which reside in $OA_HTML/WEB_INF .. (there were some backups in our case)

*Remove any nonEBS-related files which reside in $OA_HTML (some backup JSP files were there)

*Recompile JSP files.

cd $FND_TOP/patch/115/bin
$ perl -x ojspCompile.pl --compile –flush

*Restart the Oacore Services (managed servers) and that's it.

Some References:

java.lang.NullPointerException at weblogic.application.utils.ClassFinderMetaDataEnumaration.hasMoreElements During The Deployment Of Applications At The Server Startup (Doc ID 1937917.1)

12.2 E-Business Technology Stack Weblogic Service Oacore Services Are Down And Will Not Start Due To Error 'ModuleException: Failed to load webapp' And 'DeploymentException: java.lang.ClassNotFoundException: csrfguard.servlet.JavaScriptServlet' (Doc ID 2533082.1)

Wednesday, March 22, 2023

Oracle Systems Event -- "Maximizing The Gains and Carrying Them Into The Future with Oracle Systems", Exadata, PCA, ODA, ZFS, PMEM, PMEM Commit and Data Accelerators, ROCE, KVM, OCI, ATP and more.

At yesterday's event, I shared my experience on positioning Oracle Systems on mission critical customer environments ((including Core Banking and ERP). Health checks, assessments, migration strategies, replatforming, design, failback scenarios, bundling, readiness, improvements and execution...

I supported the benefits that I underlined and explained throughout the presentation, with 4 real life cases at the end of my talk. 4th use case (as a bonus --considering it was an Oracle Systems event) was related with Oracle Cloud Infrastructure (OCI). It was about the Auto Scale Core Banking solution (both scale-out and scale-in capabilities), that we implemented on OCI. (by using a multi node IAS Weblogic Cluster, some cloud level customizations/coding and an Autonomous Transaction Processing Database).

This Oracle Event was inWyndham Grand Istanbul Kalamis Marina Hotel, Istanbul.

All the attendees were Turkish (mostly tech managers & directors). So that, the invitation and the slides were mostly in Turkish, but I will still share them. Good news is, even if you don't know Turkish, I think you still get the idea once you check the slides.(as always; even the main language of the presentation is Turkish, there are still lots of English words out there.) Anyways; I will share the slides at the end of this post, but I'm sharing the event invitation right away, as a memory ->

The subject of my presentation was "Maximizing The Gains and Carrying Them Into The Future with Oracle Systems".

I started by introducing our service areas, including Oracle Consultancy and Advanced Customer Support and them jumped into the real, existing topic, which was the Oracle Systems and the innovations offered with them. (and seeing them in action with our success stories).

I introduced the new (still considered new) Exadata X9-2M and PCA X9 machines, and also explained what else we have in Oracle System family, by giving the whole picture from Exadata to ODA and Oracle Servers. I also shared my experience on ODA and Supercluster(EOF but still in the field) engineered systems.

"The Oracle Way":

Attention was mostly on Exadata and ODA. Event attendees found the new and innovative features of Exadata very interesting. That part was also my favorite. While explaining the PMEM (persistent memory - Intel Optane), Persistent Data Accelerator and Persistent Commit Accelerator, I felt like I was talking about Quantum Computing:) -- Yes, I admit I'm exaggerating a bit :)

ROCE (RDMA Over Converged Ethernet) wasn't new for Exadata users, but explaining how ROCE supports the super fast PMEM I/O Path, was a pleasure, as well as mentioning how well the Oracle Database integrates with the newest hardware. Write through PMEM cache, fast LGWR I/O - write, best fit for OLTP, database utilizing the PMEM in AppDirect mode using RDMA transparently, no need for configuring or managing anything, 1.5 TB PMEM per cell, the improvements in existing Exadata unique features.. Well...These topics were already enough to win the hearts.

Private Cloud Appliance (PCA) also got people excited.. We as GTech already completed a big Exadata and PCA replatforming project recently.. We consolidated lots of FMW apps in a single PCA X8 machine in that project. Sharing our experience along with the capabilities of the machine and shedding some lights on the motivations for positioning this machine helped a lot for getting the attention.

Being able to run both the modern apps technology (containers, Kubernetes, microservices and all that) and the traditional apps (like java apps on Weblogic Clusters) on the same platform, being ready for the cloud (especially for OCI), the things like GiraalVM, the ZFS storage inside, isolation and virtualization, ease of implementation and management, fault tolerancy and performance... These were more than enough for justifying the latest Oracle Apps Engineered System known as PCA.

I spoke almost for 40 minutes in this presentation and it is difficult to summarize it all here. Don't worry I have lots of good news :) This time the good news is, I'm sharing the slides below :)

Wednesday, March 15, 2023

RDBMS -- Handy App for Checking Oracle Database Features and Licensing

Following is an introduction for the Oracle Database Features and Licensing Application.

It is a web/Apex application. It is fast, easy-to-use, up-to-date, and quite handy. You can get all the details about Oracle features and options.. Licensing is also included.

Suppose you are wondering what is delivered with Advanced Security Option. Well, just click on the Licensing button, Choose Advanced Security Option which will be in the right pane and get all the information you need.

What I mean by "all the information" is actually the availability (release availability), the functional category (for instance: security) and the option details (the features delivered with the option -- with some additional notes)

Just check the screenshots below to see the details of the delivered information.

So using these app, you won't be looking for your Oracle Salesman much. (at least for non-commercial matters)

You can also use this app during your presales/sales activities. You can even populate your slides by using the information you get from this app.

Here is a screenshot that I took while checking the Oracle Advanced Security option from the licensing perspective.

Here is another screenshot that I got while checking the PMEM Support.

Links:

Oracle Database Features

https://apex.oracle.com/pls/apex/r/features/dbfeatures/home

Erman Arslan's Oracle Forum / FEB 20 - MAR 15, 2023 - "Q & A Series"

Supporting the Oracle users around the world. Let's check what we have in the last 23 days.

Ask Me Anything -> Oracle 20 FEB - 15 MAR 2023

Remember, you can ask questions and get remote support using my forum. Just click on the link named "Erman Arslan's Oracle Forum is available now". Click here to ask a question", which is available on the main page of Erman Arslan's Oracle Blog. A total of 2133 questions were asked and here is the links to the latest questions from the last period.

-- or just use the direct link:

https://ermanarslan.blogspot.com/p/forum.html

Bandwidth calculation for Data Guard log shipping

KVM CPU settings

Upgrade legacy and_EBS

DB performance question

APEX question

VNCR question

Invalids after upgrade to R12.2.10

Change NLS LANG

Upgrade EBS R12.1.3 to R12.2.10

Connectivity to EBS database from hadoop

Purge Obsolete Workflow Runtime Data For Applications

Concurrent managers down EBS R12.2 - 19c

txkCfgUtlfileDir.sh 19c EBS database

RUP 10 -> skip worker does not work

Adop RUP 10 - Skip worker

Unable to open database - flashback

Upgrade R12.1.3 to R12.2

WorkFlow Mailer notification test

Upgrade R12.1.3 to R12.2

DBCA hangs on step 9

Rerun dbca question

19c cloning- pluggable database naming failed.

Renaming PDB error out R12.2 19c

Enable trace for FORMS question

xhost + problem

post clone app error

Forms issue EBS R12.2

Tuesday, February 21, 2023

Quantum Computing -- Qiskit Terra - Unexpected probabilities (due to misinterpretation of the result)

These days, I'm trying to use the things I learned from physics in the field of computer science. That is, I'm using IBM Quantum Computing Lab and Composer to run Python codes utilizing the Quantum Computing algorithms and I do it on real Quantum Computers (and Quantum Simulators). It is very exciting and I find it (I mean Quantum Computing) very enjoyable, but I must admit that it is difficult at first, -- after all it is different than classical computing. I mean, we still have those operators, those quantum gates (somewhat similar with the classical logic gates), but we exploit the power of the quantum computing by using those two phenomena called superposition and entanglement, and these are a little bit confusing.. It is an interesting journey, made me go back to the Linear algebra, Matrices, Tensor Products, Logic gates and stuff like that.

Well.. I will go a little quick and concise in this one. So, we have QBITs, and we do the measurements at the right place to get our results.(measuring collapses the wave function (according to the Copenhagen Interpretation of Quantum Mechanics) and make us lose those quantum states -- that's why it is important) . Actually I have lots to say about this subject but this will be the subject of a future blog post, so let's concentrate on this particular issue that I 've faced while using Qiskit Terra. (Note that-- when I try a similar code with similar logical expressions with Qiskit Aqua(which is deprecated), the expected outputs are produced).

As you may already sensed, actually it wasn't an issue. I mean the thing I identified as an issue is actually by design.. So, I will just share some information, something I learned on the way running Grover's Algorithm. In short; I was doing some quantum computing gymnastics, and I saw (thought) that some outputs are not aligned with the truth tables.

Lets see;

##Importing what I need

from qiskit import *
from qiskit.tools.visualization import plot_histogram
from qiskit.circuit.library import PhaseOracle
from qiskit.algorithms import Grover, AmplificationProblem
from qiskit.visualization import plot_histogram

##Creating my Oracle (this has nothing to do with Oracle Corporation :) Just check Grover's algorithm to understand the concept..

oracle = PhaseOracle('((A & C) | (B & D)) & ~(C & D)')

problem = AmplificationProblem(oracle=oracle, is_good_state=oracle.evaluate_bitstring)

backend = Aer.get_backend('qasm_simulator')

grover = Grover(quantum_instance=backend)

result = grover.amplify(problem)

##Printing the expected result in a chart and get the result as text (as a list actually) as well.

print(result.circuit_results[0])

{'1101': 251, '0111': 274, '0011': 231, '1100': 268}

plot_histogram(result.circuit_results[0])

Well.. As you may understand from the code above, I was trying to find the correct results ( aligned with the expected truth table) for my logical expression ((A & C) | (B & D)) & ~(C & D)') and I used Grover for doing that.

Qiskit Aqua's LogicalExpressionOracle, sorts the variables alphabetically, so an expected output which is A & B& D should have been there in the outputs 1011, but it was not there. This is the issue (not an issue actually) that I mentioned in the first paragraph.. I must admit that I spent lots of time for this and finally learned that, Qiskit Terra sorts the variables by the order of their appearance!

So in this case, the order is A then C then B and finally D. So, for instance we should see 1101 as an expected result (A & B & D) and, as you see above, it is there. So no problem with Terra. I was the one who had a lack of information :)

AC is also a correct result , and it corresponds to 0011, and it is already seen in the results..

The truth table with a format of ABCD (alphabetically sorted from left to right) can be converted to Terra's style truth table like the following;

ABCD DBCA

0101 - > 1100 *

1010 - > 0011 *

1101 - > 1101

1110 - > 0111

As you see, it is completely consistent with the output generated with Qiskit.

Just wanted to share this with you.. Maybe it will be of use to someone.

Monday, February 20, 2023

EBS / Chrome -- Jar Resources in JNLP file are not signed by the same certificate -- After Enabling JWS on Multi-Node

As you may already know, Chrome users must enable Java Web Start(JWS) on their EBS instance, in order to be able to run Oracle EBS applications on their browsers. (Otherwise, they encounter java problems while opening the forms apps)

Chrome hasn't not supported the Java plug-in, since the release 45 and therefore, Java Web Start is a key for the EBS Chrome users. The Forms applications and regular Java applets as part of the Oracle E-Business Suite can only run with Java Web Start. Note that, we have EBS Forms Plugin for Chrome and we were using it in the past, but it is not feasible anymore..

In this post, I will not go into the details of enabling the JWS. It is already addressed on MOS - "Using Java Web Start with Oracle E-Business Suite (Doc ID 2188898.1)". But, I will give you a solution to a particular problem that you may face with, after enabling the JWS.

This issue may be encountered especially in the multi apps node environments.

As you see above, the exception is "JAR Resources In JNLP File Are Not Signed By Same Certificate". Files related with the jar signing should be checked, and in order to solve this, we do the following:

We just stop service on the second node.

Backup the adkeystore.dat and adsign.txt (located in $APPL_TOP_NE/ad/admin), and copy those files from the primary node. -- overwriting the current ones.

Lastly, we regenerate the jar files with force option and start the services..

Note that, we don't expect this issue in the EBS environments configured with shared JAVA_TOP. In those environments, we have the files shared by the nodes already. This issue is expected only on non-shared EBS Application filesystem environments.. With that, we have no problem to believe that those copy operations are appropriate , and can be considered as the solution to this problem..

That's it. That was the quick tip of the day. I hope you 'll find it useful.

Erman Arslan's Oracle Forum / 1 JAN - FEB 20, 2023 - "Q & A Series"

2023 started fast, we had interesting issues. I tried hard to find time to answer all the questions despite the load of my work (leading my department, mission critical support activities & projects + intellectual gymnastics - physics and stuff like that ) ! :)

-- or just use the direct link:

https://ermanarslan.blogspot.com/p/forum.html