Monday, March 23, 2026

26 AI -- Let’s Talk About True Cache

I mentioned True Cache in a couple of blog posts ago, and here I'm writing more details about it. In fact, a short while ago, we installed and configured it in our lab. We did some tests and I saw that it can do what is supposed to do, what it promised to do. Once again, Oracle used its own technological advantage by using a part of its proven technologies in different context to offer a solution for something outside of its usual area, Apps Tier-like caching. So, we see True Cache is available among Oracle Solutions.

True cache is a actually a database instance, a synchronized (Asynchronous Redo Apply) read-optimized database instance. It uses redo stream replication, allowing read workloads to be offloaded without requiring application redesign. It sits near the primary database (logically) and leverages Oracle's redo transport technologies (like the Dataguard) for data replication, for getting the changes from primary. The cache data is put in memory  of True Cache and served from there. It is database-level caching architecture that maintains synchronized data using redo apply while serving high-volume read operations independently from the primary system.

What True Cache does is to, increase total system throughput, stabilize latency, reduce primary load and enable horizontal read scaling. It is not a replacement for disaster recovery, or it is not something that will help in write-heavy workloads. It is database-native read scaling architecture designed for modern hybrid workloads.

It doesn't store any data in the disk, in the datafiles.( There is no persistent user datafiles in True Cache) It stores all the cached data in its memory.
When it is first started, the cache is empty. Once the apps or sessions that are directed to True Cache encounter cache/block misses, True cache starts to be filled. (Missing blocks are fetched directly from the primary DB ) 

Only the hot data ( data that is queried by the sessions that are connected to True Cache, maintained by modified LRU and touches counts) is put in the True Cache. So, Oracle's redo transport services runs all the time from primary to the cache, but only the data that is hot, that is in the memory of true cache is fed from transported changes. If the data that is needed by query of a session that is connected to True Cache is not in the cache, then it is received from primary and served to the query of the session that is requesting it. Subsequent queries that needs the same data will be served directly from the True Cache itself, without a need to go and get the data from primary. As you may guess, after that point, any redo that changes that data, will be applied into the True Cache version of that data. So redo replication is always  and already there, but application of the redo is done to the data that is already present in the True Cache. 

As you may already understand, True Cache is filled automatically once you query it. The queries shape the cache. So, a true cache that is serving the sales app / sales app sessions automatically becomes a cache dedicated to sales, likewise a true cache that is is serving the finance app/finance sessions all the time, automatically becomes the cache of finance data.

Okay we draw the big picture. Details are there if you need it but I think we are good to go with a test; with a validation. 

Well, to validate Oracle AI Database 26ai True Cache behavior from an application perspective, we did some tests with a simple Java application and Oracle JDBC Thin Driver. Our goal was to simulate a real application connection and measure query performance directly at the application layer.

The application connected to the True Cache instance using standard JDBC connectivity:

jdbc:oracle:thin:@//192.168.1.113:1522/FREE_TC

The session was configured as read-only, consistent with the True Cache architecture where read workloads are served by the cache instance while the primary database remains responsible for transactional operations. Connection validation confirmed that queries were executed against a database with the role TRUE CACHE.

Okay.. first performance testing was performed by executing the same SQL query twice within a single session:

Before going forward, a Note on the Test Methodology: >

I know what the skeptics might be thinking: Is a SELECT COUNT(*) really enough to benchmark a new caching layer?' Well, let's be clear.. This wasn't intended to be a definitive enterprise benchmark. Instead, consider this a structural validation or a smoke test. 

Our goal was to observe the fundamental transition from a cold start to a warm state. While a COUNT(*) might primarily hit index blocks or metadata depending on the optimizer's path, it perfectly demonstrates how the instance learns the workload, fetches the necessary blocks from the primary, and eliminates latency on the subsequent pass. It’s about proving how it works before we start measuring the flow rate of a complex production system.

SELECT COUNT(*) FROM SYS.TC_TEST;




As seen above; the first execution acted as a warmup run, allowing data to be populated into the cache. The second execution represented a cache-hit run, where results were expected to be served directly from True Cache memory structures. Execution time was measured within the application using high-resolution timers.

Well.. The performance metrics speak for themselves, but as with any enterprise-grade technology, these results are naturally contingent on a variety of environmental factors (correct SGA sizing, network speed and so on) and specific use cases. 

Next, we tested the true cache performance with a join query; (5000 records in each table)

Again performance testing was performed by executing the same SQL query twice within a single session: 

SELECT c.city, p.category, COUNT(*) order_count,
SUM(oi.quantity * oi.unit_price) total_revenue
FROM SYS.CUSTOMERS c
JOIN SYS.ORDERS o ON o.customer_id = c.id
JOIN SYS.ORDER_ITEMS oi ON oi.order_id = o.id
JOIN SYS.PRODUCTS p ON p.id = oi.product_id
GROUP BY c.city, p.category
ORDER BY total_revenue DESC;

Again, the first execution acted as a warmup run and the second execution represented a cache-hit run, where results were expected to be served directly from True Cache memory structures.




Okay good. More realistic :) Primary - Buffer Cache warmup is a little faster here, which may be expected. But once the True Cache is warmed up / once that data is fetched from primary, this time True Cache query performs a little faster :) Again, these results are naturally contingent on a variety of environmental factors (correct SGA sizing, network speed and so on) and specific use cases. 

What’s clear, however, is that Oracle True Cache introduces a fascinating new layer to the stack. When configured correctly and aligned with the right workload, it offers a tangible path to offloading the Primary and accelerating data delivery.

We have only scratched the surface here, and the real-world potential of this true caching layer is something I look forward to exploring further. Stay tuned for more deep dives into specific configuration strategies and performance tuning in upcoming blog posts..!

I almost forgot, one piece of information regarding the license: Oracle True Cache is a database feature included in your Oracle AI Database 26ai Enterprise Edition license at no additional cost.

Wednesday, March 11, 2026

Troubleshooting JSP Compilation Hangs during EBS 12.2 CPU JAN 2026 Patches - AD-TXK Delta 16 Upgrades

Patching Oracle E-Business Suite 12.2 is often a journey full of surprises. Recently, a member of my forum reached out regarding a common but frustrating issue encountered while applying the 2026 CPU  JAN patches and upgrading AD/TXK from Delta 13 to Delta 16.

In this post, I’ll summarize the case and the solution, as it serves as a great reminder of how specific patch dependencies can break an entire upgrade process.

Okay let's start;

The environment was running EBS 12.2.10 with a 19c Database. While applying CPU JAN 2026 patches using adop, the process appeared to hang, or let's say, it slowed to a crawl with a massive failure rate during the JSP compilation phase.

The logs showed the list of jsp translation records as below;

translating jsps...  3% complete: 200/6003 [failed: 200] ETA: 1d0h
  translating jsps...  6% complete: 400/6003 [failed: 400] ETA: 23h17m
  translating jsps...  9% complete: 600/6003 [failed: 600] ETA: 22h24m
  translating jsps... 13% complete: 800/6003 [failed: 800] ETA: 21h34m
  translating jsps... 16% complete: 1000/6003 [failed: 1000] ETA: 20h43m
  translating jsps... 19% complete: 1200/6003 [failed: 1200] ETA: 19h52m
  translating jsps... 23% complete: 1400/6003 [failed: 1400] ETA: 19h1m
  translating jsps... 26% complete: 1600/6003 [failed: 1600] ETA: 18h10m
  translating jsps... 29% complete: 1800/6003 [failed: 1800] ETA: 17h21m
  translating jsps... 33% complete: 2000/6003 [failed: 2000] ETA: 16h29m
  translating jsps... 36% complete: 2200/6003 [failed: 2200] ETA: 15h39m
  translating jsps... 39% complete: 2400/6003 [failed: 2400] ETA: 14h48m
  translating jsps... 43% complete: 2600/6003 [failed: 2600] ETA: 13h58m
  translating jsps... 46% complete: 2800/6003 [failed: 2800] ETA: 13h8m
  translating jsps... 49% complete: 3000/6003 [failed: 3000] ETA: 12h18m
  translating jsps... 53% complete: 3200/6003 [failed: 3200] ETA: 11h29m
  translating jsps... 56% complete: 3400/6003 [failed: 3400] ETA: 10h39m
  translating jsps... 59% complete: 3600/6003 [failed: 3600] ETA: 9h50m
  translating jsps... 63% complete: 3800/6003 [failed: 3800] ETA: 9h0m
  translating jsps... 66% complete: 4000/6003 [failed: 4000] ETA: 8h11m
  translating jsps... 69% complete: 4200/6003 [failed: 4200] ETA: 7h22m

When you see a such big failure rates in the translation phase and an ETA that spans days, you know it's not just a slow server. You understand that something is fundamentally broken in the environment's ability to compile JSPs.

My initial advice in these situations is always to return to the basics: Prerequisites. When adop hangs or JSPs fail to compile  you must verify: OS-level packages and ensure all required libraries for the Oracle stack are present. You must also ensure the software's internal environment is correctly pointing to the right JDK and compiler paths. Patch Prerequisites should also be verified ( AD/TXK upgrade path doesn't have missing dependencies.)

In this case however; the the solution was specific. The Patch:  38502365, - JSP COMPILATION FAILING AFTER APPLYING PATCH 38501349:R12.CAC.C (Patch)....

Once this specific patch was applied, the JSP compilation proceeded as expected, and the upgrade to Delta 16 was successfully completed. That's it.

So let's conclude our article by linking this point as follows;

as we all know; in the world of EBS, one patch fixes what another one breaks is a reality we often live with. This is why tools like the EBS Critical Patch Update Checker are vital, they help us identify what's missing, but even then (as seen in this case) , keeping an eye on the latest patches and support notes is key.

Remember, knowledge is the only treasure that grows when shared. If you encounter similar hangs, don't just wait for the ETA to count down; check your patches and ensure your prerequisites are rock solid.

Stay technical.

Friday, March 6, 2026

EBS -- Cannot Login & FNDCPASS Error in password verification for APPS /FND_VAULT & FND_ORACLE_USERID and all that

In the world of Oracle E-Business Suite, an upgrade completion is often seen as the finish line. However, as many seasoned APPS DBAs know, the real challenges sometimes begin the moment you try to secure the newly upgraded environment. Recently, a complex case surfaced on my forum that perfectly illustrates this. Following an upgrade from 12.2.9 to 12.2.14, the system wouldn't allow logins due to a bug, but the patch to fix the bug couldn't be applied because the applications patching utility - adop itself was unable to authenticate. Exploring the underlying FND_VAULT data inconsistencies, data fixes and the out-of-the-box workarounds required to break the cycle. As I always say, knowledge is the only treasure that increases when shared! 

Let’s get into the technical heart of the issue. Recently I got a support call in my forum. It was about APPS password verification and it required lots of efforts to be fixed. That's why I thought this issue was worth sharing. So, here I'm blogging it.

Here is the forum url for the ones who want to read the original thread: http://erman-arslan-s-oracle-forum.124.s1.nabble.com/Working-Error-in-password-verification-for-APPS-td13177.html

The issue started after upgrading EBS from 12.2.9 to 12.2.14. Actually, once the upgrade was finished,  everything was working one, but the issue started after changing the APPS password.

Apps password could be changed successfully, but after that, EBS login was not working and at that point; re-changing the password was not a fix, because it was also failing.

FNDCPASS was throwing -> Error in password verification for APPS (generic error), and Oracle Support said that this was due to a bug,  but in order to fix it, we needed to patch the system, but adop prepare was also failing.. (cannot complete application logon you may have entered an invalid application passwords , or there may have been a database connect error. internal manager status could not be determined)

In that kind of a broken password situation, these things were expected actually.

When my follower (owner of this issue) asked it to Oracle Support, they suggested to apply 38207032, FND and AD fixes for APEX Schema user integration (Patch). (Note that, APEX configuration was also there, in this system) But! that patch could not be applied, because adop couldn't login due to this issue. adop was trying to connect to the application and submit the relevant program for online patching but it was failing...

Additional symptoms:

$FND_TOP/bin/FNDSVCRG STATUS

Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Internal Concurrent Manager status could not be determined.

---
$ adcmctl.sh status <apps_user>/<apps_password>

Cannot complete applications logon. You may have entered an invalid applications password, or
there may have been a database connect error.

Internal Concurrent Manager status could not be determined. adcmctl.sh: exiting with status 0

---
FNDCPASS system/***** USER sysadmin *****
+----------------------------------------------------------------------------+
Working...
Error in password verification for APPS

After analyzing, I thought that this may be related with the FND_VAULT. It seemed like a FND_VAULT datafix was required, and I found a patch and updated the thread with the following;

If you could trace your login issues, you could maybe see the FND_VAULT is failing in the backend..

For instance; in the following note, the same patch is also recommended;

Login Doesn't Work After R12.ATG_PF.C.delta.12 (34776645) Patch And GUEST Password Change, KB291738

Here the note says - > On EBS 12.2 version Login doesn't work after R12.ATG_PF.C.delta.12 (34776645) patch and a GUEST password change. The customer has applied patch 34776645 along with the latest AD and TXK patches. They also changed GUEST password, then they can't login to the applications.
fnd_web_sec.validate_login returns Y for the new GUEST password.
Response in Google Chrome tracing suggest that the application may still be using the old GUEST password which is invalid.
If they change to the old GUEST password, they can login fine, but Forms do not launch.
They tried clearing tmp and cache dir as mentioned in the Doc ID 3013989.1, but it did not help.

So, that patch is a cure for login issues.. There we see.

Also see the following -> https://ebs-dba.com/wp/blog/2025/07/20/12-2-14-changing-passwords-and-extending-ebs-with-apex-a-critical-warning/

There is also a reference to that patch in the article above.

And see this MOS note -> Unable To Change Password With Hard To Guess Profile, KB320427

There Oracle says:

1. Please download and review the readme for <Patch 36961988:R12.FND.C> AFTER PATCH 36857308 : AFPASSWD AND FNDCPASS ALLOWING SPECIAL CHARACTERS IN APPS PASSWORD
2. Please apply <Patch 36961988:R12.FND.C> in a Test environment initially before application to any Production instance.
3. Please retest the issue.
4. If the issue is resolved, please migrate the solution as appropriate to other environments.

We needed to be sure about the corruption in FND_VAULT related data, all the tables related to it, FND_ORACLE_USERID and so on, so I suggested;

That statement is given in MOS: "Applying 12.2.13 RUP(34776655) Failed With ORA-06512: AT APPS.FND_VAULT" -> "The issue is encountered in environments where the FND_VAULT table contains inconsistent or invalid data, such as plain text values instead of the expected RAW format."

So, FND_VAULT_KEYMASTER and FND_VAULT_DATA tables are in the play.

In the patch, we have a datafix -> 36324608 FND_VAULT DATA FIX

That fix is however applicable in cases where you have FND_VAULT initialization problems.

If you have FND_VAULT related corruption, you should see them in the some of logs at least.

Note  ORA-20001: Please Review FND_VAULT.INITIALIZE Output Before Continuing -> says :

"Manually run the script from the patch 36842768:R12.FND.C
In 12.2.13 we discovered a data corruption issue with FND_Vault, the script looks for the indicators of this corruption and fixes known issues.
In this case this is not a known entry for us so we need to verify with the data owner (ISG) if this is simple ASCII data or is it raw binary data.
If this is a simple text password string we can fix the corruption with the script supplied by the patch 36842768:R12.FND.C"


Scripts are under -> p36842768_R12.FND.C_R12_GENERIC (1).zip\36842768\fnd\patch\115\sql but! this is risky. You should know what you are doing.

Anyways, please review the logs and try to find some clues ( clues about the FND_VAULT.initilize or other clues unrelated to that..) In any case, I would consider applying this patch.

However; my follower didn't want to take the risk (which was quite normal in this case), and uploaded all the info to the Oracle Support SR.

Oracle SR provided the datafix via the script ->  WebSecFix.sql 119.0. That script was updating FND_ORACLE_USERID table. It was decrypting and encrypting the contents of that table.  It seemed it was be fixing the entire chain, ending with re-encrypted fresh passwords.

Okay now, FNDCPASS was working fine, and could change the password(s), but! this time adop cutover phase was failing :)

Update from my follower (the owner of this issue): 

when i do adop phase=cutover i get
fndsvcrg.log ------

cannot complete application logon you may have entered an invalid application password or there may have been a database connect error . internal concurrent manager status could not be determined .

I am searching if there is any possible issue .

Application login is working fine FNDCPASS is working fine ,


Patch below was already applied:
the MOS note: "EBS Upgrade From 12.1.3 to 12.2.12 Fails at Adop Cutover with Internal Concurrent Manager status could not be determined" , KB757279 is describing the same exact thing.  
(This was my recommendation)

At this point, setting the patch environment and running cutover again worked as a workaround, that patch (patch is recommended by Oracle in the beginning - 38207032) was applied and everything started working fine. However; while these were happening; Oracle has released final patch for this issue 38965426. 

That is, 38207032:R12.FND.C is replaced by 38965426:R12.FND.C. So my follower decided to use this patch instead, and said: "I will reclone the staging system and reapply again before applying on production. This is the bug in oracle which has been linked with apex user creation. Now this bug is registered and patch has been released and published , Thanks alot for your support and cooperation ."

Well... That's the issue of the day.. Knowledge is the only treasure that increases when shared!