EBS -- Cannot Login & FNDCPASS Error in password verification for APPS /FND_VAULT & FND_ORACLE_USERID and all that

In the world of Oracle E-Business Suite, an upgrade completion is often seen as the finish line. However, as many seasoned APPS DBAs know, the real challenges sometimes begin the moment you try to secure the newly upgraded environment. Recently, a complex case surfaced on my forum that perfectly illustrates this. Following an upgrade from 12.2.9 to 12.2.14, the system wouldn't allow logins due to a bug, but the patch to fix the bug couldn't be applied because the applications patching utility - adop itself was unable to authenticate. Exploring the underlying FND_VAULT data inconsistencies, data fixes and the out-of-the-box workarounds required to break the cycle. As I always say, knowledge is the only treasure that increases when shared! 

Let’s get into the technical heart of the issue. Recently I got a support call in my forum. It was about APPS password verification and it required lots of efforts to be fixed. That's why I thought this issue was worth sharing. So, here I'm blogging it.

Here is the forum url for the ones who want to read the original thread: http://erman-arslan-s-oracle-forum.124.s1.nabble.com/Working-Error-in-password-verification-for-APPS-td13177.html

The issue started after upgrading EBS from 12.2.9 to 12.2.14. Actually, once the upgrade was finished,  everything was working one, but the issue started after changing the APPS password.

Apps password could be changed successfully, but after that, EBS login was not working and at that point; re-changing the password was not a fix, because it was also failing.

FNDCPASS was throwing -> Error in password verification for APPS (generic error), and Oracle Support said that this was due to a bug,  but in order to fix it, we needed to patch the system, but adop prepare was also failing.. (cannot complete application logon you may have entered an invalid application passwords , or there may have been a database connect error. internal manager status could not be determined)

In that kind of a broken password situation, these things were expected actually.

When my follower (owner of this issue) asked it to Oracle Support, they suggested to apply 38207032, FND and AD fixes for APEX Schema user integration (Patch). (Note that, APEX configuration was also there, in this system) But! that patch could not be applied, because adop couldn't login due to this issue. adop was trying to connect to the application and submit the relevant program for online patching but it was failing...

Additional symptoms:

$FND_TOP/bin/FNDSVCRG STATUS

Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Internal Concurrent Manager status could not be determined.

---
$ adcmctl.sh status <apps_user>/<apps_password>

Cannot complete applications logon. You may have entered an invalid applications password, or
there may have been a database connect error.

Internal Concurrent Manager status could not be determined. adcmctl.sh: exiting with status 0

---

FNDCPASS system/***** USER sysadmin *****
+----------------------------------------------------------------------------+
Working...
Error in password verification for APPS

After analyzing, I thought that this may be related with the FND_VAULT. It seemed like a FND_VAULT datafix was required, and I found a patch and updated the thread with the following;

If you could trace your login issues, you could maybe see the FND_VAULT is failing in the backend..

For instance; in the following note, the same patch is also recommended;

Login Doesn't Work After R12.ATG_PF.C.delta.12 (34776645) Patch And GUEST Password Change, KB291738

Here the note says - > On EBS 12.2 version Login doesn't work after R12.ATG_PF.C.delta.12 (34776645) patch and a GUEST password change. The customer has applied patch 34776645 along with the latest AD and TXK patches. They also changed GUEST password, then they can't login to the applications.
fnd_web_sec.validate_login returns Y for the new GUEST password.
Response in Google Chrome tracing suggest that the application may still be using the old GUEST password which is invalid.
If they change to the old GUEST password, they can login fine, but Forms do not launch.
They tried clearing tmp and cache dir as mentioned in the Doc ID 3013989.1, but it did not help.

So, that patch is a cure for login issues.. There we see.

Also see the following -> https://ebs-dba.com/wp/blog/2025/07/20/12-2-14-changing-passwords-and-extending-ebs-with-apex-a-critical-warning/

There is also a reference to that patch in the article above.

And see this MOS note -> Unable To Change Password With Hard To Guess Profile, KB320427

There Oracle says:

1. Please download and review the readme for <Patch 36961988:R12.FND.C> AFTER PATCH 36857308 : AFPASSWD AND FNDCPASS ALLOWING SPECIAL CHARACTERS IN APPS PASSWORD
2. Please apply <Patch 36961988:R12.FND.C> in a Test environment initially before application to any Production instance.
3. Please retest the issue.
4. If the issue is resolved, please migrate the solution as appropriate to other environments.

We needed to be sure about the corruption in FND_VAULT related data, all the tables related to it, FND_ORACLE_USERID and so on, so I suggested;

That statement is given in MOS: "Applying 12.2.13 RUP(34776655) Failed With ORA-06512: AT APPS.FND_VAULT" -> "The issue is encountered in environments where the FND_VAULT table contains inconsistent or invalid data, such as plain text values instead of the expected RAW format."

So, FND_VAULT_KEYMASTER and FND_VAULT_DATA tables are in the play.

In the patch, we have a datafix -> 36324608 FND_VAULT DATA FIX

That fix is however applicable in cases where you have FND_VAULT initialization problems.

If you have FND_VAULT related corruption, you should see them in the some of logs at least.

Note  ORA-20001: Please Review FND_VAULT.INITIALIZE Output Before Continuing -> says :

"Manually run the script from the patch 36842768:R12.FND.C
In 12.2.13 we discovered a data corruption issue with FND_Vault, the script looks for the indicators of this corruption and fixes known issues.
In this case this is not a known entry for us so we need to verify with the data owner (ISG) if this is simple ASCII data or is it raw binary data.
If this is a simple text password string we can fix the corruption with the script supplied by the patch 36842768:R12.FND.C"

Scripts are under -> p36842768_R12.FND.C_R12_GENERIC (1).zip\36842768\fnd\patch\115\sql but! this is risky. You should know what you are doing.

Anyways, please review the logs and try to find some clues ( clues about the FND_VAULT.initilize or other clues unrelated to that..) In any case, I would consider applying this patch.

However; my follower didn't want to take the risk (which was quite normal in this case), and uploaded all the info to the Oracle Support SR.

Oracle SR provided the datafix via the script ->  WebSecFix.sql 119.0. That script was updating FND_ORACLE_USERID table. It was decrypting and encrypting the contents of that table.  It seemed it was be fixing the entire chain, ending with re-encrypted fresh passwords.

Okay now, FNDCPASS was working fine, and could change the password(s), but! this time adop cutover phase was failing :)

Update from my follower (the owner of this issue): 

when i do adop phase=cutover i get
fndsvcrg.log ------

cannot complete application logon you may have entered an invalid application password or there may have been a database connect error . internal concurrent manager status could not be determined .

I am searching if there is any possible issue .

Application login is working fine FNDCPASS is working fine ,

Patch below was already applied:
the MOS note: "EBS Upgrade From 12.1.3 to 12.2.12 Fails at Adop Cutover with Internal Concurrent Manager status could not be determined" , KB757279 is describing the same exact thing.  (This was my recommendation)

At this point, setting the patch environment and running cutover again worked as a workaround, that patch (patch is recommended by Oracle in the beginning - 38207032) was applied and everything started working fine. However; while these were happening; Oracle has released final patch for this issue 38965426. 

That is, 38207032:R12.FND.C is replaced by 38965426:R12.FND.C. So my follower decided to use this patch instead, and said: "I will reclone the staging system and reapply again before applying on production. This is the bug in oracle which has been linked with apex user creation. Now this bug is registered and patch has been released and published , Thanks alot for your support and cooperation ."

Well... That's the issue of the day.. Knowledge is the only treasure that increases when shared!

From Readers to Algorithms: Seeing My Oracle Work in the Anthropic Copyright List

 Today I received an email about the Anthropic Copyright Settlement, which was an unexpected reminder of how AI training data can directly affect individual authors. After checking the official settlement website, I saw that one of my works (especially Practical Oracle E-Business Suite, The Book, Apress 2017) appears in the lookup list. It is a strange feeling to see your work surface this way, through a legal notice rather than through readers. Cases like this show how urgently the relationship between creators, copyright, and AI still needs clearer boundaries.

Oracle Database 26AI -- Vector Search (Giving Mass to Information) , JSON-RDBMS Duality (Particle-Wave Duality of Data) and True Cache (The Observer at Near-Light Speed)

This post will be about the key features of Oracle Database 26ai. Actually, about the features introduced in 23ai and enhanced in 26ai. Again as the name suggest the vector search (the vector and embedding capabilities actually) will be revisited, JSON-Relational duality as well, but more than that, I will focus on the True cache. 

In order to make this unique, and fun, I will start with physics and built my metaphorical bridge to 26ai, Embeddings & Vector Search, JSON Relational duality and True Cache. This post is for giving you a new perspective to assess the value that 26ai brings to the table for the data, and the database layer.

Okay. Let's start with neutrinos.(You weren't expecting this, you'll say "what's the connection?", but read on...)  

I will suppose that you already know what a neutrino is and that's why I will skip giving background information part ( but I'll still give you basic info of what a neutrino is), and directly jump into the interesting staff about it. 

Don't panic , I will make 26ai and True cache be on the spotlight in the middle of this article, and finalize accordingly. Note that, my next blog post will be technical, I will make you understand the True cache via a short demo. But! With this post, I will make it interesting first and whet our appetites.

Let's really start now. The Neutrino, a ghostly, nearly massless subatomic particle with no electric charge that can zip through an entire planet without ever touching an atom.

For a long time, we thought neutrinos were massless. But if they were truly massless, they would travel at the speed of light. According to Einstein’s Special Relativity, for anything traveling at the speed of light, time stands still. (Actually, the concept of time ceases to be valid for an observer moving at the speed of light, but let's not get into that now.) 

An object with no internal clock cannot change; it cannot evolve. A massless neutrino would be frozen in whatever state it started in. 

However; it was discovered that Neutrinos oscillate, they change their flavor as they travel. So it meant and proved that they have mass. Because they have mass, time flows for them. Because time flows for them, they can transition.

So let's bind this to Oracle 26ai :) As you may already guess, 26ai is giving mass and speed to the enterprise database layer. Now, with 26ai our data in our core databases transform from being massless --static-- to being intelligent -- evolving.

In this context, AI-related capabilities (like the vector space, embeddings and vector search) in Oracle Database is what gives mass to information. 

In traditional RDBMS, data was often like a massless neutrino. A string was just a string, a number was just a number. It moved fast, but it didn't evolve or carry meaning within the database itself. It had no internal clock for context.

With Oracle Database 26ai (and earlier with 23ai), we add mass (intelligence) to our data. By representing data as vectors , the database now understands the semantic weight of information. Just as a neutrino’s mass allows it to change flavor, Vector capabilities of 26ai allows our data to transition from bits into meaningful insights. 

**Technically, it's the embedding process -- the model that generates the vectors-- that adds the meaning, and we got that inside the Oracle Database.. (Oracle AI Database 26ai includes an ONNX runtime engine for running embedding models directly inside the database.. We have VECTOR_EMBEDDING function and all that...)

Our database is no longer a static observer; it understands the closeness of notions of data, just as gravity acts on objects with mass.

Well.. Let's continue with the True cache. The speed of light problem in our data architectures. I mean, we want our applications to be fast. But especially in distributed systems, as the distance between APP and DB increases, latency increases too.. Latency in this context takes us away from being fast. (Like in physics; the further the observer, the more latency ...)

This is where caching comes in, but! we will go with Oracle True Cache here!

In our context, in our analogical view, we said AI Vector Search brings mass. If it is so, True Cache brings the velocity. True Cache is an in-memory cache. It is eventually consistent with the primary at a very tight lag window. It is transparent cache that logically sits in front of the primary database. It is beneficial both in standalone environments and in distributed environments. True Cache can be clustered, or it can also be positioned as a sharded cache (one true cache for finance data for the finance app, one true cache for sales data from the sales app, or sales module of the main app) Besides, True Cache is easy to manage (almost auto managed), built on top of proven Oracle Technologies, and tightly integrated with the Oracle Database. (I will get into that in more details in my next blog post)

So far so good. Well, in physics, your perception of time depends on where you are. In IT, your perception of current data depends on your cache. Unlike traditional side-caches that have serving risk of stale data, and managed explicitly; true Cache is for managed by Oracle. 

It’s like having a neutrino detector right at the edge of the galaxy. It allows the application to read at RAM speed while staying perfectly synchronized with the proper time (SCN) of the primary instance. 

Of course we must have the necessary tech investment to ensure that. I mean, we shouldn't wait so long to get the data we want to be present on True Cache. True cache have the technology. But! the it shouldn't wait that data to be delivered to its memory structure for so long. The True Cache shouldn't wait the compute, the computational power to be utilized to it for serving the data we need (network and compute I mean...)

True Cache ensures that even though you are moving fast, your clock (data consistency) remains valid. You get the speed of a nearly massless particle (It's just a cute analogy, of course:) without losing the identity of the data. (like without losing the flavor of a neutrino :)

At this point, I want you to note that, I wanted my claim to be appealing .. I mean the one about the speed of True Cache and the analogy about the data consistency..

 But! I also don't want to make physicists angry. That's why, I feel the need to clarify that, in relativity, increasing velocity dilates time.. So it slows our clock relative to a stationary observer, whereas True Cache does the opposite: it reduces perceived latency while keeping the clock (SCN) synchronized.. Well, analogies are hard and risky :) again here we see :) 

Before finalizing, I want to remind you the JSON Relational duality. Again with the metaphorical bridge to physics. Through the duality views, applications can access (read or change) the same data as either a collection of JSON documents or as a set of related tables and columns. 

So basically, we may think it like, store the data in relation tabular format, but if needed serve it in JSON. Apps that need document mode, can use document APIs to get the data, apps like analytical engines can use languages such as SQL, PL/SQL, C, and JavaScript and get the data in relational mode, directly as rows and columns.

So this make it possible to store the same data in two states at once. Like the "Particle-Wave Duality" of data :), isn't it?

Okay. We pushed hard for the metaphors, now let's conclude it with the finalization.

How did we start? We started with the subject of neutrinos and connected it to the Vector capabilities of Oracle Database 26ai, so our conclusion for that is : "If a neutrino had no mass, the universe would be a static, boring place where nothing ever changes. If our database has no AI mass, our data will remain static, and our applications will eventually lose their flavor in the face of modern requirements." :)

We already pushed hard with the metaphors for explaining the value of True Cache and JSON Relation duality, so I will not give a conclusion for those :) 

My final words will be for us, the DBA(s), APPS DBA(s) and Architects. We are no longer just administrators or architects. We are the observers of a data universe that is finally beginning to behave according to the laws of intelligence and speed. 

So, we should stay curious, keep learning, and we should not be afraid to add a little mass to our career :)

Until we meet again...

SuperCluster M8 - EOSL March 2026 (Targeted for End of Support Life effective next month)

You may remember my warning about Oracle SuperCluster M8's End Of Life (EOL) & End Of Support (EOS) dates.  (link to that: https://ermanarslan.blogspot.com/2024/03/oracle-supercluster-m8-end-of-life-eol.html)

Now what I want you to remind that, "SuperCluster M8 will reach its End of Support Life in March 2026".

Since these systems have reached EOSL, you can no longer even perform a support renewal. In fact, the window for migration and replatforming from these platforms is already long overdue. Therefore, let’s keep this in mind and evaluate it with urgency:

Considering these risks, it's recommended to migrate from Oracle Supercluster M8 to a supported system (Like EXADATA or PCA or EXADATA + PCA, or ODA + EXADATA or Oracle/Sun Traditional Servers + ZFS.. and KVM virtualization..)  Choice depends on the context actually.. These are some other on-prem options as well .

If you currently own a SuperCluster and are considering replatforming—especially within the Oracle ecosystem—I would be honored to assist you. We possess deep expertise and extensive experience in this domain. Please do not hesitate to reach out. From initial hardware setup to comprehensive database and application migrations, we provide a professional approach that includes both strategic guidance and hands-on execution. Feel free to contact me whenever needed.

FMW 12C - Upcoming Premium Support End Date & Support TimeLine

 This is for FMW 12C users, or solution provider who built their solution on top of FMW 12C product family. ( It is a big family by the way).

We checked the support lifecycles (Premier → Extended → Sustaining) of Oracle FMW  components against current Oracle documentation. This work confirmed that components in the Oracle Fusion Middleware 12c line will complete their Premier Support phase in December 2026, their Extended Support phase in December 2027, and then enter the Sustaining Support phase.

Below, we share the scope of the support types, the status of the products we use (this is from our perspective, I mean the family is wider) , and the actions that need to be taken in one go.

Feature	Premier Support	Extended Support	Sustaining Support
Scope	Comprehensive: Security patches (CPU), bug fixes, and new certifications.	Limited: Critical patches and maintenance (restricted scope).	No new patches/fixes: Access to existing knowledge base and previous patches only.
Duration	Generally ~5 years	Generally +3 years (varies by product)	Indefinite / Perpetual
Fees	Standard annual maintenance (~22%)	Additional fee (typically a 10–20% uplift/premium applied)	Standard maintenance fee; no new fixes included

Exception to "Market-Driven Support"

If a very old version is still being used (for example, Database 11.2.0.4 or 12.1.0.2), Oracle sometimes offers a fourth layer called Market-Driven Support even after Extended expires. This is a very expensive "lifeline" service that only covers the most critical vulnerabilities.

About "Extended Support fee"

Extended Support fees vary depending on Oracle products/family; field licensing guides indicate a range of 10–20%. so Extended Support is an additional fee, and especially in environments using embedded licenses, it is crucial to clarify whether the relevant extended support fee is included in the existing embedded support agreements.

The following table is prepared for the FMW family products in our focus.. As I mentioned earlier, the whole family is bigger than that, and there are some extras (like ODI 11G) in the list, this is because ODI 11G is still used in the environments around us.

Oracle Product	Version	Premier Support End	Extended Support (Start–End)	Sustaining Support (Starts)	Required Action
WebLogic Server	12.2.1.4	December 2026	December 2026 – December 2027	December 2027 → Indefinite	14c upgrade plan should be prepared (before moving to Sustaining).
Oracle Data Integrator (ODI)	12.2.1.4	December 2026	December 2026 – December 2027	December 2027 → Indefinite	14c migration plan should be prepared.
Oracle Data Integrator (ODI)	11g	December 2018	2018 – 2021	Since December 2021	Urgent: Must be consolidated by upgrading from 12c → 14c.
BI Publisher	12.2.1.3	September 2022	September 2022 – June 2023	Since June 2023	Urgent: Must be replaced with OAS (Oracle Analytics Server).
Oracle Analytics Server (OAS) (incl. BI Publisher capabilities)	2025 (8.2) * based on FMW 12.2.1.4	Subject to OAS release cycle; Premier 2026 / Extended 2027 for FMW 12.2.1.4 line.	—	After December 2027 Sustaining for FMW 12c	Plan migration to the FMW 14c base with the next version of OAS.

Note that, OAS 2025 (8.2) version requires FMW 12.2.1.4; the corresponding FMW component is supported throughout the OAS version support lifecycle. However, for standalone WLS usage, the general FMW 12c roadmap (2026/2027) remains applicable. [community.oracle.com], [docs.oracle.com].

Assessment of the Current Situation:

December 2026: FMW 12c products will transition out of Premier Support. [oracle.com], [docs.oracle.com]

December 2027: Extended Support concludes. Beyond this date, during the Sustaining Support phase, no new security patches (CPU), bug fixes, or new OS/Browser certifications will be provided. [oracle.com]

From a financial regulation and audit perspective (SA, Symphony Core, Statutory Reporting), remaining on Sustaining Support poses a significant risk. [oracle.com]

Conclusion and Recommendation: 

The Oracle FMW 12c product line is approaching its critical 2026/2027 thresholds. It is essential to finalize the 14c migration plan on a per-product basis and establish a clear implementation roadmap.