Friday, March 6, 2026

EBS -- Cannot Login & FNDCPASS Error in password verification for APPS /FND_VAULT & FND_ORACLE_USERID and all that

In the world of Oracle E-Business Suite, an upgrade completion is often seen as the finish line. However, as many seasoned APPS DBAs know, the real challenges sometimes begin the moment you try to secure the newly upgraded environment. Recently, a complex case surfaced on my forum that perfectly illustrates this. Following an upgrade from 12.2.9 to 12.2.14, the system wouldn't allow logins due to a bug, but the patch to fix the bug couldn't be applied because the applications patching utility - adop itself was unable to authenticate. Exploring the underlying FND_VAULT data inconsistencies, data fixes and the out-of-the-box workarounds required to break the cycle. As I always say, knowledge is the only treasure that increases when shared! 

Let’s get into the technical heart of the issue. Recently I got a support call in my forum. It was about APPS password verification and it required lots of efforts to be fixed. That's why I thought this issue was worth sharing. So, here I'm blogging it.

Here is the forum url for the ones who want to read the original thread: http://erman-arslan-s-oracle-forum.124.s1.nabble.com/Working-Error-in-password-verification-for-APPS-td13177.html

The issue started after upgrading EBS from 12.2.9 to 12.2.14. Actually, once the upgrade was finished,  everything was working one, but the issue started after changing the APPS password.

Apps password could be changed successfully, but after that, EBS login was not working and at that point; re-changing the password was not a fix, because it was also failing.

FNDCPASS was throwing -> Error in password verification for APPS (generic error), and Oracle Support said that this was due to a bug,  but in order to fix it, we needed to patch the system, but adop prepare was also failing.. (cannot complete application logon you may have entered an invalid application passwords , or there may have been a database connect error. internal manager status could not be determined)

In that kind of a broken password situation, these things were expected actually.

When my follower (owner of this issue) asked it to Oracle Support, they suggested to apply 38207032, FND and AD fixes for APEX Schema user integration (Patch). (Note that, APEX configuration was also there, in this system) But! that patch could not be applied, because adop couldn't login due to this issue. adop was trying to connect to the application and submit the relevant program for online patching but it was failing...

Additional symptoms:

$FND_TOP/bin/FNDSVCRG STATUS

Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Internal Concurrent Manager status could not be determined.

---
$ adcmctl.sh status <apps_user>/<apps_password>

Cannot complete applications logon. You may have entered an invalid applications password, or
there may have been a database connect error.

Internal Concurrent Manager status could not be determined. adcmctl.sh: exiting with status 0

---
FNDCPASS system/***** USER sysadmin *****
+----------------------------------------------------------------------------+
Working...
Error in password verification for APPS

After analyzing, I thought that this may be related with the FND_VAULT. It seemed like a FND_VAULT datafix was required, and I found a patch and updated the thread with the following;

If you could trace your login issues, you could maybe see the FND_VAULT is failing in the backend..

For instance; in the following note, the same patch is also recommended;

Login Doesn't Work After R12.ATG_PF.C.delta.12 (34776645) Patch And GUEST Password Change, KB291738

Here the note says - > On EBS 12.2 version Login doesn't work after R12.ATG_PF.C.delta.12 (34776645) patch and a GUEST password change. The customer has applied patch 34776645 along with the latest AD and TXK patches. They also changed GUEST password, then they can't login to the applications.
fnd_web_sec.validate_login returns Y for the new GUEST password.
Response in Google Chrome tracing suggest that the application may still be using the old GUEST password which is invalid.
If they change to the old GUEST password, they can login fine, but Forms do not launch.
They tried clearing tmp and cache dir as mentioned in the Doc ID 3013989.1, but it did not help.

So, that patch is a cure for login issues.. There we see.

Also see the following -> https://ebs-dba.com/wp/blog/2025/07/20/12-2-14-changing-passwords-and-extending-ebs-with-apex-a-critical-warning/

There is also a reference to that patch in the article above.

And see this MOS note -> Unable To Change Password With Hard To Guess Profile, KB320427

There Oracle says:

1. Please download and review the readme for <Patch 36961988:R12.FND.C> AFTER PATCH 36857308 : AFPASSWD AND FNDCPASS ALLOWING SPECIAL CHARACTERS IN APPS PASSWORD
2. Please apply <Patch 36961988:R12.FND.C> in a Test environment initially before application to any Production instance.
3. Please retest the issue.
4. If the issue is resolved, please migrate the solution as appropriate to other environments.

We needed to be sure about the corruption in FND_VAULT related data, all the tables related to it, FND_ORACLE_USERID and so on, so I suggested;

That statement is given in MOS: "Applying 12.2.13 RUP(34776655) Failed With ORA-06512: AT APPS.FND_VAULT" -> "The issue is encountered in environments where the FND_VAULT table contains inconsistent or invalid data, such as plain text values instead of the expected RAW format."

So, FND_VAULT_KEYMASTER and FND_VAULT_DATA tables are in the play.

In the patch, we have a datafix -> 36324608 FND_VAULT DATA FIX

That fix is however applicable in cases where you have FND_VAULT initialization problems.

If you have FND_VAULT related corruption, you should see them in the some of logs at least.

Note  ORA-20001: Please Review FND_VAULT.INITIALIZE Output Before Continuing -> says :

"Manually run the script from the patch 36842768:R12.FND.C
In 12.2.13 we discovered a data corruption issue with FND_Vault, the script looks for the indicators of this corruption and fixes known issues.
In this case this is not a known entry for us so we need to verify with the data owner (ISG) if this is simple ASCII data or is it raw binary data.
If this is a simple text password string we can fix the corruption with the script supplied by the patch 36842768:R12.FND.C"


Scripts are under -> p36842768_R12.FND.C_R12_GENERIC (1).zip\36842768\fnd\patch\115\sql but! this is risky. You should know what you are doing.

Anyways, please review the logs and try to find some clues ( clues about the FND_VAULT.initilize or other clues unrelated to that..) In any case, I would consider applying this patch.

However; my follower didn't want to take the risk (which was quite normal in this case), and uploaded all the info to the Oracle Support SR.

Oracle SR provided the datafix via the script ->  WebSecFix.sql 119.0. That script was updating FND_ORACLE_USERID table. It was decrypting and encrypting the contents of that table.  It seemed it was be fixing the entire chain, ending with re-encrypted fresh passwords.

Okay now, FNDCPASS was working fine, and could change the password(s), but! this time adop cutover phase was failing :)

Update from my follower (the owner of this issue): 

when i do adop phase=cutover i get
fndsvcrg.log ------

cannot complete application logon you may have entered an invalid application password or there may have been a database connect error . internal concurrent manager status could not be determined .

I am searching if there is any possible issue .

Application login is working fine FNDCPASS is working fine ,


Patch below was already applied:
the MOS note: "EBS Upgrade From 12.1.3 to 12.2.12 Fails at Adop Cutover with Internal Concurrent Manager status could not be determined" , KB757279 is describing the same exact thing.  
(This was my recommendation)

At this point, setting the patch environment and running cutover again worked as a workaround, that patch (patch is recommended by Oracle in the beginning - 38207032) was applied and everything started working fine. However; while these were happening; Oracle has released final patch for this issue 38965426. 

That is, 38207032:R12.FND.C is replaced by 38965426:R12.FND.C. So my follower decided to use this patch instead, and said: "I will reclone the staging system and reapply again before applying on production. This is the bug in oracle which has been linked with apex user creation. Now this bug is registered and patch has been released and published , Thanks alot for your support and cooperation ."

Well... That's the issue of the day.. Knowledge is the only treasure that increases when shared!