EBS R12 -- APP-FND-01926 ORA-06508 ORA-01403 errors in Clone evironment

In a new clone environment, while opening the forms, you may encounter something like the following;

Yesterday such a problem was escalated to me.
Lets look what I did for diagnosing the problem, finding the actual cause and fixing it.

Reviewed notes:

APP-FND-01926, ORA-06508 (Doc ID 797242.1) --> Custom.pll and plx was in place, recompiled and didnt see any problems.. Also, no environment variable problems were seen

"App-Fnd-01926" and "ORA-01403" Errors Occur While Initializing Forms (Doc ID 1290156.1) --> checked the profile "FND_MO_INIT_CI_DEBUG" .. It was not set..

Could not Find Program Unit when Connecting to Apps (Doc ID 229399.1) --> did not fix..compiling all forms and all libraries did not fixed..

Client Gets App-Fnd-01926 Every Time After Bounce Apps 11i Forms Service (Doc ID 555409.1) --> no pll , plx , fmx or fmb were located in the alternate directories.. Not related

Troubleshooting ORA-01403, ORA-06508 and FRM-40735 in Forms (Doc ID 459005.1) --> saw some diffences between pll and plx files.. (custom.pll and custom.plx versions were ok bytheway)

Also tried the following;

apps compile

Regenerate jar files.

Relink Applications

Restarted Application Tier..

--> Did not help..

Checked the forms server logs:

Nothing much there.. No errors . No warnings..

Checked the Clients java console log:(also enabled trace and debug for java in the control panel)

Nothing much in java console log..

It was loading the images, no errors.. It was starting the applet, and waits.. Seemed like forms server is the one that generates the error.
 

basic: Loaded image: jar:http://ermanhost:8087/OA_JAVA/oracle/apps/fnd/jar/fndaol.jar!/oracle/apps/media/FNDIEROR.gif

basic: Applet started

basic: Told clients applet is started

network: CleanupThread used 7826080 us

Checked the CUSTOM.pll and CUSTOM.plx headers.. Also compared with the Live environment.. Their versions were the same..

Opened the custom.pll using Forms Builder and checked the libraries attached to it, there were several libraries there.. So checked all the pll and plx  version and saw several pll and plx files which differ in version..

Used the scripts in the document:

Troubleshooting ORA-01403, ORA-06508 and FRM-40735 in Forms (Doc ID 459005.1)

Found several files.

Then decided to compile all the plls using frmcmp_batch..

Prepared a script and executed it..
But, this was a good try, but did not solve the problem...

Copied AU_TOP/resource from the LIVE environment once again..
Didnt solve..

After all, checked the FRD trace which is present in FORMS_TRACE_DIR again and found the following:

Error Message: FRM-40815: Variable GLOBAL.IGI_INSTALLED_GLOBAL_VAR does not exist.

Error Message: FRM-40815: Variable GLOBAL.IGI_INSTALLED_GLOBAL_VAR does not exist

Error Message: FRM-40815: Variable GLOBAL.IGI_PREV_ORG does not exist.

Error Message: FRM-40815: Variable GLOBAL.IGI_INSTALLED_ORG_VAR does not exist.

Error Message: FRM-40815: Variable GLOBAL.OPM_GML_INSTALLED1 does not exist.

Okay. at this point; at least we have seen the errors..
There were bunch of documents for these specific error, but I have started to feel that this blogpost

will never end :) That's why I w ll keep it short..

SOLUTION:

The problem was in CUSTOM.pll ..

For the solution - > Deployed a backup CUSTOM.pll , compiled it and restarted the forms server. This was the fix!

On the other hand; even with the old CUSTOM.pll , there were the same FRM-40815 errors in the forms server logs..So these FRM-40815 errors were not the actual causes of the problem.. The exact cause was the CUSTOM.pll...

To find the actual cause, I analyzed the forms trace file..

Forms server was saying "Comleted call_all_libraries" for the CUSTOM.pll that was working properly ... For the problematic pll, ,Forms Server was saying "CUSTOM_EVENT_EXCEPTION ORA-06508 ORA-01403" in the same lines..

So, it was obvious that the problematic CUSTOM.pll had some plsql which executes some sqls and encounters ORA-01403..

Actually, there was a problematic plsql call in the CUSTOM.pll .. That problematic plsql was belong to another custom.pll(lets say blabla.pll) and the problem was caused because of it, not the CUSTOM.pll actually..