EBS 11i - INVIDITM / Master Items hangs/not opening -- case study & lessons learned

After a period of time, I put my Apps DBA's hat once again :)
This time, I needed to investigate a strange issue in a customer environment.
(let's say a new customer environment :)

Well.. I had to investigate and solve an issue in a mission critical EBS 11i Production environment.

This EBS environment was on Solaris and the database tier of it was 12cR1.

The problem was the Master Items form.. It was not opening at all..

When the users click on Master Items icon, the form was not opening and the java on client side was just hanging.. (even the java console was crashing)

I did lots of diagnostic works to investigate the root cause of it, but I couldn't determine any usable information..

Following is the list of the things; that I did to investigate and solve the issue;

• Analyzed the tcpdump between the client and application server side (the traffic between the forms applet and the application server side)
• Disabled the client side java cache and cleared the java cache in client, just in case where the client side/cached forms jars are corrupt (didn't solve)
• Enabled the FRD trace and saw that the problematic form was hanging on a specific point, without any errors. (actually I saw signal 14 in FRD trace, but it was misleading.. Form server could have signal 14 in lots of cases)
• Enabled Java Plugin trace ( when the problem appeared, the plugin itself was crashing, so couldn't get any usable info from this trace)
• Opened the problematic form with Forms Builder and analyzed the part of the code where the form hanging.. no obvious issues were there..
• Disabled the APPS Tier SSL and retested the issue, just in case..(didn't solve)
• Tried with different java plugins and browsers according to the certified Java-Browser matrix.. (didn't solve)
• Recompiled/regenerated everything using adadmin... forms, flexfields , jar files and everything.. (didn't solve)
• Checked the db session and the alert log (no weirdness, no errors)
• Recompiled the Apps schema. (actually there was no important invalid objects there..)
• Increased the heap sizes of the java plugin in the client side.. (didn't solve)
• Requested functional admins to check the functional folders that are defined on EBS. (no problems were there)
• Wanted the Apps DBAs to apply Patch 14009893 to Forms Oracle Home. They applied it.. But they could not open any forms after applying it.. (maybe there was a failure during this patch application process, I didn't check it actually...)

So after these diagnostics and tries, I decided to gather the change log of this EBS environment to understand what was done/changed on this environment recently..

The most important change, that I have pointed out was the EBS's database upgrade.. That is, the database of this EBS environment was recently upgraded to 12cR1 and unfortuneatly, there was no user acceptence test report for that problematic form...

With this in mind, I analyzed the upgrade process..

The upgrade process was based on the document named "Interoperability Notes Oracle EBS 11i with Database 12cR1 (12.1.0.1) (Doc ID 1968807.1)".

This interop document was redirecting to the "Upgrading Developer 6i with Oracle Applications 11i (Doc ID 125767.1)"  by saying : "If your patch set level is earlier than patch set 19, apply the latest certified patch set. See Upgrading Developer 6i with Oracle Applications 11i on My Oracle Support".

So when I checked the document named "Upgrading Developer 6i.." ; I saw that it was pointing to the patch 22709024 , which was a merged patch, developed for fixing several forms server bugs.

Patch 22709024: MERGE REQUEST ON TOP OF 6.0.8.28.0 FOR BUGS 21671403 22351071

Well.. This patch was not applied to the environment. So we applied it and that was it ! It fixed the issue! 

Actually, this patch was the "new version" of the patch that I requested the Apps DBA team to apply in the first place.

Patch change log:

Replaced MLR Patch 21671403 with the latest MLR Patch 22709024 in the Download Additional Developer 6i Patches table.
Sep 23, 2015
Replaced MLR Patch 19444825 with the latest MLR Patch 21671403 in the Download Additional Developer 6i Patches table .
Aug 11, 2015
Replaced MLR Patch 16699473 with the latest MLR Patch 19444825.
May 31, 2013
Replaced MLR Patch 16414360 with the latest MLR Patch 16699473.
Mar 18, 2013
Replaced MLR Patch 14615390 with the latest MLR Patch 16414360.
Oct 19, 2012
Replaced MLR Patch 14009893 with the latest MLR Patch 14615390.
Jun 12, 2012
Replaced MLR Patch 13384700 with MLR Patch 14009893.
Jan 05, 2012
Replaced Windows MLR Patch 9436629 with Patch 13384700.
Added one off Patch 13384700 for HP Tru64.

At the end of the day; the lessons learned were the following : 

• If you are a dealing with an issue in a new environment (new to you), request the change log.
• Always make your detailed diagnostics. Gather technical info about the problem by doing detailed diagnostics works.
• Search knowledge base for the error. Consider implementing your findings (first in TEST, then in PROD)
• If you can't find anything and if you can't see anything during your diagnostics works, search for patches for the related technology.. (forms in this case)
• Based on the change log of the environment, check if there are any missing patches. 
• Open an Oracle Support SR :)
• If you find a patch, which seems related, check its newer versions.. Check the compliance between your EBS version and the patch before applying it.
• Always document the things you checked and the things you tried. By doing so, you can narrow down the list of things that can be done to solve the issue.
• If it is a developer related problem , contact your development team to get help, to make them analyze the code.
• Lastly, don't trust the things that are said to you. Always analyze the problem with your own eyes :) for example: if the env is recently upgraded, check the things that need to be done during the whole upgrade path. Don't accept the statements like "we upgraded this env properly, applying all the patches" :) Do you own checks and then decide ..

sds