EBS 11i - INVIDITM / Master Items hangs/not opening -- case study & lessons learned

After a period of time, I put my Apps DBA's hat once again :)
This time, I needed to investigate a strange issue in a customer environment.
(let's say a new customer environment :)

Well.. I had to investigate and solve an issue in a mission critical EBS 11i Production environment.

This EBS environment was on Solaris and the database tier of it was 12cR1.

The problem was the Master Items form.. It was not opening at all..

When the users click on Master Items icon, the form was not opening and the java on client side was just hanging.. (even the java console was crashing)

I did lots of diagnostic works to investigate the root cause of it, but I couldn't determine any usable information..

Following is the list of the things; that I did to investigate and solve the issue;

• Analyzed the tcpdump between the client and application server side (the traffic between the forms applet and the application server side)
• Disabled the client side java cache and cleared the java cache in client, just in case where the client side/cached forms jars are corrupt (didn't solve)
• Enabled the FRD trace and saw that the problematic form was hanging on a specific point, without any errors. (actually I saw signal 14 in FRD trace, but it was misleading.. Form server could have signal 14 in lots of cases)
• Enabled Java Plugin trace ( when the problem appeared, the plugin itself was crashing, so couldn't get any usable info from this trace)
• Opened the problematic form with Forms Builder and analyzed the part of the code where the form hanging.. no obvious issues were there..
• Disabled the APPS Tier SSL and retested the issue, just in case..(didn't solve)
• Tried with different java plugins and browsers according to the certified Java-Browser matrix.. (didn't solve)
• Recompiled/regenerated everything using adadmin... forms, flexfields , jar files and everything.. (didn't solve)
• Checked the db session and the alert log (no weirdness, no errors)
• Recompiled the Apps schema. (actually there was no important invalid objects there..)
• Increased the heap sizes of the java plugin in the client side.. (didn't solve)
• Requested functional admins to check the functional folders that are defined on EBS. (no problems were there)
• Wanted the Apps DBAs to apply Patch 14009893 to Forms Oracle Home. They applied it.. But they could not open any forms after applying it.. (maybe there was a failure during this patch application process, I didn't check it actually...)

So after these diagnostics and tries, I decided to gather the change log of this EBS environment to understand what was done/changed on this environment recently..

The most important change, that I have pointed out was the EBS's database upgrade.. That is, the database of this EBS environment was recently upgraded to 12cR1 and unfortuneatly, there was no user acceptence test report for that problematic form...

With this in mind, I analyzed the upgrade process..

The upgrade process was based on the document named "Interoperability Notes Oracle EBS 11i with Database 12cR1 (12.1.0.1) (Doc ID 1968807.1)".

This interop document was redirecting to the "Upgrading Developer 6i with Oracle Applications 11i (Doc ID 125767.1)"  by saying : "If your patch set level is earlier than patch set 19, apply the latest certified patch set. See Upgrading Developer 6i with Oracle Applications 11i on My Oracle Support".

So when I checked the document named "Upgrading Developer 6i.." ; I saw that it was pointing to the patch 22709024 , which was a merged patch, developed for fixing several forms server bugs.

Patch 22709024: MERGE REQUEST ON TOP OF 6.0.8.28.0 FOR BUGS 21671403 22351071

Well.. This patch was not applied to the environment. So we applied it and that was it ! It fixed the issue! 

Actually, this patch was the "new version" of the patch that I requested the Apps DBA team to apply in the first place.

Patch change log:

Replaced MLR Patch 21671403 with the latest MLR Patch 22709024 in the Download Additional Developer 6i Patches table.
Sep 23, 2015
Replaced MLR Patch 19444825 with the latest MLR Patch 21671403 in the Download Additional Developer 6i Patches table .
Aug 11, 2015
Replaced MLR Patch 16699473 with the latest MLR Patch 19444825.
May 31, 2013
Replaced MLR Patch 16414360 with the latest MLR Patch 16699473.
Mar 18, 2013
Replaced MLR Patch 14615390 with the latest MLR Patch 16414360.
Oct 19, 2012
Replaced MLR Patch 14009893 with the latest MLR Patch 14615390.
Jun 12, 2012
Replaced MLR Patch 13384700 with MLR Patch 14009893.
Jan 05, 2012
Replaced Windows MLR Patch 9436629 with Patch 13384700.
Added one off Patch 13384700 for HP Tru64.

At the end of the day; the lessons learned were the following : 

• If you are a dealing with an issue in a new environment (new to you), request the change log.
• Always make your detailed diagnostics. Gather technical info about the problem by doing detailed diagnostics works.
• Search knowledge base for the error. Consider implementing your findings (first in TEST, then in PROD)
• If you can't find anything and if you can't see anything during your diagnostics works, search for patches for the related technology.. (forms in this case)
• Based on the change log of the environment, check if there are any missing patches. 
• Open an Oracle Support SR :)
• If you find a patch, which seems related, check its newer versions.. Check the compliance between your EBS version and the patch before applying it.
• Always document the things you checked and the things you tried. By doing so, you can narrow down the list of things that can be done to solve the issue.
• If it is a developer related problem , contact your development team to get help, to make them analyze the code.
• Lastly, don't trust the things that are said to you. Always analyze the problem with your own eyes :) for example: if the env is recently upgraded, check the things that need to be done during the whole upgrade path. Don't accept the statements like "we upgraded this env properly, applying all the patches" :) Do you own checks and then decide ..

sds

Oracle VM Server-- Licensing & Cpu pinning

This is an important info for you, especially if you want to decrease your license costs using Oracle Vm Server's Capacity on Demand feature.

We see the Oracle VM Server utilized for the database environments, as well as for the application server environments and we see the Cpu configurations for the guest machines are properly aligned with the licenses on hand.

However; there is one more thing that should be configured for the capacity-on-demand feature to work. I mean it is required in order to make Oracle accept the license fee which is decreased by using the capacity-on-demand feature.

That is the Cpu pinning.

What CPU pinning does is basically, assigning physical CPUs to a VM.

This is also called hard partitioning, as it is used for binding a virtual machine CPU to a physical CPU or core, and preventing it from running on other physical cores than the ones specified. This is done for Oracle CPU licensing purposes, since Oracle VM is licensed on a per-CPU basis.

So, unless you do this, you may have to pay the licenses for all the cores that you have on your physical server.

Implementing this kind of a configuration is very easy though..

You can check the Oracle VM Server documentation for the detailed steps.

Exadata -- Elastic config, adding one compute and one cell node to Exadata X4-2

This post will be about Exadata, but it is different than my other Exadata blog posts. Because, in this post, I will fully concantrate on the hardware related part of the work.

We recently added 1 compute node and 1 cell node to an Exadata X4-2 Quarter Rack.. So, after these actions, the machine became Elastically configured, as it was not an X4-2 Quarter Rack anymore.

If you are asking about the support thing, my answer is yes! it is supported for X4-2 as well.
If you are asking about the whole process , here is the link : Extending Oracle Exadata Database Machine

The versions of the compute node and the cell node that we added, were X7-2.

Note that, the newly added nodes had 18.1.0.0.0 image versions installed on them.  So we planned to upgrade the image versions of the existing nodes to 18.1.0.0.0, as it is recommended to use the same  image version for all the nodes of an Exadata machine.

We planned this upgrade, but the first thing that we needed to, was to install this nodes physically into the Exadata machine and today's blog post is specially about that.

Okay.. Let's take a look at what we have done for physically installing the nodes and building an Elastic Configuration.

We first took the new servers/nodes out of their boxes .
In order to attach the new nodes to the Exadata Rack, we installed server rack rails and server slide rails that come with these new nodes.
After installing the rails, we installed the cable arms/cable organizers into the new nodes.
After installing the rails and cable organizers, the new nodes became ready , so that we installed them into the Exadata Rack easily.

After physically installing these new servers, we first put the power cables.
Note that, we put 2 power cables into each node (high available) and we connected each of these cables into different PDUs (Power Distribution Unit) , PDU-1 and PDU-2.

We didn't installed the infiniband cables and left this work to be done in the image upgrade part of the work. On the other hand, we installed 2 SFP cards into the compute nodes. ( to be used for backups)

After these hardware related installation actions were taken, we connected to the new nodes using serial port. Using the serial port connection we configured ILOM net interfaces of these new nodes by executing the following commands;

set /SP/network pendingipdiscovery=static

set /SP/network pendingipaddress=<some_ip_address>

set /SP/network pendingipgateway=<gateway_ip_address>

set /SP/network pendingipnetmask=<netmask>

set /SP hostname=<somename-ilom>

set /SP/clients/ntp/server/1 address=<ntp_server_ip_address>

set /SP/clients/dns nameserver=<dns_server_1_ip>,<dns_server2_ip>

set /SP/network/ commitpending=true 

Next, we connected related network cables to the ILOM ports of these newly installed nodes and lastly, we powered these new nodes on and checked their connectivity.. (ILOM)