EBS R12/Linux -- frmweb , segfault error 4 in libsosdw.so.0 --investigation

In a EBS R12 environment, segfaults for frmweb may appear in sys log (/var/log/messages) .
I have faced with this errors in a EBS 12.1.3 environment, which was running on Oracle Linux 6.5 64 bit.
Anyways, the reason that makes me write this blog post, is not the segfault itself. I m writing this blog post to show you the method to investigate these types of errors further..
Okay.. The error itself does not give us detailed information.  We need to have at least a call stack of frmweb while it was encountering the segfault.

• Here ; the error is reported in /var/log/messages;

kernel: frmweb[9727]: segfault at 6974655e ip 00000000f758f2c1 sp 00000000ffdd9b58 error 4 in libsosdw.so.0[f758b000+b000]

• But the coredump can not be produced because of the configuration of abortd;

Jan 5 17:38:02 erpr12app abrt[20281]: Saved core dump of pid 29744 (/u1/oracle/R12PROD/apps/tech_st/10.1.2/bin/frmweb) to /var/spool/abrt/ccpp-2015-01-05-17:38:01-29744 (58826752 bytes)
Jan 5 17:38:02 erpr12app abrtd: Directory 'ccpp-2015-01-05-17:38:01-29744' creation detected
Jan 5 17:38:02 erpr12app abrtd: Executable '/u1/oracle/R12PROD/apps/tech_st/10.1.2/bin/frmweb' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Jan 5 17:38:02 erpr12app abrtd: 'post-create' on '/var/spool/abrt/ccpp-2015-01-05-17:38:01-29744' exited with 1

• So we configure the abort daemon to create the coredump for the unpackaged applications like frmweb.

To accomplish that; we modify the file /etc/abrt/abrt-action-save-package-data.conf file as follows;
ProcessUnpackaged = yes

Then, we restart the abrtd ..-> service abrtd restart 

• After the restart of abrtd, we see that coredumps are generated for frmweb segfaults.

Jan  5 22:30:54 erpr12app kernel: frmweb[22841]: segfault at 6974655e ip 00000000f75d42c1 sp 00000000fff34e98 error 4 in libsosdw.so.0[f75d0000+b000]

Jan  5 22:30:55 erpr12app abrt[24120]: Saved core dump of pid 22841 (/u1/oracle/R12PROD/apps/tech_st/10.1.2/bin/frmweb) to /var/spool/abrt/ccpp-2015-01-05-22:30:54-22841 (47157248 bytes)

Jan  5 22:30:55 erpr12app abrtd: Directory 'ccpp-2015-01-05-22:30:54-22841' creation detected

• We use gdb(GNU debugger) to reach the callstack stored in the coredump;

cd  /var/spool/abrt/ccpp-2015-01-05-22:30:54-22841
gdb /u1/oracle/R12PROD/apps/tech_st/10.1.2/bin/frmweb coredump
(gdb) bt

-->Program terminated with signal 11, Segmentation fault.
#0 0xf75b12c1 in siehjmpterm () from /u1/oracle/R12PROD/apps/tech_st/10.1.2/lib/libsosdw.so.0
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.i686 libXp-1.0.0-15.1.el6.i686 libXt-1.1.3-1.el6.i686 libgcc-4.4.7-4.el6.i686 libuuid-2.17.2-12.14.el6.i686 multi_lib32-4.8.2-1_Oso.x86_64 openmotif21-2.1.30-11.EL6.i686
(gdb) bt
#0 0xf75b12c1 in siehjmpterm () from /u1/oracle/R12PROD/apps/tech_st/10.1.2/lib/libsosdw.so.0
#1 <signal handler called>
#2 0xf77d742e in __kernel_vsyscall ()
#3 0xf6a825d3 in __read_nocancel () from /lib/libpthread.so.0
#4 0xf758fe00 in ixhgr_GenRead () from /u1/oracle/R12PROD/apps/tech_st/10.1.2/lib/libixw.so.0
#5 0xf758f059 in ixhrdh_ReadHeader () from /u1/oracle/R12PROD/apps/tech_st/10.1.2/lib/libixw.so.0
#6 0xf758fa5e in ixhhsm_HTTPStateMachine () from /u1/oracle/R12PROD/apps/tech_st/10.1.2/lib/libixw.so.0
#7 0xf758f51e in ixhhrd_HTTPRead () from /u1/oracle/R12PROD/apps/tech_st/10.1.2/lib/libixw.so.0
#8 0xf7595a76 in ixncrd_Read () from /u1/oracle/R12PROD/apps/tech_st/10.1.2/lib/libixw.so.0
#9 0xf759614b in ixncru1_ReadUB1 () from /u1/oracle/R12PROD/apps/tech_st/10.1.2/lib/libixw.so.0
#10 0xf7594054 in ixncrme_ReadMsg () from /u1/oracle/R12PROD/apps/tech_st/10.1.2/lib/libixw.so.0
#11 0xf7595004 in ixncrq_ReadQueue () from /u1/oracle/R12PROD/apps/tech_st/10.1.2/lib/libixw.so.0
#12 0xf7595328 in ixncpl_PutList () from /u1/oracle/R12PROD/apps/tech_st/10.1.2/lib/libixw.so.0
#13 0xf76bc53c in iifwcpl_PutList () from /u1/oracle/R12PROD/apps/tech_st/10.1.2/lib/libiifw.so.0
#14 0xf4b9dfa6 in afm_msg_send () from /u1/oracle/R12PROD/apps/apps_st/appl/fnd/12.0.0/bin/fndfmxit.so

• After obtaining the call stack, we check the Oracle Support .. We check the bugs, and Knowledge base to find a document or bug record that matches the call stack.
• In this particular case; the solution was applying patch: 8940272 , as the general error definition and call stack declared in the related document and bug record was the same as ours.