EBS 12.2 -- The Late Charges Report (XDODTEXE) java.lang.OutOfMemoryError in post processing // allocLargeObjectOrArray

Couple of days ago, a concurrent program that suppose to produce an XML output encountered an java.lang.OutOfMemoryError.
I have found it useful to write about this incident  as it was an unsual OutofMemoryError, because it was actually failing in "allocLargeObjectOrArray".

When I checked the OPP log, I found the error stack as follows;

Caused by: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [Ljava/lang/Object;, size 16400
at oracle.xdo.parser.v2.XMLDocument.xdkIncCurrentId(XMLDocument.java:3020)
at oracle.xdo.parser.v2.XMLNode.xdkInit(XMLNode.java:2758)
at oracle.xdo.parser.v2.XMLNode.<init>(XMLNode.java:423)
at oracle.xdo.parser.v2.CharData.<init>(CharData.java:118)
at oracle.xdo.parser.v2.XMLText.<init>(XMLText.java:168)
at oracle.xdo.parser.v2.XMLDocument.createNodeFromType(XMLDocument.java:2869)
at oracle.xdo.parser.v2.XMLDocument.createNodeFromType(XMLDocument.java:3000)
at oracle.xdo.parser.v2.XMLDocument.createTextNode(XMLDocument.java:735)
at oracle.xdo.parser.v2.DocumentBuilder.characters(DocumentBuilder.java:553)
at oracle.xdo.parser.v2.NonValidatingParser.reportCharacters(NonValidatingParser.java:1680)
at oracle.xdo.parser.v2.NonValidatingParser.parseText(NonValidatingParser.java:1391)
at oracle.xdo.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1268)
at oracle.xdo.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:338)
at oracle.xdo.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:285)
at oracle.xdo.parser.v2.XMLParser.parse(XMLParser.java:289)


So the failing code was trying to allocate an object or array with a size of 16400, but it was failing because of some JVM limits.

Okay. this was not related with xmx, xms (heap related params) or perm size, but It was related with something else.
Anyways; I will keep it short.

XXtlasize and XXlargeobjectlimit was the cause.. Increasing xmx was also a good idea.

So, I updated the fnd_cp_services for opp as follows;

J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx3096m -XXtlasize1024k -XXlargeobjectlimit1024k

Also update the problematic concurrent program's arguments as;

-Xms2048M -Xmx3096M

Note that : different jvm versions uses different argument names for the same thing. You will see that some arguments become obsoleted when you use a higher level jdk..

Note that: The JVM used in EBS 12.2 was 32 bits. Even if the EBS is 64 bit, it uses 32 bit jvm for these processes.  The Java version used by OPP in this environment is 1.6.. So you may encounter errors if using the above params in 1.7. Note that -> these XX parameters are for Jrockit only.

/u01/oracle/TEST/fs2/FMW_Home/jrockit32/bin/java -version
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.2.0-79-146777-1.6.0_29-20111005-1807-linux-ia32, compiled mode)

Note that : In 12.2 by default;

Application Tier Location	JDK Version
<AS 10.1.2 Oracle Home>/jdk/bin	1.7.0_25 (32-bit)
<FMW_HOME>/webtier/jdk/jre	1.6.0_29 (64-bit)
<COMMON_TOP>/util/jdk32	1.7.0_25 (32-bit)
<COMMON_TOP>/util/jdk64	1.7.0_25 (64-bit)

Database Tier Location	JDK Version
<11gR2 Oracle Home>/jdk	1.5.0_30 (64-bit)
<11gR2 Oracle Home>/appsutil/jre	1.7.0_25 (64-bit)

Okay.. After updating the command line java arguments for both OPP and Concurrent Program , I restarted the apps tier, and rerun the Concurrent Program..

This time, It did not failed because of the outofmemory error. On the other hand, it failed while waiting the OPP to do its work.. Actually, the concurrent program encountered a timeout while waiting for OPP response..

Checked the OPP process , and saw that it was running with my params;

/u01/oracle/TEST/fs2/FMW_Home/jrockit32/bin/java -DCLIENT_PROCESSID=10685 -Dhttp.proxyHost= -Dhttp.proxyPort= -Dhttp.nonProxyHosts= -Djava.awt.headless=true -server -Xmx384m -Doracle.apps.fnd.common.Pool.leak.mode=stderr:off -verbose:gc -mx3096m -XXtlasize1024k -XXlargeobjectlimit1024k -Ddbcfile=/u01/oracle/TEST/fs2/inst/apps/TEST_erptest/appl/fnd/12.0.0/secure/TEST.dbc -Dcpid=17165 -Dconc_queue_id=6269 -Dqueue_appl_id=0 -Dlogfile=/u01/oracle/TEST/fs_ne/inst/TEST_erptest/logs/appl/conc/log/FNDOPP17165.txt -DLONG_RUNNING_JVM=true -DOVERRIDE_DBC=true -DFND_JDBC_BUFFER_MIN=1 -DFND_JDBC_BUFFER_MAX=2 oracle.apps.fnd.cp.gsf.GSMServiceController

Note1 : there were 2 mx defined in the command line , but mx3096 was overwritting the first one , so there were no problem. 

Note2 : If you dont want to see the first mx ; 

Edit the contextfile; the following line and make an autoconfig..  But as I said -> no need
<AFJSMARG oa_var="s_afjsmarg" osd="LINUX_X86-64">-server -Xmx384m -Doracle.apps.fnd.common.Pool.leak.mode=stderr:off -verbose:gc</AFJSMARG>

Then I realized that ; the output size of the program was 511 MB!

This was a huge output for OPP to process.

For the solution , I updated the timeout profile and the arguments of OPP concurrent manager(from the Forms screens) as follows ;

1)Increased the related timeouts(Concurrent:OPP Process Timeout and Concurrent:OPP Response Timeout -> 20000 almost 5 hours)

2)Increased the max thread count of OPP (oracle.apps.fnd.cp.opp.OPPServiceThread:2:0:max_threads=10)

I wanted to put OPP into maximum performance mode :)

Anyways; after these modifications, I rerun the concurrent program.

Note that : the problematic program was a standart concurrent program..(The Late Charges Report (XDODTEXE))

So, after making these changes, program did not encounter any errors.. On the other hand, it is still continuing..

This was normal, as It is a huge output for OPP to process and the server is a VM..

Later , it turned out that ; this program must have been executed by giving some arguments.. So , if an input argument would have been given to this program, it would complete in 3 seconds! :) Tested and verified that..

In conclusion,  I have spent 1 day in total for this problem, but it was a pleasure to work with OPP in this kind of advanced level.