EBS 12.2 -- adop phase=prepare pasteConfig.sh hangs java.lang.OutOfMemoryError

While applying a little 5kb patch using online patching (which is a must bytheway), you may encounter a hand situation.. Spending hours to find out the causes and correcting them during a Project implementation, is not something we want to deal.. Also inability to apply a 5kb patch even in 1 hour is does not seem good from the Business perspective..

The problem that I will mention in the below paragraps is a good example for the unwanted situations we as Apps Dba may encounter in EBS 12.2.3..

Lets review the online patching in EBS 12.2.. So , to apply a patch, we start a patching cycle using adop phase=prepare and after that we continue with phase=apply, phase=finalize, phase=cutover and phase=cleanup.

This problematic situation arises in the "prepare" phase, as we cant complete the prepare phase in hours..
There is definetely a hang situation , but we have a lot of place to collect the data for our diagnosis..
We have some clues though.. For example : adop log files..
Anyways, when you see a line starting with pasteConfig.sh in the tail of the related log file, you are there where I was 1 day ago :)

Lets clarify the situation..

In pasteConfig , adop tries to pack and unpack Weblogic domain to supply the Weblogic consistency accross the patches. It deals with files and it runs java programs against those files..

These java programs often encounter OutOfMemoryError ... sometimes it is about java heaps, sometimes you ll not able to see the problematic memory area, but here is what you need to do if you see OutOfMemoryError in the related log file while running PasteConfig.sh during adop prepare session.

Note that: we make the controls using the logfile located in admin/clone directory .. 

For example:  /u01/oracle/TEST/fs2/inst/apps/TEST_erptest/admin/log/clone/wlsT2PApply log

Following lines from the related log file describes the real problem that makes adop to hang ;

Configuring the Server -  AdminServer
Configuring the Server -  oacore_server1
Configuring the Server -  forms_server1
Configuring the Server -  oafm_server1
Configuring the Server -  forms-c4ws_server1
Configuring the Machine -  erptest
Configuring the Cluster -  forms_cluster1
Configuring the Cluster -  oacore_cluster1
Configuring the Cluster -  oafm_cluster1
Configuring the Cluster -  forms-c4ws_cluster1
Configuring the Datasource -  EBSDataSource

Exiting WebLogic Scripting Tool.
INFO : Dec 13, 2014 7:25:31 PM - CLONE-23283  Unpack Domain.
<< read template from "/tmp/CLONINGCLIENT8433880607085395265/packed_template.jar"
>>  succeed: read template from "/tmp/CLONINGCLIENT8433880607085395265/packed_template.jar"
<< set config option ServerStartMode to "prod"
>>  succeed: set config option ServerStartMode to "prod"
<< write Domain to "/u01/oracle/TEST/fs1/FMW_Home/user_projects/domains/EBS_domain_TEST"
Exception in thread "Thread-1" java.lang.OutOfMemoryError

So , as you see there is an java.lang.OutOfMemoryError there .. It is encountering the unpacking phase which confirms the thing that I mentioned in the 3th step of the below action plan.

ACTION PLAN:

1)export CONFIG_JVM_ARGS="-Xms1024m -Xmx2048m"(OR MAYBE MORE) in your terminal and restart the adop phase=prepare

2)Edit the related scripts in COMMON_TOP/adopclone_hostname/bin, and change the values from 600M to 2000M in the related files. (you can use grep -R 600M * in the above directory to find the files that contains 600M xmx limit )

3) Delete the old weblogic logiles; and use /dev/null > to the current weblogic logfiles .. Because; your log size may be increased recently.. pasteConfig creates a CLONINGCLIENT***/packed_template.jar firstly , and it unpacks it .. When this file becomes bigger, the risk to get outOfmemoryError increases.

You will see that when you delete the logs, this file will be smaller and you wont get outOfmemory error.. So dont save up big weblogic files in any of the Managed Server directories. Backup them then delete them if you need to store them..

4)export TIMEDPROCESS_TIMEOUT=-1 ->to disable the timeout. Just in case ,as we dont want a timeout.

5)restart the adop phase=prepare 

These action will do the job( they are required actions+ some extras).. If the issue persists; feel free to comment.

Also see :

EBS 12.2 -- ADOP /Online Patching Common Errors --script timed out,resource unavailable,outofMemory and more..