EBS 12.2 -- ADOP /Online Patching Common Errors --script timed out,resource unavailable,outofMemory and more..

Logic behind the Adop utility and Online Patching is complex, even if we cant feel it from the surface..

A lot of actions are taken by the patching utility and there seem to be a lot of dependencies.
Because of these complexity; sometimes applying a little patch becomes a throuble.. 

I have seen this in adop's prepare or apply phases mostly.. 

I have seen a lot of resource related errors .. Especially in Virtual environments..
Improperly added Custom Tops may also trigger errors during the patching cycle.

For more details about adop; check the following document for the infomation about adop..

http://ermanarslan.blogspot.com.tr/2014/03/ebs-122-applications-dba-online.html

These kind of little problems can make you spend more effort while performing patching operations in EBS 12.2.
Anyways; I m bringing these kind of error together.. 

These errors will make this article to be a good thing to check before creating a patch cycle.

"Like Bruce Lee said: The less effort, the faster and more powerful you will be." :)

Error : Script timed out.. Script Executed in 7200113 milliseconds, returning status -1

This error may be reported during a fs_clone or prepare.. During the pasteConfig.sh run.

Note that: The script may take some time depending on the server resources (CPU and RAM).

May be encountered in Low Cpu and Low Memory environments..

Running /u01/oracle/TEST/fs2/FMW_Home/oracle_common/bin/pasteConfig.sh -javaHome /u01/oracle/TEST/fs1/EBSapps/comn/adopclone_erptest/FMW/t2pjdk -al /u01/oracle/TEST/fs1/EBSapps/comn/adopclone_erptest/FMW/WLS/EBSdomain.jar -tdl /u01/oracle/TEST/fs2/FMW_Home/user_projects/domains/EBS_domain_TEST -tmw /u01/oracle/TEST/fs2/FMW_Home -mpl /u01/oracle/TEST/fs1/EBSapps/comn/adopclone_erptest/FMW/WLS/plan/moveplan.xml -ldl /u01/oracle/TEST/fs1/inst/apps/TEST_erptest/admin/log/clone/wlsT2PApply -silent true -domainAdminPassword /u01/oracle/TEST/fs1/EBSapps/comn/adopclone_erptest/FMW/tempinfo.txt
Script Executed in 7200113 milliseconds, returning status -1
Script timed out.

Solution: Increasing the timeout will fix this.

1)Open a new terminal.
2) Execute :
export TIMEDPROCESS_TIMEOUT=-1
3. Execute :
adop phase=fs_clone or adop_phase=prepare  -- according to your requirements..

Error : ERRORMSG: Adsplice action did not go through successfully. 

Inside evalADPATCHStatus()...
message_status: ERROR
Adsplice action did not go through successfully.
*******FATAL ERROR*******
PROGRAM : (/oracle/product/prp_apps/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME : Fri Sep 19 16:37:58 2014
FUNCTION: main::execADSPLICE [ Level 1 ]
ERRORMSG: Adsplice action did not go through successfully.

SQL Command: SELECT status||',' FROM ad_adop_session_patches WHERE node_name = 'erptest' AND applied_file_system_base = '/u01/oracle/TEST/fs1' AND patch_file_system_base IS NULL AND bug_number = 'ADSPLICE_YOURCUSTOMTOP' ORDER BY TO_CHAR(end_date,'YYYY.MM.DD:HH24:MI:SS') DESC
patch_status = Y
Updated patch_status = Y
EXIT STATUS: 1

Solution: This is because of an erroneous Custom top adding operation .. There is an improper Custom top definition in the System .

1)Remove the following entries from the adxmlctx.tmp file:
<oa_customized>
<c_fss oa_var="c_fss" scope="CUSTOM" oa_type="PROD_TOP" oa_enabled="FALSE" default="/u01/sim2/fs1/EBSapps/appl">%c_fss%</c_fss>
</oa_customized>
</oa_context>
2. Run autoconfig .
3. Re-run fs_clone.

If this does not fix the error, add your custom again/the same custom top, but this time properly.

For adding the custom top: http://ermanarslan.blogspot.com.tr/2014/05/ebs-122-add-custom-top.html

Error:  Exception in thread "Thread-1" java.lang.OutOfMemoryError: Java heap space

Exception in thread "Thread-1" java.lang.OutOfMemoryError: Java heap spaceat java.util.Arrays.copyOfRange(Arrays.java:2694)

at java.lang.String.(String.java:203)
at java.io.BufferedReader.readLine(BufferedReader.java:349)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at com.oracle.cie.common.util.CRLF.readData(CRLF.java:129)
at com.oracle.cie.common.util.CRLF.processFile(CRLF.java:67)
at com.oracle.cie.common.util.CRLF.process(CRLF.java:57)
at com.oracle.cie.domain.util.stringsub.SubsScriptHelper.processCRLF(SubsScriptHelper.java:167)
at com.oracle.cie.domain.DomainGenerator.generate(DomainGenerator.java:478)
at com.oracle.cie.domain.script.ScriptExecutor$2.run(ScriptExecutor.java:2992)

Solution: The Xms and Xmx parameters of java are not enough for the adop..

1. Run the following command before executing adop .. In the same terminal... export CONFIG_JVM_ARGS="-Xms1024m -Xmx2048m"
2. Run adop again.

If this does not fix the problem; increase the parameters a little more.

1. Run the following command before executing adop .. In the same terminal...
export CONFIG_JVM_ARGS="-Xms1500m -Xmx3000m"
2. Run adop again.

Error:  AutoPatch error: Missing file format id in file {YOUR _APPL_TOP LOCATION}/admin/applcust.txt

This version of AutoPatch requires a file format id on the first line of the file.
Error reading customized files file
Freeing includes hash table
Freeing fixes hash table
Freeing basedons hash table
Freeing entities hash table

Solution: Rename the applcust.txt file : mv applcust.txt applcust_BAK.txt , and retry the patching operation.
If renaming the file wont fix the error; check out following Support note: Doc ID 1227113.1

Error: [ERROR][thread ] Could not start thread ExecuteThread: '4' for queue: 'default'. Resource temporarily unavailable

Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
Connecting to t3://yourhost:7002 with userid weblogic ...
[ERROR][thread ] Could not start thread weblogic.timers.TimerThread. Resource temporarily unavailable
This Exception occurred at Wed Nov 05 21:54:26 EET 2014.
java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1096).
Attempting to allocate 4G bytes
There is insufficient native memory for the Java
Runtime Environment to continue.
Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit

Solution: Restart your system to be a clean enviroment.. We need free memory here , or clear your env to have more free memory. Also you may decrease your Xmx using CONFIG_JVM_ARGS="-Xms1024m -Xmx2048m" before starting adop.

Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Disable compressed references (-XXcompressedRefs=false)