EBS 12.2 -- adstrtal.sh hangs -- can not start managed servers oacore_server1, forms_server1, oafm_server1 etc..

We use adstrtal.sh to start all of the EBS application tier components .. This script is used in eBS 11i, 12.0, 12.1 and now in EBS 12.2..

Last week, I have encountered an interesting problem in starting EBS 12.2.3 Application services ..
The problem was in adstrtal.sh..
It happened when I realized that adstrtal.sh script could not start the application services entirely..
It was waiting after starting the Weblogic Admin Server..

Normally, it should start the Weblogic Managed Servers like oacore , forms and etc, but it could not.. I was waiting ..
When I analyzed the situation and check the log files for diagnosis; I could not find any clue.. I checked admin server, node manager logs , managed server logs and etc, but they were all clean..
Checked the related java processes and adstrtal itself, they were all clean , too..

Executing service control script:
/u01/apps/fs1/inst/apps/VIS_erpdemo/admin/scripts/adadminsrvctl.sh start -nopromptmsg
Timeout specified in context file: -1 second(s)
script returned:
****************************************************
You are running adadminsrvctl.sh version 120.10.12020000.7
Starting WLS Admin Server...
Refer /u01/apps/fs1/inst/apps/VIS_erpdemo/logs/appl/admin/log/adadminsrvctl.txt for details

adadminsrvctl.sh: exiting with status 0
adadminsrvctl.sh: check the logfile /u01/apps/fs1/inst/apps/VIS_erpdemo/logs/appl/admin/log/adadminsrvctl.txt for more information ...
.end std out.
.end err out.

****************************************************

---  It was waiting in here..

When this was happening , the adstrtal.log was displaying the followings;

Checking the status of the AdminServer...

.end std out.

.end err out.

Checking/updating the JVM argument values for the managed servers ...

Validated the passed arguments for the option ebs-set-allmanagedsrvargs

Deploying application: ebs-forms on forms_cluster1

Location = /u01/apps/fs1/EBSapps/comn/shared-libs/ebs-forms

Connecting to t3://erpdemo.takasdom.takasbank.com.tr:7001 with userid weblogic ...

Successfully connected to Admin Server 'AdminServer' that belongs to domain 'EBS_domain_PRELIVE'.

Warning: An insecure protocol was used to connect to the

server. To ensure on-the-wire security, the SSL port or

Admin port should be used instead.

-->>> So no clue again... Maybe we should wait for some time .. Maybe more than 10 mins:)

Anyways, when I checking the things in Weblogic admin console , I saw  I saw that a configuration lock was held... Some things were changed by an admin but the changes were not activated or rolled back.. Thus there was a domain lock.. 

For the solution, I have released the lock by activating the changes , and managed servers have started without any problems..

Then I checked the adstrtal.sh log again and saw the following;

Deploying application from /u01/apps/fs1/EBSapps/comn/shared-libs/ebs-forms to targets forms_cluster1 (upload=false) ...

[Deployer:149140]The task cannot be processed further until the current edit session is activated. When this occurs, task processing will continue. The user can exit the deployer tool without affecting the task..................................................................................................................................................................................................................................................................................Completed the deployment of Application with status completed

So , a latecomer clue  :) 

Anyways, releasing the configuration lock have fixed the problem..

If we dig deeper;

When we executed adstrtal.sh; it executes admanagedsrvctl.sh script to start our managed servers.

admanagedsrvctl.sh uses adProvisionEBS.pl with ebs-nmstart-managedsrv argument to start our managed servers. 

  { echo $wlspass; } | perl /u01/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl -contextfile=/u01/apps/fs1/inst/apps/VIS_erpdemo/appl/admin/VIS_erpdemo.xml ebs-nmstart-managedsrv -wlsadminuser=weblogic -wlsadminport=7001 -managedsrvname=$SERVER_NAME -domain=EBS_domain_PRELIVE -domainhome="/u01/apps/fs1/FMW_Home/user_projects/domains/EBS_domain_PRELIVE" -nmhome="/u01/apps/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1" -nmport=5556 -nmtype=Plain -machinelistenaddress="erpdemo.takasdom.takasbank.com.tr" -promptmsg=hide -logfile=$LOG

adProvisionEBS.pl is a wrapper that is used to call the Provisioning API oracle.apps.ad.tools.configuration.EBSProvisioner to provision the E-Business Suite.

It uses ebs_deploy_app to make the deployments in every startup.  I didnt have enough time to analyze deeper , but at least I can say that It makes the deployments for forms_cluster in every startup.

"Initiating deploy operation for application, ebs-forms to targets forms_cluster1"

So when it comes to this deployment, it acquires a lock, and waits if that lock is held..

Something like the following runs there;

startEdit() --> it waits here..

if startswith forms_

Deploy!

Lets look at the defition of startEdit wlst command;

startEdit([waitTimeInMillis], [timeoutInMillis], [exclusive])

Argument	Definition
waitTimeInMillis	Optional. Time (in milliseconds) that WLST waits until it gets a lock, in the event that another user has a lock. This argument defaults to 0 ms.
timeoutInMillis	Optional. Timeout (in milliseconds) that WLST waits to release the edit lock. This argument defaults to -1 ms, indicating that this edit session never expires.
exclusive	Optional. Specifies whether the edit session should be an exclusive session. If set to true, if the same owner enters the startEdit command, WLST waits until the current edit session lock is released before starting the new edit session. The exclusive lock times out according to the time specified in timeoutInMillis. This argument defaults to false.

You see ; timeoutInMillis -> This argument defaults to -1 ms, indicating that this edit session never expires. :)

Okay, we have satisfied our curiosity ..

Lastly, I suggest you to take a look at the following presentation, it explains Fusion Middleware in 

E-Business Suite 12.2 very well..

http://www.aioug.org/sangam14/images/Sangam14/Presentations/201587_Sai_Subramanian_ppt.pdf

Linux -- SQLPLUS backspace (^H) problem & fix - vt100,vt220

When you working with sqlplus in Linux , you will see that you cant use backspace to erase characters..
"sqlplus" does not handle the backspace character as it should , and displays ^H characters instead..
So, If you want to delete characters in sqlplus , you need to use shift+backspace ...

[appvis@erpdemo scripts]$ sqlplus apps/apps
SQL*Plus: Release 10.1.0.5.0 - Production on Fri Jan 2 11:29:23 2015
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> select * from v$instance;^H

To able to use only the backspace button to erase characters in sqlplus,

you basically need to add; stty erase ^H in your .bash_profile

stty gives you to opportunity to change the terminal line settings..

So , when you issue stty erase ^H in your .bash_profile (this file sourced everytime you login or switch user), you make your shell to erase the last character when it sees ^H  typed.

Okay, setting erase char as explained above is a good workaround for you.

Lets dig a little deeper;

env|grep TERM

TERM=vt100

ORACLE_TERM=vt220

So our terminal uses vt100 right ...  vt100 terminal sends '^H' (0x08) when the backspace key is hit..

In vt220 , The X-term emulation is different..

Look what is said; X-term behaviour was that control-H could not be used by applications just like other control-shortcuts by GUI applications ..

VT220

The disadvantage of the X-term behaviour was that control-H could not be used by applications just like other control-shortcuts by GUI applications (e.g. control-F for find, or perhaps control-H for help).

For this reason, the behaviour was changed again, to the following mapping, which is now the default in most Linux distributions:

Keystroke	Glyph on keyboard	ASCII Character(s)	TTY representation	Expected application behaviour
Control-H		BS (8, 0x08)	^H	(Passed on to application)
Backspace	⌫	DEL (127, 0x7F)	^?	Erase to the left
Delete	⌦	ESC (27, 0x1B) + [3~ ("\e[3~")	^[[3~	Erase to the right

So VT220 handles ^H diffrently...

Lets find the files that has vt220 inside them;

cd $ORACLE_HOME

 grep -R vt220 *

Apache/Apache/php/lib/php/build/shtool:            xterm|xterm*|vt220|vt220*)

Binary file bin/frmbld matches

Binary file forms/admin/terminal/fmrcvt220.res matches

inventory/filemap/forms/admin/terminal/files.map:fmrcvt220.res::{"component","oracle.developer.forms.compiler","10.1.2.0.2"},

inventory/Components21/oracle.developer.forms.compiler/10.1.2.0.2/fastCopyLog.xml:    <FILE STAGE_LOCATION="fmrcvt220.res"> %ORACLE_HOME%/forms/admin/terminal/fmrcvt220.res </FILE>

Binary file lib/libsosdw.so matches

Binary file lib/libsosd.a matches

Binary file lib/libfrmjapi.so.0 matches

Binary file lib/libsosdw.a matches

Binary file lib/libsosd.so.0 matches

Binary file lib/libfrmjapi.so matches

Binary file lib/libsosd.so matches

Binary file lib/libsosdw.so.0 matches

Hmm.. We have some so files which have vt220 inside, but unfortnetaly none of them are used by the sqlplus.. In fact, when I use strace to see the syscalls that sqlplus made, I dont see a syscall or file read that might affect the terminal setting.
So in this case; we need to speak the langage what sqlplus understands..

Okay... Now comes the solution :)

The solution comes from the terminal software ( SSH Secure Shell client in this case) 

Here, I check "Backspace sends Delete" checkbox and this fixes the problem.

Why does "Backspace sends Delete" fix the problem?

When I check "Backspace send Delete",  my SSH Secure Shell client starts to send "^?" not "^H", that 's what fixes the problem..

Happy New Year!!

EBS 12.2 -- Create Accounting error ORA-14411

Concurrent Programs like Create accounting may encounter ORA-14411: The DDL cannot be run concurrently with other DDLs errors.
Like the following Journal Import;

General Ledger: Version : 12.0.0
Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved.
GLLEZL module: Journal Import
+---------------------------------------------------------------------------+
>> gluddl() 26-DEC-2014 20:45:58
-- gluddl ad.do_ddl error buffer begin
do_ddl(APPLSYS, SQLGL, 17, $statement$, GL_INTERFACE): private_do_ddl(APPS, APPLSYS, GL, 17, $statement$, GL_INTERFACE): do_at_tab_a_seq_acd_ind(GL, $statement$): : do_apps_ddl(GL, $statement$): : substr($statement$,1,255)='Alter table GL_INTERFACE add (code_combination_id_interim NUMBER(15))'
-- gluddl ad.do_ddl error buffer end
-- gluddl Message Dictionary Start:
do_ddl(APPLSYS, SQLGL, 17, $statement$, GL_INTERFACE): private_do_ddl(APPS, APPLSYS, GL, 17, $statement$, GL_INTERFACE): do_at_tab_a_seq_acd_ind(GL, $statement$): : do_apps_ddl(GL, $statement$): : substr($statement$,1,255)='Alter table GL_INTERFACE add (code_combination_id_interim NUMBER(15))'
APP-FND-01388: Cannot read value for profile option GL_GLLEZL_ARCHIVE_ROWS in routine &ROUTINE.
APP-FND-01388: Cannot read value for profile option GL_RECORDS_TO_PROCESS in routine &ROUTINE.
APP-FND-01388: Cannot read value for profile option GL_JI_ALWAYS_GROUP_BY_DATE in routine &ROUTINE.
APP-FND-01388: Cannot read value for profile option GL_DEBUG_MODE in routine &ROUTINE.
 
-- gluddl Message Dictionary End
<x gluddl() 26-DEC-2014 20:45:58
ORA-14411: The DDL cannot be run concurrently with other DDLs
ORA-0651
<x gllcnt() 26-DEC-2014 20:45:58

Error in: gllcnt
Function return status: 0
Function Err Message: gluddl failed
....

These concurrent programs do ddl operations using their routines like gluddl , do_ddl, do_apps_ddl and etc.. Normally, these ddl operations are database ddl operations, in other words , these operations are done in the database . Thus they are dependet to the rules of the underlying database.. (For EBS this database is Oracle ,as known)

So , the ORA-14411 error arises when multiple sessions try to do the ddl on the same table..

In such situation, you need to connect to the database, and look for the active sessions to find the session that does the ddl on relevant tables, thus preventing your concurrent programs to do their jobs..

Once you find out the problematic session, we identify its source and take the corrective actions..

I m saying "we need to identify its source" , because we don't usually do ddls on this kind of Standart tables!

What I m trying to say is; the session that does the ddl on the standart table and blocks the ddls against that standart table may belong to anohter Create Accounting  process which left behind or forgotten.. Maybe the problem is running the same process again and again concurrently..  So , If you Identify such a session, contact your Functional Team, there may be someting functionally broke.

Lastly, once the cause of issue is found, just kill the session or  cancel the concurrent request which the blocker session belongs.

EBS R12 -- 5-byte UTF8 encoding not supported.

Using EBS XML reports, you may encounter problems because of junk characters..
You may see  "5-byte UTF8 encoding not supported." in your OPP logfile..
When you encounter such a problem, you may see that your PDF or RTF output is not produced.. You may see only an XML output.. In other words; this problem will make OPP not be able to produce the desired output..
Also , the functionals or developers will directly send this problem to us (Apps Dbas).. They will redirect this problem to us, as they may think that it is related with configuration..
Here I m writing this post to state the real thing that causes theses type of  problems. As mentioned in my first sentence, the problem arises because of junk characters..
Junk characters such as �..
That �.. for example corresponds to 11111000 in ISO 8859-1 character set , and this creates the problem as when the bytes are converted to UFT8..

I mean the parser sees 11111000 and then thinks that it is a start of a 5 byte UTF8 sequence.

The solution can be replacing this character in Sql .. Using char() ..

That 's it.. I hope you will find this useful.

EBS R12/LINUX -- 32 bit Application tier with large memory -- OMM-KILLER problem

Another 32 bit problem, this time OMM-killer comes into play and kills processes without knowing the importance of them..  Sometimes , OMM-killer kills an application processes and make the Application Web Service to be down, and sometimes it kill a backup process and affect production system backups..
The reason behind the utilization of OMM-killer is using an 32 bit Linux OS and installing a big sized memory(like 64 GBs)..

 The problem arises in Oracle/Redhat Linux 5 and 6 32 bit, as those operating systems can not handle the memory stably when the installed memory is more than 16GB..

16 GB memory however , can be used with Oracle/Redhat Linux 32 bit Operating Systems without any problems..
The reason behind this incapability is not being able to use HugePages.. As you may know, starting with Oracle Linux 5 for example; hugemem is not option..

So even if you can use PAE to address a big memory, and even if you can address a big memory like 64 bit, you will end up with OMM-killer killing your processes...
I mean even our UEK kernel supports PAE and can address those big sized RAMs, we 'll still have problems in runtime..
Note: Checking PAE support:

Go to /boot ;

cat config-2.6.32-300.10.1.el5uek. (your kernel) | grep CONFIG_X84_PAE;

If a line like CONFIG_X86_PAE=y is returned, yhen you kernel supports PAE and it is active right now.

What I try to say is;
OMM-killer kill your processes because it see a shortage in LowMem area. Hugepage option in earlier Linux releases address this problem by making us able to use a 4-4 split rather tan 1-3 memory split.
You will have a better idea , when you read my related post about Linux 32 bit memory addressing ,->
http://ermanarslan.blogspot.com.tr/2013/11/linux-ebs-11i-uek-kernel-pae-support32.html

Okay, lets list the recommended solution for this particular environment;

1) Why not go for 64 bit? EBS Release 12 supports 64 bit in Apps Tier too!
2) Deinstall those RAMs, and use 16GB Ram in your PAE enabled Linux 32 bit envrionment
       - Also, increase the lowmem zone protection ..
       For Oracle Linux 5/Rhel 5 and higher:
add the following line into /etc/sysctl.conf

vm.lowmem_reserve_ratio = 256 256 250
Also execute following for current setting.
echo "256 256 250" > /proc/sys/vm/lowmem_reserve_ratio     
       For earlier releases :
echo "250" > /proc/sys/vm/lower_zone_protection
3)You may try with OMM disabled: (not recommended) --just in caseAdd a line to /etc/sysctl.conf and reboot 
vm.oom-kill = 0

EBS -- Analyzing Concurrent Processing --CP analyzer

CP analyzer is a great tool to analyze Concurrent Processing environment in EBS instances.. It gives you a detailed and clear picture in HTML format.. Thus, you gather almost all the information you need to investigate the distinct problem causes  in Concurrent Processing environment, without struggling with OS commands or complicated sql statements..

I strongly recommend to use CP analyzer in making Health Checks for EBS environments.

CP analyzer is available via the following Oracle Doc:
Concurrent Processing - CP Analyzer for E-Business Suite (Doc ID 1411723.1)

CP analyzer can be used in any EBS envrionment with version 11.5.10.2 to 12.2.3 [Release 11.5 to 12.2]

It does te following analysis and provide an HTML for you to view those analysis;

• Immediate Analysis and Output of Concurrent Environment
  
  • E-Business Applications Concurrent Processing Analyzer Overview
    • Total Purge Eligible Records in FND_CONCURRENT_REQUESTS
    • E-Business Suite Version
    • Concurrent Processing Database Parameter Settings
    • Applied ATG Patches
    • Known 1-Off Patches on top of Rollups
  • E-Business Applications Concurrent Request Analysis
  • Long Running Reports During Business Hours
  • Elapsed Time History of Concurrent Requests
  • Requests Currently Running on a System
  • FND_CONCURRENT_REQUESTS Totals
  • Running Requests
  • Total Pending Requests by Status Code
  • Count Pending Regularly Scheduled/Non Regularly-Scheduled Requests
  • Count of Pending Requests on Hold/Not on Hold
  • Listing of Scheduled Requests
  • Listing of Pending Requests on Hold
  • Listing of Pending Requests Not on Hold
  • Volume of Daily Concurrent Requests for Last Month
  • Identify/Resolve the Pending/Standby Issue, if Caused by Run Alone Flag
  • Tablespace Statistics for the fnd_concurrent tables
  • E-Business Applications Concurrent Manager Analysis
  • Concurrent Managers Active and Enabled
  • Concurrent Manager Processes by Workshift
  • Active Manager for Applications that are not Installed/Used
  • Total Target Processes for Request Managers (Excluding Off-Hours)
  • Request Managers with Incorrect Cache Size
  • Concurrent Manager Request Summary by Manager
  • Check Manager Queues for Pending Requests
  • Check the Configuration of OPP
• Identifies Concurrent System Setup and configurations
• Identifies and recommends Concurrent Best Practices
• Easy to add Tool for regular Concurrent Maintenance
• Execute Analysis anytime to compare trending from past outputs

EBS 12.2-- Output Post Processor(OPP) Performance in RTF and Excel reports.

The need to investigate the performance of OPP have arised by a long running Report in a Production EBS 12.2 instance. The name of report was Chart of Accounts - Segment Values Listing (XML)..
The issue was there for all the XML reports actually.. Especially, XML ereports which produce big sized RTF and Excel outputs..

In this blog post;

I will directly jump to the conclusino and then give the diagnostics & actions whic was taken to fix the issue. I m giving the conclusion first because; the conclusion is not differ according to the our actions, whatever the actions are...

Okay.. This is a limitation problem.
I mean, as OPP is a Java Concurrent Manager dedicated to BI Publisher, and as BI Publisher is all about java processing; and because Concurrent Tier uses 32 Bit JVM in EBS 12.2; we are under control of the limits in 32 Bit JVM ..

Altough, EBS 12.2 uses 10.1.3.4.2 version of BI Publisher,  it is still uses a 32 bit JVM in XML processing.
So, in such a large RTF processing/producing , there is nothing that can be done about the performance of OPP....

Note that:
These are the facts about the BIP output formats from an RTF layout template type:
- PDF output has the most efficient file size
- RTF output generates several times larger than pdf for the same xml data set size
- Excel output (which is really XHTML format) can be 10X larger than the pdf for the same xml data set size

What are the performance characteristics of 64-bit versus 32-bit VMs?

Generally, the benefits of being able to address larger amounts of memory come with a small performance loss in 64-bit VMs versus running the same application on a 32-bit VM. This is due to the fact that every native pointer in the system takes up 8 bytes instead of 4. The loading of this extra data has an impact on memory usage which translates to slightly slower execution depending on how many pointers get loaded during the execution of your Java program. The good news is that with AMD64 and EM64T platforms running in 64-bit mode, the Java VM gets some additional registers which it can use to generate more efficient native instruction sequences. These extra registers increase performance to the point where there is often no performance loss at all when comparing 32 to 64-bit execution speed.
The performance difference comparing an application running on a 64-bit platform versus a 32-bit platform on SPARC is on the order of 10-20% degradation when you move to a 64-bit VM. On AMD64 and EM64T platforms this difference ranges from 0-15% depending on the amount of pointer accessing your application performs

Anyways.. Lets explore what I have done;  when the issue was escalated to me..
Note that: the cause is :32 bit JVM limitations in here but still; I m giving these info to you, because it might help you diagnosing similar issues in EBS 12.2

I have done  the following diagnostics;

• Traced the process(the running thread inside of it) using strace;

            strace -p 8527 (the main process)
           Process 8527 attached - interrupt to quit
          [ Process PID=8527 runs in 32 bit mode. ]
          futex(0xf7743bd8, FUTEX_WAIT, 8530, NULL

         strace -p 8530 (the thead that main process was waiting)
        Process 8530 attached - interrupt to quit
       [ Process PID=8530 runs in 32 bit mode. ]
         futex(0xf7507644, FUTEX_WAIT_PRIVATE, 1, NULL

• Looking at the load of the server

There is no load on the server. Java HotSpot(TM) Server VM (build 23.25-b01, mixed mode)
[appldev@erpdev scripts]$ top
top - 17:55:21 up 21:27, 9 users, load average: 2.06, 1.49, 1.21
Tasks: 413 total, 2 running, 411 sleeping, 0 stopped, 0 zombie
Cpu(s): 14.6%us, 1.1%sy, 0.0%ni, 78.0%id, 6.1%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 12198304k total, 12032184k used, 166120k free, 39948k buffers
Swap: 16383992k total, 318672k used, 16065320k free, 6002308k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8527 appldev 20 0 822m 193m 19m S 99.8 1.6 15:46.39 java


• Java version:
Java version is

/u01/apps/fs1/EBSapps/comn/util/jdk32/bin/java -version
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) Server VM (build 23.25-b01, mixed mode)


• The java program itself:
Problematic process is :

/u01/apps/fs1/EBSapps/comn/util/jdk32/bin/java -Dxdo.xliff.source=EBS -DCLIENT_PROCESSID=3969 -Dhttp.proxyHost= -Dhttp.proxyPort= -Dhttp.nonProxyHosts= -Djava.awt.headless=true -server -Xmx384m -Djava.security.egd=file:///dev/urandom -Doracle.apps.fnd.common.Pool.leak.mode=stderr:off -verbose:gc -mx600m -Ddbcfile=/u01/apps/fs1/inst/apps/DEV_erpdev/appl/fnd/12.0.0/secure/DEV.dbc -Dcpid=11756 -Dconc_queue_id=6269 -Dqueue_appl_id=0 -Dlogfile=/u01/apps/fs_ne/inst/DEV_erpdev/logs/appl/conc/log/FNDOPP11756.txt -DLONG_RUNNING_JVM=true -DOVERRIDE_DBC=true -DFND_JDBC_BUFFER_MIN=1 -DFND_JDBC_BUFFER_MAX=2 oracle.apps.fnd.cp.gsf.GSMServiceController


• OS , hostname and the kernel info:
uname -a
Linux erpdev 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Dec 13 06:58:20 EST 2013 x86_64 x86_64 x86_64 GNU/Linux


• Concurrent processing analysis using CP Analzer:

            CP Analyzer and upload the output - Doc ID 1411723.1

• OPP manager log:

           OPP.log file was as follows; 
          The problematic thing was GLXRLSEG_XML
          It starts at 10:06 , still continuing at 11:40

          [12/12/14 10:06:11 AM] [11816:RT636361] XML Publisher post-processing action complete.
          [12/12/14 10:06:12 AM] [11816:RT636361] Completed post-processing actions for request 636361.
         [12/12/14 10:11:28 AM] [OPPServiceThread0] Post-processing request 636387.
        [12/12/14 10:11:28 AM] [11816:RT636387] Executing post-processing actions for request 636387.
        [12/12/14 10:11:28 AM] [11816:RT636387] Starting XML Publisher post-processing action.
      [12/12/14 10:11:28 AM] [11816:RT636387]
      Template code: GLXRLSEG_XML

• The running java code from thread dump:

        java.lang.Thread.State: RUNNABLE 
      at oracle.xdo.parser.v2.XSLExprValue.mergeOrderedList(XSLExprValue.java:551)

• Java threads stack Dump using jstack:

          The running runnable process:

Worker-3" daemon prio=10 tid=0x23a00c00 nid=0x41b7 in Object.wait() [0x21669000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at oracle.ias.cache.TaskQ.waitForWork(TaskQ.java:255)
- locked <0x3354ad50> (a oracle.ias.cache.TaskQ)
at oracle.ias.cache.TaskQ.getTask(TaskQ.java:138)
- locked <0x3354ad50> (a oracle.ias.cache.TaskQ)
at oracle.ias.cache.WorkerThread.run(ThreadPool.java:303)

"9786:RT639072" daemon prio=10 tid=0x24002c00 nid=0x3a28 runnable [0x216b9000]
java.lang.Thread.State: RUNNABLE
at oracle.xdo.parser.v2.XSLExprValue.mergeOrderedList(XSLExprValue.java:551)
at oracle.xdo.parser.v2.XPathAxis.getDescNodeList(XPathAxis.java:200)
at oracle.xdo.parser.v2.XPathStep.evaluate(XPathStep.java:405)
at oracle.xdo.parser.v2.PathExpr.evaluate(XSLNodeSetExpr.java:873)
at oracle.xdo.parser.v2.ComparisonExpr.evaluate(XSLExpr.java:1457)
at oracle.xdo.parser.v2.XSLExprBase.testBooleanExpr(XSLExprBase.java:298)
at oracle.xdo.parser.v2.AndExpr.evaluate(XSLExpr.java:441)
at oracle.xdo.parser.v2.XSLExprBase.testBooleanExpr(XSLExprBase.java:298)
at oracle.xdo.parser.v2.XPathFunctionCall.evaluate(XPathFunctionCall.java:695)

• Os thead analysis:

OS Thread analysis: ps -eLo pid,ppid,tid,pcpu,comm | grep 19686

19686 17301 19686 0.0 java
19686 17301 19687 0.0 java
19686 17301 19688 0.0 java
19686 17301 19689 0.0 java
19686 17301 19690 0.0 java
19686 17301 19691 0.0 java
19686 17301 19692 0.0 java
19686 17301 19693 0.0 java
19686 17301 19694 0.0 java
19686 17301 19695 0.0 java
19686 17301 19696 0.0 java
19686 17301 19697 0.0 java
19686 17301 19698 0.0 java
19686 17301 19699 0.0 java
19686 17301 19700 0.0 java
19686 17301 19701 0.0 java
19686 17301 19703 0.0 java
19686 17301 19718 0.0 java
19686 17301 19719 0.0 java
19686 17301 19720 0.0 java
19686 17301 19721 0.0 java
19686 17301 19722 0.0 java
19686 17301 19723 0.0 java
19686 17301 19724 0.0 java
19686 17301 19725 0.0 java
19686 17301 19726 0.0 java
19686 17301 19727 0.0 java
19686 17301 19728 0.0 java
19686 17301 19729 0.0 java
19686 17301 19730 0.0 java
19686 17301 19733 0.0 java
19686 17301 19734 0.0 java
19686 17301 20212 99.4 java The only thread that does the work is this.
19686 17301 20302 0.0 java

Okay, after collecting the diagnostics info, I have taken following actions to tune OPP, and I acutally tuned the OPP process but these did not fix the problem.

1)
In XML Publisher Administrator responsibility >Administration > Configuration
- under : Properties -> General -> set a temporary directory
This directory should be at least 5 times larger than the largest anticipated data file.
Support recommends the directory to be at least 5GB.

2)
XML Publisher Administrator responsibility ->
Navigation Path : XML Publisher Administrator responsibility
- query the data definition
- on the View Data Definition page select Edit Configuration button
- set 'Use XML Publisher's XSLT processor' to True
- set 'Enable scalable feature of XSLT processor' to True
- set 'Enable XSLT runtime optimization' to True
Note that these property values, when set at the Data Definition level, takes precedence over values set at the Site level.

3)
in System Administrator  > System
- set Concurrent:OPP Process Timeout = 43200
- set Concurrent:OPP Response Timeout = 43200
(these values are for 12 hours in seconds)

4)

update FND_CP_SERVICES
set DEVELOPER_PARAMETERS = 'J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx2048m'
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = 'FNDCPOPP');

5)

rebooted the server (for futex wait)

6)

Changed the java process to use urandom instead of random

7)

Stopped the NTP

/etc/init.d/ntp stop
then;
date `date +"%m%d%H%M%C%y.%S"`

8)

Changed OPP thead counts, made it to run single thead; then made it to run in several thread.. It was just like a Guessing Game

The actions above could at least made the report to successfully complete in 1 hours..To decrease this completion time; one should use BI Publisher Enterprise 11g product is 64-bit as it will use the 64-bit JVM which is exponentially larger.

By the way, I said using BIP 64 is a solution, but BIP 64 bit is a separate application/rquired to be installed separatley. BIP 64 bit  can not be embedded into EBS, as it is not 32 bit..
BI Publisher 32 bit however,can be embedded into EBS.. It comes embedded in EBS. Oracle used the libraries from it to EMBED it into EBS, to replace/upgrade the aging 5.6.3 version.
They were able to do this because 10.1.4.3.2 is still 32-bit.

EBS 12.2 -- Weblogic Aware Administration

In EBS 12.2 , Weblogic is a core component for us as Apps Dbas..
To have such a capable but also complex Application Server in our Technology Stack brings us the requirement to have advanced administration knowledge about it.. Also , our forms server, oacore server ,oafm and more reside in Weblogic.. So if our Weblogic is down, then we can say that EBS is down from the users perspective.. (http server, listeners, concurrent managers and the database is up actually:) )
Okay, I said advanced , because I have faced the situations which made me spend times to solve..
I mean, if you say "this is EBS , Weblogic is just a component comes built-in, so I manage the Weblogic using EBS tools, and I solve the problems using autoconfig and similar operations in conjuction with Oracle support document" , you will have hard times.. You need to follow oracle support doc, that is for sure but there are times that we need to take deeps dive in to the Weblogic to solve some of our problems, or lets say : to diagnose the problems..

I have seen this, I have done this.. Look at my posts about EBS 12.2.
Weblogic is involved in patching, in cloning, in restarting, in java deployments , in performance problems and so on.. Weblogic itself is also need to be patched rarely..
Sometimes, you connect in to the Weblogic using admin console to take necessary actions and sometimes you wont find an admin console to connect to, that's why using manual approaches..
Anyways, In this post I tried to draw attention to Weblogic in EBS 12.2..  Java knowledge is also important, one should understand the java error stack , one should able to deal with java command line parameters like xmx,xms,permsize so on..

One last thing... If you are not in TXK and AD.C delta 5 patch levels, you will be a Weblogic specialist one day, as there are bugs that you need to diagnose (especially performance)and solve in the previous patch levels.

Essential Weblogic related Bugs fixed in for AD Delta 5 and TXK Delta 5

• Performance improvements have been made for managing application tier services, including implementation of the Managed Server Independence Mode feature (-msimodeparameter to adstrtal.sh) to allow application tier services to be started or stopped without the WebLogic Administration Server running.
• On a multi-node application tier system configuration, remote connectivity is no longer required for packaging the Oracle E-Business Suite WebLogic Server domain.
• JVM heap size (-Xms and -Xmx) has been increased to 1 GB for the WebLogic Administration Server and all managed servers.

EBS 12.2 -- adop phase=prepare pasteConfig.sh hangs java.lang.OutOfMemoryError

While applying a little 5kb patch using online patching (which is a must bytheway), you may encounter a hand situation.. Spending hours to find out the causes and correcting them during a Project implementation, is not something we want to deal.. Also inability to apply a 5kb patch even in 1 hour is does not seem good from the Business perspective..

The problem that I will mention in the below paragraps is a good example for the unwanted situations we as Apps Dba may encounter in EBS 12.2.3..

Lets review the online patching in EBS 12.2.. So , to apply a patch, we start a patching cycle using adop phase=prepare and after that we continue with phase=apply, phase=finalize, phase=cutover and phase=cleanup.

This problematic situation arises in the "prepare" phase, as we cant complete the prepare phase in hours..
There is definetely a hang situation , but we have a lot of place to collect the data for our diagnosis..
We have some clues though.. For example : adop log files..
Anyways, when you see a line starting with pasteConfig.sh in the tail of the related log file, you are there where I was 1 day ago :)

Lets clarify the situation..

In pasteConfig , adop tries to pack and unpack Weblogic domain to supply the Weblogic consistency accross the patches. It deals with files and it runs java programs against those files..

These java programs often encounter OutOfMemoryError ... sometimes it is about java heaps, sometimes you ll not able to see the problematic memory area, but here is what you need to do if you see OutOfMemoryError in the related log file while running PasteConfig.sh during adop prepare session.

Note that: we make the controls using the logfile located in admin/clone directory .. 

For example:  /u01/oracle/TEST/fs2/inst/apps/TEST_erptest/admin/log/clone/wlsT2PApply log

Following lines from the related log file describes the real problem that makes adop to hang ;

Configuring the Server -  AdminServer
Configuring the Server -  oacore_server1
Configuring the Server -  forms_server1
Configuring the Server -  oafm_server1
Configuring the Server -  forms-c4ws_server1
Configuring the Machine -  erptest
Configuring the Cluster -  forms_cluster1
Configuring the Cluster -  oacore_cluster1
Configuring the Cluster -  oafm_cluster1
Configuring the Cluster -  forms-c4ws_cluster1
Configuring the Datasource -  EBSDataSource

Exiting WebLogic Scripting Tool.
INFO : Dec 13, 2014 7:25:31 PM - CLONE-23283  Unpack Domain.
<< read template from "/tmp/CLONINGCLIENT8433880607085395265/packed_template.jar"
>>  succeed: read template from "/tmp/CLONINGCLIENT8433880607085395265/packed_template.jar"
<< set config option ServerStartMode to "prod"
>>  succeed: set config option ServerStartMode to "prod"
<< write Domain to "/u01/oracle/TEST/fs1/FMW_Home/user_projects/domains/EBS_domain_TEST"
Exception in thread "Thread-1" java.lang.OutOfMemoryError

So , as you see there is an java.lang.OutOfMemoryError there .. It is encountering the unpacking phase which confirms the thing that I mentioned in the 3th step of the below action plan.

ACTION PLAN:

1)export CONFIG_JVM_ARGS="-Xms1024m -Xmx2048m"(OR MAYBE MORE) in your terminal and restart the adop phase=prepare

2)Edit the related scripts in COMMON_TOP/adopclone_hostname/bin, and change the values from 600M to 2000M in the related files. (you can use grep -R 600M * in the above directory to find the files that contains 600M xmx limit )

3) Delete the old weblogic logiles; and use /dev/null > to the current weblogic logfiles .. Because; your log size may be increased recently.. pasteConfig creates a CLONINGCLIENT***/packed_template.jar firstly , and it unpacks it .. When this file becomes bigger, the risk to get outOfmemoryError increases.

You will see that when you delete the logs, this file will be smaller and you wont get outOfmemory error.. So dont save up big weblogic files in any of the Managed Server directories. Backup them then delete them if you need to store them..

4)export TIMEDPROCESS_TIMEOUT=-1 ->to disable the timeout. Just in case ,as we dont want a timeout.

5)restart the adop phase=prepare 

These action will do the job( they are required actions+ some extras).. If the issue persists; feel free to comment.

Also see :

EBS 12.2 -- ADOP /Online Patching Common Errors --script timed out,resource unavailable,outofMemory and more..