EBS 12.2 -- adstrtal.sh hangs -- can not start managed servers oacore_server1, forms_server1, oafm_server1 etc..

We use adstrtal.sh to start all of the EBS application tier components .. This script is used in eBS 11i, 12.0, 12.1 and now in EBS 12.2..

Last week, I have encountered an interesting problem in starting EBS 12.2.3 Application services ..
The problem was in adstrtal.sh..
It happened when I realized that adstrtal.sh script could not start the application services entirely..
It was waiting after starting the Weblogic Admin Server..

Normally, it should start the Weblogic Managed Servers like oacore , forms and etc, but it could not.. I was waiting ..
When I analyzed the situation and check the log files for diagnosis; I could not find any clue.. I checked admin server, node manager logs , managed server logs and etc, but they were all clean..
Checked the related java processes and adstrtal itself, they were all clean , too..

Executing service control script:
/u01/apps/fs1/inst/apps/VIS_erpdemo/admin/scripts/adadminsrvctl.sh start -nopromptmsg
Timeout specified in context file: -1 second(s)
script returned:
****************************************************
You are running adadminsrvctl.sh version 120.10.12020000.7
Starting WLS Admin Server...
Refer /u01/apps/fs1/inst/apps/VIS_erpdemo/logs/appl/admin/log/adadminsrvctl.txt for details

adadminsrvctl.sh: exiting with status 0
adadminsrvctl.sh: check the logfile /u01/apps/fs1/inst/apps/VIS_erpdemo/logs/appl/admin/log/adadminsrvctl.txt for more information ...
.end std out.
.end err out.

****************************************************

---  It was waiting in here..

When this was happening , the adstrtal.log was displaying the followings;

Checking the status of the AdminServer...

.end std out.

.end err out.

Checking/updating the JVM argument values for the managed servers ...

Validated the passed arguments for the option ebs-set-allmanagedsrvargs

Deploying application: ebs-forms on forms_cluster1

Location = /u01/apps/fs1/EBSapps/comn/shared-libs/ebs-forms

Connecting to t3://erpdemo.takasdom.takasbank.com.tr:7001 with userid weblogic ...

Successfully connected to Admin Server 'AdminServer' that belongs to domain 'EBS_domain_PRELIVE'.

Warning: An insecure protocol was used to connect to the

server. To ensure on-the-wire security, the SSL port or

Admin port should be used instead.

-->>> So no clue again... Maybe we should wait for some time .. Maybe more than 10 mins:)

Anyways, when I checking the things in Weblogic admin console , I saw  I saw that a configuration lock was held... Some things were changed by an admin but the changes were not activated or rolled back.. Thus there was a domain lock.. 

For the solution, I have released the lock by activating the changes , and managed servers have started without any problems..

Then I checked the adstrtal.sh log again and saw the following;

Deploying application from /u01/apps/fs1/EBSapps/comn/shared-libs/ebs-forms to targets forms_cluster1 (upload=false) ...

[Deployer:149140]The task cannot be processed further until the current edit session is activated. When this occurs, task processing will continue. The user can exit the deployer tool without affecting the task..................................................................................................................................................................................................................................................................................Completed the deployment of Application with status completed

So , a latecomer clue  :) 

Anyways, releasing the configuration lock have fixed the problem..

If we dig deeper;

When we executed adstrtal.sh; it executes admanagedsrvctl.sh script to start our managed servers.

admanagedsrvctl.sh uses adProvisionEBS.pl with ebs-nmstart-managedsrv argument to start our managed servers. 

  { echo $wlspass; } | perl /u01/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl -contextfile=/u01/apps/fs1/inst/apps/VIS_erpdemo/appl/admin/VIS_erpdemo.xml ebs-nmstart-managedsrv -wlsadminuser=weblogic -wlsadminport=7001 -managedsrvname=$SERVER_NAME -domain=EBS_domain_PRELIVE -domainhome="/u01/apps/fs1/FMW_Home/user_projects/domains/EBS_domain_PRELIVE" -nmhome="/u01/apps/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1" -nmport=5556 -nmtype=Plain -machinelistenaddress="erpdemo.takasdom.takasbank.com.tr" -promptmsg=hide -logfile=$LOG

adProvisionEBS.pl is a wrapper that is used to call the Provisioning API oracle.apps.ad.tools.configuration.EBSProvisioner to provision the E-Business Suite.

It uses ebs_deploy_app to make the deployments in every startup.  I didnt have enough time to analyze deeper , but at least I can say that It makes the deployments for forms_cluster in every startup.

"Initiating deploy operation for application, ebs-forms to targets forms_cluster1"

So when it comes to this deployment, it acquires a lock, and waits if that lock is held..

Something like the following runs there;

startEdit() --> it waits here..

if startswith forms_

Deploy!

Lets look at the defition of startEdit wlst command;

startEdit([waitTimeInMillis], [timeoutInMillis], [exclusive])

Argument	Definition
waitTimeInMillis	Optional. Time (in milliseconds) that WLST waits until it gets a lock, in the event that another user has a lock. This argument defaults to 0 ms.
timeoutInMillis	Optional. Timeout (in milliseconds) that WLST waits to release the edit lock. This argument defaults to -1 ms, indicating that this edit session never expires.
exclusive	Optional. Specifies whether the edit session should be an exclusive session. If set to true, if the same owner enters the startEdit command, WLST waits until the current edit session lock is released before starting the new edit session. The exclusive lock times out according to the time specified in timeoutInMillis. This argument defaults to false.

You see ; timeoutInMillis -> This argument defaults to -1 ms, indicating that this edit session never expires. :)

Okay, we have satisfied our curiosity ..

Lastly, I suggest you to take a look at the following presentation, it explains Fusion Middleware in 

E-Business Suite 12.2 very well..

http://www.aioug.org/sangam14/images/Sangam14/Presentations/201587_Sai_Subramanian_ppt.pdf