Friday, January 2, 2015

EBS 12.2 -- adstrtal.sh hangs -- can not start managed servers oacore_server1, forms_server1, oafm_server1 etc..

We use adstrtal.sh to start all of the EBS application tier components .. This script is used in eBS 11i, 12.0, 12.1 and now in EBS 12.2..

Last week, I have encountered an interesting problem in starting EBS 12.2.3 Application services ..
The problem was in adstrtal.sh..
It happened when I realized that adstrtal.sh script could not start the application services entirely..
It was waiting after starting the Weblogic Admin Server..

Normally, it should start the Weblogic Managed Servers like oacore , forms and etc, but it could not.. I was waiting ..
When I analyzed the situation and check the log files for diagnosis; I could not find any clue.. I checked admin server, node manager logs , managed server logs and etc, but they were all clean..
Checked the related java processes and adstrtal itself, they were all clean , too..

Executing service control script:
/u01/apps/fs1/inst/apps/VIS_erpdemo/admin/scripts/adadminsrvctl.sh start -nopromptmsg
Timeout specified in context file: -1 second(s)
script returned:
****************************************************
You are running adadminsrvctl.sh version 120.10.12020000.7
Starting WLS Admin Server...
Refer /u01/apps/fs1/inst/apps/VIS_erpdemo/logs/appl/admin/log/adadminsrvctl.txt for details

adadminsrvctl.sh: exiting with status 0
adadminsrvctl.sh: check the logfile /u01/apps/fs1/inst/apps/VIS_erpdemo/logs/appl/admin/log/adadminsrvctl.txt for more information ...
.end std out.
.end err out.

****************************************************

---  It was waiting in here..

When this was happening , the adstrtal.log was displaying the followings;
Checking the status of the AdminServer...
.end std out.
.end err out.
Checking/updating the JVM argument values for the managed servers ...
Validated the passed arguments for the option ebs-set-allmanagedsrvargs
Deploying application: ebs-forms on forms_cluster1
Location = /u01/apps/fs1/EBSapps/comn/shared-libs/ebs-forms
Connecting to t3://erpdemo.takasdom.takasbank.com.tr:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'EBS_domain_PRELIVE'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

-->>> So no clue again... Maybe we should wait for some time .. Maybe more than 10 mins:)

Anyways, when I checking the things in Weblogic admin console , I saw  I saw that a configuration lock was held... Some things were changed by an admin but the changes were not activated or rolled back.. Thus there was a domain lock.. 
For the solution, I have released the lock by activating the changes , and managed servers have started without any problems..
Then I checked the adstrtal.sh log again and saw the following;

Deploying application from /u01/apps/fs1/EBSapps/comn/shared-libs/ebs-forms to targets forms_cluster1 (upload=false) ...
[Deployer:149140]The task cannot be processed further until the current edit session is activated. When this occurs, task processing will continue. The user can exit the deployer tool without affecting the task..................................................................................................................................................................................................................................................................................Completed the deployment of Application with status completed

So , a latecomer clue  :) 
Anyways, releasing the configuration lock have fixed the problem..

If we dig deeper;

When we executed adstrtal.sh; it executes admanagedsrvctl.sh script to start our managed servers.
admanagedsrvctl.sh uses adProvisionEBS.pl with ebs-nmstart-managedsrv argument to start our managed servers. 

  { echo $wlspass; } | perl /u01/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl -contextfile=/u01/apps/fs1/inst/apps/VIS_erpdemo/appl/admin/VIS_erpdemo.xml ebs-nmstart-managedsrv -wlsadminuser=weblogic -wlsadminport=7001 -managedsrvname=$SERVER_NAME -domain=EBS_domain_PRELIVE -domainhome="/u01/apps/fs1/FMW_Home/user_projects/domains/EBS_domain_PRELIVE" -nmhome="/u01/apps/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1" -nmport=5556 -nmtype=Plain -machinelistenaddress="erpdemo.takasdom.takasbank.com.tr" -promptmsg=hide -logfile=$LOG

adProvisionEBS.pl is a wrapper that is used to call the Provisioning API oracle.apps.ad.tools.configuration.EBSProvisioner to provision the E-Business Suite.
It uses ebs_deploy_app to make the deployments in every startup.  I didnt have enough time to analyze deeper , but at least I can say that It makes the deployments for forms_cluster in every startup.
"Initiating deploy operation for application, ebs-forms to targets forms_cluster1"

So when it comes to this deployment, it acquires a lock, and waits if that lock is held..

Something like the following runs there;

startEdit() --> it waits here..
if startswith forms_
Deploy!

Lets look at the defition of startEdit wlst command;

startEdit([waitTimeInMillis], [timeoutInMillis], [exclusive])

Argument
Definition 
waitTimeInMillis
Optional. Time (in milliseconds) that WLST waits until it gets a lock, in the event that another user has a lock. This argument defaults to 0 ms.
timeoutInMillis
Optional. Timeout (in milliseconds) that WLST waits to release the edit lock. This argument defaults to -1 ms, indicating that this edit session never expires.
exclusive
Optional. Specifies whether the edit session should be an exclusive session. If set to true, if the same owner enters the startEdit command, WLST waits until the current edit session lock is released before starting the new edit session. The exclusive lock times out according to the time specified in timeoutInMillis. This argument defaults to false.

You see ; timeoutInMillis -> This argument defaults to -1 ms, indicating that this edit session never expires. :)
Okay, we have satisfied our curiosity ..

Lastly, I suggest you to take a look at the following presentation, it explains Fusion Middleware in 
E-Business Suite 12.2 very well..

11 comments :

  1. Bluehost is one of the best web-hosting provider with plans for all of your hosting needs.

    ReplyDelete
  2. Hi Erman,

    Reg When we executed adstrtal.sh; it executes admanagedsrvctl.sh script to start our managed servers.
    admanagedsrvctl.sh uses adProvisionEBS.pl with ebs-nmstart-managedsrv argument to start our managed servers.

    If in case nodemanager is down,can adstrtall script start admin and managed servers?

    ReplyDelete
  3. See, http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/adstartall-sh-tp2444.html

    ReplyDelete
  4. Thanks Erman,that information is really useful.Is the same applies to admin server as well?can adstrtal.sh start admin server when node manager is down

    ReplyDelete
  5. Yes. You will get the following:

    ERROR: Cannot connect to Node Manager. Check if port 5557 is correct
    ERROR: Unable to connect to the Node Manager. The Admin Server cannot be started up.

    ReplyDelete
  6. both the scripts adadminsrvctl.sh and admanagedsrvctl.sh uses adprovisionebs.pl?then admin server as well cannot start if node manager is down.is that correct?

    ReplyDelete
  7. The lines that are used for starting the admin server, in the adadminsrvctl script is as follows;

    printf "Starting WLS Admin Server...\n"
    fi
    { echo $wlspass; } | perl /u01/install/APPS/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl -contextfile=/u01/install/APPS/fs2/inst/apps/ORAPROD_daroraappsrv1/appl/admin/ORAPROD_daroraappsrv1.xml ebs-nmstart-adminsrv -wlsadminuser=weblogic -wlsadminport=7002 -domain=EBS_domain_ORAPROD -domainhome="/u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain_ORAPROD" -nmhome="/u01/install/APPS/fs2/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1" -nmport=5557 -nmtype=plain -machinelistenaddress="ERMANSRV" -promptmsg=hide -serverstartargs="-XX:PermSize=128m -XX:MaxPermSize=384m -Xms1024m

    So, as you see, it takes the Node manager arguments there.

    adadminsrvctl.sh itself starts the node manager if it is not up.

    so the answer for your question is; " node manager must not be up, but it is able to be up, when the adadminsrvctl.sh starts)

    ReplyDelete
  8. Here is the proof for you..

    I just move the nm_password.properties file(node manager's password config file) and try to start the admin server using adadminsrvctl.sh and as you below, adadminsrvctl.sh returns error. (status 1)

    So ,as I said.. It is dependent to the node manager by design.

    WARNING: I/O error while reading domain directory
    NMProcess: java.io.FileNotFoundException: Node manager user properties file '/u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain_ORATEST/config/nodemanager/nm_password.properties' not found
    NMProcess: at weblogic.nodemanager.server.DomainManager.loadUserInfo(DomainManager.java:136)
    NMProcess: at weblogic.nodemanager.server.DomainManager.initialize(DomainManager.java:90)
    NMProcess: at weblogic.nodemanager.server.DomainManager.(DomainManager.java:60)
    NMProcess: at weblogic.nodemanager.server.NMServer.findOrCreateDomainManager(NMServer.java:284)
    NMProcess: at weblogic.nodemanager.server.NMServer.getDomainManager(NMServer.java:273)
    NMProcess: at weblogic.nodemanager.server.Handler.handleDomain(Handler.java:235)
    NMProcess: at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:117)
    NMProcess: at weblogic.nodemanager.server.Handler.run(Handler.java:73)
    NMProcess: at java.lang.Thread.run(Thread.java:724)
    NMProcess:
    Refer /u01/install/APPS/fs2/inst/apps/ORATEST_ermansrv/logs/appl/admin/log/adadminsrvctl.txt for details

    adadminsrvctl.sh: exiting with status 1

    ReplyDelete
  9. Hi


    adadminsrvctl.sh: exiting with status 1
    05/19/17-12:4
    05/19/17-12:43:07 :: adadminsrvctl.sh version 120.10.12020000.9
    Validated the passed arguments for the option ebs-get-serverstatus
    AdminServer is currently not running.

    Validated the passed arguments for the option ebs-nmstart-adminsrv
    Checking if the Admin Server is already up.
    The Admin Server is not already up.
    Checking if the Node Manager is already up..
    Traceback (innermost last):
    File "", line 1, in ?
    File "", line 123, in nmConnect
    File "", line 648, in raiseWLSTException
    WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : Connection refused. Could not connect to NodeManager. Check that it is running at scchopap1.mycompany.com.sa:5557.
    Use dumpStack() to view the full stacktrace

    Connecting to Node Manager ...

    Launching NodeManager ...

    Successfully launched the Node Manager.
    The Node Manager process is running independent of the WLST process.
    Exiting WLST will not stop the Node Manager process. Please refer
    to the Node Manager logs for more information.
    The Node Manager logs will be under /sccprod/applmgr/SCC/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1
    Node Manager starting in the background

    Traceback (innermost last):
    File "", line 1, in ?
    File "", line 123, in nmConnect
    File "", line 648, in raiseWLSTException
    WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : Connection refused. Could not connect to NodeManager. Check that it is running at scchopap1.my company.com.sa:5557.
    Use dumpStack() to view the full stacktrace
    3:07 :: adadminsrvctl.sh version 120.10.12020000.9
    Validated the passed arguments for the option ebs-get-serverstatus
    AdminServer is currently not running.

    Validated the passed arguments for the option ebs-nmstart-adminsrv
    Checking if the Admin Server is already up.
    The Admin Server is not already up.
    Checking if the Node Manager is already up..
    Traceback (innermost last):
    File "", line 1, in ?
    File "", line 123, in nmConnect
    File "", line 648, in raiseWLSTException
    WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : Connection refused. Could not connect to NodeManager. Check that it is running at scchopap1.mycompany.com.sa:5557.
    Use dumpStack() to view the full stacktrace

    Connecting to Node Manager ...

    Launching NodeManager ...

    Successfully launched the Node Manager.
    The Node Manager process is running independent of the WLST process.
    Exiting WLST will not stop the Node Manager process. Please refer
    to the Node Manager logs for more information.
    The Node Manager logs will be under /sccprod/applmgr/SCC/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1
    Node Manager starting in the background

    Traceback (innermost last):
    File "", line 1, in ?
    File "", line 123, in nmConnect
    File "", line 648, in raiseWLSTException
    WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : Connection refused. Could not connect to NodeManager. Check that it is running at scchopap1.mycompany.com.sa:5557.
    Use dumpStack() to view the full stacktrace

    ReplyDelete
  10. tried to clean /tmp locations and start but no luck.

    ReplyDelete
    Replies
    1. Please use the link and create an issue into EBS 12.2 category: http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/EBS-12-2-f6.html

      I m giving support through my Forum. "http://ermanarslan.blogspot.com.tr/p/forum.html"
      If you have any issues, use my forum. It is not efficient to use blog comments for this.

      A similar issue already solved there -> check this ->

      http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/Adop-Prepare-Has-Failed-With-adclone-pl-Error-td292.html#a954

      Delete

If you will ask a question, please don't comment here..

For your questions, please create an issue into my forum.

Forum Link: http://ermanarslan.blogspot.com.tr/p/forum.html

Register and create an issue in the related category.
I will support you from there.