Wednesday, June 11, 2014

EBS 12.2 -- adadminsrvctl.sh start -- sh: module: line 1: syntax error: unexpected end of file

After completed the fresh install of EBS 12.2 , you may encounter the following error while starting Weblogic admin server. Note that : this error does- not prevent weblogic to work..
./adadminsrvctl.sh start
sh: module: line 1: syntax error: unexpected end of file
sh: error importing function definition for `module'

Oracle explains this problem with an internal bug : internal Bug 14259166 , and suggest the following rename for the fix:

Rename /etc/profile.d/modules.sh to /etc/profile.d/modules.sh.bak.
Log out, log back in, and set the application environment.
Confirm these environment variables are not set: MODULEPATH, LOADEDMODULES and MODULESHOME.
Actually, it is not required because the problem is caused by the script /usr/share/Modules/init/bash, exactly the following line "export -f module" triggers the error. So commenting this line will fix the error.

In this post, I will try to explain the cause of this problem;

So when we have modules.sh in our profile.d folder, we have modules related environments set;
for example;

[root@erpprod profile.d]# su - applmgr
[applmgr@erpprod ~]$ echo $MODULEPATH
/usr/share/Modules/modulefiles:/etc/modulefiles

Normally when we havent any modules.sh in our profile.d folder, we dont set any modules environment variables.
[root@erpprod profile.d]# su - applmgr
[applmgr@erpprod ~]$ echo $MODULEPATH

Okay, another important thing in the error message is; it reports an error while importing a function definition but it does not display a function name..
Our error is ; sh: error importing function definition for `module'
But it must be something like; sh: error importing function definition for `module '/opt/IBM/InformationServer/Server/DSComponents/lib/libicui18n.so'

I mean it should display the function name ,and it is also lacking a ' character... It has module' at the end, but that ' character is not closed.. 
So this seems to be the problem.. This explains the EOF error, as a ' character should be there not EOF..

As we now that our .bash_profile sources bashrc and bashrc source /etc/profile.d/*.sh scripts;
 Lets take a look what is written in our modules.sh file;

shell=`/bin/basename \`/bin/ps -p $$ -ocomm=\``
if [ -f /usr/share/Modules/init/$shell ]
then
  . /usr/share/Modules/init/$shell
else
  . /usr/share/Modules/init/sh
fi

So basicall it finds our shell and executes related scripts accordingly..
So assuming our shell is bash; lets look at the file contents of the script : /usr/shares/Modules/init/bash
--------------------------------------

module() { eval `/usr/bin/modulecmd bash $*`; }
export -f module

MODULESHOME=/usr/share/Modules
export MODULESHOME

if [ "${LOADEDMODULES:-}" = "" ]; then
  LOADEDMODULES=
  export LOADEDMODULES
fi

if [ "${MODULEPATH:-}" = "" ]; then
  MODULEPATH=`sed -n 's/[       #].*$//; /./H; $ { x; s/^\n//; s/\n/:/g; p; }' ${MODULESHOME}/init/.modulespath`
  export MODULEPATH
fi

if [ ${BASH_VERSINFO:-0} -ge 3 ] && [ -r ${MODULESHOME}/init/bash_completion ]; then
 . ${MODULESHOME}/init/bash_completion
fi

So, before going into the deep in the script above;

Note that :
The modules system is based on modulefiles,which specify groups of environment settings that need to be made together. Modulefiles can be installed in a central location for general use, or in a user directory for personal use. Environment Modules modulefiles are written in the Tcl (Tool Command Language) and are interpreted by the modulecmd program via the module user interface. Environment Modules modulefiles can be loaded, unloaded, or switched on-the-fly while the user is working; and can be used to implement site policies regarding the access and use of applications.


Okay, after this little info it is clear that modules.sh sets our environment variables..
Okay lets, run the modules.sh script and check our environment for the changes.

Hmm..I see the following at the end of my enviroment (checked using command env)

module=() {  eval `/usr/bin/modulecmd sh $*`
}

I have suspected from the curly brace, it is in the second line , but I have checked it, it seems it s not a problem..
[applmgr@ermanprod ~]$ module() { eval `/usr/bin/modulecmd sh $*`;
> }
no errors..

So afer setting this module function; lets export it to our shell environment;

[applmgr@ermanprod ~]$ export -f module

no problems again...

So by declaring and extracting our shell function named module(), we are now able to use it directly from the shell..
That 's why, when we run module command in our shell, the module function declared above will run, and it will call modulecmd as it is declared so.
The command to be investigated is "/usr/bin/modulecmd sh $*"

So lets start with the meaning of $*...

"$*" is equivalent to "$1c$2c...", where c is the first character of the
              value of the IFS variable.  If IFS is unset, the parameters are separated by spaces.  If IFS is null, the parameters are joined without  intervening  separators.

So what does IFS variable mean?

The IFS is a special shell variable.
You can change the value of IFS as per your requirments.
The Internal Field Separator (IFS) that is used for word splitting after expansion and to split lines into words with the read builtin command.

In fact, I m suspecting from the following multi-line environment function, and it should be the cause because we see an EOF error in our error message..
[applmgr@ermanprod ~]$ module() { eval `/usr/bin/modulecmd sh $*`;
> }
Eventhough it does not produce any errors, it is defined there.. It is multi- line.. I guess  weblogic start routine processes the environment variable in some way that it can not handle multi-lines and encounters the following error because of this multi-line environment function(module)

In brief, with the gathered knowledge; I guess that Weblogic start process cant handle multi-line environment variables or functions.. 

No comments :

Post a Comment