ODA virtualized-- ODA_BASE, oakd , oakcli OAKERR:6002 No master node was found

You may encounter this problem when trying to use oakcli in ODA BASE nodes.
If you hit this, any oakcli command will return OAKERR:6002 , as it is caused by the lack of the oakd process.
In order to use oakcli, the oakd process must be up and running on at least one of ODA BASE nodes.
If for some reason, the oakd is stopped, then you can't run oakcli commands as shown the command outputs below;

[root@ermoravmsrv1 ~]# oakcli show vm
OAKERR:6000 Exception encountered in Module getMasterNode
OAKERR:6002 No master node was found

[root@ermoravmsrv1 ~]# oakcli show ismaster
Failed to connect to oakd.

So , the standard solution for this problem is restarting the oakd using the oakcli, but in reality, oakcli can not handle that.
Well, when this happens , we need to run oakd manually and save the day.

Here is the environment and the command that I have used for starting the oakd manually.
Note that: The environment setting is important, as without a proper environment, you can't run oakd..

export ORA_OAK_HOME=/opt/oracle/oak
export ORA_CRS_HOME=/u01/app/11.2.0.4/grid
export CONSOLE=/dev/console
export NODENUM=0
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/oracle/oak/lib:/usr/lib64/sun-ssm/storagelibs/sdks:/u01/app/11.2.0.4/grid/lib
export ORA_NLS10=/opt/oracle/oak/nls/data
export SUN_HMP_LIBS=/usr/lib64/sun-ssm/storagelibs/sdks
export OAK_MS_DEBUG=1
export PATH=$PATH:/bin:/usr/bin:/sbin:/usr/sbin
export ORACLE_HOME=/u01/app/12.1.0.2/grid

[root@ermoravmsrv1 bin]#cd /opt/oracle/oak/bin
[root@ermoravmsrv1 bin] nohup ./oakd foreground &

--wait a little(30 secs) for oakd to initialize itself.


As shown below, after starting the oakd manually from the command line,  oakcli show vm command, could do its job properly.

[root@ermoravmsrv1 bin]# oakcli show vm

          NAME                                  NODENUM         MEMORY          VCPU            STATE           REPOSITORY
   
        EBS_12_2_3_PROD_APP                     0               17384              8            ONLINE          vmrepo1                
        EBS_12_2_3_PROD_DB                      0               65536             16            ONLINE          vmrepo1                
        EBS_12_2_3_TEST_APP                     1               17384              8            ONLINE          vmrepo1                
        EBS_12_2_3_TEST_DB                      1               65536              8            ONLINE          vmrepo1                
        EBS_12_2_3_VISION                       0               48000             16            ONLINE          vmrepo1                

So, the fix is to start the oakd manually, but a proper and stable fix is actually restarting the oda_base, which requires downtime.

Without a oda_base restart, oakcli can work but it can not restart the oakd, which is manually started. That's why, we recommend restarting the oda_base to make the system run as it is supposed to run.

Why oakcli can not restart oakd , even if it can be started manually? 

The answer of this question lies on the code, because there are no logs generated when this happens. I wish I could go in to the scripts and code but had no time for reviewing the code, perl scripts and configuration files. 

So, if you encounter this problem, apply my woraround and request a downtime for restarting the oda_base.