EBS 12.2 - About Java Visual VM -- jvisualvm

Here is very interesting tool that comes by default with the EBS 12.2 installation.

It is called Java VisualVM(jvisualvm) and it is a GUI that gives us detailed info about Java applications running on a given JVM.

Using jvisualvm, we can do monitoring, troubleshooting, and profiling. For example, we can check heap sizes of oacore server, we can trigger garbage collector or we dump java threads.

In EBS 12.2, the jvisualvm is located under $COMMON_TOP.

We can use it via a Vnc session. All we have to do is to execute "$COMMON_TOP/util/jdk64/bin/jvisualvm"

                                                               

  Java Visual VM initializing...

Monitoring Tab for Oacore Server

Check Java SE documentation for further info:

http://docs.oracle.com/javase/6/docs/technotes/tools/share/jvisualvm.html

EBS -- Excessive redo/archive generation due to "Initialization SQL Statement - Custom"

A few days ago, I analyzed an excessive redo generation problem in an EBS 12.2 environment which was recently upgraded from 11i.

The problem was the redo generation, as more than 100Gbs of Redo was generated per day and this has started to become a critical problem since, the dataguard transport was saturating the network.

When we analyzed the generated redologs/archives using Log Miner, we saw the following;

/* No SQL_REDO for temporary tables */

This comment is normally seen when there is a redo generated for a temporary table, so there must be an index on it. Becauase, Oracle don't generate redo  for temporary tables unless there are indexes on them.

In order to find the table, we used the following query;

SELECT sql_redo, table_name, COUNT (*), operation_code
FROM v$logmnr_contents
GROUP BY sql_redo, table_name,operation_code
HAVING COUNT (*) > 10

--operation_code=1 -> insert
--operation_code=2 -> delete

--operation_code=3 -> update

--It was the delete operation causing excessive redo in our case...

So, at the end we concluded the table GL_BIS_SEGVAL_INT for the cause. (the count for GL_BIS_SEGVAL_INT in table v$logmnr_contents was much higher than the others)

The redologs and archives which were created during the problematic period was containing the redo for GL_BIS_SEGVAL_INT.

Using this info, we reached the document named: Table GL.GL_BIS_SEGVAL_INT Generates a Lot of Archive Redo (Doc ID 2161480.1)

The solution suggested in the document was -> setting "Initialization SQL Statement - Custom" to NULL at site level (remove value BEGIN GL_SECURITY_PKG.INIT; END;).

When we checked the  Initialization SQL Statement - Custom, we saw that we had the following set there;

BEGIN IF (fnd_profile.VALUE ('XLA_MO_SECURITY_PROFILE_LEVEL') = 0) THEN mo_global.init ('S'); ELSE GL_SECURITY_PKG.init (); mo_global.init ('M'); END IF;END;

This profile was set in order to make Discoverer work properly with MOAC (Multiple Organization Access Control) and Developers didn't want to set it null, as it would affect the Discoverer.

So, rather than setting the "Initialization SQL Statement - Custom" to null, we set it to the following
->

BEGIN IF (fnd_profile.VALUE ('XLA_MO_SECURITY_PROFILE_LEVEL') = 0) THEN mo_global.init ('S'); ELSE mo_global.init ('M'); END IF;END;

So, we only remove the GL_SECURITY_PKG.init () from the profile and with this setting; Discoverer could continue to work and the archive/redo generation dramatically decreased and the system went back to normal.

The cause seemed to be the  the GL_SECURITY_PKG.INIT, as it had a delete statement inside "delete from GL_BIS_SEGVAL_INT" and it had insert statement as well..

 sql_stmt := 'INSERT INTO GL_BIS_SEGVAL_INT( ' ||

                    'segment_column_name,' ||

                    'segment_value,      ' ||

                    'parent_segment) ' ||

                    'SELECT ''' || segment_column_name || ''',' ||

                    'flex_value, parent_flex_value_low ' ||

                    'FROM FND_FLEX_VALUES ' ||

                    'WHERE flex_value_set_id=' || value_set_id;

So , these statements were executed in every login (the profile was set in site level) and as the GL_BIS_SEGVAL_INT temporary table had index on it, we ended up with lots of redo generated..

Anways, we implemented the solution and now all is fine :) 

Interesting isn't it :) Sessions doing dml on Global tables do generate redo ( undo for table(just a small amount) + undo for indexes), and setting  "Initialization SQL Statement - Custom" profile to a standard initialization routine can create a big throuble...

RDBMS -- licensing - 10 days rule

In this blog post, I will give you an info about licensing.

It is a crucial info for some of the Oracle customers, because a misunderstanding of it, may bring them in a situation where they will find themselves lacking the database licenses.

I m writing about the "10 days rule", because I heard the people speaking about it and I witnessed that they are getting it wrong. This misconception is mostly on the Dataguard licensing side...

Anyways, 10 days rule says: "you can use your cold failover node (the passive one) as unlicensed for up to total of ten seperate days in any given calendar year".

However, this cold failover environment used in the sentence above, is actually used for describing the passive nodes in active-passive Clustered environments, such as a Windows cluster.

Morever, this rule is only applicable for the cluster environments,on which the nodes have access to one Single Storage/SAN. Well... 10 days rule has nothing to do with dataguard/standby licensing, even if the standby in question is what it is called as a cold standby, which is not used as the primary at all.

In summary, if you have a standby database, then you need to get license for it. (even if you don't use it as a primary database at all)

Reference: http://www.oracle.com/us/corporate/pricing/data-recovery-licensing-070587.pdf

ODA -- quick tip for CPU & VCPU counts + Named User Plus Licensing

In this blog post, I will give you 2 tips. 1) explanation of CPU and VCPU s in an OD Aenvironment 2) Named user Licensing.

Let's start with the first one...

I have seen that this virtual cpu thing in ODA Virtualized environments can be a little bit confusing sometimes.

It is an important subject though, because it is crucial for licensing. (especially the ODA BASE side)

Yes, it may be confusing on paper, but it is very easy to understand when we do the following analysis.

Note: the system that I used for the analysis in this note is an ODA X4.

We go to dom0 and check if hyper threading is enabled;

[root@odavm1 ~]# dmidecode -t processor | grep HTT
HTT (Hyper-threading technology)
HTT (Hyper-threading technology)

-- It is enabled. why 2 lines? Because there are 2 cpus. (you'll see in the next paragraph)

Anyways, we use the following command to check cpu core and socket counts. 

--on dom0 ->

[root@odavm1 ~]#  dmidecode -t 4 | egrep -i "Designation|Intel|core|thread"

        Socket Designation: P0 -- this is the 1 st cpu

        Manufacturer: Intel

                HTT (Hyper-threading technology)

        Version: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz

        Core Count: 12

        Core Enabled: 12

        Thread Count: 24 --as a result of hyper threading

        Socket Designation: P1 -- this is the 2 nd cpu

        Manufacturer: Intel

                HTT (Hyper-threading technology)

        Version: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz

        Core Count: 12

        Core Enabled: 12

        Thread Count: 24   --as a result of Hyper Threading

So, we see a single ODA node has 2 CPUs, each of these CPUs has 12 cores and 24 threads (because of the HTT)

In summary;

1 CPU = 12 core
2 CPU = 24 core
(1 core = 2 VCPU (hyperthreading))

Total VCPU per  ODA node is 24x2 VCPU = "48 VCPU"

Total VCPU of an ODA X4 HA machine is = "96 VCPU "

Well.. We have 96 VCPUs to distribute along our virtual machines. 

We can assign these VCPUs to our ODA Base and all other VM Machines running on our VM user domains.

The licensing, however; is based on the physical cores.

1 VCPU equals to  1/2 physical core and that's why, when we assign 8 cpu cores for our ODA Base nodes, we see 16 Virtual cpus assigned to each of our ODA Base nodes.

[root@odavm1 ~]# oakcli configure oda_base

Core Licensing Options:
1. 2 CPU Cores
2. 4 CPU Cores
3. 6 CPU Cores
4. 8 CPU Cores
5. 10 CPU Cores
6. 12 CPU Cores
7. 14 CPU Cores
8. 16 CPU Cores
9. 24 CPU Cores
Current CPU Cores :8

....

......

[root@odabasenode1 oak]# cat /proc/cpuinfo |grep processor

processor : 0
processor : 1
processor : 2
processor : 3
processor : 4
processor : 5
processor : 6
processor : 7
processor : 8
processor : 9
processor : 10
processor : 11
processor : 12
processor : 13
processor : 14
processor : 15

In other words, we license our database running on our ODA Base according to the value that we see/set using  "oakcli configure oda_base" command.

Another confusing thing about licensing is the Named User licensing (Named User Plus).

We may think that, we can enable all of the CPU cores if we have a Named User Plus License .

(i.e if we have 25 named user license and we can run our database on any server ( 24 core, 48 core etc..) -> "This is actually wrong!!!"

Here is the reference for this. (It is from Oracle) :

(note that this is for Enterprise Edition)

The Enterprise Edition requires a minimum of 25 Named User Plus per Processor licenses or the total number of actual users, whichever is greater. Example: A customer who wants to license the Database Enterprise Edition on a 4-way box will be required to license a minimum of 4 processors * 25 Named User Plus, which is equal to 100 Named User Plus. 

So, there is a relation between User Named Plus and Cpu based licensing.

When licensing our database by a User Named Plus license, we also need to take our CPU cores into account.

For instance; if we have 200 Named user license, we can use up to 8 Processors.

Likewise,  if we have 100 Named User License, our database server can have a maximum of 4 Processors.
In the case of Intel CPUs, we divide this processor count to 2 for deriving the cpu licenses... In the examples above, having 8 processor requires 4 cpu licenses (in case of Intel). Likewise, having 4 processor requires 2 cpu licenses (in case of Intel)

You can read more on licensing -> http://www.oracle.com/us/corporate/pricing/databaselicensing-070584.pdf

Support -- about "Erman Arslan's Oracle Forum"

I have written this blog since 03.14.2013. I didn't only try to write as much as I can, but also be picky in choosing my topics. I tried to write unique articles to add a value to Oracle documentation, general community and support. In August 2014, I created a forum to give a voluntarily remote support. Firstly, it was like an extension of my blog, but as the time passes, it has become a very important motivation of mine.
I must admit that; since the last 5-6 months, I have started to spend a lot of time for this forum and remote support. (maybe more than the time I spent for writing blog posts)
Although it is hard to mantain this kind of a continous support in parallel to my work, I m pleased to see the increased count of solved issues.
More than 380 issues solved already!
The quality of this remote support , which has increased as the time passes, is also another motivation of mine.
So, it is working!!
You can ask questions and get remote support using my forum.
Just click on the link named "Erman Arslan's Oracle Forum is available now. Click here to ask a question", which is available on the main page of Erman Arslan's Oracle Home or just use the direct link: http://ermanarslan.blogspot.com.tr/p/forum.html

EBS 12.2 -- Tip of the day: ORA-00955 Creating materialized view, Edition Based Redefinition

Recently an interesting issue reported to me.
There was a case, where ORA-00955 was encountered while creating a custom MV on a EBS 12.2 environment.

The issue was reported through my forum: (Here is the link of the issue : http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/Another-interesting-issue-td1918.html)

I guess the environment in question was managed by following the new deployment standards of EBS 12.2. I mean the EBR and online patching related rules of the EBS 12.2 deployment processes were taking into account while deploying the customization and after an online patching session, it seems they started to get ORA-00955 while trying to create a custom MV.
The object in question was not in dba_object, but it was there sys._CURRENT_EDITION_OBJ with type# 10.
I wanted to be sure about the existence of the object and directly recommended to enable trace for it.

alter session set events '955 trace name errorstack level 10';
alter session set events '10046 trace name context forever, level 12';

The problem interestingly solved after running a full cleanup (adop phase=cleanup cleanup_mode=full), which taught us the following;

"adop full cleanup does all the cleanup tasks for objects and it is not only required after aborting an online patching cycle, but also in some other situations like the one described in this blog post"

It is better to keep this in mind...

ODA -- Replacing a failed Disk, missing disk diagnotics, Firmware Upgrade required or not?

The environment that is the subject of this article is a Bare Metal ODA X5 machine.
In this blog post, I will go through a real life case, where we needed to replace a failed hard drive of an ODA X5 machine.

The cause that made me write this blog post, is actually a lack of a crucial info. You will understand what I mean when I will go in to the details of our case, but first  take a look at the process of changing a disk drive in an ODA machine.

The disk replacement on ODA is easy.

We basically follow the MOS document named: How to Replace an ODA (Oracle Database Appliance) FAILED/ PredictiveFail Shared Storage Disk ( Doc ID 1435946.1 )

In case of ODA, we(customers or consultants) are replacing the disks , no need for Oracle Field engineers for that. That's why, the disk components are called CRU's (Customer Replacable Units)

The actions to be taken for replacing a failed disk can be summarized as follows;

• Identify the failed disk (do the diagnostics)
• Take out the failed fisk
• Wait 2-3 mins
• Attach the new disk
• Check the conditions and take post actions if necessary

Physically replacing the disk is so easy, that's why I will not concantrate on that in this article.

However; the steps: identifying the failed disk and the post actions are interesting. So here in this post; I will give some details about them.

Before the replacement, we just check the failed disk and be sure that it is in failed status (STATE_DETAILS=DiskRemoved or PredictiveFail)

We use oakcli show disk command for checking the disks and their status.

Note that, the failed disk may not be there in the output and if this happens is totally fine. 

It means the failed disk is removed.

In this example, I m doing my check for a failed disk drive on slot 10 ( the slot number can be obtained by looking at the machine rack itself as well).  

Here, first I use the oakcli show disk command and see the disk in slot 10 is not there at all.

[root@ermanoda0 ~]# oakcli show disk

        NAME            PATH            TYPE            STATE           STATE_DETAILS

        e0_pd_00        /dev/sdc        HDD             ONLINE          Good           

        e0_pd_01        /dev/sdd        HDD             ONLINE          Good           

        e0_pd_02        /dev/sde        HDD             ONLINE          Good           

        e0_pd_03        /dev/sdf        HDD             ONLINE          Good           

        e0_pd_04        /dev/sdg        HDD             ONLINE          Good           

        e0_pd_05        /dev/sdh        HDD             ONLINE          Good           

        e0_pd_06        /dev/sdi        HDD             ONLINE          Good           

        e0_pd_07        /dev/sdj        HDD             ONLINE          Good           

        e0_pd_08        /dev/sdaa       HDD             ONLINE          Good           

        e0_pd_09        /dev/sdac       HDD             ONLINE          Good

---Attention no output for e0_pd_10--           

        e0_pd_11        /dev/sdag       HDD             ONLINE          Good           

        e0_pd_12        /dev/sdai       HDD             ONLINE          Good           

        e0_pd_13        /dev/sdak       HDD             ONLINE          Good           

        e0_pd_14        /dev/sdam       HDD             ONLINE          Good           

        e0_pd_15        /dev/sdao       HDD             ONLINE          Good           

        e0_pd_16        /dev/sdab       SSD             ONLINE          Good           

        e0_pd_17        /dev/sdad       SSD             ONLINE          Good           

        e0_pd_18        /dev/sdaf       SSD             ONLINE          Good           

        e0_pd_19        /dev/sdah       SSD             ONLINE          Good           

        e0_pd_20        /dev/sdaj       SSD             ONLINE          Good           

        e0_pd_21        /dev/sdal       SSD             ONLINE          Good           

        e0_pd_22        /dev/sdan       SSD             ONLINE          Good           

        e0_pd_23        /dev/sdap       SSD             ONLINE          Good       

As I see the disk is not there in the output, I m doing more checks to be sure that the disk is not seen by OS or any other software component on ODA system.

--I do my checks on both of the ODA nodes..

First, checking the multipath devices;

multipath -ll output:

HDD_E0_S10_992975636 (35000cca23b2f9b14) dm-14 

size=7.2T features='0' hwhandler='0' wp=rw

"no disk paths listed here"

Well. The failed disk's multipath device name should be dm-14, as there is no disk paths listed for it.

I also don't see any slaves for it..

cd /sys/block/dm-14/slaves

ls -al

total 0

drwxr-xr-x 2 root root 0 Aug  1 10:28 .

drwxr-xr-x 8 root root 0 Mar  1  2016 ..

No devices...

This should give the real device names..

For ex: working a dm:

cd /sys/block/dm-14/slaves

ls -al

total 0

drwxr-xr-x 2 root root 0 Aug  1 10:28 .

drwxr-xr-x 8 root root 0 Mar  1  2016 ..

lrwxrwxrwx 1 root root 0 Jan  3 18:19 sde -> ../../sde

lrwxrwxrwx 1 root root 0 Jan  3 18:19 sdo -> ../../sdo

[root@ermanoda0 mapper]# ls -lrt|grep S10

brw-rw---- 1 grid asmadmin 252,  14 Mar  1  2016 HDD_E0_S10_992975636

brw-rw---- 1 grid asmadmin 252,  49 Jul 24 17:26 HDD_E0_S10_992975636p2

brw-rw---- 1 grid asmadmin 252,  36 Sep 24 09:59 HDD_E0_S10_992975636p1

Well.. the multipath devices are there,. But these are multipath devices right? They are not physical. So it is not a bad thing.

Continuing my diagnostics..

Next, I check the OAK logs;

log/ermanoda0/oak/oakd.l45:2016-12-15 11:22:08.886: [CLSFRAME][4160715072]{0:35:2} payload=|OAKERR : 9009 : Couldn't find the resource: e0_pd_10||

log/ermanoda0/oak/oakd.l45:2016-12-15 11:22:08.886: [CLSFRAME][4160715072]{0:35:2} String params:CmdUniqId=|ServiceName=e0_pd_10|pname=Error|

log/ermanoda0/oak/oakd.l45:2016-12-15 11:22:08.886: [   OAKFW][4160715072]{0:35:2} PE sending last reply for: MIDTo:1|OpID:1|FromA:{Relative|Node:0|Process:35|Type:2}|ToA:{Relative|Node:0|Process:0|Type:1}|MIDFrom:4|Type:1|Pri2|Id:8:Ver:2Value params:payload=|OAKERR : 9009 : Couldn't find the resource: e0_pd_10||String params:CmdUniqId=|ServiceName=e0_pd_10|pname=Error|Int params:ErrCode=0|MsgId=4359|flag=2|sflag=64|

OAK says, I can't find the e0_pd_10 resource, which actually corresponds to our failed disk. So this is normal.

In OAKCLI logs, I see the Disk Removed state for our failed disk, which is totally expected.

log/ermanoda0/client/oakcli.log:2016-08-01 11:02:51.999: [  OAKCLI][2575886656]     e0_pd_10        /dev/sdae       HDD             FAILED          DiskRemoved  

I check the physical device name, and it is not there (as expected)

--> ls-al /dev/sdae

ls: /dev/sdae: No such file or directory

I also check the fishwrap logs and see that the failed disk is deleted.

log/fishwrap/fishwrap.log:Sun Jul 24 17:27:39 2016: deleting an old disk: /dev/sg17

In Fishwrap log: Sun Jul 24 17:27:39 2016: Slot [10] sas-addr = 5000cca23b2f9b16

Sun Jul 24 17:27:39 2016: fwr_scsi_tree_topology_update finish, device num = 51

In this log, I see that, the SCSI deviced count was 53 before the failed disk. After the failed disk it have become 51. (this is also expected)

EARLIER:

Tue Mar  1 14:13:37 2016: Number of SCSI device found = 53, existing = 53 

NOW:

Sun Jul 24 17:27:38 2016: Number of SCSI device found = 51, existing = 53

Sun Jul 24 17:27:38 2016: fwr_scsi_tree_topology_update: expander update start

I execute a Storage Diagnostics and see the following in its output;

 8  : fwupdate

          [INFO]: fwupdate does not see disk from both controllers

   9  : Fishwrap

          [INFO]: Fishwrap not able to discover disk

  10  : Check for shared disk write cache status

          [INFO]: Unable to find OS devices for slot 10

  11  : SCSI INQUIRY

          [INFO]: Unable to run scsi inquiry command on disk as OS device are absent

          [INFO]: Unable to run scsi inquiry command on disk as OS device are absent

  12  : Multipath Conf for device    

           multipath {

             wwid 35000cca23b2f9b14

             alias HDD_E0_S10_992975636

       }

  13  : Last few LSI Events Received for slot 10

          [INFO]: No LSI events are recorded in OAKD logs

  14  : Version Information

          OAK              :  12.1.2.4.0

          kernel           :  2.6.39-400.250.6.el5uek

          mpt2sas          :  17.00.06.00

          Multipath        :  0.4.9  

  15  : OAK Conf Parms

          [INFO]: No scsi devices found for slot 10

In summary; 

I only see the Multipath device names are present for the failed disk and all the other things that are related with the failed disk are removed.

That 's why, I conclude that the failed disk is eliminated from OS and OAK.

So, the disk is ready to be replaced and the environment was totally in an expected state. 

I did such diagnostics to ensure the disk is removed properly ,because I could not see the failed disk with a failure status in oakcli output ... (Remember oakcli show disk doesn't report this disk at all)

So, oak removed the disk and it is not listing any more. Maybe it was the behaviour of it after a reboot, but anyways; there is a thing that needs to be added to the documentation right? :)

Well. After the disk is replaced online, every OS and OAK related thing is done automatically and transparently. Here is the status of the checks;

fwupdate list disk

===============

ID        Manufacturer   Model               Chassis Slot   Type   Media   Size(GiB) FW Version XML Support

-----------------------------------------------------------------------------------------------------------

c2d0      HGST           H7280A520SUN8.0T    0       0      sas    HDD     7325      P554       N/A        

c2d1      HGST           H7280A520SUN8.0T    0       1      sas    HDD     7325      P554       N/A        

c2d2      HGST           H7280A520SUN8.0T    0       2      sas    HDD     7325      P554       N/A        

c2d3      HGST           H7280A520SUN8.0T    0       3      sas    HDD     7325      P554       N/A        

c2d4      HGST           H7280A520SUN8.0T    0       4      sas    HDD     7325      P554       N/A        

c2d5      HGST           H7280A520SUN8.0T    0       5      sas    HDD     7325      P554       N/A        

c2d6      HGST           H7280A520SUN8.0T    0       6      sas    HDD     7325      P554       N/A        

c2d7      HGST           H7280A520SUN8.0T    0       7      sas    HDD     7325      P554       N/A        

c2d8      HGST           H7280A520SUN8.0T    0       8      sas    HDD     7325      P554       N/A        

c2d9      HGST           H7280A520SUN8.0T    0       9      sas    HDD     7325      P554       N/A        

c2d10     HGST           H7280A520SUN8.0T    0       10     sas    HDD     7325      P9E2       N/A        

c2d11     HGST           H7280A520SUN8.0T    0       11     sas    HDD     7325      P554       N/A        

c2d12     HGST           H7280A520SUN8.0T    0       12     sas    HDD     7325      P554       N/A        

c2d13     HGST           H7280A520SUN8.0T    0       13     sas    HDD     7325      P554       N/A        

c2d14     HGST           H7280A520SUN8.0T    0       14     sas    HDD     7325      P554       N/A        

c2d15     HGST           H7280A520SUN8.0T    0       15     sas    HDD     7325      P554       N/A        

c2d16     HGST           HSCAC2DA4SUN400G    0       16     sas    SSD     373       A29A       N/A       

c2d17     HGST           HSCAC2DA4SUN400G    0       17     sas    SSD     373       A29A       N/A       

c2d18     HGST           HSCAC2DA4SUN400G    0       18     sas    SSD     373       A29A       N/A       

c2d19     HGST           HSCAC2DA4SUN400G    0       19     sas    SSD     373       A29A       N/A       

c2d20     HGST           HSCAC2DA6SUN200G    0       20     sas    SSD     186       A29A       N/A       

c2d21     HGST           HSCAC2DA6SUN200G    0       21     sas    SSD     186       A29A       N/A       

c2d22     HGST           HSCAC2DA6SUN200G    0       22     sas    SSD     186       A29A       N/A       

c2d23     HGST           HSCAC2DA6SUN200G    0       23     sas    SSD     186       A29A       N/A  

[root@ermanoda0 ~]# oakcli show disk

        NAME            PATH            TYPE            STATE           STATE_DETAILS 

        e0_pd_00        /dev/sdc        HDD             ONLINE          Good           

        e0_pd_01        /dev/sdd        HDD             ONLINE          Good           

        e0_pd_02        /dev/sde        HDD             ONLINE          Good           

        e0_pd_03        /dev/sdf        HDD             ONLINE          Good           

        e0_pd_04        /dev/sdg        HDD             ONLINE          Good           

        e0_pd_05        /dev/sdh        HDD             ONLINE          Good           

        e0_pd_06        /dev/sdi        HDD             ONLINE          Good           

        e0_pd_07        /dev/sdj        HDD             ONLINE          Good           

        e0_pd_08        /dev/sdaa       HDD             ONLINE          Good           

        e0_pd_09        /dev/sdac       HDD             ONLINE          Good           

        e0_pd_10        /dev/sdp        HDD             ONLINE          Good           

        e0_pd_11        /dev/sdag       HDD             ONLINE          Good           

        e0_pd_12        /dev/sdai       HDD             ONLINE          Good           

        e0_pd_13        /dev/sdak       HDD             ONLINE          Good           

        e0_pd_14        /dev/sdam       HDD             ONLINE          Good           

        e0_pd_15        /dev/sdao       HDD             ONLINE          Good           

        e0_pd_16        /dev/sdab       SSD             ONLINE          Good           

        e0_pd_17        /dev/sdad       SSD             ONLINE          Good           

        e0_pd_18        /dev/sdaf       SSD             ONLINE          Good           

        e0_pd_19        /dev/sdah       SSD             ONLINE          Good           

        e0_pd_20        /dev/sdaj       SSD             ONLINE          Good           

        e0_pd_21        /dev/sdal       SSD             ONLINE          Good           

        e0_pd_22        /dev/sdan       SSD             ONLINE          Good           

        e0_pd_23        /dev/sdap       SSD             ONLINE          Good          

Even the multipath.conf file is updated automatically;

    multipath {

      wwid 35000cca2604a90ac   --> this is the wwid of the newly added disk

      alias HDD_E0_S10_1615499436

}

[root@ermanoda0 etc]# lsscsi

[0:2:0:0]    disk    LSI      MR9361-8i        4.23  /dev/sda 

[7:0:0:0]    disk    ORACLE   SSM              PMAP  /dev/sdb 

[8:0:0:0]    enclosu ORACLE   DE2-24C          0018  -       

[8:0:1:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdc 

[8:0:2:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdd 

[8:0:3:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sde 

[8:0:4:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdf 

[8:0:5:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdg 

[8:0:6:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdh 

[8:0:7:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdi 

[8:0:8:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdj 

[8:0:9:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdl 

[8:0:10:0]   disk    HGST     H7280A520SUN8.0T P554  /dev/sdn 

[8:0:12:0]   disk    HGST     H7280A520SUN8.0T P554  /dev/sdr 

[8:0:13:0]   disk    HGST     H7280A520SUN8.0T P554  /dev/sdt 

[8:0:14:0]   disk    HGST     H7280A520SUN8.0T P554  /dev/sdv 

[8:0:15:0]   disk    HGST     H7280A520SUN8.0T P554  /dev/sdx 

[8:0:16:0]   disk    HGST     H7280A520SUN8.0T P554  /dev/sdz 

[8:0:17:0]   disk    HGST     HSCAC2DA4SUN400G A29A  /dev/sdab

[8:0:18:0]   disk    HGST     HSCAC2DA4SUN400G A29A  /dev/sdad

[8:0:19:0]   disk    HGST     HSCAC2DA4SUN400G A29A  /dev/sdaf

[8:0:20:0]   disk    HGST     HSCAC2DA4SUN400G A29A  /dev/sdah

[8:0:21:0]   disk    HGST     HSCAC2DA6SUN200G A29A  /dev/sdaj

[8:0:22:0]   disk    HGST     HSCAC2DA6SUN200G A29A  /dev/sdal

[8:0:23:0]   disk    HGST     HSCAC2DA6SUN200G A29A  /dev/sdan

[8:0:24:0]   disk    HGST     HSCAC2DA6SUN200G A29A  /dev/sdap

[8:0:25:0]   disk    HGST     H7280A520SUN8.0T P9E2  /dev/sdp 

[9:0:0:0]    enclosu ORACLE   DE2-24C          0018  -       

[9:0:1:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdk 

[9:0:2:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdm 

[9:0:3:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdo 

[9:0:4:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdq 

[9:0:5:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sds 

[9:0:6:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdu 

[9:0:7:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdw 

[9:0:8:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdy 

[9:0:9:0]    disk    HGST     H7280A520SUN8.0T P554  /dev/sdaa

[9:0:10:0]   disk    HGST     H7280A520SUN8.0T P554  /dev/sdac

[9:0:12:0]   disk    HGST     H7280A520SUN8.0T P554  /dev/sdag

[9:0:13:0]   disk    HGST     H7280A520SUN8.0T P554  /dev/sdai

[9:0:14:0]   disk    HGST     H7280A520SUN8.0T P554  /dev/sdak

[9:0:15:0]   disk    HGST     H7280A520SUN8.0T P554  /dev/sdam

[9:0:16:0]   disk    HGST     H7280A520SUN8.0T P554  /dev/sdao

[9:0:17:0]   disk    HGST     HSCAC2DA4SUN400G A29A  /dev/sdaq

[9:0:18:0]   disk    HGST     HSCAC2DA4SUN400G A29A  /dev/sdar

[9:0:19:0]   disk    HGST     HSCAC2DA4SUN400G A29A  /dev/sdas

[9:0:20:0]   disk    HGST     HSCAC2DA4SUN400G A29A  /dev/sdat

[9:0:21:0]   disk    HGST     HSCAC2DA6SUN200G A29A  /dev/sdau

[9:0:22:0]   disk    HGST     HSCAC2DA6SUN200G A29A  /dev/sdav

[9:0:23:0]   disk    HGST     HSCAC2DA6SUN200G A29A  /dev/sdaw

[9:0:24:0]   disk    HGST     HSCAC2DA6SUN200G A29A  /dev/sdax

[9:0:25:0]   disk    HGST     H7280A520SUN8.0T P9E2  /dev/sdae

[root@ermanoda0 etc]# oakcli show disk e0_pd_10

Resource: e0_pd_10

        ActionTimeout   :       1500           

        ActivePath      :       /dev/sdp       

        AsmDiskList     :       |e0_data_10||e0_reco_10|

        AutoDiscovery   :       1              

        AutoDiscoveryHi :       |data:43:HDD||reco:57:HDD||redo:100

                                :SSD||flash:100:SSD|

        CheckInterval   :       300            

        ColNum          :       2              

        DependListOpr   :       add            

        Dependency      :       |0|            

        DiskId          :       35000cca2604a90ac

        DiskType        :       HDD            

        Enabled         :       0              

        ExpNum          :       0              

        IState          :       0              

        Initialized     :       1              

        IsConfigDepende :       false          

        MonitorFlag     :       0              

        MultiPathList   :       |/dev/sdae||/dev/sdp|

        Name            :       e0_pd_10       

        NewPartAddr     :       0              

        OSUserType      :       |userType:Multiuser|

        PrevState       :       UnInitialized  

        PrevUsrDevName  :       HDD_E0_S10_1615499436

        SectorSize      :       512            

        SerialNum       :       001634PA07WV   

        Size            :       7865536647168  

        SlotNum         :       10             

        State           :       Online         

        StateChangeTs   :       1483596721     

        StateDetails    :       Good           

        TotalSectors    :       15362376264    

        TypeName        :       0              

        UsrDevName      :       HDD_E0_S10_1615499436

        gid             :       0              

        mode            :       660            

        uid             :       0              

}

It is also seen in OS logs... The disk is discovered properly by OS;

/var/log/messages: 

Jan  5 08:10:29 ermanoda0 kernel: mpt3sas0: detecting: handle(0x0024), sas_address(0x5000cca2604a90ae), phy(10)

Jan  5 08:10:35 ermanoda0 kernel: scsi 9:0:25:0: Direct-Access     HGST     H7280A520SUN8.0T P9E2 PQ: 0 ANSI: 6

Jan  5 08:10:35 ermanoda0 kernel: scsi 8:0:25:0: Direct-Access     HGST     H7280A520SUN8.0T P9E2 PQ: 0 ANSI: 6

Jan  5 08:10:35 ermanoda0 kernel: scsi 8:0:25:0: SSP: handle(0x0024), sas_addr(0x5000cca2604a90ae), phy(10), device_name(0x5000cca2604a90af)

Jan  5 08:10:35 ermanoda0 kernel: scsi 8:0:25:0: SSP: enclosure_logical_id(0x5080020001ecf27e), slot(80)

Jan  5 08:10:35 ermanoda0 kernel: scsi 8:0:25:0: serial_number(001634PA07WV        VLHA07WV)

Jan  5 08:10:35 ermanoda0 kernel: scsi 8:0:25:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)

Jan  5 08:10:35 ermanoda0 kernel: scsi 9:0:25:0: SSP: handle(0x0024), sas_addr(0x5000cca2604a90ad), phy(10), device_name(0x5000cca2604a90af)

Jan  5 08:10:35 ermanoda0 kernel: scsi 9:0:25:0: SSP: enclosure_logical_id(0x5080020001eceb7e), slot(80)

Jan  5 08:10:35 ermanoda0 kernel: scsi 9:0:25:0: serial_number(001634PA07WV        VLHA07WV)

Jan  5 08:10:35 ermanoda0 kernel: scsi 9:0:25:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)

Jan  5 08:10:35 ermanoda0 kernel: sd 8:0:25:0: Attached scsi generic sg17 type 0

Jan  5 08:10:35 ermanoda0 kernel: sd 8:0:25:0: [sdp] Enabling DIF Type 1 protection

Jan  5 08:10:35 ermanoda0 kernel: sd 8:0:25:0: [sdp] 15362376264 512-byte logical blocks: (7.86 TB/7.15 TiB)

Jan  5 08:10:35 ermanoda0 kernel: sd 8:0:25:0: [sdp] 4096-byte physical blocks

Jan  5 08:10:35 ermanoda0 kernel: sd 9:0:25:0: [sdae] Enabling DIF Type 1 protection

Jan  5 08:10:35 ermanoda0 kernel: sd 9:0:25:0: [sdae] 15362376264 512-byte logical blocks: (7.86 TB/7.15 TiB)

Jan  5 08:10:35 ermanoda0 kernel: sd 9:0:25:0: [sdae] 4096-byte physical blocks

Jan  5 08:10:35 ermanoda0 kernel: sd 8:0:25:0: [sdp] Write Protect is off

Jan  5 08:10:35 ermanoda0 kernel: sd 9:0:25:0: [sdae] Write Protect is off

Jan  5 08:10:35 ermanoda0 kernel: sd 9:0:25:0: [sdae] Write cache: disabled, read cache: enabled, supports DPO and FUA

Jan  5 08:10:35 ermanoda0 kernel: sd 8:0:25:0: [sdp] Write cache: disabled, read cache: enabled, supports DPO and FUA

Jan  5 08:10:35 ermanoda0 kernel:  sdae:

Jan  5 08:10:35 ermanoda0 kernel: sd 9:0:25:0: Attached scsi generic sg32 type 0

Jan  5 08:10:35 ermanoda0 kernel:  sdp:

Jan  5 08:10:35 ermanoda0 kernel: sd 9:0:25:0: [sdae] Attached SCSI disk

Jan  5 08:10:35 ermanoda0 kernel: sd 8:0:25:0: [sdp] Attached SCSI disk

On the other hand, there is one thing that still needs to be cleared and it is the firmware of the newly added disk.

This part is actually the part named "Check the conditions and take post actions if necessary".

Here, I use the oakcli show version -detail command to see the installed and supported firmware version of the ODA Components.

[root@ermanoda0 device]# oakcli show version -detail

Reading the metadata. It takes a while...

System Version  Component Name            Installed Version         Supported Version        

--------------  ---------------           ------------------        -----------------        

12.1.2.4.0                                                                                   

                Controller_INT            4.230.40-3739             Up-to-date               

                Controller_EXT            06.00.02.00               Up-to-date               

                Expander                  0018                      Up-to-date               

                SSD_SHARED {                                                                 

                [ c2d20,c2d21,c2d22,      A29A                      A122                     

                c2d23 ]                                                                      

                [ c2d16,c2d17,c2d18,      A29A                      A122                     

                c2d19 ]                                                                      

                             }                                                               

                HDD_LOCAL                 A720                      Up-to-date               

                HDD_SHARED {                                                                 

                [ c2d0,c2d1,c2d2,c2d      P554                      Up-to-date               

                3,c2d4,c2d5,c2d6,c2d                                                         

                7,c2d8,c2d9,c2d11,c2                                                         

                d12,c2d13,c2d14,c2d1                                                         

                5 ]                                                                          

                [ c2d10 ]                 P9E2                      P554                     

                             }                                                               

                ILOM                      3.2.4.42 r99377           Up-to-date               

                BIOS                      30040200                  Up-to-date               

                IPMI                      1.8.12.0                  Up-to-date               

                HMP                       2.3.2.4.1                 Up-to-date               

                OAK                       12.1.2.4.0                Up-to-date               

                OL                        5.11                      Up-to-date               

                GI_HOME                   12.1.0.2.4(20831110,      Up-to-date               

                                          20831113)                                          

                DB_HOME {                                                                    

                [ OraDb12102_home1 ]      12.1.0.2.4(20831110,      Up-to-date               

                                          20831113)                                          

                [ OraDb11203_home1 ]      11.2.0.3.15(20760997      Up-to-date               

                                          ,17592127)                                         

                             }  

As seen in the output above, while the firmware of all the other disks are P554, the newly added disk has P9E2 firmware.
Well... The column named "supported" is showing the firmware P554.
So here is the question which needs to be answered? Is P9E2 newer than P554?

This question needs to be answered. That is, if the firmware version of the newly added disk is older than the supported one, then a Firmware upgrade needs to be done using "oakcli update -patch <patch_bundle_version> --infra" command

Reference from note "How to Replace an ODA (Oracle Database Appliance) FAILED/ PredictiveFail Shared Storage Disk ( Doc ID 1435946.1 )"

"If the newly replaced disk has older firmware from what the ODA Software is expecting, you will need to update the firmware on this disk.
If the disk has newer firmware from the existing disks, this is fine, and the firmware does not need to be downgraded to match existing disks."!!!

However, there is no info about P9E2 on Internet or Oracle Support. 

Actually, there is an info about P901..

Here in note Oracle Database Appliance FAQ: Common Questions Regarding Firmware Versions on ODA (Doc ID 2119003.1), it says: 

The P901 firmware is a newer firmware version for the 8 TB drive, and firmware P554 is the one we released in 12.1.2.5.

So p901> p554 right? I mean 901 is a greater value.

In the same logic p9E2 is greater than p901, but still we create an SR for it.

Here is what Oracle Support(SR) says:

The FirmWare P9E2 of the replaced shard storage disk in slot 10 is the latest FW which has been released with ODA image 12.1.2.8.0 in Sept 2016.

So, the problem solved. No need for firmware upgrade..

At the end of the day, we see the FAQ is not up-to-date and there is no document for getting info about ODA HDD firmware versions (a version compatibility between ODA versions and Component Versions).

Fortunetaly, we at least have Oracle Support :)

ODA & RDBMS-- A look to the prerequisites for "oakcli database upgrade" command and some tips for the RAC Standby Databases

As you already may know, oakcli has a database upgrade feature as well.
That is , 'oakcli upgrade database'  command can be used for doing one command database upgrades on ODA.

Command Syntax:

oakcli upgrade database [-db db_names | -from source_home] -to destination_home [-h]

However, this command requires some prereqs.

That is, we need to use the correct source and target homes. Also, our database should be registered in CRS and /etc/oratab etc..

In this blog post,  I will be consantrated on CRS registry part.

Well... In order to be able to use oakcli upgrade database command, we need to have our database registered with the CRS on ODA.

That is, we should be able to see our database when we issue oakcli show databases command... Well...In order to see our database with oakcli show databases command, we need our database be registered in the cluster registry. (CRS/srvctl configuration)

In other words; our database and instance/instances should be added to the cluster repository.

Once our database is added , we can use oakcli upgrade database command for upgrading it.

Here is a demo for adding a standby to the cluster registry on ODA.

In this example, I m adding a standby database which is residing on an ODA X5 to the cluster registry and then use the oakcli show databases command to see it.
(note that, in this example; I only add a single standby instance...)

At first, the standby database is not registered at all;

[root@marsdb1 bin]# oakcli show version -detail
[root@marsdb1 bin]# oakcli show databases -detail
Name Type Storage HomeName HomeLocation Version HostName
----- ------ -------- -------------- ----------------- ----------- --------------

Then I register the standby database as a physical standby database in to cluster registry as follows;

[oracle@marsdb1 dbs]$ srvctl add database -d PROD -o /u01/app/oracle/product/11.2.0.3/dbhome_1 -p '+DATA/spfilePROD' -r PHYSICAL_STANDBY -s MOUNT

[oracle@marsdb1 dbs]$ srvctl add instance -d PROD -i PROD2 -n marsdb1

[oracle@marsdb1 ~]$ srvctl config database -d PROD

Database unique name: PROD
Database name:
Oracle home: /u01/app/oracle/product/11.2.0.3/dbhome_1
Oracle user: oracle
Spfile: /u01/app/oracle/product/11.2.0.3/dbs/initPROD2.ora
Domain:
Start options: mount
Stop options: immediate
Database role: PHYSICAL_STANDBY
Management policy: AUTOMATIC
Server pools: PROD
Database instances: PROD2
Disk Groups:
Mount point paths:
Services:
Type: RAC
Database is administrator managed

Now, I execute "oakcli show databases" command again and this time it shows me my standby database as seen below;

[root@marsdb1 bin]# oakcli show databases -detail

Name Type Storage HomeName HomeLocation Version HostName
----- ------ -------- -------------- ----------------- ----------- --------------
PROD RAC OraDb11203_home1 /u01/app/oracle/product/11.2.0.3/dbhome_1 11.2.0.3.15(20760997,17592127)

At this point; my standby database can be upgraded using oakcli upgrade database command.. (supposing all the other prereqs are in place)

I want to remind you that, I have used a single instance standby database in this example.

On the other hand, ODA let us have a multi-instance RAC standby database as well.

Remember the following if you are implementing a RAC standby database ;

1) All the standby instances can be started in mount mode (physical standby mount mode), but only one of them can be in recovery mode.

2) Standby Redolog files should be added to the Standby database, for each thread. (multi instance)

3) Primary instances( Primary database environment) must be configured to send redo data to the standby database. (to all the standby database instances or to one of the standby database instances, which can receive the redo data --RFS).. So if we want to send redo data to all of the standby instances, then there should be an address list for connecting to all of the standby nodes/instances in Prod environments tnsnames.ora file.
Note that, even if we configure the address lists on tns files of our primary nodes,  RFS will start only on one of the nodes. (the node which is in the first line of the address list).
In addition, MRP will be started manually by us on the instance of our choice.
That is, MRP will run on one of the instances. (currently MRP can't be enabled on multiple instances -- maybe in the future it will be... maybe in RDBMS 12.2..)

4) On a multi instance (RAC) Standby environment, a node failure may require a human intervention. (if not using DG Broker) . I mean, if we lose the node on which the RFS is running , then we need to do log_archive_dest_state_% = > defer/enable in primary and if we lose the node on which the MRP is running, then we need to start managed recovery on the surviving node.

5) The archives (archive dest) in the standby environment should be reachable by all of the instance.

Oopps, this is going beyond the scope.  I have started to write things about Dataguard configuration.. RAC-standby related info in this level should be the topic of another blog post, right?
Besides, RAC-standby deserves a seperate article.
Well...  I m putting a dot to this blog post, and start writing the new one (RAC-Standby related one) then :)

Linux -- How to mount a Windows Share on Linux using NFS (not CIFS - not SMB)

This blog post will be about mounting Windows shares on Linux Operating Systems.
Actually, I have used this method explained in this post, recently for mounting a Windows Share to a virtualized ODA machine (mounting directly to the Linux ODA Base nodes).
Those who follow me know that I m trying to share unique articles and try to give you the info that can't be easily found on the internet or Oracle Support...
This is also true for this article, although its name sounds quite familiar. I mean, Windows Admins may use it frequently but when it comes to Linux, we are more concantrate on the success of mouting the Windows shares, rather the method that we are using to mount them.
Actually, the main reason that made me writing this post is the hang situation that we have faced when using Windows shares.. (I m talking mounting Windows shares using CIFS , as we have faced sudden and unexpected hang situations using it.)
Another reason is, for mounting the Windows shares on Linux environments, we are using CIFS mostly, so that's why I m writing this post to give you another method for mounting Windows Shares on Linux.

Before going forward, I want to give you the quick definitions of CIFS and NFS.

CIFS (Common Internet File System) is a form of SMB, developed by Microsoft.
Note that: Samba is an implementation of SMB written for UNIX. So Samba allows Unix clients to access CIFS shares. Samba and Smb are two different thing in this manner.
CIFS is used by Windows operating systems for file sharing.
It is based on a Client Server model and it is using TCP/IP.
With CIFS, servers share the storage and clients use or map the storage share.
The most common SMB/CIFS server for Linux is Samba.

NFS is the "Network File System" . NFS was originally developed by Sun and it is often used with Unix operating systems.
NFS has various versions like Nfsv2, v3 and v4. All these versions can use TCP/IP.
Note that, v2 and v3 can use UDP as well.(less protocol overhead than TCP)
With NFS, servers export the storage and clients mount the storage.
The most common Linux NFS server is "nfsd".

Windows Shares supports both CIFS and NFS.
Wel... After this quick info, let's take a look at  what we need to do for sharing our Windows directories using NFS and mounting them on Linux Operating systems.

Fist, we install services for NFS components in our Windows Server.

Click Start, point to Administrative Tools, and then click Server Manager.
In the left pane, click Manage Roles.
Click Add Roles. The Add Roles Wizard appears.
Click Next. The Select Server Roles options appear.
Select the File Server check box and click Next.
The File Server screen appears. Click Next to view the Role Services options.
Select the Services for Network File System (NFS) check box and click Next.
Confirm your selection and click Install.
When the installation completes, the installation results will appear. Click Close.

After we install the services, we share our folder as NFS Service.

Note that, if we just mount the folder (without completing the NFS enablement of our share), we get the error represented in example below:

mount.nfs: mounting 10.34.10.184:/backupca failed, reason given by server: No such file or directory

We open the Share and Storage Management  and click on the Provision Share;

Then we select the folder that we want to share;

We choose NFS for the share protocol, type the name of our share and click Next;

In the NFS authentication window, we just accept the default selections and click Next.

In the NFS Permissions window, we click edit and change permissions to Read-Write. We also check the Allow root access ( not recommended ) checkbox and click Next. (note that, these configuration is just for our specific example, you may do other configurations according to your needs)

As our last work in Windows side, we click Create button in the Review Settings and Create Shared window.

Well.. At this point, our work in Windows side is finished. So, we go to our Linux client and mount the NFS share from Linux using the standard mount commands as shown in the example below;

Mounting NFS share on Linux

We just run command below and mount our Windows Nfs share.

mount -v -t nfs servername:/nfssharename /mountpoint

Example:

mount -v -t nfs 10.34.10.184:/backupca /mnt/

Easy isn't it? 

What about the difference between NFS and CIFS? Which one is better? Which one to use? Which one to use when? 

It is important to mention that, these comparisons should not only be done from the performance perspective but also from the Fault tolerancy and consistency perspectives.

While the main difference seems to be -> "NFS is stateless (except NFSv4) and CIFS is stateful (not stateless)", these questions are all for another topic.. 

But I hope, I have explained what I wanted to explain to you with this post.

If you are interested with the answers of the question above; I suggest you to read the following article;

https://www.kernel.org/doc/ols/2007/ols2007v1-pages-131-140.pdf, A New Network File System is Born: Comparison of SMB2, CIFS, and NFS

ODA - "Investigating an OAKCLI Problem", OAKERR:6000 Exception encountered in Module getMasterNode OAKERR:6002 No master node was found

Recently investigated an oakcli problem, which was encountered in a virtualized ODA X4.
The problem was obvious and it was blocking the administration.
All the oakcli commands were encountering the following errors;

OAKERR:6000 Exception encountered in Module getMasterNode
OAKERR:6002 No master node was found

I have did a detailed investigation and fixed the error.
The cause was interesting and that's why I want to share it with you.

Note: There is no document or article written for this specific issue in Oracle Support.

I first looked at the odabaseagent.log..

ODA_BASE /opt/oracle/oak/log/$HOMENAME/odabaseagent/odabaseagent.log

Howerer; there was nothing there. That is, I executed a failing command (actually all the oakcli commands were failing) "oakcli show vm", but the odabaseagent.log was idle.

Then, I checked the oakcli.log, /opt/oracle/oak/log/ODA_NAME/client/oakcli.log and saw the following there.

2017-01-02 20:42:03.199: [ OAKCLI][10737392] OakCli started.... !
2017-01-02 20:42:03.233: [ OAKCLI][10737392] SERVER IPC CONNECT STR: (ADDRESS=(PROTOCOL=IPC)(KEY=__OAKD_CONNECT_STR))
2017-01-02 20:42:03.257: [ OAKCLI][10737392] Cli's version is: 2
2017-01-02 20:42:03.293: [CLSFRAME][10737392] Inited lsf context 0xf11da0
2017-01-02 20:42:03.329: [CLSFRAME][10737392] Initing CLS Framework messaging
2017-01-02 20:42:03.347: [CLSFRAME][10737392] New Framework state: 2
2017-01-02 20:42:03.365: [CLSFRAME][10737392] M2M is starting...
2017-01-02 20:42:03.390: [ CRSCOMM][10737392] Ipc: Starting send thread
2017-01-02 20:42:03.420: [ CRSCOMM][1468736] Ipc: sendWork thread started.
2017-01-02 20:42:03.586: [ COMMCRS][10590528]clsc_connect: (0x7f53f80009a0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=__OAKD_CONNECT_STR))

2017-01-02 20:42:03.619: [ CRSCOMM][10737392][FFAIL] IpcC: Could not connect to (ADDRESS=(PROTOCOL=IPC)(KEY=__OAKD_CONNECT_STR)) ret = 9
2017-01-02 20:42:03.637: [CLSFRAME][10737392] Failure at IPC connect to server:2
2017-01-02 20:42:03.655: [CLSFRAME][10737392] Unable to start module-to-module comms: 1
2017-01-02 20:42:03.674: [ OAKCLI][10737392] OakCli unable to connect to the server
2017-01-02 20:42:03.692: [ OAKCLI][10737392] OAKD is exiting with exit code: -1

There was a connection problem recorded in the oakcli log. It was an IPC connection and oakcli was compilaning about a listener, but that listener must not have been  a regular database listener.

Then I modified the scripts(mostly perl) residing in the oak directory(/opt/oracle/oak....) and put some debug lines in them to make my own debugging.

With this move, I  could see the flow was actually as follows...

Note that: in parallel, I also checked the log files of the programs that are visited during the flow and had a clearer view of the flow and the endpoint of it.

"oakcli show vm" command --> executes doakcli.pl -> connects to the dcliagent.py listening on the port 8881 -> dcliagent.py tries to reach the "oakd", but it can not reach the oakd and failing with: "DEBUG Failed to connect to oakd"

oakcli could reach the dcliagents running on both of the ODA BASE nodes (listening from the private interconnect ip address and port 8881 ), as it could be seen from the following doakcli.logs. The error OAKERR.:6002 was also in doakcli logs but it was the result of an exception..

2017-01-02 20:47:45,928 doakcli_log DEBUG trying to get the master node
2017-01-02 20:47:45,928 doakcli_log DEBUG Trying to contact http://192.168.16.28:8881
2017-01-02 20:47:46,373 doakcli_log INFO 1 call to isMaster() returned false
2017-01-02 20:47:46,373 doakcli_log DEBUG Trying to contact http://192.168.16.27:8881
2017-01-02 20:47:46,845 doakcli_log INFO 0 call to isMaster() returned false
2017-01-02 20:47:46,845 doakcli_log ERROR OAKERR:6000 Exception encountered in Module getMasterNode
OAKERR:6002 No master node was found

The errors we saw, were actually the results.

The connections to the dcliagents were successful. However, the oakd, which was the core of the process, was not running. 

 Because oakd could not be reached, the agent code were breaking and the agent was saying that "No master node was found"..

It should say oakd was not running.

Anyways, I started to search the answer  for the question "why".. Why wasn't oakd running? We are talking about an Engineered System, right? So why wasn't such an important process running?

In order to be sure, I used the various "ps" commands and concluded that oakd was really not running on any of the ODA Base nodes.

So I tried to start it..

I used oakcli restart oak but it didn't help. Nothing happened, no logs generated.. Probably it was also dependent to oakd.

Then I checked all the files in /etc directory, because I knew that oakd was started automatically during the boot process of ODA Machines.

I used a simple but handy "grep" for this.

cd /etc

grep -R oak *

The grep command displayed lots of lines and most of them were irrelevant.. But seeing the relevant file was not so hard.

The file in question was inittab (/etc/inittab).

Info about inittab: https://docs.oracle.com/cd/E19683-01/817-3814/6mjcp0qgh/index.html 

The inittab file was containing a line similar to the following;

h3:35:respawn:/etc/init.d/init.oak run >/dev/null 2>&1 </dev/null

I will not go in to the details about inittab (actually I want to but it is out of the scope and I m very tired :) 

However, I must say : what the line above means is, "init.oak script is run automatically during the boot process and relevant runlevel changes. init.oak was configured to be run during by the boot process automatically and it was under the control of Linux.  So in case of a failure, it was configured to be restarted automatically by Linux." 

The interesting thing was, the init.oak file was empty!

It was containing zero lines, zero characters.

I checked another ODA (in another customer environment) and saw that it was filled with the code, the code/script to start/stop/restart the oakd process.

At that moment, I concluded that, the problem was in init.oak. It was empty!

This was also explaining why I could start oakd manually.. That is, I could start oakd manually for working around this error as I documented in the following post almost 1 year ago. 

Workaround for the same error: http://ermanarslan.blogspot.com.tr/2016/02/oda-virtualized-odabase-oakd-oakcli.html

This workaround was written for the same issue .. The same issue in the same ODA...

Thanks to my customer, who didn't accept this workaround, I analyzed the problem deeper and I m here today for giving you the fix:)

As for the fix,  I copied the init.oak from another ODA.. 

I copied it to both of the ODA BASE nodes (to /etc/init.d directories) and then run "/etc/init.d/init.oak run" command in these ODA BASE nodes.

The issue dissapeared without a need to reboot ODA BASE domain. Moreover, oakd could start without any problems after rebooting the ODA BASE nodes.

Note: the reason why init.oak was empty, is still a mystery.. Who or what deleted its contents is uncertain.. No trace or logs for it.. I will also share it with you , if I 'll find that reason..