ODA -- "Patch your ODA with ODA Patch Bundle" - Stay away from PSUs, CPUs and/or other patches

I want to inform you on an important subject.. It is about patching Oracle Database Appliance.
You may already know this, but in some cases you may want to take the risk and apply one-off patches, PSUs or CPUs on your Oracle Database Appliance environments..

Firstly, I want you to know that, this is not a good idea!

Only in special circumstances, you may consider applying one-offs on ODA, but you still need to get the approval of Oracle Support by creating a SR.

But generally no PSU, no CPU!

Only the ODA Patch bundle...  The one-button patch specifically designed to upgrade Oracle Database Appliance firmware, OS, Grid Infrastructure, and Database PSUs.

Do you want to upgrade your GRID PSU or Database PSU on ODA? Then find a ODA Patch bundle that delivers those PSUs and go on with that..

For instance;

If you have Oracle Database Appliance 12.1.2.10.0 release, then we can say that your Oracle Database version is 12.1.0.2 or 11.1.0.4 .. Moreover, we can also say that your PSU level is 12.1.0.2.170117 (PSU) or 11.2.0.4.161018.

Well, if you want to upgrade your PSU version, then you should upgrade your ODA release to a newer ODA release, such as 12.1.2.12.0.

12.1.2.12 delivers Oracle Database Bundle Patch (BP) 12.1.0.2.170814 and Oracle Database Patch Set Update (PSU) 11.2.0.4.170814.

By doing such an upgrade, you also get an upgrade in the other layers and components such as OS, kernel, BIOS and etc.. (These are nice to have and but also required... Keep that in mind..)

Check the documentation for ODA software releases and all the related details... (Just select your ODA release from drop down menu and check the documentation of it) -> 

https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/index.html

Also, now we have Oracle Support patch tab to find our patches for ODA..

Earlier it was not available.. 

Well, I have dealed with this machine almost since its birth, but then I took a break. So maybe it wasn't available at least for some years ago..

So let me tell you a quick real story about the effect of applying PSUs on ODA directly.

Yes, I have seen this.. 

I have seen a real bad effect of applying GRID and Database PSU into ODA X6-2 HA..

After patching GRID and RDBMS with traditional PSUs, at first everyting seemed okay.

However; the nightmare started when the database was opened.. Interestingly, one of the node rebooted itself, when the database was opened. After the reboot, the issue continued.. It was like an infinite power cycle..!

 

Very interesting, right? No clues in the OS side, no clues in CRS, ASM or ACFS logs.. No errors in agent logs, nothing in the database.. The node was rebooting itself directly when the database was started. 

The environment was a standard one.. A Bare Metal ODA X6-2 HA.. It was ACFS-based. So, if ACFS is present in the environment, it is even more dramatic.. That is, facing with this kind of an issue is becoming even more potential.

I still think that, the reboot problem was related with ACFS.. I mean, everytime when the database was opened and did some stuff on ACFS, the OS was crashing and we saw a reboot.. ( without any clues, believe me...) 

The only thing that I could see was in the "last" command output.. It was a crash.. Probably it was caused by a failing ACFS kernel module.. ( a fault in a kernel module may bring the system down, may result a direct crash just like the one I faced) 

That fail was probably an unexpected one, because even the dmesg command output was clean.. Normally, just a simple printk(KERN_ALERT ... would do the work for informing us.. So this must be an unexpected one.

Ofcourse, we could reproduce and trace it at OS level, but we didn't have that amount of time.. 

So, as suggested by the title of this blog post, Go on with ODA Patch Bundle. Stay away from PSUs...

Read the following MOS note for more->

• ODA Support Guidelines for Using Existing Interim Patches or Requesting New Interim/Merge Patch Requests (Doc ID 2081794.1)
• Oracle Database Appliance FAQ (Doc ID 1463638.1)
• ODA Patching FAQ : 18.3 and Lower (Doc ID 1546944.1)