ODA -- All MMONs die at random times and DBs down / investigation

I've been working on a case last week. In fact, this case was escalated to me (usually as the end point of escalation)

The environment was an ODA X6-2 HA and it was installed as bare metal.

The problem was about the RAC databases.. There were critical production databases running on this machine and they were encountering errors. All the instances in one of the nodes were getting terminated..

The issue was on node 1, on the first node of the ODA.. Since the databases were configured as RAC, there was no major business impact. But it was still annoying of course.

Well, in order to see the real cause behind this, I decided make a full stack analysis. (note that, in this environment, it is not allowed to run scripts and collect diag data, so manual diagnostics were required.)

I looked up from the hardware and took a look at the stack. (I counted myself lucky as it was not a virtualized environment :) From top to bottom, we had databases, GRID infrastructure(+ASM +ACFS), Oracle Linux Operating Systems and the Oracle Servers (the hardware itself).. Bytheway, this is not a surprise, of course. I am writing this because there may be some who do not know ODA.

I concantrated on the last time the database failed.

My findings were; 

*There were 18C and 12CR2 databases on this ODA. The GRID was 18C. OS was Oracle Linux 6.10.

*The load was getting higher during that period.. I saw a peak in the sar outputs.  (Note that, the instances were failed at 10:16 PM)

10:10:01 PM         9      3877      6.22      7.30      7.76

10:20:01 PM        63      3583     66.82     44.44     24.37

10:30:01 PM        12      3614      6.10     12.03     16.17

*I saw the following errors in the syslog (/var/log/messages), but they are ignorable though;
/sys/bus/pci/devices/0000:af:00.1/virtfn0/uevent failed! 2BROADCOM[87864]: ERROR 

*Node 1 was evicted.. (so some things were going on cluster wide..)

*The first error that was seen in one of the database instances was: "ORA-00445: background process "m000" did not start after 120 seconds"

*All the databases were getting terminated. (18C, 12CR2 all of them) -- I though that there might be an ASM or ACFS issue.. On new ODA environments , we have ACFS right..

*MMON of ASM instance was also terminated. The error stack was;

ksedst1()+110 call kgdsdst() 7FFE764BE710 000000002

                                                   7FFE764B8A50 ? 7FFE764B8B68 ?

                                                   7FFE764BE230 ? 000000082 ?

ksedst()+64          call     ksedst1()            000000000 000000001

                                                   7FFE764B8A50 ? 7FFE764B8B68 ?

                                                   7FFE764BE230 ? 000000082 ?

dbkedDefDump()+2385  call     ksedst()             000000000 000000001 ?

9                                                  7FFE764B8A50 ? 7FFE764B8B68 ?

                                                   7FFE764BE230 ? 000000082 ?

ksedmp()+593         call     dbkedDefDump()       000000003 000000002

                                                   7FFE764B8A50 ? 7FFE764B8B68 ?

                                                   7FFE764BE230 ? 000000082 ?

dbgexPhaseII()+2130  call     ksedmp()             0000003EB 000000002 ?

                                                   7FFE764B8A50 ? 7FFE764B8B68 ?

                                                   7FFE764BE230 ? 000000082 ?

dbgexExplicitEndInc  call     dbgexPhaseII()       7F9FB6F726D0 7F9FB1D59528

()+609                                             7FFE764C28B0 7FFE764B8B68 ?

                                                   7FFE764BE230 ? 000000082 ?

dbgeEndDDEInvocatio  call     dbgexExplicitEndInc  7F9FB6F726D0 7F9FB1D59528

nImpl()+695                   ()                   7FFE764C28B0 ? 7FFE764B8B68 ?

                                                   7FFE764BE230 ? 000000082 ?

ksbsrvn_opt()+4562   call     dbgeEndDDEInvocatio  7F9FB6F726D0 7F9FB1D59528

                              nImpl()              7F9FB6FB2980 7FFE764B8B68 ?

                                                   7F9FB6FB2980 000000082 ?

ksbsrv_opt()+45      call     ksbsrvn_opt()        7F9FB6F726D0 ? 7FFE764C63C8

                                                   7F9FB6FB2980 ? 7FFE764B8B68 ?

                                                   7F9FB6FB2980 ? 000000082 ?

ksvspawn()+889       call     ksbsrv_opt()         7F9FB6F726D0 ? 7FFE764C63C8 ?

Ofcourse, there were a lot of other findings in CRS, ora agent logs, ASM alert logs, ACFS logs and so on.. However; they didn't affect the result very much, so I am not including them here.

Anyways; I was concantrated on the MMON traces.. MMON could give me a clue to solve this problem. 

In fact, MMON processes were already giving clues .. MMON was the first failing process.. More interestingly, just before the issue, MMON processes of "all" instances (including ASM) were getting similar errors.

Well, ORA-00445 was generic, but the error stack wasn't. The call for the dbgeEndDDEInvocatio and all the other calls under it was directing me to the MOS note ->  ORA-00445: background process "m000" did not start after 120 seconds (Doc ID 2679704.1).

The cause was ; "Bug 30276911 - AIM: BACKGROUND PROCESS "M000" DID NOT START AFTER 120 SECONDS - KEBM_MMON_SCHEDULE_SLAVE (duplicate of Bug 29902299 - ORA-445 - KEBMSS_SPAWN_SLAVE)"

The solution was applying Patch 29902299 or upgrading the databases. "But, wait a sec.. This is a database patch, but ASM was also getting this MMON error. There could be another reason for this common problem.. " After saying that, I have connected to the ILOM of node 1.. (I should have done this before), and saw the following;

A physical memory problem.. Acutally multiple memory problems.. Yeah, I know those faults may be cleared from ILOM, but look; altough they seem correctable (by ECC - Error Correcting Code),  why Are they produced in the first place ? (an answer to this question: Electrical or magnetic interference inside a computer system can cause a single bit of dynamic random-access memory (DRAM) to spontaneously flip to the opposite state.) And why for this RAM ?

Additional info about the error:

Description : Multiple correctable ECC errors on a memory DIMM have been detected.

Response : The affected page(s) of memory associated with the faulty
memory module maybe immediately retired by the operating
system to avoid subsequent errors.

Impact : The system will continue to operate in the presence of this
fault. The memory DIMM is still in use and is not disabled.
If the DIMM_CE_MAP_OUT policy is enabled, the memory DIMM is
disabled on next system reboot and will remain unavailable
until repaired. System performance may be impacted slightly
due to retired memory pages.

Action : Please refer to the associated reference document at
http://support.oracle.com/msg/SPX86A-8002-XM for the latest
service procedures and policies regarding this diagnosis.

Linux was still using that memory.. (probably it would discard that memory in the first reboot) Somehow, when ASM and DBs started to use the memory space corresponding to this memory, the problem was arising and all of them were getting errors. Note that, these findings also showed us that, ASM and all the database instances of node1 were all starting to use this area at the same time. I didn't need to dig anymore. After all, there was a memory problem and the customer was dealing with meaningless instance terminations.

Yes.. this was the reason behind.. As a matter of fact, the call stack in the MOS note 2679704.1 did not exactly fit the one I got from the environment. So especially, this one -> "Call Stack must contain: kebm_mmon_schedule_slave" We didn't have it! The error stack was similar, but we didn't have a call for kebm_mmon_schedule_slave.

After further research, I found the MOS note named "Top ASM Instance Crash Issues (Doc ID 2247412.1)". The issue named "Issue #2: ASM Instance Terminated with ORA-00445 and ORA-29743" was similar to the issue I was dealing with.. The high load was documented, IPC Send timeout was also documented there and the call stack was similar.. (not similar to the call stack of MMON , but it was similar to the CJQ's call stack.. CJQ was the first process that failed after MMON). The note was stating that the cause as "BUG 20134113 - ORA-445: BACKGROUND PROCESS "M000" DID NOT START AFTER 120 SECONDS". The solution offered was "Applying the Interim Patch 20134113 by downloading from MOS or logging a SR to get the patch from Oracle Support".. This little patch was designed to change kjxgna.o. But, it was weird, because the patch was only for database tier.. However; ASM instances are running on GRID, so definetely, a SR was  required there..

I checked all the logs, all the notes , all the Oracle Support and didn't find anything more which is directly related ( except Note 2679704.1 and 2247412.1), and then I checked the ILOM and found the failing RAM. Replaced the RAM and the issue didn't come up again.. yet!.. :) 2 patches I mentioned above are still in question and Oracle SR will guide us after this point. End of the story :)

Note that; a RAM failure may cause an instance crash a process crash.. The database node does not have to crash completely. ( Reference : Exadata DB Crash Due To Failing Memory DIMM (Doc ID 2512283.1)

ODA as a Weblogic Appliance ? Or is it better to consider PCA?

I was recently designing an architecture for enterprise applications and databases. 

Yes, we don't have any confusion when it comes to the database tier :) Especially for the enterprise databases, which requires high-end performance, resiliency and fault tolerancy. Independent of the type of workload, the platform that we recommend is Exadata. 

However, recommending a platform for application tier is another subject. Just like for the database tier; for large scale mission critical application environments, which mostly rely on Oracle Weblogic and FMW products, enterprise customers prefer engineered systems. 

Supposing the needs of the database tier are already satisfied with Exadata, the first platform that comes to mind for this kind of an application tier is Exalogic, which provides extreme performance, reliability and scalability for Oracle, Java and other business applications. However; currently I don't see a price for on-prem Exalogic.. I mean, according to the Oracle Engineered Systems Price List -September 25, 2020, Exalogic is not even on sale. Currently this exalogic link ( https://www.oracle.com/engineered-systems/exalogic/index.html) redirect us to https://www.oracle.com/engineered-systems/private-cloud-appliance. 

Yes! the link redirect us to Private Cloud Appliance (PCA). But before we get to that point; we must ask ourselves -> why not Oracle Database Appliance (ODA)?  There are strong E-Business Suite (EBS) references for ODA and I personally migrated several EBS customers to ODA, and Oracle still have several customers using EBS on ODA..  Well, mostly for the mid size EBS customers, we use ODA as a consolidation platform for hosting several EBS Application and databases.

Besides, we know that; although the name implies that it is a Database Appliance, ODA can also function as an Oracle WebLogic Appliance.

However, this context may change based on the real life stories and needs. I mean regardless of having EBS or other applications that rely on Weblogic and FMW, we may want to position an Engineered System for our applications and middleware, and only for them! (considering our database layer may be on Exadata..) 

So is ODA the solution for hosting or consolidating application environments and application environments only ?!?

Well, in fact, even if ODA may be suitable for this scenario, it does not make much sense to use ODA in such a scenario. Especially when we consider the storage mechanism provided for virtual environments on ODA. I mean the GRID, ASM and ACFS that are used for providing it .. These things are pretty simplified on ODA. Correct, but still we find ourselves in the DBA world for hosting Application environments, and for hosting only the applications!

Think about it... You just put applications on Guest VMs of ODA, you don't have any database residing on ODA, but still you maintain GRID, ASM and ACFS.

In addition to that, ODA can not scale out well. That is, when you reach the limits of its capacity, there is the possibility of vertical growth, but unfortunately there is no possibility of horizontal growth.

So this may be a problem for enterprise application environments, especially for large companies.

Okay, now we are here, the new engineered system ( at least it is new for me, as I didn't make any implementation on it, yet..).

Private Cloud Appliance comes into play at this point. It is avaible for on-premises and allows customers to efficiently consolidate business critical middleware and application workloads.

It has an integrated ZFS Storage and Oracle X8-2 servers for management and compute nodes.

It is scalable (up to 1200 cores,  3.3 Pb disk capacity).

It is high available and cloud ready. 

Moreover, PCA can be direct attached to Exadata to provide the lowest latency between Middleware and DB tier.

Virutalization technology used in PCA is OVM. However; we are expecting to have KVM be supported on PCA soon. (Remember what I wrote earlier about this OVM and KVM thing ->https://ermanarslan.blogspot.com/2020/09/end-of-premier-support-of-ovm-it-is.html)

Virtualization on PCA supports Oracle Solaris, Oracle Linux, Red Hat Enterprise Linux and Microsoft Windows Server for Guest Vms.

There is even a tool for automating the migration of virtual machines from VMware vSphere to Oracle VM.

This machine also has strong references.

I can keep writing about PCA, but it's more helpful to have a look yourself.

Check -> https://www.oracle.com/servers/technologies/private-cloud-appliance.html

Read the Private Cloud Appliance FAQs. 

Check the youtube video -> https://www.youtube.com/watch?v=Gtv6Mssbnp0 ( Oracle Private Cloud Appliance a.k.a PCA X8)

Very likely, you will think what I think and we will come to the same point.

This is the platform we are looking for large and critical Oracle application environments. 

An Engineered System that can be used to host and consolidate layers to our applications and can bring us many new features and capabilities.

Have a nice weekend :)

Erman Arslan's Oracle Forum -- Questions and Answers Series - September 2020

Let's start with the following question and answer. 

PS, this question and answer reveals what my motivation is. ->

Question: How much time do you spend/lose? 

Answer: Well, how much time I gain? :) 

In September, again I tried to answer all the questions. I gave advices when necessary, and provided guidance for the solutions when I had enough info about those problems and the environments where those problems arise. 

Take a look at the issues and related topics in Erman Arslan's Oracle Forum. Collect the harvest you can gather from the support and technical directions provided!

Erman Arslan 's Oracle Forum September 2020 -> 

EBS backup issue - Completely zero block found during validation

When last time DB objects was accessed/ modified? 

Linked Directory - EBS 12.2 - Invalid directory path 

Autoconfig Failing with error on Stansby node (EBS 12.2.7 with db 19c) 

Failed while upgrading from 12.2.8 to 12.2.9 

Effect of DC change on running database server

Listener issue while DR setup (EBS 12.2 with 19c database) 

rman target / not connects to correct DBID and database 

Datafiles redistribution across mountpoints 

Unable to implement Digital signature In R12.2 

WF mailer and oAuTH2.0 and office365 

Solaris - License Question 

Invalid APPS password after EBS 12.2 19C Upgrade 

Fs_clone which is failing on the master node  

javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake in R12 Inst_Top purging in R12.1.3 

Recommendation - Programming Language for a DBA 

Question - Big Data Architecture 

Question - Database Benchmark 

Goldengate big data initial load replication error 

Data Guard Setup for EBs 12.2.7 running with 19c CDB and PDB databases 

GoldenGate for Big data 19c and GoldenGate 18c compatability  

Create PDB Descriptor Failed 

Flashback cleanup 

Custom SSO / Login to OBIEE from a 3rd party app. By sending a POST request.. This works even when the LightWeightSSO is enabled!

In one of my previous blog post (https://ermanarslan.blogspot.com/2020/09/obiee-sso-integrating-with-third-party.html), I shared a third party SSO integration method for OBIEE.

We were just passing the user and password info as url arguments and it was working.

On that blog post, there was the following sentence; 

That is -> We make our OBIEE to get the user and password through the OBIEE url. (on-the-fly login using url arguments).. Note that this is the simplest way of doing this work.. Ofcouse, customer's ability to post the usernames and passwords using any other method than this one, will make us change/improve the design of this login flow.

Anyways, this was one of the ways, but today we realized something else.. Something else that is refuting that way.

That is, if we login to OBIEE and then try to reach ODV from there,  we find ourself in a login dialog, where we should enter our user and password information once again. Yes.. This is not cool..

Fortuneatly; we have a solution for this too!

The solution is to enable LightweightSSO. Sound simple right? But wait a sec, LightweightSSO is not compatible with our 3rd party integration method , I mean -> Logging into the OBIEE from a third party app by passing user and password as arguments in OBIEE URL...

Remember, in that blog post, I already mentioned that when 12.2.1.3 LightWeightSSO is ON, NQPwd/User(I mean the URL method) won't work for OBIEE login.. So, as I mentioned in that earlier blog post, we disabled LightWeightSSO to be able to pass user and password info through url.

However; when the LightWeightSSO is disabled, we can't directly reach ODV from OBIEE.. I mean, ODV requires us to re-enter our user and password info as I just mentioned. 

So it is not acceptable. 

This means we need to enable LightWeightSSO to make  automatic SSO integration between OBIEE and ODV work.. Ofcourse, this time (when the LightWeightSSO is enable), our OBIEE login (through url arguments user and password) will not work..

Well, this is what makes me write this blog post.

The question : How can we login to OBIEE from a 3rd application automatically in a custom SSO-like way, even when the LightWeightSSO is enabled?

In order to answer this, we take a look at the OBIEE login flow, I mean we do a technical login mechanism analysis. 

I don't mean a code analysis, but we use our browser (For instance Chrome-> F12-> Network tab) to analyze the http requests, http headers and the form data.. We need to check the required the arguments.

Once we do those analysis; we can see that, when the LightWeightSSO is enabled, the login page changes. 

Our login page becomes login.jsp. Login.jsp get the user and password info from the user and authenticates it using "login" (without .jsp suffix). 

So when we check that "login", we see that it is designed to receive some POST request arguments. j_username, j_password and so on. 

So if we can make a HTTP POST request to "login" directly from our 3rd party app, it should work.. 

This way, we will be able to pass the username and password info to OBIEE and OBIEE will let us in automatically. (even when the LightWeightSSO is enabled!)

So, we create a simple html to test this..

Note that the values that you see below are just examples.  -> 

<html>

<form id='redirectForm' method='POST' action='https://oiee_host:obiee_port/bi-security-login/login'>

<input type='hidden' name='j_username' value='weblogic'/>

<input type='hidden' name='j_password' value='erman'/>

<input type='hidden' name='j_msi' value='none'/>

<input type='hidden' name='j_language' value='en'/>

<input type='hidden' name='j_redirect' value='L2FuYWx5dGljcy9zYXcuZGxsP2JpZWVob21lJnN0YXJ0UGFnZT0xJmhhc2g9RlEyeDZFaGp3cnJHQXNzbmVWOWtSeVVuYmxVQjYyczZMR0JESFEtR3F5ZEoxcXh2bjMyMmxKaUlwU1R4VFIxMA'/>

</form>

<h1><a href="#" onclick="document.getElementById('redirectForm').submit()">GO!!</a></h1>

</body>

</html>

Please note the hidden input names -> j_username, j_password, j_msi, j_language and j_redirect..

j_redirect is the url that OBIEE will redirect us after the login process. It is in the base64 form. (in this case it is basically set to -> /analytics/saw.dll?bieehome&startPage=1)

So, we open this html with our browser and click GO! Guess what? We found ourselves in the OBIEE home page! (logged in automatically in the backend by posting user and password info) So it works! 

At the end; we pass this html to the developers of the third party application as a reference and they modify their OBIEE login code and that's it :) We login to OBIEE automatically from a 3rd app automatically even when the LightWeightSSO is enabled.

I 'm not finished! :)

If the third party app requires a form, and if it doesn't like the form of the login.jsp. (because it is doing its work with javascript probably) , I mean if the 3rd party app requires a submit button, then we create a wrapper html like the one below and deploy it to our Weblogic (or any webserver that we have)..

Want to deploy it to a Weblogic? -> here is the way ->  "How To Publish a Static HTML Page To WebLogic Server and Request Through Oracle HTTP Server 11g (Doc ID 1192439.1)" -- Part 1 is enough..

With this action, we actually put a middle man between our 3rd app and  OBIEE login and make the 3rd app to post to OBIEE login using that middle man :) This works too!

So the flow becomes;  "3rd pary app -> Wrapper html -> OBIEE Login"

<html>

    <form  name="loginform" method='POST' 

        action='/bi-security-login/login' 

        style="visibility:hidden">

    <input type='hidden' name='j_username' value=''/>

    <input type='hidden' name='j_password' value=''/>

    <input type='hidden' name='j_msi' value=''/>

    <input type='hidden' name='j_language' value=''/>

    <input type='hidden' name='j_redirect' value=''/>

    <input type='submit' value='Login'/>

</form>

</body>

</html>

That is it for today:) I hope this will help you.

Upcoming End date of OVM Premier Support. It is time to consider KVM + OLVM (especially for the new projects)

This is for the ones who is considering a virtualization solution for new projects.

Especially for the ones considering Oracle Virtıalization... 

Usually when we say Oracle virtualization, we mean Oracle VM Server, but actually that was in the past. Now we have an alternative to OVM..  It is KVM (Kernel Based Virtual Machine).

KVM is actually an open source virtualization technology that turns Linux into a hypervisor. 

I firstly used this KVM in ODA X7-2 S and M environments. At that time, we had some limitations though.. For instance, there were no capacity-on-demand option for databases and application that are running on Guest KVM Machines.

However; now we have the cpu pinning/ hard partitioning / capacity-on-demand option in KVM!

Note that,  using hard partitioning to limit Oracle product software licensing still adds some restrictions such as live migration and scheduling policies available on Oracle Linux Virtualization Manager.

If you want to check out some of my advantures on ODA and KVM, you can read the blogposts pointed by the following urls.  :)

https://ermanarslan.blogspot.com/2017/07/oda-kvm-virtualization-for-oda-x6-2sx6.html

https://ermanarslan.blogspot.com/2017/10/oda-x6-2m-virtualization-with-kvm-real.html

https://ermanarslan.blogspot.com/2017/11/oda-x7-2-closer-look-oracle-database.html

https://ermanarslan.blogspot.com/2017/09/oda-x6-2m-virt-manager-display.html

Also note that, Oracle Linux KVM is the same hypervisor used in Oracle Cloud. You can read the related  article in the following url to get more info : https://www.oracle.com/a/ocom/docs/olvm-datasheet-nov2019.pdf

At the moment, KVM has a support contract advantage too.. KVM is offered under Oracle Linux Virtualization Manager (OLVM). Note that OLVM is the virtualization management platform that can be easily deployed to configure, monitor, and manage KVM environments.

So, KVM is offered under Oracle Linux Premier limited support.  So no need to purchase support for the virtual layer seperately.. 

Another thing that motivates us to prefer KVM is the upcoming end date of OVM premier support.

As per Lifetime Support Policy document; Oracle VM Premier Support period will end in March, 2021. Therefore, the users of OVM would need to buy additional extended support. 

As I mentioned, alternative to OVM is KVM, offered under Oracle Linux Virtualization Manager (OLVM). 

Here is the key benefits of using KVM and OLVM -> 

• Complete server virtualization and management solution with zero license cost
• Single software distribution for Oracle Linux OS or Oracle Linux KVM 
• Speeds application deployment with Oracle Virtual Appliances
• Ksplice integration to patch kernel, QEMU, and user space libraries with no service interruption
• Hard Partitioning support enables efficient Oracle application software licensing
• Full Stack Management with Oracle Enterprise Manager
• Path to Oracle Cloud Infrastructure with a common hypervisor

So, we advise our customers to use KVM which is offered under Oracle Linux Premier limited support, especially for their new virtualization projects. This is the purpose of this blog post actually. Final decision is still yours :)

RDBMS / ASM / Exadata - "Smart Rebalance" / Seems like the %15 free space rule (or %9 free space rule) is becoming history

Yesterday, I published a blog post about the %15 rule. I shared my thoughts on %15 free size rule, which states that, in order to be in the safe side in case of a cell/or disk failure in Exadata/ASM environments, we need to have some free space in the relevant diskgroups. This actually guarantees the rebalance, which should be done after a disk or cell failure, to be succesful. The rule states that, at least %15 of a diskgroup should be free. 

I found this magic number, or let's say this magic percentage (%15) a little interesting and felt the need to write a publish a blog post about it.

You can access that blog post via the url below;

http://ermanarslan.blogspot.com/2020/09/asm-grid-my-thougths-on-15-free-disk.html

Yestarday night, I was still curious and checking the documents to learn something new about this subject and I finally found the thing I was looking for. That was exactly what was expected. "Smart Rebalance"

Smart rebalance used with Oracle ASM which eliminates the need for free space for Grid Infrastructure 19c and Higher using high redundancy diskgroups.

In other words;  no need for free space!   If there is not enough space to rebalance at the time of failure, offline the disk! Upon replacement, efficiently repopulate it from partner disks automatically! 

This eliminates the need to reserve free space for rebalance when using high redundancy. It provides  seamless repair without the risk of out of space errors..

Currently there is no internal info about it, but you may visit the following the url to see  Exadata MAA slides.. Slide 58 introduces the smart rebalance and shows the dissaperance of %15 rule gradually :)

https://www.oracle.com/a/tech/docs/exadata-maa.pdf

I will give you more information on this topic when I have a little more detail.

EBS - Attention ! Workflow mailer & oAuTH2.0 and office365 - Microsoft / End of Support for Basic Auth - "Deadline has been pushed to the second half of 2021"

Thanks to the community in my forum (Erman Arslan's Oracle Forum), we realized something important and fortuneatly, it is still not to late to report this!

Thanks Laurel for pointing it out in the following thread :) -> 

http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/WF-mailer-and-oAuTH2-0-and-office365-td8771.html

First, Microsoft announced, that they will stop supporting Basic Authentication for Exchange online on October 13, 2020.  EAS, POP and IMAP..

But then, they changed the deadline .. That is, the Basic Auth is "not" going to be disabled on October 13, 2020. Due to the COVID... That deadline has been pushed to the second half of 2021. 

So, it seems we still have time. Good news, right? :)

Here is the statement about this new deadline; https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-and-exchange-online-april-2020-update/ba-p/1275508

Microsoft says: In response to the COVID-19 crisis and knowing that priorities have changed for many of our customers we have decided to postpone disabling Basic Authentication in Exchange Online for those tenants still actively using it until the second half of 2021. We will provide a more precise date when we have a better understanding of the impact of the situation.

Anyways, when that time will come, workflow Mailer IMAP with Office 365 basic authentication will not be supported and probably it will just not work.(Basic authentication will be turned off)

EBS customers will have to use OAuth 2.0 token based authentication for IMAP.

So, EBS customers who are using Workflow mailer with office365 may be in trouble , and I think Microsoft is ready for this -> 

https://developer.microsoft.com/en-us/outlook/blogs/announcing-oauth-2-0-support-for-imap-smtp-client-protocols-in-exchange-online

They say : We’re announcing the availability of OAuth 2.0 authentication for IMAP, SMTP AUTH protocols to Exchange Online mailboxes. If you have an existing application that reads or sends email using one or more of these two protocols, the new OAuth authentication method will enable you to implement secure, modern authentication experiences for your users. This functionality is built on top of Microsoft Identity platform (v2.0) and supports access to email of Microsoft 365 (formerly Office 365) users.

Oracle address this situation by the following document ;

EBS Workflow Mailer Configuration with OAuth 2.0 Token-Based Authentication for Cloud-Based Email Services (Gmail, Yahoo, Office365, etc) (Doc ID 2650084.1)

Note that, this document is not up-to-date...

We have also a bug record, an Enhancement Request for it. 

Bug 30505419 : WORKFLOW MAILER SUPPORT OF OAUTH2 - GENERIC PLATFORMS

Unlike the document, the enhancement requests seems up-to-date. Oracle seems working on this subject as I see some recent updates on the bug record;

*** 09/11/20 08:45 am ***
*** 09/11/20 09:18 am RESPONSE ***

The Enhancement Request is in Internal Review status, meaning not approved nor denied.

However, currently we have no ETA for this. 

In any case, I think the solution/patch will be for EBS 12.2.x.. So, I think, upgrading to the latest version (12.2.10)  should not be a must.

But still, you need to design and implement your backup solution, because the ATG fix may not be ready until the second half of 2021.. ( Actually, I think there is enough time to deliver the fix, but still we need to be prepared)  

Especially EBS 12.1.3 customers should be careful and ready. 12.1.3 is also subject to restriction on new patches starting Dec 1, 2021 and a solution for 12.1.3 cannot be guaranteed until the final solution for EBS 12.2 connecting to Office365 will be developed.

In order to be in the safe side, customer should just create a local mail server, test it and be ready for activating it on the second half of 2021.. (just in case)

I will continue to follow this subject and keep you updated.

ASM / GRID -- My Thougths on %15 free size rule --- rebalance, imbalance, calculations, bugs and all that

The rule states that, in order to be in the safe side in case of a cell/or disk failure in Exadata/ASM environments, we need to have some free space in the relevant diskgroups. This actually guarantees the rebalance, which should be done after a disk or cell failure, to be succesful. The rule states that, at least %15 of a diskgroup should be free.

Well, I found this magic number, or let's say this magic percentage (%15) a little interesting and that's why I want to share my thoughths on this with you.

Normally, we have a metric named USABLE_FILE_MB as you may already know. It may depend on the version but, normally this metric gives us the safe allocatable size considering a case of a disk failure.. In the old versions, this was reporting the safe allocatable size, a value which can be taken as a reference for being safe even in a cell failure.

In simple logic, we can say that; we have no risks, ofcourse if the USABLE_FILE_MB has a positive value and if we think it will stay positive even when we consider potential new future allocations.

Moreover, USABLE_FILE_MB is derived by considering the REQUIRED_MIRROR_FREE_MB, which is the required size for a rebalance operation to complete in the worst case scenario.

The formulas are as follows;

Normal Redundancy
USABLE_FILE_MB = (FREE_MB – REQUIRED_MIRROR_FREE_MB) / 2

High Redundancy
USABLE_FILE_MB = (FREE_MB – REQUIRED_MIRROR_FREE_MB) / 3

If USABLE_FILE_MB is a negative value, then we can directly say that the normal redundancy environments are in danger, but in any case we can still check FREE_MB. If the value that we see in FREE_MB is bigger than the disk size (if the disk sizes are equal.. If they are not equal, then FREE_MB should be bigger than the largest disk size), we can still rebalance in case of a disk failure. 

So far so good. These are all related with disk failures. (as I mentioned earlier, we need to check the version and conclude what the USABLE_FILE_MB reports to us.. Usable file mb even in the case of a disk failure or Usable file mb even in the case of a cell failure)

Of course, if we lose a cell and if the USABLE_FILE_MB considers only the disk failures, the situation is different. We need to multiple the USABLE_FILE_MB with the count of disks in the cell.

It is independent from the redundancy being normal or high; for instance , if the USABLE_FILE_MB is 10 and it reporting us the usable file mb in the case of disk failures and if we have 12 disks in a cell, then we have to  multiply that value 10 with 12. This makes 120 and that 's minimum usable file mb that we need to see in USABLE_FILE_MB in order to be safe even in  a case of a cell failure.

At this point and in this context, following article of Emre Baransel might be nice for reading.

https://www.doag.org/formes/pubfiles/8587254/2016-INF-Emre_Baransel-A_Deep_Dive_into_ASM_Redundancy_in_Exadata-Manuskript.pdf 

Until here, if you notice, I have never mentioned the 15% rule.  So I have explained  the subject ignoring this rule, but actually this rule must not be ignored.

Now it is time to explain that rule:)

Well, we first revisit the MOS note named, "Understanding ASM Capacity and Reservation of Free Space in Exadata (Doc ID 1551288.1)".

In MOS note, we have a script that calculates the reserve space and capacity for the disk failure coverage and it has a reserve factor of 0.15 and that's where the %15 rule comes in..

When we examine the script, we can say that, it directly multiplies the raw total disk size by %15 and then, it substract that value from the raw total disk size.

In my opinion, it shouldn't be that way.. I mean, there shouldn't be a %15 rule and I think this subject is a little buggy.

Note that, at the moment;  we need to consider the %15 rule and we must follow it!

Anways; if we reserve %15 of space , are we safe ? Well, probably.. But, the following bug says that, even if we have %15 reserve space ,we still may have problem during rebalance..

Bug 21083850  ORA-15041 during rebalance despite having free space -> Bug 21083850 - ORA-15041 during rebalance despite having free space (Doc ID 21083850.8)

The cause of this bug is probably the imbalance during rebalance -> 

When a disk is force dropped, its partners lose a partner.
As a result, the partners of its partners get more extents relocated to them, causing an imbalance.
This imbalance results in the ORA-15041, because some disks run out of space faster than others.

In the document above, we see a patch is addressed. However, in another Oracle script, we see a comment like the following -> "Use the new 15% of DG size rule for single disk failure, regardless of redundancy type (Bug 21083850)" 

This makes me think that this subject is buggy :) The %15 rule is there not only to address that specific bug, but it is there due to other bugs as well.  In my opinion, these kinds of rules are there because of other problems.. In this specific case, probably because of imbalance, or let's say it is probably due to the ASM extents not being distributed properly.

Normally, when we lose a disk, ASM will distribute the mirror extents of that failing/lost disk to the other disks that are available on the relevant diskgroup (ofcourse, according to the redundancy type).. That comes from the logic of disk mirroring. However, probably, ASM distributes these extents not evenly and overloads some discs in some cases and that's where we get ORA-15041.

This situation can also be explained by ; having those disks already overloaded even before the rebalance.. So as you may guess, if ASM uses them aggressively during the rebalance they get full and the rebalance code returns an error.

Ofcouse, imbalance  may be normal in some cases.. For instance when we have fail groups .. 

That is; when we have a fail group configuration, ASM will have a more difficult job during the rebalance.. I mean, when we have fail groups; ASM will have less choices for distributing the mirror extents when a disk is dropped.. Still, I don't think that these kinds of causes should not be enough to reveal such a  rule (%15 rule)

Well, these are my thought on this subject... Please feel free to comment and correct me if I'm wrong. Please share your thoughts on this subjects by commenting to this blog post.

ACE Virtual Happy Hour - The Great Gathering

A Virtual Happy Hour. An ACE get-together. 111 ACEs & Oracle members in call.  Thanks to Jennifer Nicholson and Oracle Ace Program for organizing this. Being an Oracle ACE has always been an honor for me, and as I see these valuable people who are experts in their fields together, my curiosity for new technologies, my passion for Oracle and my motivation for research continue to increase even today. 

PS: Ace video and song was great :) It's a good memory.

OBIEE - SSO -- Integrating with a third party login with AD authentication / passing user and pass in URL

Implementing SSO or Windows Native authentication in OBIE is something we do frequently.

Basically, we integrate OBIEE with Microsoft Active Directory and obtain centralized password management. 

This is also more secure and easy for the users.. They don't have to remember or manage their OBIEE usernames and passwords as they already have more important usernames and passwords, I mean their domain users and passwords. 

This gain we make by implementing SSO is actually in convenience, manageability and security. This benefit or gain is provided by all types of SSO configurations. In single password implementations, users login with their domain usernames and passwords. In windows native authentication, they don't even login as we get the credentials (or lets say the auth info) from the client OS on the fly in the backend transparently to the user :) It's been a long sentence :)

Anways, this is what we do in OBIEE and even in EBS envirıonments. In EBS, we use OID and OAM as well. (Things get complicated there but that 's true :)

So I guess all of us are familiar with this single sign-on (single password or no password)  concepts already. 

However; what I want to share in this blog post is something, which is a little different than a standard configuration.. A scenario, a real life story, a workaround; you name it :)

That is, suppose your customer wants your OBIEE to authenticate the users with AD usernames and passwords but suppose the customer has a custom web page which is in front of the OBIEE and the customer wants to get the usernames and passwords through that custom web page. The custom web page must authenticate the users and then redirect to OBIEE..

So what we do?

Well, we implement SSO in OBIEE side. This is what we need to do in the first place. 

Then we make our OBIEE to get the user and pass through the OBIEE url. (on-the-fly login using url arguments).. Note that this is the simplest way of doing this work.. Ofcouse, customer's ability to post the usernames and passwords using any other method than this one, will make us change/improve the design of this login flow.

At this point, we pay attention to the following;

"12.2.1.3 LightWeightSSO is ON by default and NQPwd/User wont work." 

This means, the news versions of OBIEE won't let you in with the usernames and passwords supplied through the url.

So, what we do? 

If that is a must.. I mean if the page can't post the username and password information to OBIEE using any other method than the url method, then we disable the LightWeightSSO. 

In other words; If we must use the NQUser and NQPassword login url parameters, we must disable lightweight SSO using the WLST disableSingleSignOn command. 

The following document will help us for that;

OBIEE 12c: Using NQUser and NQPassword in URL, Fails to Login When Single Sign-On (S
SO) or Lightweight SSO (LWSSO) Is Enabled (Doc ID 2316810.1)

Once we configure our OBIEE side, we tell the customer to make the necessary modifications in the custom webpage to make it pass the username and password information to OBIEE during the login process. That is it..

We reached our goal.. Clients will use the custom web page to enter their AD/domain usernames and passwords and then the custom web page will make OBIEE to authenticate it in the backend and the clients will see his/or her BI dashboards without authenticating again.

Before finishing, 2 important reminders ->

Don't forget to implement a full path SSL for the HTTPS communication.

Consider implementing  LDAPS for the ldap traffic between OBIEE and AD.

This is the story of the day :) I hope you will find it useful.