EBS / RDBMS -- High Latch Free Waits --> "Result Cache: RC Latch"

I have dealt an interesting performance issue recently.  The environment was an EBS R12 ( 12.2.6 ) .. EBS database was an Oracle 12.1.0.2 running on Solaris SPARC.

The issue was appearing in the office hours. 

Every business day around 9:30 AM, there were Latch Free waits and this situation was making the end user face a performance degradation and complain..

We had the Enterprise Manager, AWR and ASH reports. The cause aboveground was Latch Free waits, but the reason behind was the Result Cache. (Actually the improper use of the  Result Cache...)

If you don't have the necessary background info about the Result Cache, please read the following blog post for a quick enablement-> 

https://ermanarslan.blogspot.com/2019/10/rdbms-things-to-know-about-result-cache.html

(RDBMS -- things to know about the Result Cache / Result Cache : RC latch)

In this case, the following PL/SQLs were waiting on the Result Cache: RC Latch;

*FND_AOLJ_UTIL.is_Valid_ICX(:2,NULL,'Y',TRUE,TRUE,NULL,NULL,NUL)

*BEGIN   fnd_global.bless_next_init('FND_PERMIT_0006');   

 *FND_GLOBAL.APPS_INITIALIZE(    user_id=>:1    ,resp_id=>:2    ,resp_appl_id=

  >:3    ,security_group_id=>:4    ,server_id=>:5);END;

Look like a login issue right? I mean, it looked like, there was a login storm which was expected in the morning and it was the one that triggers the issue. (and actually it was the one that is becoming the victim of it..) Why do I say that? Because I saw those PLSQLs that were used in session validation, apps initialization and so on.. And! we were seeing the issue in the office office around 9 AM..

In these types issues, customers chase the quick wins at first..

So what might be the solution?

*Increasing the Result Cache Size?

*Disabling the custom code? (there was a Custom SSO on the login) 

*Gathering fixed object stats and dictionary stats?  - ref: Some Queries Are Causing Performance Issues in ISG (Doc ID 2396223.1

*Setting some hidden parameters -> alter system set "_optimizer_dsdir_usage_control" = 0;  -- ref: High Latch Free Result Cache RC LATCH On Object Linked Views (Doc ID 2245996.1) 

*Some more hidden parameters -> *alter system set "_optimizer_ads_use_result_cache" = FALSE; High "Latch Free" Waits with Contention on 'Result Cache: RC Latch' when RESULT_CACHE_MODE = MANUAL on Oracle 12c (Doc ID 2002089.1)

*Doing some other improvements -- for decreasing the latching -- moves like changing the cursor_sharing parameter --- not related actually..

*Enabling session trace for EBS users? -- doesn't tell us much in this case.

Log onto System Administrator Responsibility

Navigate: Profile > System

Search for the profile option Initialization SQL Statement - Custom (USER level. )

Enter the following sql statement into the User Level profile option. 

--modify USERID to the actual username of the user.

BEGIN FND_CTL.FND_SESS_CTL('','', '', 'TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER='||''''||'USERID' ||''''||' EVENTS ='||''''||' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '||''''); END;

*Patches? -- It was a good try..

Patch 26436717  (Database/RDBMS Home Patch) and Patch 17305006 - SEVERE PERFORMANCE SLOWDOWN WHEN RESULTS CACHE IS THRASHING is needed.

Patch 27508480: CONNECTION LEAKS FROM WSH.SERVLET.REQUESTRATES değerlendirilebilir. 

PAY122:CONNECTION LEAKS WHEN ACCESSING ONLINE W-2 IN 12.2(Patch 22684327:R12.PAY.C) 

Patches 28831943 and 26599059 for FND 

Patches 25969099 and 20141119

BNE Is Leaking Connections In12.2.4 --- AD/TXK 7, ATG 5 And HRMS PC.C.7 (Doc ID 2141923.1)

21771287: CONNECTION LEAKS FROM GWY.JSP - SIMILAR TO BUG 20446583 -- değerlendirilebilir

R12.ATG_PF.C.delta.6 Patch 21900895 or greater has been applied which includes the latest changes for BNE memory allocation.

I could keep going , but actually no need. Well... That's because, we have considered all of the things above and we actually did some of them, but we couldn't get any benefit..

-->This time, we were not looking for a needle in a haystack. (that is actually pretty easy, because a needle and a haystack are very different).. We were looking for a particular needle in a stack of needles :) -- I like this phrase by the way.

We were almost sure that, the solution should be directly related with the result cache or to those (some EBS code) using it.

Disabling the Result Cache in the database level will resolve the issue, but we were not so into that.

What we did was the following; we first made the result cache off by using the bypass mode.

Then we flushed the Result Cache and made in on again.

Once the Result Cache is flushed, we started monitoring it..

For checking the current result cache

-----------------------------------------

SELECT dbms_result_cache.status() FROM dual;

To enable the bypass mode and flush:

-----------------------------------------

BEGIN

   DBMS_RESULT_CACHE.BYPASS(TRUE);

   DBMS_RESULT_CACHE.FLUSH;

END;

/

To disable the bypass mode

-----------------------------------------

exec DBMS_RESULT_CACHE.BYPASS(FALSE);

We saw that the Result cache is occupied by the FND_PROFILE mostly..

There was no space pressure on the cache but things like FND_PROFILE.GET_PROFILE_APPL_ID and FND_PROFILE.GET_PROFILE_ID was there on the top..

Query used for Result cache monitoring:

select substr(name, 1,100) name,

cache_id,

count(*) result_count,

round(avg(scan_count)) avg_scan_count,type,status

from gv$result_cache_objects

where type = 'Result'

group by name, cache_id,type,status

A culprit from the monitoring output:

NAME                                                                            

--------------------------------------------------------------------------------

CACHE_ID                                                                        

--------------------------------------------------------------------------------

RESULT_COUNT AVG_SCAN_COUNT TYPE       STATUS                                   

------------ -------------- ---------- ---------                                

"APPS"."FND_PROFILE"::11."GET_PROFILE_APPL_ID"#32fb3b6bdac49c05 #245            

85njc349fpj4gc54hq8ca2u41q                                                      

         691         272415 Result     Published                                

                                                                                

"APPS"."FND_PROFILE"::11."GET_PROFILE_ID"#32fb3b6bdac49c05 #216                 

6m1zcmdsv8v40c201cvbaa3y8u                                                      

         721         261123 Result     Published                                

                                                                                

"APPS"."FND_PROFILE"::11."GET_LEVEL_ENABLED_FLAG"#ba365d4c80ddb2d5 #275         

4b86jg87f0gkxafawmtmr29dh4                                                      

        4746         139188 Result     Published                                

                                                                                

"APPS"."FND_PROFILE"::11."GET_HIERARCHY_TYPE"#a0e3bfe28f4de20f #326             

1q2vx7pv7jvc18uksjffyf6vb9                                                      

         678         139164 Result     Published                        

Then , we jumped into the code.. The standard code and found that FND_PROFILE was written in a way that is implicitly use the result cache.. We saw the usage of cross-session PLSQL Fuction Result Cache in multiple places of the code. (the RESULT_CACHE Clause) 

Those FND_PROFILE calls were done mostly during the login and we concluded that, when there is a login storm and when there is a DML that may affect the data/FND_PROFILE related data that is already stored in the result cache, those waits for the Result Cache Latch start appearing..

Note that, we also had some cached things that belongs to the WF_CORE, but this wasn't the issue..

Anyways, as as workaround we wrote a script (job, cron job etc.. -- you can think whatever you like) to flush the result cache every morning just before the issue happens.. (The issue was mostly seen after 9 am..)

We could also blacklist the relevant parts of the result cache, as a more elegant workaround.

Blacklisting those caches:

Using the following query, identify the Cache_IDs that has high RESULT_COUNT.

SQL> select substr(name, 1,100) name,

cache_id,

count(*) result_count,

round(avg(scan_count)) avg_scan_count,type,status

from gv$result_cache_objects

where type = 'Result'

group by name, cache_id,type,status

Use Black_list_Add to ban them..

exec dbms_result_cache.Black_List_Add(xxxxxxxxxx, TRUE);

But what about the fix? Well... Fix is in the code.. The code is written such a way to use the Result Cache, but the processes in the ERP changes the data that is cached by the code and that is too bad!

Either the code or the processes (ERP processes that is updating the data cached by FND_PROFILE) should be revisited..

Nevertheless, we have 2 workarounds! :)  Again -> Be careful when using the result cache!

Well.. I can't stop myself to give some background info here.. Here it is ;

Result cache in these kinds of scenarios brings the data faster than the buffer cache, as it has the data in the result format and it just copies it to our session/process 's buffer.

However; if we try to use the result cache in a dynamic environment, things change.

Things change, because the invalidation ( when the data and objects that builds the contents cached in the result cache change) means "Writing the Result Cache"

Writing is not a very bad thing? Yes for 1 write or 10 writes maybe it is so, but consider 1000 concurrent writes..

Why writing to result cache concurrently is a bad thing then?

It is because , the result cache is protected by a single latch.

In order to write to the cache, it is required to get an exclusive latch.

This exclusive latch is need to be get when there is a cache miss or cache invalidation..

So cache hit doesn't require the session to get that exclusive latch.

This exclusive latch, on the other hands; is a single one, and it has no child.

It is basically, when you want to write to cache, you need to get it -> concurrent write access -> Problem

However, if you read the cache (cache hit), you need to get a shared latch -> no contention -> NO Problemo :)

Here is what Oracle Support says about it ->

The result cache was designed to cache small results sets for a short period of time and once it gets full there is an overhead of handling the data in it. When no empty blocks are found in the RC we attempt to make room first by removing any invalids, then requesting a new chunk to be allocated and finally by removing using LRU. Currently, there is a cost associated with making room in a full result cache, which increases with size of the cache, especially for LRU processing and this is unlikely to change.

If you indirectly or directly put an object or let's say a result into the result cache and if the data/object that produces that result is dynamically changing; then guess what?

You will need to write to the cache..

-- By the word directly, I mean using hints -> SELECT /+result_cache/

You will have Result Cache related activity.

You will see a peak for the latch named Result Cache: RC latch.

Okay.. It is a perfect place to end.. See you in the next post :)

EBS -- poor performance in Autoconfig -- due to Java and unnecessary waits on Syscalls -- FUTEX_WAIT & FUTEX_WAIT_PRIVATE

Recently upgraded an EBS database from 11.2.0.4 to 19C. The source apps tier was an EBS 12.1.3 and the database was an Oracle Ent. Edition 11.2.0.4. Both applications and database was running on Oracle Linux 7 Operating Systems (OEL 7.9). 

Due to the low application version, we took lots of patching actions before doing the actual database upgrade and the multi tenant conversion.

The source environment was a multi-node E-Business Suite. It was consisting of 2 SSO enabled apps nodes (Non-shared Application Filesystem), and a 2-nodes RAC database.

The production iteration was our third iteration in this project (note that we do 3 iterations at least..). 

So we already had perfected our operation. We had a detailed SOP and a Runbook. We had a detailed upgrade schedule and we estimated the duration of the upgrade tasks carefully.

But! the production 19C upgrade took longer than we thought.

The problem was in the execution of adpatch actions and autoconfig.

We had lots of patches to apply to the apps tier nodes and we had to run autoconfig several times just to make the apps tier ready for the 19C upgrade. ( you know these things changes from env to env.. In this env, there were several apps tier patches required and needed to be in place before upgrading the database to 19C and converting it to a multi-tenant  -- 1CDB-1PDB)

The real cause that made the autoconfig and adpatch run slower was an inactive wait.. We kept following the autoconfig logs during those executions for instance... And we saw that autoconfig was waiting even during the JDBC connections.. There were no issues with the database connections but autoconfig was just waiting in these kinds of basic operations and that's why the total run time of autoconfig was extending.. The total duration of an autoconfig was almost 15 mins and this was an issue that can not be ignored.. Adpatch was not doing any better than autoconfig by the way..

Anyways, we used strace to get the system calls of those running java processes.. You know... In apps, we start with sh, we continue with perl and most of the time we end up with java.. :) So we traced the java processes while they were being executed by the autoconfig (or adpatch)

We saw that the java processes were waiting on FUTEX_WAIT system calls while we were seeing those unnecessary waits.. 

Futex / Fast Mutexes are just the locking mechanisms that are used for basic locking, or as a building block for higher-level locking abstractions such as semaphores and POSIX mutexes or condition variables. This is why, we thought that there was probably a contention on a memory location, which was protected by one of those mutexes (operated by calling futexes :)  

Following is from the Man page:

long syscall(SYS_futex, uint32_t *uaddr, int futex_op, uint32_t val, const struct timespec *timeout, /* or: uint32_t val2 */ uint32_t *uaddr2, uint32_t val3);

The futex() system call provides a method for waiting until a certain condition becomes true. It is typically used as a blocking construct in the context of shared-memory synchronization. When using futexes, the majority of the synchronization operations are performed in user space. A user- space program employs the futex() system call only when it is likely that the program has to block for a longer time until the condition becomes true. Other futex() operations can be used to wake any processes or threads waiting for a particular condition.

The uaddr that we saw in the FUTEX_WAIT system calls was always the same, not changing.

So same processes were waiting on FUTEX_WAIT and the uaddr was always the same. .(this might be normal & expected, because of the implementation of virtual memory..)

Note that, I just mentioned FUTEX_WAIT so far but FUTEX_WAIT_PRIVATE is not so different than that. It seems, FUTEX_WAIT_PRIVATE is just the product of an optimization done by linux glibc to make futexes faster when they're not shared between processes. -- so just wanted to shed a light on this one.. Lets continue;

So at first we thought that those FUTEX_WAITs were caused by a contention. In order to see the contention and find the blocker ,we installed "stap" and used the script that was provided by Redhat.. (note that, the problem was on an Oracle Linux but they are almost the same right :)

The installation of stap was a little throublesome, but we installed it and used the procedure given in the Redhat article which is publically available in the following url > 

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/systemtap_beginners_guide/futexcontentionsect

--IDENTIFYING CONTENDED USER-SPACE LOCKS

Suprisingly, there was no contention reported, even when the autoconfig was waiting for getting an established jdbc connection.

Then, we made a deep dive, checked some futex examples, wrote some code to implement futex waits and guess what we found; the contention was not the only possible cause of futex waits. A poor written code or a blocking task (slow I/O, CPU shortage , high load) could also indirectly cause this kind of an issue.

Okay now; just suppose I m the main thread and your are the child thread.. Now suppose, I ( the main) was just written in such a way that, I just acquire a lock (semaphore, mutex, futex you name it) and then create "you" / the threads and then do some other fast stuff (some things that require I/O) and only then I release that lock. 

Suppose;  you the threads are written in such a way, that you need to get that lock to start doing your actual work. Okay not a dead lock, but it is a lock!

Now suppose those fast stuff that I just mentioned were blocked.. So I m waiting for their return and that's why I don't release the lock.. Well, you will "wait"..

Okay, I need to stop this story telling, because it just started getting weird :) And I felt like we should speed up a bit. Therefore, I am now heading towards the conclusion.

So, we thought that OS or JVM might be the real cause that was making us wait, but we also thought that a poorly written code might be the cause.. So we might need a patch for it. (autoconfig patch, a tech patch or something like that..)

Java thread dump didn't help much. By the way, we got it by following the MOS note below;

*How to Obtain a Thread Dump (Stack Traces) from a Java Process or from a Core File of a Java Process on Linux (Doc ID 1282871.1)

Then, we tried dropping the fs caches of Linux.. Because some example codes, that were written in a way to read some files and import the contents of those files into the database; were running very fast in their very first executions, but they were start waiting on FUTEX_WAIT in their subsequent exeuctions. So, we thought that this might be caused by a misbehaviour of Linux FS caching and we just tried to drop the fs caches and retest.. 

It was a nice try but it didn't solve the issue :)

Okay, I m speeding up!

Well, finally we found the cause..

It was /dev/random & I was there , done that! :)

Read -> https://ermanarslan.blogspot.com/2014/07/weblogic-weblogic-java-processes-hang.html

Read -> https://ermanarslan.blogspot.com/2020/05/entropy-linux-kernel-csprngs-devurandom.html -- this is one my favorites.. "Entropy, Linux Kernel - CSPRNGs, /dev/urandom vs /dev/random and all that"

Some background info about /dev/random and /dev/urandom:

In Linux, we have /dev/urandom and /dev/random for this. These are character devices and they look like files. We read them like we read files and when we read 100 bytes from them, they actually run CSPRNG on the entropy pool and give us the random number we need.

These tools provide us limited and uniform random bytes when we need. Moreover, the source they are fed, is populated by the unpredictable events.

But, as you may ask, we have two devices, right? /dev/random and /dev/urandom.. So which one should be used in which case? This is definitely the question that one may ask.

Well, first describe the difference between these tools, so that maybe we can make a decision depending on those differences.

The main difference between /dev/random and /dev/urandom is that, /dev/random tracks the entproy that we have in the entropy pool and it blocks when the entropy is low. (remember the entropy that I mentioned in the first part of the blog post).. It is basically implemented in a way to block itself when it thinks that the unpredictability is low.

Reference for the above: Erman Arslan's Oracle Blog :)

Entropy and Claude Shannon again! :)

We tested it by providing /dev/urandom using a command line argument to those java programs ..Djava.security.egd=file:/dev/./urandom did the job and we saw that this action cleared the waits.

However; we had to generalize it and somehow made it system-wide. 

We used the following for that ->

• Open the java.security file of the related JDK/JRE.. (ex: JAVA_HOME/jre/lib/security/java.security)
• Change the line: "securerandom.source=file:/dev/random" to "securerandom.source=file:/dev/./urandom"
• Note that, we need to change the line to /dev/./urandom. Otherwise, java ignores it... For instance java ignores /dev/urandom.. (the one without /./ is ignored!) 
• Save the changes.. 

Ref: https://docs.oracle.com/cd/E40972_01/doc.70/e40973/cnf_jvmrand.htm

Okay. That 's it :) This was for all my followers, and those ones who work in Oracle Application Technology & Oracle Linux Support..

I hope it will be useful.. 

EBS 12.2 -- custom OAF Deployment doesn't realize the new code -- due to misplaced files/codes under OA_HTML

We had an issue with the custom OAF deployment in an newly upgraded EBS 12.2.10 environment.

We just couldn't make the new codes be activated and no matter what we did, the application was still using the old codes and we were not able to deploy the changed OAFs.

Actually, our deployments were fine, but the changes that were supposed to be delivered with those deployment, were not reflected to the system. 

Even when we deleted a custom code from JAVA_TOP, the page that was related with that code was still working without any problems.. So it was a clue.. That is, somehow EBS was getting the code from elsewhere.. Note that, we were facing this issue for almost all custom developments under JAVA_TOP. 

Before finding the real cause, we did the following, but didn't help;

Stopped all the application services, recompiled JSPs, started the application services

Cleared the java cache using Functional administrator

Cleared the browser cache

Cheared any networking cache ( including the Load Balancer)

Then, we saw that there were 2625 classes in the directory named $OA_HTML/WEB-INF/oracle.

The custom classes were also there..

So, we removed; any code (except for _pages) that reside in the directory named $OA_HTML/WEB-INF/oracle; and the issue disappeared.. 

As you may guess; the real cause was the misplacement of code.. 

That is, when the custom classes are deployed to OA_HTML/WEB-INF/oracle, they take precedence over customall,jar generated by adcgnjar under $JAVA_TOP. 

So no matter what we deploy to JAVA_TOP, it doesn't matter.. I mean, if the code we deploy to JAVA_TOP exists in OA_HTML/WEB-INF/oracle then, the code that resides in OA_HTML/WEB-INF/oracle is executed. So we can't see our changes, we can't activate our changes..

That was an interesting issue, and the question "who put the codes there?" -> is still a mystery :)

Erman Arslan's Oracle Forum / March 2022 - "Q & A Series"

This month, we had interesting issues. We were chatty :) and I tried hard to find time to answer all the questions despite the load of my work (leading my department, mission critical support activities & projects + intellectual gymnastics - physics and stuff like that ) ! :)

This month, we (GTech) had also a mission critical EBS 19C database upgrade task. Besides, I had 2 presentations .. One of them was in a physical Oracle event and other one was in online webinar.. So it was a busy month, but things went well, and I am here blogging again :)

Remember, you can ask questions and get remote support using my forum. Just click on the link named "Erman Arslan's Oracle Forum is available now". Click here to ask a question", which is available on the main page of Erman Arslan's Oracle Blog.

-- or just use the direct link:

https://ermanarslan.blogspot.com/p/forum.html

Come on, let's see what we've been up to in March 2022.

#amaerman

WorkFlow Mailer PROCESS file by big

Rpm 32 bit ebs by satish

Form not open after upgrade to 19c by raiq1

Autoconfig failed, Non supported character set by raiq1

Migrate pluggable database from filesystem to ASM by raiq1

R12.2 Login page freezes by big

memory by Roshan

hcheck error 19c upgrade by raiq1

ap_holds_pkg.insert_single_hold by big

SSL termination - load balancer - R12.2 by satish

DMZ r12.2 admin server by satish

Sizing concurrent manager processes by satish

Migration to new server by satish

Rapidwiz nonshared appltop by satish

Patching actions in shared appltop by satish

Upgrade R12.2.0 to R12.2.10 fresh installation by satish

send email from oracle by Arsalan

RMAN recover error by Roshan

move redolog by big

App and db on single node - migration of ebs to new server by satish

Status diagram by big

Tools for migrations by satish

Workflow System Administrator by big

Rpm 12.1.3 EBS by satish

supplemental log group by Roshan

How to check custom forms in EBS R12.1.3 by soumya

Start Stop Apache by big

Cannot Launch Forms by big

certification OCI by Roshan

JDK for R12.2 by big

oracle cloud by Roshan

RAC node eviction by Roshan

row lock by Roshan

Migration of ebs R12.1.3 by satish

R12.1.3 migration from el5 32 bitto 64bit by satish

audit_file_dest by big

RDBMS -- Oracle Database 19C & 21C - Release Support Schedule, Innovations, RU and RURs, Patching & Upgrade + some background info

This event will be like the continuation of our previous (physical) event, where we presented our knowledge for almost all of these subjects (given in the title of this blog post) to a private group of Oracle Users (key accounts).

In this event we will have a closer look on the new features of 21C, the latest innovation release. We already started our tests in our Lab, and we will try to show those new features in action, with some appropriate contexts. We will also give you the needs for using these features along with the benefits of using them.. Of course, we will also give the definitions and some background information for making the audience understand the database innovations, that are delivered to us, better..

We will go through the following subjects;

• Oracle Databases Current Releases, Future Releases, and Maintenance Statuses
• Brief information about Bugfix/Patch and Upgrade processes
• Oracle Support Processes -- short background
• New Features of Latest Innovation Version 21C

As you may already know, Oracle is on its way to becoming a converged database. (or let's say an autonomous convered database).. So the new features are exciting for that reason as well.

This will be an online event. It will be in Turkish and it will be done with the contributions of Oracle and Techdata. This event will be held on 31 March 2022 10:00 AM (Istanbul Time).

Note that, I will update this blog post when the Sign Up link will be activated.

UPDATE --> here it is ;

https://gtech-tr.zoom.us/webinar/register/4016469968976/WN_52UymQOXTcq13kX128sANw

Başlık:

GTech ile Oracle Veri Tabanı Webinarı

Tanım:

Güncel Sürümler, Yeni Özellikler ve Bakım Süreçlerinde Anahtar Stratejiler

31 Mart 2022 tarihinde gerçekleşecek webinarımızda, Oracle Veri Tabanı mevcut sürümleri, bakım durumları ve güncel sürümlerin yeni özellikleri hakkında bütün detayları aktaracağız.

İçerik:

Oracle Veri Tabanları Mevcut Sürümler,

Beklenen Sürümler ve Bakım Durumları

Bugfix/Yama ve Upgrade Süreçleri

Oracle Support Süreçleri

Son Innovasyon Sürümü 21C 'nin Yeni Özellikleri

Konuşmacılar:

- Erman Arslan, GTech, Veritabanı ve Sistem Yönetimi Kıdemli Direktörü

- Özer Evci, GTech, Veritabanı ve Sistem Yönetimi Müdürü

Oracle ACE sertifikalı GTech uzmanlarından, Oracle veri tabanları hakkında bütün detayları öğrenmek için webinarımızı kaçırmayın!

Tarih: Mar 31, 2022 10:00 AM in Istanbul

RDBMS -- Oracle Database Event in İstanbul with Oracle, GTech and Techdata / Releases, Patching, Upgrades, 21C & 23C

Last week, we (GTech, Oracle and Techdata) organized a database event in Istanbul..  

In this event, I shared our expertise (along with the key strategies) in areas like database release management, database support, patching and upgrade. Although it was a private event for the key customers, the hall was full. There was a great  interest in our event.

It was a physical event and we found that opportunity to meet the Oracle customers face to face. We discussed the future of Oracle and the things that "we" (oracle users) need to do for being aligned with the latest enhancements in the RDBMS area..

I have also gone through the key new features of 21C and mentioned that, the next long term support release will be 23C.

I shed a light on new patching and upgrades methods and presented some new methods on these contexts.. Things like zero downtime patching,  and zero downtime upgrade... 

I tried to give some insights for explaining the methodology behind the scenes to make the community be familiar with the latest enhancements in the area of database patching and upgrade.. (After all, being in the latest code level requires patching & upgrade. These things just start with patching...)

I also tried to give some background info for the bugfix and development processes that are operated in the development side of Oracle.. The main code line, the branches, the delivery of Oracle releases, the one-offs, merge patches, the release updates, revisions and all that.

I found some room in my speech to include Oracle Support related things as well.. (escalations, requests for backport, inclusion and stuff like that)

We know that 23 is not released yet. That 's too bad but! we know those new features of 21C are the harbinger of the features of 23C..

I will not go any further and explain the things I just mentioned in more details here, because I already decided to write a separate blog post about these subjects.. (-- just after finishing this one:)

Okay.. Sharing the pictures below as memories;

Erman Arslan's Oracle Forum / January and February 2022- "Q & A Series"

I still did not break the tradition (even in 2022!) and continued to support customers and the community despite the load of my work (leading my department, mission critical support activities & projects + intellectual gymnastics - physics and stuff like that ) ! :) 

Remember, you can ask questions and get remote support using my forum. Just click on the link named "Erman Arslan's Oracle Forum is available now". Click here to ask a question", which is available on the main page of Erman Arslan's Oracle Blog.

-- or just use the direct link:

https://ermanarslan.blogspot.com/p/forum.html

Come on, let's see what we've been up to in January and February 2022.

#amaerman

Jan & Feb 2022 Issues:

row lock by Roshan

FIPH_WS_GBCP_PKG.pls by big

Migration 2 node rac to new server by satish

R12.1.3 migration from el5 32 bitto 64bit by satish

Standard to enterprise edition by satish

19c upgrade with EBS 12.1.3 environment by raiq1

Password file 12c by satish

OEM 13C by Arsalan

CONTEXT_FILE when cloning by big

Separate Remittance Advice by big

Add target agent by Arsalan

COLMAP by Roshan

session character set in SQL DEV by big

windows11 by serdar

User who read dictionary tables by big

large lreg trace file by big

SSO by Zakia

FRM-92101: there was a failure in the forms server during startup by baig

Request Header for Html pages by big

Note Java 1530033.1 and JRE on 19c DB tier by Laurel

restore rman backup from windows to linux by Arsalan

Patch 21916712:R12.AP.C by big

Workflow mailer not coming up. by baig

CVE Jan 2022 by big

HTTP 400 by big

NFS slowness by Roshan

R12.2 Installation by satish

Last PSU by big

Change Server Management to another VLAN by kuMan

config.xml and connection-filter-rule by Laurel

Secure Headers Normalized TLS by big

EBS_system schema by Laurel

row cache lock by Roshan

Form & report upgrade by raiq1

Hyperion -- Failure of Webserver Bridge -- After SSL/TLS implementation

A short story for today:)

An issue on a SSL/TLS enabled Hyperion 11.2.7 environment reported to me.. This issue was directly related with the SSL/TLS implementation. This is because it was reported that the issue started to appear just after enabling SSL/TLS on Hyperion.

Okay.. I will keep it short this time.. 

This is related with the wrong config in OHS-Weblogic..

OHS logs and Weblogic Managed Server logs told us the details, but as it has started happening after SSL/TLS config, it was already clear that, the cause was something related with the configuration.. 

(Note that, these types of issue may also caused by some incompatibilities between Hyperion tech. and the SSL/TLS versions used in implementation.. Also note that, there are few bugs around these things..

Anyways, It is good to go through MOS documents below.. (of course after analyzing the logs..)

After Applying OHS/Web-Tier Patch Set 11.1.1.9 : SSL Connection to WLS fails with "Failure of Web Server bridge" (Issue 1 - SSL Protocols) (Doc ID 2087174.1)

"Failure of Web Server bridge" If Mod_wl_Ohs Is Configured to Talk SSL with WLS and Strong Ciphersuites Are In Use (Doc ID 1604551.1)

After Configuring SSL between OHS and WLS, the Application from the Browser Fails with "Failure of Web Server bridge" (Doc ID 2198073.1)

After Upgrading Oracle HTTP Server to 11.1.1.9 SSL Connection to Weblogic Fails with Error "Failure of Web Server bridge" and "wl_ssl_open failed. Failed to initialize secure connection" (Doc ID 2573369.1)

*In our case, the solution was found while reviewing the setup with the MOS Note -> "Steps to Setting Up SSL Offloading with OHS Webserver From EPM 11.1.2.x (Doc ID 1530169.1)"

Just wanted to share.. This is a direct info for a specific case I know, but it may save you time! (at least the context & the error & MOS document mappings will ease your work for diagnosing these issues on Hyperion :)

ODA X8-2 -- Quick Info for Presales + Backup VMs using KVMBR on Virtualized environments

This blog post is about ODA X8-2, the latest generation of Oracle Database Appliance, which is an engineered system family that I have written about lots of times earlier. But! this time it is a little different, as this post is a little defensive :) I m actually writing to defend ODA against the doubts on virtual machine backup & recovery.

Just a little background information, before getting straight to the point;

ODA engineered system family gives us multiple options in terms of  machine size and scalability. We have the HA model for high availability and we have M (Medium) and S(small) models for having fast and reliable single node database (and apps) environments.

The purpose of this blog post is not to give you a detailed information about ODA X8 model family.. 

I will just give a quick, general info about ODA X8 HA and then I 'll get straight to the point.. My actual purpose is to give you some insights that you can use for defending ODA in the presales activities.

In general, if it's the case; I mean if the sizing tells us something like "Exadata is a little too much for the customer environment", we may consider ODA to be placed for the customer's needs. 

For hosting PROD, TEST and DEV environments, we may recommend the ODA X8-2HA. In such a scenario, where we have ODA X8-2 HA for hosting the PROD, ODA X8-2M can be placed as the DR platform.. -- as long as things like computational power and disk capacity of the proposed ODA solution is aligned with the outputs of sizing, and the requirements of the customer.

With its virtualized structure, X8-HA can run both database and application layers in clustered mode (active active or active passive). X8-2M &S can also host both database and applications. (in a non-clustered manner)

Production environments along with the test environments, can be hosted on the ODA machine.. In such an environment, Production environments can be isolated  from the  TEST/DEV/UAT environments (if desired).. 

As for lowering the costs, ODA X8-2M or S can be placed in the DR for hosting standby databases and standby application nodes.

Following is a quick info for giving you the benefits quickly;

ODA X8-2 HA and ODA X8-2 M/S

I am attaching the data sheets of the products. I briefly list the advantages of ODA below;

ODA X8-2 HA Data Sheet

ODA X8-2 S/M Data Sheet

I briefly list the advantages of ODA below;

• Simple, Optimized, Affordable Database System with Built-in Automation (Capabilities and resources for hosting application servers as well.)
• Integrated patching facility. (HW + SW)
• Oracle Database and Application server Licensing with Pay As You Grow and Get Capacity On Demand approach
• Cost-Effective Consolidation Platform for Databases and Applications
• High performance and modern hardware
• Easy installation and setup
• Single point support.. (Oracle supports hardware, OS, DB and Apps tiers)
• On-Premises Database and Application Platform aligned with the cloud (easy pathway for the Cloud) 
• Simplified operations via the easy-to-use Appliance GUI
• Stable, cost-free and  easy-to-manage virtualization (with Oracle Linux KVM)
• Solutions for Database & Guest VM backup recovery

The virtualization solution used with the ODA X8-2 is Oracle Linux KVM. 

KVM is the virtualization used in Oracle Cloud Infrastructure as well. 

Oracle invests heavily in Linux and KVM areas. In addition to the Linux Guests, we can also run Windows Guest VMs on ODA/on KVM  -- if we desired to do so..

Of course, ODA shouldn't be thought as a general purpose, an enterprise level consolidation solution. 

It is not a replacement for VMware. But! in some cases, it can replace VMware-like environments.. Especially in smaller project based cases... 

For instance, we can consolidate an Oracle EBS platform as a whole on a virtualized ODA X8-2 HA.. 

EBS TEST, PROD, UAT, DEV database and apps tiers can be hosted on this platform isolated from each other.. 

But! Don't expect all the tools and capabilities that you have on Vmware-like environments to be available on ODA. 

Once again; our real purpose with ODA is not to replace big Vmware-like environments.. 

Our purpose is something different.. It is having a low cost, high tech, reliable and robust engineered system for meeting  the platform requirements of our mid-sized projects or our product based solutions..  Remember the EBS examples that I gave above...

Let's get to the main topic now.. The main reason why I wrote this blog post... It is the backup & recovery on ODA. 

Yes, we have RMAN, RSYNC and ACFS snapshot feature which can be used on ODA environments. 

RMAN (with or without all integrated 3rd party backup tools) can be used for DB backups. 

The filesystem agents of those 3rd party backup tools can also be used for backing up application filesystems and Oracle Homes if needed.

I don't see a need for giving details for RMAN, RSYNC and other agent-based backup solutions, as we all already know those things perfectly well.

Snapshot feature is a little interesting but it is not a big news for us, because it's been around for a long time already. 

Check following blog post, where I demonstrate ACFS snapshot-based database backup & recovery on ODA -> 

https://ermanarslan.blogspot.com/2017/02/oda-x6-2-using-acfs-snapshots-for.html

ACFS Snapshots can also be used for Virtual Machine (KVM Guest) backup & recovery on ODA.

Being able to use such a feature, I mean the ACFS snapshot feature for Virtual machine or Database backup & recovery is nice to have, but some customers (especially the ones using Hyper-V or Vmware) may want to be able to backup their Virtual Machines using a similar technique that they are used to.. 

I mean, they may want to use a Veeam-like way for backing up their virtual machines, rather than using a snapshot technology. In other words; they may want a real backup for their virtual machines.

This type of a requirement can give us a hard time but! luckily, we have KVMBR tool which can be used on ODA/KVM. (This one is the actual reason that made me write this blog..) 

So, we have an answer for the ones questioning the online VM guest backup capabilities of ODA!

KVMBR the the tool for those purposes.. I mean to to backup/restore KVM domains running from ACFS/LVM or a generic folder repository. 

We can execute our backups without downtime and with limited CPU/IO impact as the file operations are executed with nice/ionice.

KVMBR supports Guest VM repositories on Oracle ACFS, LVM and generic folders.

Please see the following MOS notes for more information on KVMBR and ACFS snapshot method for backing up KVM guest machines on ODA environments.

KVMBR - Tool to backup/restore KVM domains (Doc ID 2688239.1)

Backup of KVM guests on ODA 19.9 BM and later (Doc ID 2779329.1)

That's it for today :) See you in my next post..