Erman Arslan's Oracle Forum -- May 1, 2025 - Feb 6, 2026 - "Questions and Answers Series"

Question: How much time do you spend/lose?

Answer: Well, how much time I gain? :)

Remember, you can ask questions and get remote support using my forum.

Just click on the link named "Erman Arslan's Oracle Forum is available now.
Click here to ask a question", which is available on the main page of Erman Arslan's Oracle Blog -- or just use the direct link: 

Erman Arslan's Oracle Forum

More that 2400 questions have been posed, sparking over 10,000 comments. Explore the latest discussions and see what valuable knowledge awaits!

Oracle EBS, Cloud, Exadata, ODA, KVM, Oracle Database, OS and all that.

Supporting the Oracle users around the world. 

Let's check what we have for the period Between May 1, 2025 - . Feb 06 2026.

Extract Generating huge trail files due to JAI_TAX_DET_FACTORS table by kvmishra

Timezone upgrade - R12.2 by satish

Unable to update Supplier or Customer Address after upgrading from 12.2.11 to 12.2.15 by UbaidurRehman

Auditing - Oracle 19c database by satish

Peak concurrent users in R12 EBS by satish

Patches in oracle 19c by Arsalan

DBCS - ASM over provisioned by satish

Disk group with external redundancy by satish

RTF Footer Issue by Junaid.Niazi

Issue with RTF If Condition in Footer by Junaid.Niazi

fs_clone fails on secondary nodes with unable to start OHS component error by ky_17

Issue with open form ebs by NguyenThienAn

Migration of physical to vm - licensing by satish

Many SSL errors in OHS and opmn log files by DBAFR62

PDB CDB by Laurel

Concurrent Requests Completing with Error Status in EBS R12 by Rabia

SSO asks password on any connection by big

ADOP Prepare [UNEXPECTED]Patch edition does not exist. by Aldair Anzh

many large sid_p000n.trc by big

SQL not accepting SQL Profile by VinodN

Invoice print requests not completing by VinodN

Custom concurrent program querying FND tables by VinodN

Issue with SYSTEM Tablespace Size in ebs 12.1 by Yahia

EBS 11i migration from aix 6 to oel 6 on oci by soumya

Duplicate by Erman fails by big

Auditing only SQL Developer commands by VinodN

Adop Apply partial patch not restarting by Ahsan0001

AppsLogin/AppsLocalLogin are not working. by Mansoor8810

Internal manager doesn’t start by Samia

adsplicer fails: This is not a valid application id by big

EBS on VM by satish

golden gate replication by Roshan

EBS Database upgrade from 12.1.0.2 to 19c by soumya

Oracle Forms and Reports Cluster in windows by Waseem

RDF based report output is not coming by kvmishra

The Gravity of Context: From Transformers’ Attention to Oracle 26ai

This post will be about AI. We 'll start by going back to early days of transformers, the 2017 moment, the attention mechanism and all that, and in the final we 'll use a metaphorical bridge to connect the Transformer math with Oracle’s in-database applied AI.  Our metaphorical bridge will be "the internal attention mechanisms". A little bit of autonomous database , a little bit about 23AI / 26AI, but we 'll get there... Okay. Let's start and we'll see :)

Let's start with a specific point in time we call the "2017 moment". I'm talking about the release of the Google paper "Attention Is All You Need." Attention mechanisms actually existed as early as 2014 (Bahdanau) , but the 2017 Moment was the declaration of independence. 

*To the architect of the 2017 moment / Ashish Vaswani: thank you for proving that attention is indeed all we need to reshape the world of AI.

While Bahdanau added attention as a support for the old RNNs in 2014, the 2017 paper made the bold claim that we didn't need the RNNs at all -> Attention was all we needed. 

Here we note that; actually, the industry is now moving towards hybrid architectures, to "Attention where you need it." But, I will not get into that. Let's get straight to the point without causing too much confusion.

The paper "Attention Is All You Need" did exactly what the title said. Before that, we (of course not the actual "we", the AI models) were stuck in the linear age, processed one word at a time, left to right, and it was slow. The memories these models had was short, and subject-specific processing couldn’t be parallelized. They couldn't scale it. But! the 2017 paper showed we could build an entire architecture using only attention, and it fundamentally reset the AI timeline. It provided the ability to seeing sentences as a big, interconnected web of context.

I will explain this with my favorite example. The excuse of the young students (not those from Turkey, but most likely those from America.. An old one, there must be some modern versions now :)

That is, the sentence "The Dog ate my homework"

So, just like in the old days of RNNs, models tried to carry the entire past context in a single, ever-changing hidden state. However, as the sentence grew, that memory faded, and the model would often lose track of the subject by the time it reached the end. If the model fails to explicitly link the word my back to dog across that distance, an error becomes almost unavoidable...

I mean let's imagine we give the input "The dog ate my" to the model and want it to tell us what's the next word should be... The word "my" itself doesn't make the model to choose the word "homework" easily. 

In fact the model gets lost here. So what I want to emphasize is that, in order to be able to guess the correct word,  ("homework" in this case), the model should  attend back to the word "dog". So, it is like catching the context.. Without the context, you generate wrong answers, or you get lost. You need to have a complete understanding of the sentence and somehow indirectly grasp the context from it.

Okay we begin with the transformer, but what is it? Well.. The Self-Attention Mechanism is the key. The Transformer is essentially just a efficient frame built specifically to let the attention mechanism run properly at scale. 

With Attention, transformer  becomes a machine that can see every word in a sentence simultaneously. And what's going on in the background is mathematics.. The cool math tricks made these things appear as they are.. In the example above, the Transformer creates a direct mathematical link between the word my and the word dog instantly.

Here, it's more about vector math. The vectors are being massaged... The vectors corresponding to the words are being related and massaged together, and the final vector is no longer the same vector it was when it started; it somehow becomes related to the previous vectors, and from this point on the next word is predicted more easily and accurately. That's basically what's happening.

Let's go a little far, let’s look under the hood to see how it actually works  The logic of Transformer is based on the high-speed dance of three vectors: Query (Q), Key (K), and Value (V).

Query: When the Transformer processes the word "my", it uses the vector generated for that token. So the Query vector asks: "Who in this sentence is relevant to me?" Every Key vector answers: "Here's what I offer." The dot product measures compatibility..!

Key : Every other word ("The", "dog", "ate" ...) has a Key vector. Its like an identity card.. These are for describing what they offer to the sentence.

So, we calculate a dot-product (a well known matrix operation. note that there are other methods as well) between query and key (Hint: Query vector of token “my” is compared against key vectors of all tokens) and run the scores through a Softmax function, which squashes the numbers into probabilities. 

If the word 'dog' gets a high score, the model focuses its attention there. But! this does not mean the model looks only at the word dog.. But we can say that, the value vector of “dog” contributes the most to the resulting representation.

Dot-products measures how close vectors are. So if they are close/aligned in vector space, a large number, high score is obtained from dot-product.

Softmax converts these to probabilities. I mean, it takes these numbers and compresses them all between 0 and 1. It makes their sum equal to 1.

An example for score(s) and softmax finale:

-----------------------

"My" is the focus, so we calculate dot-product of "my" with other words.

Scores: [The: 1.2, dog: 5, ate : 3.5, my: 2] 

Actually what happens is; Query of "My" -Dot Product- Key of "Dog" = 5 (High! Because "my" is looking for an owner, and "dog" is an object that can be owned.).. Note that, this is just for the analogy, actually transformers do not encode semantic intent like ownership, the model does not know why vectors align... :)

After Softmax : [The: 0.02 , dog: 0.75 , ate: 0.18 , my: 0.05]

Decision: I need to focus on the word "dog" 75% of the time (This is the attention!), when processing the word "my".

-----------------------

As for the finalization, we multiply these scores by the value vectors.. So, to get the contextualized vector for "my", we multiply each value vector by its score and sum them all up. The result is a new vector and "dog" contributes the most to to it. I mean dog contributes the most to the contextualized representation..

(remember -> 75% after softmax). 

So, the word "my" doesn't go alone when it moves to the next layer. 

The model takes the value vector of the word "dog," multiplies it by 0.75 (meaning it takes the value vector of dog with almost full strength), and then adds the value vector of "ate," scaled by 0.18. It even looks at the word "the" with its tiny weight of 0.02, and the result becomes : "my" is no longer just "my"; its like having my' (my prime) in hand, a dynamic, contextualized vector that has been mathematically pulled toward the dog and influenced by the act of eating.

This allows the model to predict the word "homework" in the next step, knowing that the word "my" is related to "dog" and the act of eating.

Okay.. This is like massaging, and it happens over and over through multiple layers. Each layer refines the meaning, making the vector more context-aware as it moves toward the final prediction. (Note that: this is not like the biological reasoning; it's an iterative mathematical refinement where each layer assigns better statistical weights based on the surrounding context )

By the time data flows through the transformer layers, the word "my" has been literally reshaped by the context of the "dog"'. The Transformer is the architecture that allows this massage to happen across billions of words at the same time. It doesn't care about the order of the words initially. It processes the entire context at once. However, it uses positional encoding to understand the sequence and keep the story straight, And it doesn't just do one message at a time; it runs multiple versions of this attention (Multi-Head Attention) in parallel, layers deep, to refine the meaning until the alignment is spot-on.

So far so good. Let's take a look at the training phase a little bit. There we see weights.. In the training phase, the model doesn't store the final vectors; it stores weights. When those weights are hit by a word, they produce the vectors. So, training adjusts the weights so that the resulting vectors point in the right direction. Training isn't just about guessing; it’s about adjusting the weights so that the resulting vectors point in the right direction at runtime. 

Okay.. The weights are important, they indirectly result vectors and result things like where they point and their alignment. Vectors are there in the beginning, but they are not static. 

In the training, weights are learned that will produce good vectors. So during the runtime, we get input, we already have weights, and using these two, contextual vectors are computed fresh.

For instance; the word "bank" transforms into completely different vectors in the case of the following 2 sentences; "I went to the bank to withdraw money" and "I sat on the river bank".

These new, contextual vectors are calculated at runtime. There's no pre-existing vector of "riverbank" in memory; the model constructs that meaning instantly, using the weights of the surrounding words.

In other words; the word "bank" doesn't have a stored "river" version or "money" version. The weights know how to construct the right vector based on context at runtime.

Anyways; during the training of the model, the model guesses answers.. Cross-Entropy Loss function takes place here.. If the probability that is given by the model for the correct answer is low, the loss is high. At this stage, back propagation kicks in by regarding the loss (the loss tells you how much it missed the target) and It takes actions to update weights so that future vector representations align more closely with the desired outputs. So, in other words, indirectly, those actions are taken to bring vectors that need to be close together closer together. ( this is very high level, but! I won't go any deeper here)

Now we have arrived at the point where we will cross our metaphorical bridge, where we will connect our subject to the Oracle.

Oracle didn't just watch the 2017 revolution, they built the new database with the result of it. The ability to perform AI tasks directly within the database was introduced in Oracle 23AI, and further improved in 26AI. 

In fact, we conducted RAG (Retrieval-Augmented Generation) studies with 23AI, created Sales Assistant solution using Oracle Vector Space and RAG, implemented software solutions for speech-to-SQL demos.

*To the pioneer of RAG / Douwe Kiela: thank you for giving AI a reliable memory and leading the charge as the ultimate hallucination reducer.

Since Oracle Database 23AI , the Vector is a native data type. That similar math and approach we just explained is now happening directly inside the database kernel.

By performing In-Database Vector Search, we can feed the transformer’s context window with rock-solid facts, effectively becoming Hallucination Killers through RAG (Retrieval-Augmented Generation). Note that, actually RAG doesn't eliminate hallucinations but reduces them :)

Anyways; I mean by implementing RAG on the Vector data that sits inside the Oracle Database, we basically say: "Do not guess! Use this data!"

It is an important unification of AI (Vectors, and DB integrated GEN-AI with RAG) , Graph and Native JSON in the context of Converged Oracle Database. With 23 AI and now with the evolved version: 26AI, Oracle goes beyond traditional database logic and has the ability to run inference-related AI operations directly within the database. 

This means that instead of constantly moving data to external AI environments ( or to things like vector databases), we can now perform core AI operations within Oracle itself. In older versions (like 19C) , when these types of processing is required: we needed to export data to a ML environment, run Python, TensorFlow, or OCI Data Science jobs externally.  Then we needed to push the results back into Oracle. In 26AI (actually started with 23AI), this changes completely.

Now the database itself understands vector data, offers AI-oriented SQL syntax, and includes built-in packages for inference, embeddings, and vector search.

In addition to that, Oracle 26 AI offers support for open table formats, reinforcing its converged database strategy. Actually, this is not a new thing, but 26AI i is the version that matures and standardizes this feature. That is,  Oracle can natively integrate with Apache Iceberg and Apache Parquet, allowing you to store or access data in these universal formats. You can create a table (connected to the Iceberg in the backend) using DDL as part of the Oracle database and query iceberg data directly inside an Oracle database session using SQL. (Interesting, isn't it?).

So, Oracle can seamlessly integrate Iceberg by directly querying it without the need for importing or moving the data. We can even join this external Iceberg table with a relational Oracle table.

For instance;

Oracle Session / SQL:

SELECT
    c.customer_id,
    c.first_name,
    c.last_name,
    c.email,
    s.segment,
    s.risk_score
FROM sys.customers_iceberg c --> data retrieved from iceberg / queries iceberg table.
LEFT JOIN sys.customer_segments s --> data retrieved from relational Oracle database table
  ON s.customer_id = c.customer_id

ORDER BY c.customer_id;

This means significantly reduced data movement and less complex ETL processes. The philosophy "query the data where it lives" is fully applied here, making it a truly AI-native Database. It's worth a try... (Read for more:
https://www.oracle.com/tr/database/ai-native-database-26ai/)"

Let's wrap things up, and before we finish, I'd like to remind you of Kepler.

In general, we must do what Kepler did with the stars! Kepler didn't just look at the stars and say, "If this is here today, it must be there tomorrow." Kepler changed the game by discovering the underlying laws that governed them.

In this case, having AI inside the database may be considered as the Kepler Moment for enterprise data. 

While the Transformer models are the engines of generation, Oracle provides the high-speed vector engine for retrieval. Same dot-product and similarity search operations that allow a transformer to find the 'dog' in a sentence are now happening directly inside the database. 

So, it is a Keplerian shift: the database is no longer a passive storage room; it's an active participant that understands the gravitational pull between our data points. We are moving from "What happened?" to "What is the underlying pattern?"

Of course, a new turning point is still needed. Let's see when that turning point from 2017 will come again. 

Therefore, I would like to conclude my writing with these words;

In a broader context, I think we should be waiting for a new fundamental shift, maybe another 2017 Moment, a fundamental shift that will take the AI world beyond the limits of sophisticated pattern matching. We should be looking for an architectural leap where reasoning, causal understanding, and the essence of human experience are no longer just simulated, but are inherently woven into the very fabric of the model.

Exadata X11M / X11M-Z / Naming, Elasticity, RoCE and all that

Recently dived into a new project with newest Exadata(s) inside :) and today, I wanted to share you a quick info about the newest Exadata naming based on the elasticity we have, and then I will remind you the power of Exadata by revisiting things like the Game Changer : RDMA, and the mantra : "The fastest I/O is the I/O that never happens."

Okay. First the naming, and the elasticity.. These days we see Exadata X11M-Z systems in the field.. Well, for years, we dealt with eighth rack configuration, which, let's be honest, were often just quarter racks with some cores and disks software-locked in earlier generations / historically (Note that this is true for X5, and X4.. But! for X7, and onwards Eighth Rack database servers have one of the CPUs removed, so no change for them).  

Anyways, now, all Exadata X11M systems are offered as flexible configurations, starting with at least two database servers and three storage servers. So we can start with an economical and flexible configuration using Exadata X11M-Z database and storage servers, and later if needed we can expand the system by adding more X11M-Z servers or larger X11M servers. As a result, the fixed Eighth Rack configuration is not required and is no longer available. So it is more elastic now. Of course you can built something like eight rack or quarter rack. But! know that: quarter/half/full: These names are now simply nicknames describing how full the cabin is, not official product codes.

However; still if you check the data sheet, you may see some quarter rack configuration. But! when you check it carefully, you see it there in two different ways:

Standard Quarter Rack, and Z-Series Quarter Rack.

In other words, Quarter Rack has become a term that refers to the minimum occupancy rate inside the cabinet, rather than a hardware model.

Okay, so far so good..

Alright, let's dive deep into the internals, focusing on the fast OLTP capabilities and the magic of RDMA and Persistent Memory. 

You may already know the mantra: "The fastest I/O is the I/O that never happens." But what if you have to do I/O? What if every microsecond counts? This is where Exadata isn't just fast; it’s architecturally brilliant, especially for OLTP workloads.

We often talk about Smart Scan and its analytical powers, offloading terabytes of data processing to storage. But for OLTP, where individual transactions need sub-millisecond response times, a different feature of Exadata takes place. That is the RDMA and the intelligent use of Persistent Memory (PMEM).

Most of the time, we see traditional Storage system fall short...Think about a typical OLTP transaction: a single row update or a quick lookup by primary key.. These tasks are like searching a needle in the haystack.. Actually sometimes this phrase is not enough to describe the hardness of it.. Let's say we are looking for a particular needle in a stack of needles :)

 In a traditional architecture, even for a single block read, we (in the background) issue an I/O request through the operating system, and then we wait for the storage controller to process it. The data travels over the SAN network. Then, our database server's OS receives it, handles interrupts, and context switches. Finally, we get our data in our Oracle database instance.

Each of these steps adds latency. So, thousands of transactions, these delays (although there are micro) create significant bottlenecks. This is where Exadata's features comes into play.

With RDMA / Remote DMA / Remote Direct Memory Access, what happens is, we bypass the Kernel, and we go directly to the memory...

So, Exadata completely re-engineers this critical path for OLTP I/O. Instead of the database server's CPU and OS being involved in every single I/O operation, Exadata leverages RDMA over RoCE interconnect.

With RDMA, the database server can directly access memory on the Exadata Storage Cells, and it gets the data it needs with only a minimal CPU involvement on Storage Cell's CPU or OS.

So, basically it bypasses the kernel, which means less context switching, fewer interrupts. 

Okay.. Let's visit the subject of Persistent Memory (PMEM). Well, Exadata Storage Servers come equipped with PMEM, a revolutionary technology that sits between DRAM and Flash. 

It's like DRAM but persistent like Flash. Exadata intelligently caches frequently accessed OLTP data blocks in this PMEM. When the database server needs a block that's in the PMEM cache of a Storage Cell, it uses RDMA to read it directly. You see the benefit, right? The power of Exadata is not only due to having a fast storage; it's intelligent, again! -> software and hardware engineered to work together!

In this blog post, I focused only on some of the key features and put them forward in the OLTP context, by I think we all already know the other well known features like Smart Scan and Storage Index.

These are still there, making Exadata a worlds one of the fastest database machine for dealing with hybrid workloads, as it is capable of handling both OLTP and OLAP simultaneously without compromise.

Understanding these underlying mechanisms always excites me , and these things are not just academic, it's crucial for delivering real-world results.

Until next time, keep optimizing, keep questioning, and keep digging into those internals.

Feel free to share your thoughts here and your questions on my forum: http://ermanarslan.blogspot.com.tr/p/forum.html

OLVM connectivity issue due to hosts' SSL Certificates, KVM hosts were appearing in Down status & info about the script: OlvmKvmCerts.sh

I want to share the solution that we implemented for fixing a recent issue. It was about Oracle Linux KVM and there was a misconfiguration issue with the SSL certificates of this problematic KVM environment. The certificates were attempted to be renewed manually, and the problem arose after that.

To quickly summarize the issue: KVM hosts were appearing in Down status within the OLVM (Oracle Linux Virtualization Manager) interface. Consequently, VM information and metadata were inaccessible.

During our diagnostic work, we identified issues related to the hosts' SSL Certificates. The libvirtd and vdsmd services on kvmhost2 failed to start, reporting "Authentication failed" and SASL errors. Observations noted that the host kernel version was outdated and the system had an uptime of 1877 days without a restart.

We followed the MOS Note - OLVM: How to Renew SSL Certificates that are Expired or Nearing Expiration (Doc ID 3006292.1), but! the OlvmKvmCerts.sh script was missing. So we created a SR, and got the script from Oracle Support. After that, the steps to the solution were as follows;

We renewed the certificates using the OlvmKvmCerts.sh script. (OlvmKvmCert.sh renew-all) -- executed on OLVM node.

kvmhost1 returned to an "Up" status in OLVM -- immediately following the certificate renewal. 

For resolving kvmhost2 issue, the vdsm-tool configure --force command was executed in kvmhost2 to resolve persistent configuration issues. However; the output was not that good;

Checking configuration status...
SUCCESS: ssl configured to true. No conflicts
Running configure...
Error: ServiceOperationError: _systemctlStart failed
Job for libvirtd.service failed.

At this point, we manually (re)started the following services: libvirtd, mom-vdsm, vdsmd, and supervdsmd. (via commands like: systemctl restart libvirtd mom-vdsm vdsmd supervdsmd) -- in some cases there may be a restart needed for the ovirt-engine as well.. ( on the engine: symstemcyl restart ovirt-engine)

After this restart, the issue was resolved for kvmhost2 too.. All statuses were confirmed as "Success".  We also executed the OlvmKvmCert.sh.again (just in case) for the kvmhost2 and this time OlvmKvmCert.sh was completed successfully.  This was just a check to ensure that we will be in the safe side for a possible future certificate renewal.

Note that, ovirt-log-collector helped a lot for diagnosing the issue.

Some references:

OLVM: OlvmKvmCerts - Script to Check or Renew Hypervisor Certificates (Doc ID 3008653.1)

OLVM: How to Renew SSL Certificates that are Expired or Nearing Expiration (Doc ID 3006292.1)

GoldenGate Error: OGG-02249 The Extract configuration does not support a physical standby / & the need for Downstream Capture

Recently, I saw people dealing with errors while configuring an Oracle GoldenGate Extract process against a Physical Standby database. In this post, I will share the cause of this error and the recommended architectural approach to resolve it.

*When starting the extract, the process abends with the following messages in the ggserr.log:

2025-12-20 18:13:36 ERROR OGG-02249 The Extract configuration does not support a physical standby as the source database.

2025-12-20 18:13:36 ERROR OGG-01668 PROCESS ABENDING.

This error occurs when a GoldenGate Extract is configured to pull data directly from a physical standby database that is not properly set up for such operations. By design, traditional Extract requires access to specific redo log structures and supplemental logging.

A standard physical standby (Active Dataguard / ADG) is typically in a read-only state and does not inherently support direct extract operations in the same way a primary or logical standby does.

Well. This means it is not supported to make CDC with Goldengate from an physical standby including the Active Dataguard-based ones directly.

So if we need to offload the extraction process from our primary production system to a standby environment, we should consider using downstream capture configuration. I mean; GoldenGate Integrated Extract can be configured to work with Active Data Guard using a Downstream Capture configuration. In this setup, the mining process runs on a separate database (the downstream mining database). This also offloads the CPU and I/O overhead from the primary instance.

What about using classic capture instead? Well, it is important to note that Classic Capture mode is deprecated. Therefore, moving towards Integrated Capture is not just a fix for this error, but a strategic requirement for future-proofing our environments.

**In summary, if we need to make a standby database as the source for Goldengate, we should include a data mining server in to the picture. Production will also replicate to the data mining server, which is also an Oracle Database running in Read Write mode, and then Goldengate will capture from there..

Data mining server (an Oracle Database) is not a physical standby bytheway. It is configured to receive redo from the source, but it is in read write mode. Redo transport will still be there, logs are shipped over the network from the source database to the data mining server and the log mining server in the data mining database extracts the changes from the redo log (or archive) files and serves them to the GoldenGate extract process.

That's tip for today. I hope it helps.

Monitoring Oracle Goldengate Monitoring / Oracle Enterprise Manager & the pluging & JAgent

In complex enterprise environments, the greatest danger lies in the failure.. But! the delay between a failure occurring and its detection is also very important. The longer it takes for detection to occur, the greater the impact of the problem will be. When dealing with an engine (like Oracle GoldenGate) responsible for moving crucial data, this delay translates directly into financial exposure and data divergence. Our recent project focused entirely on eliminating this blind spot, achieving a reliable, centralized view of OGG's operational state.

Monitoring is not a luxury; it is necessary for managing highly available, complex systems. Oracle  Goldengate is such a critical system. It's a bridge, where data passes between systems, and very critical tasks are then performed using that data. By implementing OEM 13.5 GoldenGate Plugin and JAgent, we can properly monitor Oracle Goldengate (OGG) landspace(s) -- even if we are using OGG in the classical mode (non-microservice mode). 

With this approach, we also have the capability of  doing the level 1 administration of our OGG environment (s) through a web-based GUI.

So, as mentioned, the tool we positioned for monitoring OGG was Oracle Enterprise Manager (OEM) Cloud Control 13.5. This is not just a dashboard; it is a central monitoring solution for our databases, middleware, and, the entire OGG environment.

We specifically deployed the OGG-EM-Plugin into OEM for this task. This plugin delivers centralized visibility into all OGG environments.

To achieve the visibility, one must establish a standardized communication path between OGG processes and management console. This path is realized through a layered agent architecture.

The OGG processes periodically generate monitoring points. These are essential data points such as status, lag, and checkpoints.

Then, these monitoring points are sent to the GoldenGate Monitor Agent, named JAgent. This agent is installed on both the source and target database servers and is kept up-to-date with the latest patch bundle(s). The JAgent's role is to collect the data from the local GoldenGate environment.

The JAgent then communicates the collected monitoring data to the standard OEM Agent. And finally, the OEM Server, empowered by the GoldenGate Plugin, receives the data.

Once the architecture is in place, the OEM Server transforms raw data into actionable intelligence. 

The GoldenGate Plugin allows us to populate dashboards with crucial performance indicators (like status, lag, and total operations) and automatically trigger alarms when metrics reach predefined thresholds. OEM also give us the capability of administering the OGG processes (this includes: initiating a stop/start action for any process, and checking alerts and errors on the processes)

We can also make configuration changes to already running parameters.

The resultant dashboards provide status updates. For instance, we can clearly see the Lag for all Extracts and Replicats, allowing quick identification of bottlenecks. By drilling down into a specific Extract, we can see fine-grained metrics, such as Checkpoint Position Delta Operations Per Second, and detailed operational counts (Inserts, Updates, Deletes).

OEM delivers us a web-based central for monitoring and administrating OGG, and it shifts our operation from a reactive to a proactive centralized model. 

We implemented this in one of our data analytics projects, and I can say it performed quite well. 

Things are a little different with the new microservice-based OGG platform. I'll examine that in another article.

Jagent / Monitoring Agent 12.2.1.2 & Oracle Goldengate 21.3 / java.lang.IllegalArgumentException: source parameter must not be null

 You are using Oracle Goldengate 21.3, the classical one (not the microservices architecture based one), and you want to monitor the activities of Goldengate?

You installed an Oracle Enterprise Manager, deployed the Goldengate Plugin, installed Jagent / Monitoring Agent 12.2.1.2 into the targets and configured them. 

You saw PMSRVR and JAGENT processes in GGSCI outputs, you started them, and they were in RUNNING status, and you told yourself: okay, so far so good.

Then, you used Oracle Enterprise Manager's auto discovery for discovering thee monitoring agents by the using relevant information like the port , host and etc..

Oracle Enterprise Manager didn't get any errors and discovery completed successfully, but! nothing changed? Couldn't Goldengate Monitoring agents be discovered?

Then you jumped into the servers where you installed those agents and checked the listen port by using netstat and everything seemed fine.

However; when you checked the log of those agents, you saw something like the following.

Could not get WSInstance Information , [[
com.goldengate.monitor.model.AgentCommunicationException: Failed to get Registry.
at com.goldengate.monitor.jagent.comm.ws.ManagerWSApi.getInstanceInfo(ManagerWSApi.java:239)
at com.goldengate.monitor.jagent.comm.ws.ManagerWSAdapter.getInstanceInfoNative(ManagerWSAdapter.java:105)
at com.goldengate.monitor.jagent.comm.impl.ManagerFacadeImpl.fillInstanceMap(ManagerFacadeImpl.java:259)
at com.goldengate.monitor.jagent.comm.impl.ManagerFacadeImpl.getInstances(ManagerFacadeImpl.java:253)
at com.goldengate.monitor.jagent.jmx.MBeansContainerImpl.createMBeans(MBeansContainerImpl.java:344)
at com.goldengate.monitor.jagent.jmx.MBeansContainerImpl$1.run(MBeansContainerImpl.java:836)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.IllegalArgumentException: source parameter must not be null
at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:119)
at com.sun.xml.internal.ws.message.AbstractMessageImpl.readPayloadAsJAXB(AbstractMessageImpl.java:135)
at com.sun.xml.internal.ws.api.message.MessageWrapper.readPayloadAsJAXB(MessageWrapper.java:171)
at com.sun.xml.internal.ws.client.dispatch.JAXBDispatch.toReturnValue(JAXBDispatch.java:88)
at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:274)
at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:289)
at com.goldengate.monitor.jagent.comm.ws.ManagerService.getRegistry(ManagerService.java:267)
at com.goldengate.monitor.jagent.comm.ws.ManagerWSApi.getInstanceInfo(ManagerWSApi.java:230)

Then you thought deleting and creating datastore commands would help, but you got invalid command errors in GGSCI ( cause those commands were already deprecated in the OGG version that you were using).. 

Is this what happened?

Well.. Here is the cure for you ->  Patch 31997138 - Oracle GoldenGate Monitor 12.2.1.2.210930 (PS2 BP8) (Server+Agent) 

That patch will solve it.

I will give the same message again: 

Never take any initiative. Take all the steps documented in Oracle documentation.

Follow the steps in the documentation exactly 

Follow the correct documentation.!   For this case it is ->  "Doc ID 2314622.1 How To Enable Monitoring For GoldenGate 12.3.x Targets Using Oracle Enterprise Manager 13c R2+"

 

That's the tip of the day..

Oracle GoldenGate and ODI Working Together for Near-Real Time Data Ingestion for DWH Projects

We're all chasing the same goal, near-real time data. You simply cannot run a modern business, especially in the Financial Services sector, with batch processes that deliver data hours late. That's where Change Data Capture (CDC) shines, and in our recent DWH project, we deployed the absolute best: Oracle GoldenGate (OGG), especially when we replicate data from Oracle to Oracle..

Let's be clear about what CDC is. It is a vital mechanism to capture only the data that has changed , ensuring minimal load on the source database. For a Data Warehouse, this is the gold standard for data ingestion and even lightweight transformations. Of Course we're not talking about querying the whole table every few minutes; we're talking about direct, surgical capture from the transaction logs. (redo logs and archived log in Oracle case)

Oracle GoldenGate is not just a CDC tool; it's a real-time replication solution, and dedicated to that. For an Oracle-to-Oracle environment, it’s arguably the best in class.

Some background info about the key capabilities of Oracle Goldengate;

Goldengate ensures continuous data synchronization and maintains high availability. If a process fails, it has the ability to resume operation after the failure.

While best with Oracle, it can handle replication across diverse environments, including PostgreSQL, MS SQL, and MySQL.
By using Goldengate, we can transform data before it hits the target, a key feature for ensuring data quality early.
The architecture supports multi-thread and/or multi-process/multi-target setups, which is critical for handling massive volumes of transactional data.

From now on; I will use OGG acronym for Goldengate. Just noted that here..

The beauty of OGG lies in its streamlined and robust process architecture.

Let's give some information about the processes of OGG, and how we used them in our recent project..

Extract: This is the capture agent. In our project, we chose the Integrated Extract. This process is configured on the source database and is responsible for capturing changes from the source database, writing all captured DML operations to a local trail file. We chose Integrated Extract because it uses Oracle server-side internal operations to get the changes, and it automatically scales with the CPU for better performance and high throughput.

Pump: In our recent project, a Pump process reads the local trail files and securely sends them over the network to the target OGG environment. This process acts as a high-speed data distributor. (note that, these path can be secured by TLS/SSL enablement) 

Replicat: This is like the the apply agent, configured on the target database. It read the remote trail files and apply the changes to the target table(s). We use parameters like ASSUMETARGETDEFS to map columns automatically and even UPDATEDELETES to treat incoming DELETE operations as UPDATEs, crucial for DWH history tracking.

Note that; This structure was successfully deployed on a massive scale, including the capture processes for a 1 TB table with a billion rows (X_TABLE).

OGG have light transformation capabilities, so at a point a crucial handover might take place. So in our case, in a recent DWH project that handover was from "OGG to ODI for Data Conversion"..

This is where the entire CDC flow comes together for our DWH project. OGG’s job is to efficiently and reliably get the changed transactional data from the the critical core PROD / source system into a stable staging layer, which we call the ODS (Operational Data Store)

The flow is explicit.

Source - > ODS  

Once the data hits the ODS, OGG's primary task is done. Of course we do some lightweight transformation here, we used SQLEXEC configurations for doing these things in OGG and we enriched the replicated data, with the operation type and timestamp to make the change history for the records to be maintained in the target / ODS & DWH. But the heavy transformation was in the next state.. The next stage involves the complex, business-rule-heavy transformations necessary to structure the data for the final DWH targets.

This is the point of handover to Oracle Data Integrator (ODI).

ODS -> ODS2 / DWH

ODI then uses its trigger-based mechanism, specifically designed like CDC, to be triggered by the changed records and read the changed records in the ODS. ODI's job is to manage the parsing and heavy transformation logic.

We are talking about journalizing here.. Journalizing in ODI is a change data capture mechanism that tracks inserts, updates, and deletes in source tables , allowing downstream processes to load only the changed data.

So OGG captures the changed data and write to the target, there ODI captures the written data, triggered with it and continues processing..

So, OGG for high-speed, real-time capture and replication into a staging area, and ODI for complex, CDC-aware transformations into the final structure. At the end; we achieve an architecture that is both highly efficient and massively scalable.

This is how you build a modern DWH fit for todays need. Real Time DWH and Data Lakehouse(s) are one step ahead.  In order to have real time DWH, all data processing logic should be done in a single streaming layer, and all data should be considered a stream. Next time, we will take a look at those things as well.

That's it for today. I hope you'll find this blogpost helpful.

Oracle Linux KVM vs. Red Hat KVM for Oracle Products

A recent discussion on my forum highlighted the importance of KVM selection for Oracle Customers.

A member of my forum created a thread and mentioned that they are moving their critical E-Business Suite (EBS) system from physical hardware to a virtual environment. They see KVM as the key to proper, aligned licensing ( due to capabilities of hard partitioning and CPU pinning).

So far so good.. But which KVM?

They were using Red Hat Enterprise Linux (RHEL), and they know that KVM is baked into RHEL. So, they were planning to install RHEL, run KVM, and use that to host their Oracle database and EBS. They knew that Oracle Products (database and EBS in this case)  were certified with RHEL. 

However; there was an important distinction there. I mean, the operating system running the application is one thing. The operating system running the hypervisor that is defining your license boundary is another. So we must differentiate...

Oracle's policy on virtualization is clear, technical, and brutally enforced. For you to claim Hard Partitioning (the ability to restrict your Oracle license count to just the cores assigned to the VM), you must use a technology that Oracle explicitly approves.

Oracle has been very specific. You cannot rely on a non-Oracle KVM vendor's implementation of CPU pinning for licensing purposes. 

Yes, Oracle Products will run on RHEL KVM , but it is important to note the following->

Oracle Products are not certified to run on virtual machines/guests provided by Xen or KVM offerin
gs by Red Hat, Novell, SUSE, Ubuntu, Citrix, Nutanix, or XenSource."
So, you may have issues if you go with Redhat and KVM, and if any issues arise, you ll be the one that will try to solve them.. (Redhat may support you but it depends according to the issue type.)

That means, if you run Oracle software on a VM hosted by Red Hat KVM, even if you technically pin the CPUs, there is still a support risk and licensing risk.!

Support Risk / Not Certified: If the issue is related to the virtualization layer, Oracle Support can, and likely will ask you to reproduce the issue on a supported platform. In such a case, you may be the one trying to solve complex kernel-level issues.

Licensing Risk: The license auditor will consider this Soft Partitioning. This means you are liable for licensing the entire physical server's capacity, regardless of your CPU pinning efforts. The cost savings you planned for are gone.

Note: in my opinion; , there is no difference between running an Oracle Linux with Red Hat Compatible Kernel and running a Red Hat Enterprise Linux, binary wise. So it is better to go with Oracle Linux in that sense as well..

That means the only way to use KVM and confidently argue for core-level licensing is to use Oracle Linux KVM.  That is the platform specifically engineered and approved to meet the hard partitioning criteria for Oracle licensing.

Here is that thread about this topic: http://erman-arslan-s-oracle-forum.124.s1.nabble.com/Migration-of-physical-to-vm-licensing-td13040.html

In summary;

Reference MOS Document ID 417770.1 -> Oracle products are not certified to run on guest virtual machines provided by Red Hat's KVM offering.

You can use Redhat KVM but you may issues with Oracle Support (if there is a need) + you may have license -cpu core alignment issues...

Use Oracle Linux KVM..

Finally, here is what Oracle says: "Only Oracle Linux KVM can be hard partitioned. No other Linux KVM can be hard partitioned because the use of Oracle software is required to implement and monitor the cpu pinning. No other Linux distribution includes the required olvm_vmcontrol utility. The OLKVM hard partitioning CFD says "The olvm-vmcontrol utility is required to set and get the CPU/vCPU bindings for a virtual machine running on Oracle Linux KVM through the Oracle Linux Virtualization Manager."

That's it. Tip of the day:)

Oracle Linux Boot Failure: A GLIBC Version Mismatch / GLIBC_2.33' not found

In this blog post I share a real production incident and its resolution. While the issue was severe, proper troubleshooting methodology and rescue media made recovery possible without data loss.

An Oracle Linux 8.10 production server suddenly became unresponsive. The system would boot but freeze indefinitely at the graphical login screen, showing only the Oracle Linux logo with a loading spinner that never completed.

No amount of waiting helped. The system was completely inaccessible through normal means. SSH connections timed out, and the console remained locked at the authentication screen.

Our initial discovery was trough the emergency shell access. I mean, to diagnose the issue, we bypassed the normal boot process using GRUB emergency parameters:

# At GRUB menu, press 'e' on the kernel line

# Add to the linux/linuxefi line:

rw init=/bin/bash

 Once we gained shell access, the true nature of the problem became apparent immediately:

[root@myprodddb01 /]# rpm --version

rpm: /lib64/libc.so.6: version 'GLIBC_2.33' not found (required by /lib64/libcrypto.so.1.1)

[root@myprodddb01 /]# yum

yum: /lib64/libc.so.6: version 'GLIBC_2.33' not found (required by /lib64/libcrypto.so.1.1)

[root@myprodddb01 /]# dnf

dnf: /lib64/libc.so.6: version 'GLIBC_2.33' not found (required by /lib64/libcrypto.so.1.1)

Every foundational system command was broken. This was not a simple misconfiguration, this was a fundamental system library corruption.

GLIBC_2.33' not found was the error. Well, lets first take a look at the GLIBC. 

GLIBC (GNU C Library) is the core system library that nearly every Linux program depends on. It provides essential functions for:

• Memory management
• System calls
• String operations
• File I/O
• Network operations

Without a working GLIBC, the system cannot function.

That's enough for the giving the background..

So, Oracle Linux 8.10 ships with GLIBC 2.28. However, our system's binaries were looking for GLIBC 2.33 and 2.34, which are part of Oracle Linux 9 (based on RHEL 9).

[root@myprodddb01 /]# /lib64/libc.so.6 --version GNU C Library (GNU libc) stable release version 2.28

The library version was correct (2.28), but the programs themselves (rpm, yum, ping, dnf) were looking for libraries from Oracle Linux 9.

How did this happen? In our case this is not certain yet, but we have some clues and here is the list of possible causes for such a catastrophic situation:

• Mixed package repositories - OL9 repos were accidentally added to an OL8 system
• Manual binary replacement - Someone copied binaries from an OL9 system
• Failed upgrade attempt - An attempt to upgrade from OL8 to OL9 went wrong
• Incorrect package installation - Installing .el9 RPMs on an .el8 system

Anyways; to fix the system, we needed rpm to reinstall packages. But! rpm itself was broken because it needed GLIBC 2.33. We couldn't use yum or dnf for the same reason. Even basic networking tools like ping were non-functional. The broken system could not fix itself.

The solution was in rescue mode recovery.

We booted from Oracle Linux 8 ISO, and entered Rescue Environment. The rescue environment automatically detected and mounted our system to /mnt/sysimage. The rescue environment provided working tools with correct GLIBC 2.28.

sh-4.4# /lib64/libc.so.6 --version GNU C Library (GNU libc) stable release version 2.28 Copyright (C) 2018 Free Software Foundation, Inc.

sh-4.4# lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

sda 8:0 0 80G 0 disk

├─sda1 8:1 0 1G 0 part

├─sda2 8:2 0 1G 0 part

└─sda3 8:3 0 70G 0 part

└─root--vg-root--lv 253:0 0 70G 0 lvm /

sh-4.4# mount /dev/mapper/root--vg-root--lv /mnt/sysroot 

We then identified the corrupted packages using the rescue environment's working rpm: ,

sh-4.4# rpm --root=/mnt/sysroot -qa | grep "\.el9"

This command listed all Oracle Linux 9 packages installed on our OL8 system.

And, we copied the GLIBC 23 libraries (and libcrypto) from rescue to our broken system.

cp -fv /lib64/libc-2.28.so /mnt/sysroot/lib64/

cp -fv /lib64/libc.so.6 /mnt/sysroot/lib64/

cp -fv /lib64/libm*.so* /mnt/sysroot/lib64/

cp -fv /lib64/libpthread*.so* /mnt/sysroot/lib64/

 cp /lib64/libcrypto.so.1.1.1k /mnt/sysroot/lib64/

After these actions, we chrooted into the system to verify and tested the foundational commands, they were all run successfully

chroot /mnt/sysroot

rpm --version        

ping -c 2 8.8.8.8   

yum --version 

We verified the GLIBC version.

rpm -q glibc

-- Expected: glibc-2.28-251.0.1.el8.x86_64

We rebooted and tested.

exit

sync

reboot

This fixed the issue but during the emergency shell access, we also reset the root password:

In emergency mode (init=/bin/bash);

mount -o remount,rw /

passwd root

# Enter new password

sync

reboot

Well, we fixed the issue and learned a few important things.

• Never install packages from OL9 (or any EL9) on an OL8 system, even if they seem compatible. ---Major version boundaries exist for a reason.
• Always verify yum repository configuration.
• Before installing RPMs, always check the version.
• Keep Rescue Media Ready ( to save time and be ready).
• Take snapshots before: System upgrades, Repository changes, Mass package installations.
• Monitor Package Origins : Set up alerts for package installations from unexpected repositories.