ZDLRA -- Zero Data Loss Recovery Appliance "Fast. Integrated. Zero Data Loss" Engineered for Data Protection !

We can't underestimate the importance of backup and recovery in our business lifes. Actually not only for business, but for personal life, as well..

In this blog posts, I will be concantrated on the business side of course :) I will introduce you an end to end solution for the business side, specifically for the Oracle database administrators.

As you know, backup and recovery are the most important things for DBAs.
As a DBA, you can't just say "I don't have any backups to restore or used for recovery".

If the backup and recovery is in our responsibility, and if there's a problem with the backups of production environments when there is a need to restore, then it means we are in a real throuble.

That is, if it is our responsibility to manage the Oracle database backup and recovery, then we don't have any such luxuries like not having yesterday's backup, or having a corrupted backup when it is required to be used.

This is also very important for IT managers and directors, as backup and recovery is their responsibility in the first place.

Recovery is also important as much as having backups. It must be ensured that our backups can be restored and used for the recovery purposes quickly to comply with the SLAs.

Backups should be taken continously and the checks should also be continous for ensuring the recoverability all the time.

In order to fullfil these important requirements , we use backup recovery tools.. So we need to have them, and we need to use them properly, efficiently.

We need  to schedule our backups, store them on disks, later on tape and restore them from there when necessary. We should be able to integrate several 3rd party backup solutions, we need to make them integrate with RMAN scripts, database storage policy, retention policies and so on.

We should install the tools, configure them, integrate them,  use them, and manage them and etc :) Lots of things to do and to manage right? :)

From this perspective, we need to be both responsible and well skilled.

However; eventhough we do all that we can, we may still have hard times when there is an urgent need to restore our backups.. As we can't backup continously , we may have gaps in our backups and we live with that risk time to time.

Even if everyting is in place, we may have a performance impact in our production database systems, while following these strictly defined backup policies.

In this backup tasks, and in this data lifecycle management mechanism, there are also invisible areas for DBAs. As a DBA and system admin, most of the time we can't even answer to the basic questions like where is your backup stored?  Which tape do you need to use for recovering last week's backup and so on..

There are many more things to consider, but I will stop now :)

I know I m a little pessimistic about these subjects, but these are the facts when using the traditional backup solutions.

Okay.. Here comes the solution to all these problems : Oracle Zero Data Loss Recovery Appliance,
by short name : ZDLRA.

ZDLRA is engineered for Data Protection! It is based on Exadata X8! -- Think about Exadata 's Flash.. Exadata 's I/O throughput and Exadata internal network bandwith (we have even RDMA over converged ethernet now)

Almost all the database versions are supported -> Supports 11G, 12C, 18C and 19C.
It eliminates Data Loss by providing real-time protection .. That is, it acts like a dataguard destination and continously backup the redo changes. (This is called Continous Redo Transport)

ZDLRA eliminates the production impact and reduces both the backup restore/restore times and complexity..  It backs up the changed data only and stores it in an intelligent way , so it restores efficiently. We can also offload our tape backup processes to this machine. (we can even deploy backup agents on it)

ZDLRA Delivers Cloud scale protection... That is, ZDLRA has the ability to serve data protection to thousands of Databases. It has the ability to scale without downtime.
We use a Policy based data protection in ZDRLA and we have an end to end visibility + control over the protected databases.

ZDLRA protects from disasters. It can replicate data real-time to remote ZDLRA environments and cloud.. This feature is an addition to the tape archival.. These replications are done transparently and in the background.

Lots of things to mention when it comes to ZDLRA. The technologies like delta push , delta store.. The incremental forever strategy, and more..

Well. So far so good... I think I have done an adequate introduction and now I leave you alone with my presentation about ZDLRA :) 

EBS -- EBS 12.1.3 / Oracle Database 19C Upgrade

It is time to write about EBS. Here we are, talking about upgrading EBS databases to 19C.

In these days, we are mostly dealing with 11.2.0.4/12C databases in EBS world, so it is the time to upgrade those databases.

A big majority of EBS customers are still using EBS 12.1.3, my blog post will be about upgrading an EBS 12.1.3 databases to 19C.

In this blog post, I will try to give you the facts about these kinds of projects and then will give you a consolidated action plan for just a quick overview.

Facts :

• EBS 12.1.3 is certified with Oracle Database 19C (as of Sep 2019)
• EBS 12.1.3 is certified with Multitenancy, 1 CDB and 1 PDB. So if you are considering 19C upgrade in EBS environments, you should know that your 19C EBS database will be a multitenant.. Oracle certifies EBS and 19C only with this condition and Oracle won't certify EBS with a non-CDB 19C database even in the future.
• With the 19C upgrade, UTL_FILE_DIR becomes obsoleted, so UTL_FILE_DIR based file access mechanisms, should be replaced with the Database Directories-based file operations.
• Uprade environment should be tested very carefuly after the Test iterations.. (custom code should be reviewed..) Performance problems may appear and if they appear they should be resolved .. Especially for the ISG(Integrated SOA Gateway).
• For Linux customers (most of our customers are on Linux), 19C requires Oracle Linux 7 or Redhat Linux 7.  So you need to consider OS upgrade if you are on a lower version.. Note that, OEL 7 upgrade can be done in-place if you are on a certain OS software level, but I find it risky. Anyways, this item must be carefully considered.
• EBS 12.1.3 apps tier is also certified with OEL&RHEL 7. Linux upgrade for apps nodes is not a must, but it is a nice-to-have thing.. Besides, this makes single node environments (Both Apps and DB in the same server) to be upgraded without a need for splitting the services to multiple nodes.
• I can't give a certain downtime requirement for such an upgrade. That's why I recommend it to be measured during the Test iterations.
• As EBS 19C upgrades require CDB-based db tier, we need to have a high level of understanding of CDB-PDB / multitenant Oracle databases. Actually, we need to learn it by hearth, or as the germans say "wir ( APPS DBAs) alle müssen es auswendig lernen" :)
• It seems Oracle made our job easy again.. I mean, things like converting non-PDB to PDB is done using perl scripts :)

Okay, let's take a look at the action plan from the surface:

Preperation :

• Applying Prerequisites Patches for the Upgrade. (19C interop,AD/TX Delta patches and etc.).
• Applying Warehouse Builder patches (optional)
• Applying Autoconfig patches.
• Applying   Patch 6400501: APPSST11G:1203:NOT ABLE TO COMPILE FORMS LIBRARRY WITH 11G DB (For Linux)
• Applying Patch 12964564:R12.FND.B - Enabling the Oracle Database 11g Case-Sensitive Password Feature for Oracle E-Business Suite Release 12.1.1+ (optional, for enabling case sensitive passwords)
• Creating the initialization parameter setup files (running txkOnPremPrePDBCreationTasks.pl)
• Install 19C RDBMS Software (Software only)
• Create Nls9i data (running $ORACLE_HOME/nls/data/old/cr9idata.pl)
• Applying 19C RDBMS Home Patches (almost 15 DB patches)
• Creating a new appsutil.zip and copying it to the required folders/servers.
• Copying orai18n.jar file to the required folders.
• Create a CDB (without any PDBs)
• Patching CDB, synching it with the 19C home (running datapatch)
• Creating CDB MGDSYS schema(running catmgd.sql)
• Creating CDB TNS Files(running txkGenCDBTnsAdmin.pl)
• Configure Transparent Data Encryption for CDB (conditional/optional)
• Shutdown CDB
• UTL_FILE_DIR(for the required UTL_FILE migration)
• Shutting down Application services and application tier listener in source
• Drop SYS.ENABLED$INDEXES (conditional)
• Disabling Database Vault (conditional)
• Exporting OLAP analytical workspaces (conditional)
• Removing the MGDSYS schema (conditional -- running catnomgdidcode.sql)

Upgrade + Conversion :

• Upgrading the DATABASE (11.2.0.4 to 19C)
• Backing up database
• Upgrading database using DBUA
•  Performing post-upgrade tasks
• Running patch post-install instructions (for the patches applied in earlier steps)
• Compiling PL/SQL code natively (optional)
• Importing OLAP analytical workspaces (conditional)
• Running adgrants.sql
• Granting create procedure privilege on CTXSYS(adctxprv.sql)
• Compiling invalids
• Granting data store access
• Validating the WF rulesets (wfaqupfix.sql)
• Gathering SYS stats
• Creating the new MGDSYS schema (conditional) -- running catmgd.sql
• Creating Demantra privileges (conditional)
• Exporting Master Encryption Key (conditional)
• Converting the upgraded database to Multitenant
• Creating the PDB descriptor
• Disabling the ENCRYPTION_WALLET_LOCATION sqlnet.ora entry (conditional)
• Updating the CDB initialization parameters
• Checking for PDB violations (review and resolve the errors if there are any)           
• Creating the PDB (run txkCreatePDB.pl)
• Running the post PDB script (txkPostPDBCreationTasks.pl)

Post-Upgrade tasks :

• Modify initialization parameters (according to the EBS-Database initialization parameters MOS document)
• Run Apps Tier autoconfig (with some additional context file modifications)
• Apply post-upgrade WMS patch (Patch 18039691)
• Recreate custom database links
• Apply latest RDBMS Release Update (Database Release Update 19.5.0.0.0, OJVM Release Update Patch 19.5.0.0.0 and other 19.5.0.0.0 patches)
• Restart application services

Post-Upgrade Support :

• Babysitting
• Error correction & Throubleshooting

Okay, we are at the end of this blog post.. Lastly, I m sharing the MOS document which should be followed line by line to do such an upgrade.. You should follow it closely.. It may redirect you to some other documents when necessary, but you will come back to it after taking the actions on those documents. So your main document is ;

"Interoperability Notes: Oracle E-Business Suite Release 12.1 with Oracle Database 19c (Doc ID 2580629.1)"