EBS 12.1 -- Apps + DB Failover test / Instructions

In a mission ciritcal environment, we have at least one disaster site for EBS.  

In order to make it work, we use dataguard to replicate the db tier, and we generally use rsync or any similar tool or method (such as storage replication, vm replication and so on) to replicate the apps tier.

Normally as a core dba, a disaster failover test means a dataguard fail over operation, however; in EBS world it is a little different, as we have an apps tier and an apps schema to manage in addition to the database tier.

Actually, I wrote several blog posts about these subjects, but in this one; I want to give you some direct instructions to accomplish such a failover test.. 

This tests can also be done while implementing the business continuity in EBS environments, just to be sure that the disaster implementation is accurate.

The scenarios for these tests may vary, however; I will give you the instructions for a specific one. for hte scenario which is mostly used one actually...

Suppose, you have an EBS environment. (multi node, RAC or single node, non-RAC it doesn't matter)

You have a vm replication for the apps tier, and we have a dataguard configuration for the db tier.

Now, you want to test this disaster environment without touching the PROD.. I mean, your PROD will continue running, while you do your test in your disaster environment.. 

In this case, the plan should be as follows; 

• Stop the data guard and apps tier replication.
• Save the state of the standby db tier. (guaranteed restore point)
• Backup tnsnames.ora, listener.ora in your standby db tier, to revert back the changes when your tests are done.    
• Open the standby db tier read/write.
• Build a new context file for the standby db tier (we suppose that the hostname of the standby db tier is different from PROD env.)
• Run fnd_conc_clone.setup_clean;
• Run autoconfig on standby db tier. 
• Manually edit the context file of the standby apps tier according the standby database tier and its listener
• Run autoconfig on standby apps tier. (if the hostname of standby apps tier is different from the Prod apps tier -> run adcfgclone.pl instead)
• Do the following updates (if hostname of the standby apps tier is different from the prod apps tier -> just in case... )

            SQL > update apps.fnd_concurrent_requests set logfile_node_name = <new application tier node>, outfile_node_name = <new application tier node> where logfile_node_name = <old application tier node> and outfile_node_name = <old application tier node>;

            SQL>update apps.fnd_conc_req_outputs set file_node_name=<new applications tier node>

where file_node_name=<old applications tier node>;

• Change instance specific profile options if necessary
• Change printer settings if necessary
• Change workflow configuration settings if necessary
• APPLCSF variable if necessary
• Start standby apps services

Do your test! That 's it. Once your test are done, 

• Shutdown the standby apps services
• Restore standby database to the guaranteed restore point
• Resume dataguard flow from where it left off. (configure the standby tier's listener and everything back to their originals , according to the standby role)
• Resume apps tier replication / vm replication.

Note that, this is the method we follow most of the time .

It is the required method when you don't follow the exact steps while you are building your EBS Disaster environments .. I mean the steps documented in  Oracle Support (MOS).. Those are the actual steps you need to follow while you are creating/configuring the EBS disaster environments.

I see that, most of the time, EBS disaster configurations are made without following these steps line  by line..

I'm actually talking about documents like the following;

Business Continuity for Oracle E-Business Suite Release 12.1 Using Oracle 12c Physical Standby Database (Doc ID 1900663.1)

Business Continuity for Oracle E-Business Suite Release 12.2 Using Oracle 12c (12.1.0.2) Physical Standby Database (Doc ID 1963472.1)

Look we have those kinds of documents even for 19C  :) -> 

Business Continuity for Oracle EBS 12.1 on Oracle Database 19c Using Physical Host Names (Doc ID 2567091.1)
Business Continuity for Oracle EBS Release 12.2 on Oracle Database 19c Using Physical Host Names (Doc ID 2617787.1)

Normally, we must follow the Business Continuity documents for implementing EBS Disaster Recovery environment, and if we follow those documents line by line; the failover and switchover steps are easier.. 

For instance, we don't need to run adcfgclone.pl even if the hostname of the standby apps tier is different from the prod apps tier.