EBS -- Active/Passive Cluster + DR minimal configuration

In this post, I will share some Active/Passive EBS 12.2 Architectures extended with simple DR solutions.
These configurations can support new EBS 12.2 environments up to 100 users and  are scaled in a way that requires minimum hardware and software fees.

Note that, this configuration can be modified , extended , scaled up or down and scaled vertically according to the environmental needs. Sizing may also vary according to the expectations, needs and the EBS modules that are planned to be implemented.

First of all our hardware resource can be sized as the following;

Note that, these are based on a standard EBS, that includes HR and standard Financial modules.

In the paragraphs below , I give active/passive + DR architecture as this post is aimed at this type of configuration. (cheap, redundant, stable in a way and fast enough)

Source site:

2 servers in the source site. 1 server active, 1 server passive.
Both of the source servers are connected to a shared storage (fiber/HBA connection preferred)
Failover operations can be done manually, or using a clustering software(Oracle Restart may be a solution, or in cases of IBM, HACMP will work)

EBS apps and db tier is placed on the same server, altough it is not recommended. (we are aimed to be minimal)

DRC site.

DRC is one server.
The replication can be storage based or dataguard based or VM based, It depends on the architecture.
If using dataguard, the target server/DR server should be licensed as well.
Application filesystem synchronization can be done using an utility like rsync or using similar tools which do the same job.
Storage replication, on the other side, does not require any Oracle licenses or an utility for filesystem replication.
However, storage replication /snapshot /snapmirror utilities require their own licenses. (so this should be kept in mind)
Also, storage based replications work in block level, so they do not understand Oracle's block format. From this perspective Dataguard is the best solution for creating and feeding Oracle Replicas/Standby/DR servers. On the other hand, in these days, Storage vendors have solutions/integrated solutions for Oracle and when these solutions are implemented, Storage replications know that they are dealing with the Oracle Databases when replicating the storage level data.

As for the sizing;

Production Site:

For the Active node:

OS: Oracle Linux 6 64 bit, disks from Storage, Local disks can be minimum(300GB-500GB), 64GB memory, 16 cores, 1x HBA(2 HBA -- preffered for multipath)

For the Passive node:

OS: Oracle Linux 6 64 bit 64GB memory, disks from Storage, Local disks can be minimum (300GB-500GB), 16 cores, 1x HBA(2 HBA -- preffered for multipath)

Storage : 

A Storage with 2 controller, 15k disks (1.2 TB disk space -- usable space , which remains after raid configuration) -- disk size is dependent on the expected % of yearly growth .
Optionally, for automatic failover -> a clustering software, and for an additional replication -> a storage in the DR site , which can do replication/snapmirror (Netapp,EMC,Oracle ZFS etc..)

DRC Site:

OS: Oracle Linux 6 64 bit, 1.2 TB disk space(local), 36GB memory, 8 cores.

Optinally a storage, in case storage replication is required, in addition to the rsync and Dataguard. If that's the case, a Storage with 1.2TB can do the job(ofcourse, it should be capable of doing replication/snapmirror kind of works)

In our first configuration we use an ODA machine for the source site. We use ODA in a way to reduce the RAC license needs. That is, we use a virtualized ODA to place our application tiers and databases.
We utilize ODA_BASE domain for placing our EBS databases, but we use only one node of ODA_BASE to keep our license fees minimal.
This configuration gives us the ability to run our EBS databases or applications in the second ODA node in case of a failure.
In case of the disaster, we use Dataguard (async). We choose our DR server to be a single  node, which has the power to handle the workload(at least may handle the workload during 1 day) in case it should be used when a planned maintanence is required in the source servers or in case of a disaster in the source site.
By using the ODA's virtualization, we use the production machine as a consolidated environment, on which PROD, TEST and DEV instances of EBS (app + db ) run.
Note that, dataguard only replicates the Database file, so Oracle binaries, Oracle Homes , Application filesystem should be replicated in to that.. (using rsync or some tool like that)

Another configuration can be based on the traditional servers in the production site;

In this configuration, we still use Dataguard and rsync for the replications , so the only difference is, this time, we use traditional servers like HP , Dell or Sun and a Storage in the Production site to provide the active/passive clustering.

As for the DR site, we still have only one server, which can even be a VM.

Another alternative, may be using a storage based replication directly without having a dataguard. This configuration can be preferred in case one doesn't want to license the target/DRC environment.

In this case, the topology can be like the following;

                                     

The network between Prod and DRC is also important. A low latency will increase the recoverability.
As for the servers, here are some notes;

Production servers are preferred to be physical (For ex: HP servers, Sun Servers)
DRC server can be a VM (Oracle VM Server , ESX etc..)
Test environments can be placed on the passive production node, or they can be placed on Virtual server outside this configuration. For every TEST environment, 32GB Memory, 750 GB disk sapce and 8 core CPU is enough.
CPU Cores specified in this note are Intel. The latest generation intel cpus are prefferred.
There can be an additional storage in the Production Site, to increase the storage level fault tolerancy , as well.

Lastly, as for the "U" sizes, here are some notes;

ODA is a 4U Rack moutnable system.
The traditional servers supporting these sources can be between 1U and 4U sizes as well and if VMs are preffered, they dont even count for the U sizes, as they are virtual.Storage for this capacity can be handled with a 4U architecture and
With a quick calculation, 8U free space is okay for the production site. DR site can be somewhere between 1U-4U according to the configuration.

Important note:  The values used for sizing are completely dependent on the environment. So it may vary accordingly. All this sizing information is given for a new EBS 12.2 environment, which is expected to have the ability to support 70 users. %10 of these users are expexted to currently access the EBS applications. The environment is expected to grow %10 in a year. The configuration is sized to support 3 years. This is for an EBS environment, on which 60000 invoices are made out in a single month, and 40000 bills invoices are received in a single month.

EBS -- Upgrading EBS 11i(11.5.10.2) to 12.2, a compilation

Here is a enterance level document to be used to have a general picture before the EBS 11i to 12.2 upgrade.
It covers the navigation for the documents that are used in the upgrade process, as well as the general steps that should be performed for upgrading EBS 11i to EBS 12.2.
Note that: This document is just a compliation of Oracle documents, but  I believe it will save you time as reading it , you will not have to go back and forth among all the related documents for determining the required upgrade tasks.

Main Document :Oracle E-Business Suite Release Notes, Release 12.2 (Doc ID 1320300.1)
    ->Section 3: Instructions for Upgrade Customers :"Oracle E-Business Suite Upgrade Guide, Release 11i to 12.2 or Oracle E-Business Suite Upgrade Guide: Release 12.0 and 12.1 to 12.2 guides"

Direct Document:
https://docs.oracle.com/cd/E26401_01/doc.122/e48834/toc.htm (Oracle E-Business Suite Upgrade Guide Release 11i to 12.2 Part Number E48834-15)
----
Upgrade Path:
11.5.10 CU2 -> Minimum Patch Requirements for Sustaining Support for 11.5.10 -> Release 12.2
------

Steps:
Before starting the process, it is important to note that it is required to have a new application tier for placing the EBS 12.2's apps tier. This is required as EBS 11i 's Apps Tier is not supported to run on Linux x86-64 bit and EBS 12.2's Apps and DB tiers are delivered as 64 bit.
So at least for Linux installations, there is a need to have a supported Linux OS for placing the EBS 12.2 's Apps Tier.

1)
Upgrading to Oracle E-Business Suite Release 12.2 requires your database to be at the minimum version 11.2.0.4.
To complete the upgrade to Release 12.2, you must upgrade your database to 11.2.0.4 or higher.
Follow the instructions in Database Preparation Guidelines for an Oracle E-Business Suite Release 12.2 Upgrade (Doc ID: 1349240.1).

2)
INFO:Completing this upgrade brings your system to the 12.2.0 release.
You must apply the 12.2.5 Release Update Pack (RUP3) or later(12.2.5) to your existing Release 12.2 system for production use.

3)
Do the sizing CPU,Memory,Disk Space, IO subsystem.

4)
Know that IDO 10g and OSSO 10g will be desupported. Think about implementing OAM and Oracle Access Gate.

5)
Apply 11i.AD.I.7 (conditional) -- if you are not on AD.I.7

6)
Run TUMS utility and reviw: Download and apply AD TUMS patch

Download patch 18342870 and apply it to the administration server node on your Release 11i APPL_TOP.
It supplies you with the script (adtums.sql) you need to generate the TUMS report (tumsr12.html).
Generate the TUMS report->
$ cd $AD_TOP/patch/115/sql
$ sqlplus <APPS username> @adtums.sql <DIRECTORY>

7)
Backup the database and apps tier including customizations (cold backup preferred)
Note that: Customization is not handled during the upgrade, so the customization should be reapplied when the upgrade is done.

8)
Prepare yourself for the customizations.

You may have customized your system for business use. Take note of these important considerations before upgrading custom database objects:
The Oracle E-Business Suite Developer's Guide contains extensive instructions about naming standards and issues related to upgrading custom database objects. Familiarize yourself with this information before you begin the upgrade.
Run several test upgrades and track their impact on your custom database objects.
Rename any custom database objects with Applications prefixes that you have created so that they do not conflict with Oracle object names.

Note: Failure to test the impact on custom database objects before the upgrade can result in a loss of functionality.
At your discretion, and depending on the customizations in your system, you should also perform the following tasks:
Preserve the CUSTOM library by making a backup copy of CUSTOM.pll. You can use this copy later in the upgrade process to migrate your CUSTOM library to Release 12.2.
If you have customized forms with Oracle Forms 6i, then upgrade them to Oracle Forms 10g after the upgrade.

9)
Convert to Multiple Organizations architecture (required).
Converting to Multiple Organizations does not require the use of multiple operating units or sets of books - it does enable you to use this functionality at any time in the future.

If you are converting from a Single Organization architecture to a Multiple Organization architecture, complete the following steps:
Create an operating unit.
Assign the operating unit you created to the profile option MO:Operating Unit.
Note: See the following documentation references: Use of Multiple Organizations in Release 12 in Use of Multiple Organizations (Multi-Org) in Release 11i (Doc ID: 210193.1), and MOAC in Oracle Purchasing (Doc ID: 404800.1). HRMS users should also see Setting Up Multiple Organizations for Oracle HRMS (Doc ID: 259546.1).

10)
Drop event alert triggers in custom schemas (conditional).
To drop all event alert database triggers in custom schemas, run the alrdtrig.sql script, located in $ALR_TOP/admin/sql (in your Release 11i system). Re-create the triggers after the upgrade is complete.

11)
Review sizes of old and new tablespaces (required).
Make sure you allocate sufficient tablespace.
For guidelines based on an upgrade of the largest Oracle production system (oraprod), see Applications Release 12 Upgrade Sizing and Best Practices (Doc ID: 399362.1).

12)
Run AD preparation scripts (conditional).

Download and unzip the new patch 13435302. Follow the instructions in the readme for running these scripts:

adgncons.sql

The tablespace model for this release (OATM) is based on database object type rather than product affiliation. The adgncons.sql script prepares adcrtbsp.sql, configures the database to hold the new products to be added during the upgrade, and switches your system to use the new tablespace model.

adcrtbsp.sql

Generated by adgncons.sql, this script creates the new tablespaces, allocates unlimited tablespace to all APPS users, updates fnd_product_installation table with correct data and index tablespace information, assigns default tablespace to all APPS users, and sets the new_ts_mode flag in fnd_product_groups to Y. Be sure to review the locations of the dbf files for each new tablespace listed in adcrtbsp.sql, and change the locations in the script as necessary. Ensure that you have adequate disk space for all the changes and additions that adcrtbsp.sql is going to make.

Note: All the tablespaces created by adcrtbsp.sql are critical and required for 12.2 as they hold critical objects. Therefore, you should not drop any of these tablespaces at any point.
Note: Oracle has tested successfully an extent size of 128 K for small systems (for example, 100 GB database or the Vision database), and, for large multi-terabyte system, an extent size of 4-10 MB has been tested successfully.

adgrants.sql :(adgrants_nt.sql for Windows) Grants SYS privileges needed by Applications, and creates required views in SYS.

13)
Migrate Existing Objects to New Tablespace Model (recommended).
Running the preparation scripts creates tablespaces and prepares the database objects only for new products.
However, it does not automatically migrate your existing objects.
Oracle strongly recommends that you use the Tablespace Migration Utility to perform this migration now.
Note that this utility is not supported for use after you enable Online Patching, so you cannot perform the migration after your environment is upgraded to Release 12.2.
If you choose not to migrate to OATM now, then you must continue to manage your tablespaces separately.
Check  the Oracle E-Business Suite Setup Guide, Release 12.2 for mor information.

14)
Convert Oracle Alert E-mail Processing to the Workflow Notification Mailer (conditional)
If you use response processing alerts and you have not already converted Oracle Alert to the Workflow Notification Mailer, convert now by applying Oracle Applications Technology 11i.ATG_PF.H Rollup 6 (patch 5903765) to your Release 11i APPL_TOP. This roll up patch causes Oracle Alert to use the Workflow Notification Mailer for new alerts, but allows you to continue to run the Alert Response Processor for incoming responses sent before the conversion. Continue to run the Response Processor until you have no more outstanding responses of this kind.
Note: See About Oracle Applications Technology 11i.ATG_PF.H Rollup 6 (Doc ID: 444524.1) for more information.

15)
Oracle XML Gateway:  In release 12.2, Oracle XML Gateway Web services depend on Oracle E-Business Suite Integrated SOA Gateway which has product dependencies on Oracle SOA Suite and Oracle Fusion Middleware Adapter for Oracle Applications (also called Oracle E-Business Suite Adapter). See Installing Oracle E-Business Suite Integrated SOA Gateway, Release 12.2 (Doc ID: 1311068.1).


16)
Ensure that the GUEST account is valid and active and that the fnd_user USER_ID for the GUEST account is set to a value of '6'.

17)
Do Product specific upgrade tasks documented in : Preparing for the Upgrade secetion of "Oracle E-Business Suite Upgrade Guide Release 11i to 12.2 Part Number E48834-15" - https://docs.oracle.com/cd/E26401_01/doc.122/e48834/T527285T527359.htm
-Customer Relationship Management Tasks
-Financials and Procurement Tasks
-Human Resource Management (HRMS)
-Lease and Finance Management
-Projects tasks
-Public Sector Tasks
-Supply Chain Management Tasks

18)
Gather schema statistics (required)
FND_STATS.GATHER_SCHEMA_STATISTICS('ALL', 10, :parallel_degree, 'NOBACKUP', NULL, 'LASTRUN', 'GATHER AUTO', 10, 'N');
Where: parallel_degree is set to the value of the database initialization (init.ora) parameter parallel_max_servers for your instance.

19)
Install JRE on the database tier (conditional)
If you are planning to run Rapid Install in Upgrade Mode by using the Use Existing ORACLE HOME option, then you must install JRE in the Database ORACLE_HOME/appsutil as follows:
Download the latest JRE 7 Update. For optimum stability, performance, scalability, and OS vendor support, use the latest available update of JRE for the Oracle E-Business Suite database tier. The JRE download location is: http://www.oracle.com/technetwork/java/javase/downloads/index.html

20)
Reset init.ora parameters (required)
Follow the instructions in Database Initialization Parameters for Oracle E-Business Suite Release 12.2 and reset the init.ora parameters as needed.

21)
Run Rapid Install (required)
Use the Rapid Install wizard to lay down the file system and install the new technology stack for your Release 12.2 Oracle E-Business Suite system.

-Start Rapid Install by typing rapidwiz on the command line. The Welcome screen lists the components that are included in, or supported by, this release of Oracle E-Business Suite. Click Next.
-On the Wizard Operation screen, select Upgrade to Oracle E-Business Suite Release 12.2. Click Next.
-On the Select Upgrade Action screen, select Create Upgrade File System.
-In the associated screen flow, enter the parameters required to set up your new environment. Then, run Rapid Install.
-Note: Oracle E-Business Suite Installation Guide: Using Rapid Install contains complete instructions for running Rapid Install for both new installations and upgrades. Chapter 3 contains the information specific to running an upgrade.

22)
Synchronize values of APPLPTMP with UTL_FILE_DIR for PL/SQL based Concurrent Requests (required)

23)
Backup the database and apps tier including customizations (cold backup preferred)

24)
At this point, the database must be 11.2.0.4 or higher. So if not upgraded earlier, upgrade the database.
If you have not already done so, you can upgrade your production database to 11g Release 2 (11.2.0.4) or higher now, before the upgrade downtime. Follow the instructions in Database Preparation Guidelines for an Oracle E-Business Suite Release 12.2 Upgrade (Doc ID: 1349240.1).
Note: If you are upgrading to 11gR2 or 12cR1 from 10g or 9i, then you MUST set the parameter sec_case_sensitive_logon = False in your init.ora file to avoid login issues with sqlplus. For more information on this parameter and other database initialization parameters, refer to Database Initialization Parameters for Oracle Applications Release 12 (Doc ID: 396009.1).
If you are upgrading to a Release 12c database, then ensure that your sqlnet_ifile.ora contains the line: SQLNET.ALLOWED_LOGON_VERSION_SERVER = 8

25)
Disable AOL Audit Trail (conditional)

26)
Shut down application tier listeners and concurrent managers (required)

27)
Update init.ora with upgrade parameters (required)

28)
Set FAILED_LOGIN_ATTEMPTS to UNLIMITED for Oracle E-Business Suite schema

29)
Disable custom triggers, constraints, and indexes (conditional)

30)
Drop MRC schema (conditional) --it is no longer needed, it can be dropped.

31)
Back up the database (recommended)

32)
Enable maintanence mode

33)
Download and apply the AD Upgrade Patch for 12.2 (patch 10117518)(required) by following the patch read me.

34)
Apply all Consolidated Upgrade Patches (CUPs) (required)
--Apply the Consolidated Upgrade Patches (CUP) and all of the pre-install patches listed in the Oracle E-Business Suite Release Notes, Release 12.2 (Doc ID: 1320300.1)

35)
Run the American English upgrade patch driver (required)
AutoPatch to run the (American English) upgrade patch driver (u10124646.drv) merged with latest Consolidated Upgrade Patch for Oracle E-Businees Suite Release 12.2 . Release 12.2 upgrade patch driver (u10124646.drv) is located under $AU_TOP/patch/115/driver on Run File System.

36)
Disable Maintenance Mode (required)

37)
Follow the instructions in Database Initialization Parameters for Oracle Applications Release 12 (Doc ID: 396009.1) and reset the init.ora parameters as needed.

38)
Backup the database and apps tier

39)
Review the recommended security processes documented in Best Practices for Securing Oracle Applications Release 12 (Doc ID: 403537.1)

40)
Configure Release 12.2 E-Business Suite instance (required)
  -create appsutil using admkappsutil.pl
  -copy appsutil to database tier.
  -install jre on the database tier.
  -create EBS/autocofig related directories in the database tier such as $ORACLE_HOME/network/admin/context_name
  -create context file in the database tier using adbldxml.pl
  -exec fnd_conc_clone.setup_clean to clean the fnd_nodes
  -Synchronize values of APPLPTMP with UTL_FILE_DIR for PL/SQL based Concurrent Requests
  -run autoconfig on db tier
  -Run AutoConfig on the Apps Tier.
  -Run Rapid Install to configure Release 12.2 E-Business Suite instance. (second time Rapid install run)

41)
Upgrade Add-on Localization products using note:  Add-on Localizations - Upgrade Consideration (Doc ID: 1491965.1)

42)
Integrate custom objects and schemas (conditional).
follow the steps in Oracle E-Business Suite Developer's Guide to reintegrate these customizations with the APPS schema.

43)
Re-enable custom triggers, constraints, and indexes

44)
Enabling Online Patching

45)configure init.ora
The parameters described in this section apply to Oracle E-Business Suite Release 12.2. For details, refer to Database Initialization Parameters for Oracle E-Business Suite Release 12 (Doc ID: 396009.1).

46)
Apply the Latest AD and TXK for Release 12.2


47)
Apply the Latest Oracle E-Business Suite Release Update Pack for Release 12.2 --"12.2.5 at the moment"

48)
Reset ORACLE schema passwords (recommended)

49)
Reset ORACLE schema passwords (recommended)

50)
Verify completion of concurrent programs (recommended)

51)
Drop Obsoleted Product Schema (optional)

Obsolete Product Schemas that can be considered for dropping include:
ABM, AHM, AMF, AMW, BIL, BIV, BIX, BSC, CSS, CUE, CUF, CUI, CUN, CUP, CUS, DDD, EAA, EDWREP, EVM, FEM, FII, FPT,
FTP, GCS, HCC, IBA, IBP, IGF, IGS, IGW, IMT, IPD, ISC, ITA, JTS, ME, MST, OKB, OKI, OKO, OKR, OZP, OZS, PFT, POA,
PSB, RCM, RHX, RLA, VEH, XNC, XNI, XNM, XNS, ZFA, ZPB, ZSA.

52)
Drop dangling synonyms by using "sqlplus APPS/****@DB @$AD_TOP/sql/adzd_drop_synonyms.sql"

53)
Install online help (recommended)

54)
Apply latest product patches (required)
Determine the latest product-specific patches. Then, download the American English patches. You must apply the patches using AD Online Patching (adop).
Note: See Patch Wizard Main Page in Oracle E-Business Suite Maintenance Guide Release 12.2.

55)
Installing NLS upgrade patch driver and NLS Online Help (conditional) by requesting requesting a translation patch,
See Requesting Translation Synchronization Patches (Doc ID: 252422.1)

56)
Update and verify custom responsibilities (conditional)

57)
Grant flexfield value set access to specific users (required) by following:  Flexfield Value Set Security chapter in Oracle E-Business Suite Flexfields Guide, Release 12.2 (Part No. E22963-07).

58)
Migrate custom development to new technologies (recommended)
Check for general custom development :Preparing Custom Development for Next Oracle Applications Release (Doc ID: 374398.1)
Check for mod_plsql custom development:Application Framework Developer's Guide (Doc ID: 1315485.1) f

59)
Migrate the CUSTOM library
Place a copy of the new CUSTOM library (CUSTOM.pll) in a safe place. It is located in the $AU_TOP/resource directory (UNIX), or the %AU_TOP%\resource directory (Windows). Then, make a copy of the old Oracle Forms CUSTOM library and place it in the new directory. Upgrade to Oracle Forms Developer 10g by regenerating the library. Or, you can cut and paste the existing custom code into the new library, and then regenerate it.

60)
Copy and re-customize modified scripts or reports (conditional)
Copy custom shell scripts or reports to the custom application directories and re-customize the copy as necessary.

61)
Copy existing custom start scripts  (for example if you have customized concurrent manager start script in 11i earlier...)

62)
Review user responsibility assignments
Although user/responsibility assignments are preserved during the upgrade, the effective permissions granted by the seeded responsibilities, menus, functions, and report security groups may have change

63)
Configure applications client software for forms applet (required).
See:Deploying JRE (Native Plug-in) for Windows Clients in Oracle E-Business Suite Release 12 (Doc ID: 393931.1)

64)
Associate organization names with custom Alert definitions

65)
Set operating unit mode for customizations (conditional)

66)
Update Status Monitor URLs (required)

67)
Do Product specific post upgrade tasks documented in : Preparing for the Upgrade secetion of "Oracle E-Business Suite Upgrade Guide Release 11i to 12.2 Part Number E48834-15" - https://docs.oracle.com/cd/E26401_01/doc.122/e48834/T527285T527359.htm
-Customer Relationship Management Tasks
-Financials and Procurement Tasks
-Human Resource Management (HRMS)
-Lease and Finance Management
-Projects tasks
-Public Sector Tasks
-Supply Chain Management Tasks

68)
Perform System Maintanence Tasks documented in : "Post-Upgrade Tasks section" of "Oracle E-Business Suite Upgrade Guide Release 11i to 12.2 Part Number E48834-15" - https://docs.oracle.com/cd/E26401_01/doc.122/e48834/T527285T527359.htm

69)
Implement New Product and Country-specific Functionality

70)
Back Up Oracle E-Business Suite

EBS - DR - Dataguard - switchover tests & flashback

Normally EBS dataguard switch over tests including the application tier switch over and switch back operations can be done using the document:

"Business Continuity for Oracle E-Business Release 12.1 Using Oracle 11g Release 2 Physical Standby Database (Doc ID 1070033.1)" , this is for R12 (12.1)

The section named "Role Transitions" is easy to implement and the documentation is accurate.

The one thing to consider is the need for running the apps tier postclone/adcfgclone.pl in case all the required steps is not done in preparing the DR environment, but except that, the documentation is perfect.

Here in this post, I will share you an alternative disaster test method, which is based on the Oracle 's Flasback database technology.

In this method, we can test our DR environments, without doing a Dataguard switchover. So it is easier to implement and it may be needed in some cases.

Here is the action plan(more or less) for accomplishing that;

On Primary DB tier:

Stop the log shipping;

alter system set log_archive_dest_state_2=defer;

On Standby DB tier:

alter database recover managed standby database cancel;

alter database flashback on;

create restore point OPEN_STANDBY guarantee flashback database;

alter database activate standby database;

alter database open;

sqlplus apps/apps_password
exec fnd_conc_clone.setup_clean;
commit;

cd $ORACLE_HOME/appsutil/bin

perl adbldxml.pl tier=db appsuser=apps

sh adconfig.sh

On Standby Apps tier:

cd $ADMIN_SCRIPTS_HOME
perl adcfgclone.pl appsTier

Start the services.

--update conc request queues and wf mailer conf. if necessary

Fail/Switch Back:
On Standby DB tier:

sqlplus "/as sysdba"
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;

FLASHBACK DATABASE TO RESTORE POINT OPEN_STANDBY;

DROP RESTORE POINT OPEN_STANDBY;

ALTER DATABASE CONVERT TO PHYSICAL STANDBY;

SHUTDOWN IMMEDIATE;
STARTUP NOMOUNT;
ALTER DATABASE MOUNT STANDBY DATABASE;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;

On Primary DB Tier:
alter system set log_archive_dest_state_2=enable scope=both;

This procedure reflects the actual work that is need to be done(more or lesss), in case of a DR test.
What we do in this procedure is, we stop the managed recovery, create guaranteed flashback restore point, then activate our standby database and then configure our application tier (standby app tier) to this newly activated database and let our functional teams to test the DR environment. After the test, we restore our restore point, start the managed recovery and get back where we start.

This routine may be changed/enhanced according to the needs of the environment, but the important thing is that you get the idea...

EBS 12.2.5 - adopmon

As a new feature of EBS 12.2.5 , we have adopmon(adop monitor, Online Patching Monitoring utility) utility for monitoring the online patching cycles. It provides a readable output, letting us monitor the events that occur in the online patching cycle phases.
The overall progress of an online patching cycle can be followed by running the new Online Patching Monitoring utility (adopmon), while seeing the various individual adop actions being taken (in a readable format).adopmon is executed after setting the run environment. It requires us to provide the apps password and that's it.
Here is an example run, captured during a cutover phase.

[root@ermanserver~]# su - applmgr

  E-Business Suite Environment Information

  ----------------------------------------

  RUN File System           : /app/oracle/PROD/fs1/EBSapps/appl

  PATCH File System         : /app/oracle/PROD/fs2/EBSapps/appl

  Non-Editioned File System : /app/oracle/PROD/fs_ne

  DB Host: ermanserver  Service/SID: PROD

Sourcing the RUN File System ...

[applmgr@ermanserver ~]$ adopmon

Running script. Press Ctrl-C to quit.

Enter the APPS password:

Validating credentials...

Printing the log statements starting from sequence #632872

Timestamp           Node name           Message Type Message Text

---------------------------------------------------------------------------------------------------------------------------

2016/05/11 15:43:55 ermanserver         EVENT        Log: /app/oracle/PROD/fs_ne/EBSapps/log/adop/13/adop_20160511_154324.l

                                                     og

2016/05/11 15:43:56 ermanserver         EVENT        Validating configuration on node: [ermanserver].

2016/05/11 15:43:56 ermanserver         EVENT        Log: /app/oracle/PROD/fs_ne/EBSapps/log/adop/13/cutover_20160511_15432

                                                     4/PROD_ermanserver

2016/05/11 15:44:17 ermanserver         EVENT        Checking if finalize required.

2016/05/11 15:44:17 ermanserver         EVENT        Finalize phase is not required

2016/05/11 15:44:18 ermanserver         EVENT        Submitting request for Internal Concurrent Manager shutdown.

2016/05/11 15:44:18 ermanserver         EVENT        Log file: /app/oracle/PROD/fs1/inst/apps/PROD_ermanserver/logs/appl/ad

                                                     min/log/adcmctl.txt

2016/05/11 15:44:18 ermanserver         EVENT        Cancelling ADZDPATCH concurrent request

2016/05/11 15:44:19 ermanserver         EVENT        Waiting for Internal Concurrent Manager to go down.

EBS 12.2 -- sync errors due to missing patch files/directory does not exists.

In Oracle documents, we see ;
"The directories where you extracted the patches applied in a given patching cycle must be retained, in the same location and with the same contents, until the next prepare phase completes. This is also a requirement for patches applied in a hotpatch session."

See what happened in the next prepare phase, after we applied a patch and then deleted it from PATCH_TOP.

You see, adop complained about the patch directory that was not exists.
This was actually an expected behaviour..

So  according to Oracle , what needed to be done is to download the missing patch from Oracle Support and restore it to the PATCH_TOP.

However, what we did in one of our test environment and want to share with you in this post is a little different.

That is, as we have fs_clone utility, we aborted the patch cycle in which the prepare phase was encountering this error and then executed fs_clone for the solution.

After the fs_clone, we saw the filesystem objects were synched and then we executed the prepare phase again. This time prepare phase completed with success.

Altough we don't suggest this approach, it might become handy one day.

executing fs_clone is actually useful some other cases as well.

As documented in  E-Business Suite - ADOP Basic Usage Training Videos [Video] ( Doc ID 2103131.1 ),  sometimes we can even execute fs_clone by our own decision to decrease the job of the next prepare phase.

Bytheway, this missing patch error, the oracle support solution and fs_clone solution are valid for both online and hot patches.

So the following scenario works; (not recommended..)

time 0: Suppose we apply a hotpatch using adop hotpatch=yes
time 1 : Suppose the patch (in hotpatch mode) applied successfully
time 2: Suppose we delete the patches (including the hotpatch at time0, that we have applied using hotpatch option) from PATCH_TOP
time 3: Development asks us to apply an online patch
-- now we need to start a new patching cycle to apply the online patch requested at time 3.
time 4: To apply the patch requested at time 3, we execute adop phase=fs_clone before executing adop phase=prepare for creating the new patching cycle , because " we deleted the patch files from PATCH_TOP, so adop prepare won't  able to sync the filesystems."

RDBMS -- enabling custom maintanence for the deployments

During the deployment cycles, we may want to have a normal database, running with all of its components including its listener. We may want to have a database in a form equal to its runtime environment and we want to be the only one that is able to connect to the database.
These kinds of desires may arise for having the deployment processes, which may require no one except the administrator to be connected on an Oracle Database, for which we need all the database functionality up&running. So we are talking about a custom maintanence mode here.

Let's see how we implement such a maintanence mode functionality

We first create a trigger(in this example, it is "LOGON_PREVENT_DURING_DEPLOY") to read the sys_context and prevent the login requests except the one coming from the administrator's machine. (in this example: it only lets the user who has "ermanarslan" as its OS user name, to connect to the database). Note that: This trigger may be modified according to our needs.

Trigger:

CREATE OR REPLACE TRIGGER SYSTEM.logon_prevent_during_deploy
   AFTER LOGON ON DATABASE
DECLARE
   v_osuser    v$session.osuser%TYPE;
   v_program   v$session.program%TYPE;
   CURSOR user_prog
   IS
      SELECT osuser, program
        FROM v$session
       WHERE audsid = SYS_CONTEXT ('USERENV', 'SESSIONID');
BEGIN
   OPEN user_prog;
   FETCH user_prog
    INTO v_osuser, v_program;
   CLOSE user_prog;

   IF LOWER (v_osuser) != ('ermanarslan')
   THEN
      raise_application_error
         (-20001,
          '<<<<< The System is in maintanence mode right now. Check again later  >>>>>'
         );
   END IF;
END;
/
ALTER TRIGGER LOGON_PREVENT_DURING_DEPLOY DISABLE;

Then we create 2 procedure for enabling and disabling the trigger. For enabling the trigger(actually enabling the maintanence mode) before we begin our deployment activities , we execute the enabling procedure and for disabling the trigger(actually the maintanence mode), we execute the disabling procedure.

Procedure for enabling the maintanance mode:

CREATE OR REPLACE procedure Enable_maintanence
 IS
 counter INTEGER := 1 ;
 count_son number;
 sqlStmt VARCHAR2(1000);
 cursor c1 is select a.sid,a.job,b.serial# from dba_jobs_running a, v$session b where a.sid=b.sid;
 jobs c1%ROWTYPE;
BEGIN
select count(*) into count_son from dba_jobs_running;
execute immediate 'alter trigger system.logon_prevent_during_deploy enable';
execute immediate 'revoke ADMINISTER DATABASE TRIGGER from dba';
execute immediate 'revoke ADMINISTER DATABASE TRIGGER from IMP_FULL_DATABASE';
loop
exit when count_son=0;
open c1 ;
fetch c1 into jobs;
DBMS_JOB.REMOVE(jobs.job);
sqlStmt := 'ALTER SYSTEM KILL SESSION ''' ||
                TO_CHAR(jobs.sid) ||
                ',' ||
                TO_CHAR(jobs.serial#) ||
                '''' ;

EXECUTE IMMEDIATE sqlStmt;

EXIT WHEN count_son=counter;
counter := counter+1;
close c1;
END LOOP;
execute immediate 'alter system set job_queue_processes=0';
END;
/

Procedure for disabling the maintanance mode:

CREATE OR REPLACE procedure Disable_maintanence
 IS
BEGIN
execute immediate 'alter trigger system.logon_prevent_during_deploy disable';
execute immediate 'grant ADMINISTER DATABASE TRIGGER to dba';
execute immediate 'grant ADMINISTER DATABASE TRIGGER to IMP_FULL_DATABASE';
execute immediate 'alter system set job_queue_processes=20';
END;

/

Of course, the enabling procedure can be modified for killing all current user sessions before performing any deployment/maintanence tasks. Lastly, you can always use the standard restricted mode of the Oracle Database (ALTER SYSTEM ENABLE RESTRICTED SESSION). This custom maintanence mode functioanlity just an alternative.

Rdbms/ASM, seperation of roles, interpretations of the linux permissions

If we want  to implement seperation of roles for Oracle on Linux, which means having two seperate users for Grid and RDBMS, then the linux permissions, file owners and group owners of ASM disk devices, as well as the oracle binaries in both Grid Home and Oracle Home becomes crucial..

Let's examine this from Linux perspective by going through a working example, where the Asm device files are owned by grid:asadmin group, oracle binary in Grid Home is owned by grid:oinstall diskgroup and where oracle binary in Oracle Database Home is owned by oracle:asmadmin.

In this working example, we have the permission of oracle binary in RDBMS_HOME/bin as follows;

permissions  Links     owner     group        file

-rwsr-s--x       1           oracle    asmadmin   oracle

What does the columns mean?

Permission column: Permission of the file and the type of the file (directory, file etc.. )

Links column: number of links, the links to that file.

Owner: Owner of the file

Group:Group owner of the file

File: file name

What does that actually mean?

oracle binary file is owned by an OS user named oracle, the group owner of the oracle binary file is asmadmin. There are no links to that file. 

When we look at the permissions, which is the most crucial part of it, it says: any Os user can execute this binary. 

The users in asmadmin group can read and execute the oracle binary, and the user oracle can read, write and execute the file oracle. 

When the file oracle is executed by any OS user who has the permission to execute it(actually anyone can execute it), it is executed with the permissions of asmadmin group and the permission of oracle user.

What basically it says to the OS operating system is; when oracle binary is executed, execute it like it is executed by the user oracle and by the group asmadmin.

The file permissions of the oracle binary located in GRID HOME is the same as the permissions of oracle binary in RDBMS_HOME, but the owner and the groups of it is different(grid:oinstall)

permissions  Links     owner     group        file

-rwsr-s--x        1           grid      oinstall  /u01/app/11.2.0.4/grid/bin/oracle

So , this basically means, everyone can execute oracle binary stored in GRID HOME and when this binary file is executed, it is executed like it is executed by grid user and by the oinstall group (in terms of permissions)

The file permissions of the super/device files that are used to reach the ASM disks, are

[oracle@daroravmsrv1 ~]$ ls -al /dev/mapper/HDD_E0_S*

permissions  Links     owner     group        file

brw-rw----      1             grid   asmadmin /dev/mapper/HDD_E0_S00_1461220864

"b" means, it is a block device which can be accessed randomly.

So, the disks are owned by grid:asmadmin and in order to able to read or write them, one should have grid user or asadmin user privileges or let's say, only the grid user or the users in asmadmin directory can read from or write to these files.

What does this configuration provides us?

By using this configuration, we  have different users(oracle and grid) which can access the ASM disks. That is, when we create a file from an Oracle Database running in the oracle home, it can access to ASM devices and create the file there. On the other hand, if we want to do a Grid administration stuff using the same user that is the owner of the Oracle Database Oracle Home, then we will not be able to. 

So, by having asmadmin as the group owner of the oracle binary located in Oracle home and as we use the suid and guid bits set for the oracle binary, our database or anything that uses the oracle binary stored in Oracle Home can reach the Asm Disk Groups without any problems. (can read and write to the ASM disk devices) 

The oracle binary in Grid Home, which is owned by grid can already access the ASM Disk groups, as the owner of the ASM disks is grid user.

So seperation of duties are in place without effecting the oracle database's mandatory ASM works.

We may ask, why we use the SUID an GUID bits in general? We see them in standard single node Non-ASM Oracle installations as well.
Well, the answer again : is they are required seperation of duties.
Suppose we want the OS user erman to be able make "sqlplus / as sysdba"  local connections in an environment, where the RDBMS owner is the OS user named oracle. In such a case, the OS user "erman" should be act like OS user "oracle" and this can be accomplished by the SUID an GUID bits of the permissions of the $ORACLE_HOME/bin/oracle file.

Without SUID and GUID bits, we end up with the following; (altough erman is in dba group)

[root@demoorcl ~]# su - erman

[erman@demoorcl ~]$ id

uid=54323(erman) gid=54322(dba) groups=54322(dba),54321(oinstall)

[erman@demoorcl ~]$ sqlplus "/as sysdba"
SQL*Plus: Release 11.2.0.3.0 Production on Tue Apr 26 15:12:32 2016

Copyright (c) 1982, 2011, Oracle. All rights reserved
ERROR:
ORA-12547: TNS:lost contact

A workaround for this is to make a TNS listener based sysdba connection like;

sqlplus sys@PROD as sysdba... This way, listener will bind us to a LOCAL=NO shadow process and we will not need the suid ang guid based permissions. (In case of a local connection; we need the permissions for executing the oracle binary because the oracle binary is executed by our processes in a local connection)
Almost the same effect will be there when we change the permission of oracle binary to "700". Local connections will end with ORA-12547, but the configuration will  actually be more secure, as no one except oracle os user will be able to oracle binary, and if someone(in dba group) wants to connect to the database using sysdba connection, he/or she should provide the password..

Here is a demo:

[oracle@demoorcl bin]$ ls -al oracle
-rwx------ 1 oracle oinstall 232891110 May 19  2015 oracle

[oracle@demoorcl bin]$ sqlplus "/as sysdba"
SQL*Plus: Release 11.2.0.3.0 Production on Fri Apr 29 19:37:11 2016
Copyright (c) 1982, 2011, Oracle.  All rights reserved
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> exit

[oracle@demoorcl bin]$ exit
logout
[root@demoorcl ~]# su - erman
[erman@demoorcl ~]$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.3.0 Production on Fri Apr 29 19:37:21 2016
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
ERROR:
ORA-12546: TNS:permission denied

[erman@demoorcl ~]$ sqlplus sys/manager@CLONE as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Fri Apr 29 19:37:33 2016
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> 

Well, this post become a little unstructured, but still it gives lost of valuable info. Hope you ll get benefit from that.

RDBMS -- standby controlfile and the db_file_name_convert, log_file_name_convert parameters

db_file_name_convert and log_file_name_convert parameters are widely used in standby environments. Their main usage is to convert the paths of the datafile and redolog files that are created in the primary during the managed recovery and make them be able to created in the standby database. (when standby_file_management is set to auto)

In this post, I will mention about a different thing, that is done using these two parameters.
That is, as you may alredy know, in case we recreate standby controlfile, the paths in the newly created standby control reflects the paths of the database and redolog files of the primary database. If the primary and standby file paths are the same, then no issue, but when they are different that we need to do some extras.
There are 3 things that we can do for this;

1)Set the filename conversion initialization parameters. (db_file_name_convert, log_file_name_convert parameters)
2)Rename the files manually using ALTER DATABASE statements.
3)Use a combination of conversion parameters and manual renames.(rename + db_file_name_convert, log_file_name_convert parameters)

So, In this post, I will mention about the first method.

Here are the quick descriptions;

DB_FILE_NAME_CONVERT:Converts primary database datafile filenames to standby datafile filenames, for example, from tbs_* to standbytbs_*.

LOG_FILE_NAME_CONVERT:Converts primary database redo log filenames to standby database redo log filenames, for example, from log_* to standbylog_*.

STANDBY_FILE_MANAGEMENT:When set to auto, this parameter automates the creation and deletion of datafile filenames on the standby site using the same filenames as the primary site. (if we want different file paths, we use db_file_name_convert and log_file_name_convert parameters) -- this parameter actually is not needed for db_file_name_convert and log_file_name_convert to convert the current filenames of the primary to the standby but it is needed when a file is added or deleted in the future)

Making Oracle to use the correct file paths in the standby environment, can be done using  db_file_name_convert and log_file_name_convert parameters easily.

That is, when db_file_name_convert and log_file_name_convert parameters are in place, oracle updates the paths that are stored in the standby controlfile accordingly and thus there will be no need to rename, or switch to copy-like rman operations. In addition Oracle creates or deletes the files in standby databases that are created or deleted in the primary according to these parameters.

As far as I can see , what happens is, during a startup operation, when Oracle understand the control file is a standby controlfile, it checks to see if the db_file_name_convert and log_file_name_convert  parameters are set and if they are set, oracle updates the paths that it will use for accessing the database files and redologfiles accordingly till the next startup. So as long as the parameters in place, oracle does this in every startup, thus there will be no need for renaming the files or using switch to copy-like rman operations.

This parameters are used when the database is a standby database, so if you start to think like; "okay, then I can do my migrations like this.. I can move the datafiles to different paths and then without the controlfile, I can open my database using db_file_name_convert and log_file_name_convert parameters.. " Well that's not true. These parameters are only honoured when used in standby databases.

Also, with this in mind, as for the switch over operations, it seems; Oracle updates the standby controlfile with the paths that are produced according to the db_file_name_convert and log_file_name_convert parameters, and then makes the standby controlfile to be a normal controlfile and opens the database. (note that, standby controlfile and a controlfile has no big difference, only a flag changes )

Note:
As for the environments where the primary and standby databases reside in ASM (managed with Oracle Managed Files), db_file_name_convert and log_file_name_convert can not be used for these kind operations.  (I mean they can be used for future adding or deleting of files but can not be used for renaming the current filepaths--after recreating the standby controlfiles, in the standby site)

This is because the filenames(not paths only) are different because of the suffixes that are added by Oracle. So, in ASM and Oracle Managed Files, it can be accomplished by using the following document: "Step By Step Guide On How To Recreate Standby Control File When Datafiles Are On ASM And Using Oracle Managed Files (Doc ID 734862.1)"

EBS 11i/R12/12.2 -- using a non-default block size for a tablespace

In this blog post, I will share you some recommendations and things to consider about using a non-default blocksize for a tablespace in Oracle EBS databases.

As you may already, the block size of an EBS database is 8K and it is a mandatory thing. However, as Oracle let us have different block size tablespace in a database, we can have bigger block sized tablespace in our EBS databases, even tough the EBS database block size is set 8K.

It is mandatory to  have 8k block sizes, in other words having the db_block_size init.ora parameter EBS databases , so this means, the standard tablespaces that comes with EBS must remain in 8K, but this also means, we can use non default tablespace block size for our custom application schemas,as having non-default block sized tablespaces is supported by Oracle.
In the early days, the ability to have non-default tablespaces may be invented for Transportable tablespaces, but as the new releases have released (11g and 12c), this ability have involved and now it is supported to be used in any case.

One use case may be when there is a need to have a new schema in this database and this new schema will be used by ETL processes / BI like reports, we can create a 32k block sized tablespace and put the db objects of this schema to this 32k tablespace to get the advantage of the following documented things;

Larger oracle block sizes (big tablespace block sizes or db_block_size) typically give fewer index levels and hence improved index access times to data. A single I/O will fetch many related rows and subsequent requests for the next rows will already be in the data buffer. This is one of the major benefits of a larger block size. Another benefit is that it will decrease the number of splits.

However, no one including Oracle can guarantee if having a 32k tablespace for a custom schema in an EBS database is a good thing or bad thing, as it depends on the load and profile of the environment.

For the standard EBS applications, it is guaranteed by Oracle , as they made 8k block size as a mandatory thing. On the hand, as for the custom applications/schemas, having a non default tablespace size may provide  3 results. That is, it may increase the performance, it may decrease the performance or the effect of it can not be felt (almost change in performance)

As for the rule of thumb, there are 3 points;

Use db_block_size = 8192 for most of the transactional processing systems.

Use db_block_size of larger than 8KB for OLTP systems in which your data structures are large. (avoiding change and migrated blocks)

Use db_block_size of larger values than 8KB for systems in which your undo generation is not a meaningful part of your workload. (Data warehouses)

So should we have non-default block sizes for custom tablespaces, which are used by reporting, BI , ETL applications then?

Well, the answer is it depends and if we can this problem as an issue, then the solution is "testing". So, it is supported, but in order to to decide whether it will be good or bad in terms of performance, the only thing that can be done is to test it.

EBS R12/12.2- HR ORG CHART transaction context is lost , Bug 19955094, Workaround: Use virtual hosts

Recently encountered this problem in a new 12.2.5 customer environment.
The first page of Hr Org Chart was displayed properly, but when we click on the Talent Profile or Manager Actions buttons/links, a web page was displaying the "Transaction Context is lost" error.

In this site, the EBS application tier and the Hr org Chart was deployed on the same server and the issue was actually expected one ,as it was already documented in "Transaction Context Is Lost when click on Org Chart from Talent Profile (Doc ID 1962008.1)"

The document was clearly saying something like:

The issue is that from the perspective of the browser, the server test.local is the same "server" regardless of both instances listening to different ports on the same host. To fix this issue standalone WLS server on which the orgchart was deployed, needs to be moved to a different host. As for the solution, Web Logic Server on which Org Chart is deployed needs to be on host different than instance host.
What we did?

As it was not applicable to install a new weblogic into a new server and deploy HR ORG Chart once again;

we did create a virtual host(a different hostname) in the Hr Org Chart weblogic server and added it to the managed server/or admin server which was running the Hr Org Chart application.

We also added a DNS entry for this new virtual hostname and updated the related EBS profiles to make them redirect us to this new virtual hostname for accessing the HR ORG CHART page. 

These actions made, the managed server, that was running the Hr Org Chart application to listen on the new virtual hostname as well. 

As we configured EBS profiles with this new virtual host(different than the EBS application server's hostname), the issue dissapeared. Note that, It took only 5 mins to activate this new virtual hostname.

This workaround or solution is not available in Oracle Support, so I hope you will find it useful.