Monday, May 30, 2016

EBS -- Active/Passive Cluster + DR minimal configuration

In this post, I will share some Active/Passive EBS 12.2 Architectures extended with simple DR solutions.
These configurations can support new EBS 12.2 environments up to 100 users and  are scaled in a way that requires minimum hardware and software fees.

Note that, this configuration can be modified , extended , scaled up or down and scaled vertically according to the environmental needs. Sizing may also vary according to the expectations, needs and the EBS modules that are planned to be implemented.


First of all our hardware resource can be sized as the following;
Note that, these are based on a standard EBS, that includes HR and standard Financial modules.
In the paragraphs below , I give active/passive + DR architecture as this post is aimed at this type of configuration. (cheap, redundant, stable in a way and fast enough)

Source site:

2 servers in the source site. 1 server active, 1 server passive.
Both of the source servers are connected to a shared storage (fiber/HBA connection preferred)
Failover operations can be done manually, or using a clustering software(Oracle Restart may be a solution, or in cases of IBM, HACMP will work)
EBS apps and db tier is placed on the same server, altough it is not recommended. (we are aimed to be minimal)

DRC site.

DRC is one server.
The replication can be storage based or dataguard based or VM based, It depends on the architecture.
If using dataguard, the target server/DR server should be licensed as well.
Application filesystem synchronization can be done using an utility like rsync or using similar tools which do the same job.
Storage replication, on the other side, does not require any Oracle licenses or an utility for filesystem replication.
However, storage replication /snapshot /snapmirror utilities require their own licenses. (so this should be kept in mind)
Also, storage based replications work in block level, so they do not understand Oracle's block format. From this perspective Dataguard is the best solution for creating and feeding Oracle Replicas/Standby/DR servers. On the other hand, in these days, Storage vendors have solutions/integrated solutions for Oracle and when these solutions are implemented, Storage replications know that they are dealing with the Oracle Databases when replicating the storage level data.

As for the sizing;

Production Site:

For the Active node:

OS: Oracle Linux 6 64 bit, disks from Storage, Local disks can be minimum(300GB-500GB), 64GB memory, 16 cores, 1x HBA(2 HBA -- preffered for multipath)

For the Passive node:

OS: Oracle Linux 6 64 bit 64GB memory, disks from Storage, Local disks can be minimum (300GB-500GB), 16 cores, 1x HBA(2 HBA -- preffered for multipath)

Storage : 

A Storage with 2 controller, 15k disks (1.2 TB disk space -- usable space , which remains after raid configuration) -- disk size is dependent on the expected % of yearly growth .
Optionally, for automatic failover -> a clustering software, and for an additional replication -> a storage in the DR site , which can do replication/snapmirror (Netapp,EMC,Oracle ZFS etc..)

DRC Site:

OS: Oracle Linux 6 64 bit, 1.2 TB disk space(local), 36GB memory, 8 cores.

Optinally a storage, in case storage replication is required, in addition to the rsync and Dataguard. If that's the case, a Storage with 1.2TB can do the job(ofcourse, it should be capable of doing replication/snapmirror kind of works)

In our first configuration we use an ODA machine for the source site. We use ODA in a way to reduce the RAC license needs. That is, we use a virtualized ODA to place our application tiers and databases.
We utilize ODA_BASE domain for placing our EBS databases, but we use only one node of ODA_BASE to keep our license fees minimal.
This configuration gives us the ability to run our EBS databases or applications in the second ODA node in case of a failure.
In case of the disaster, we use Dataguard (async). We choose our DR server to be a single  node, which has the power to handle the workload(at least may handle the workload during 1 day) in case it should be used when a planned maintanence is required in the source servers or in case of a disaster in the source site.
By using the ODA's virtualization, we use the production machine as a consolidated environment, on which PROD, TEST and DEV instances of EBS (app + db ) run.
Note that, dataguard only replicates the Database file, so Oracle binaries, Oracle Homes , Application filesystem should be replicated in to that.. (using rsync or some tool like that)


Another configuration can be based on the traditional servers in the production site;
In this configuration, we still use Dataguard and rsync for the replications , so the only difference is, this time, we use traditional servers like HP , Dell or Sun and a Storage in the Production site to provide the active/passive clustering.
As for the DR site, we still have only one server, which can even be a VM.


Another alternative, may be using a storage based replication directly without having a dataguard. This configuration can be preferred in case one doesn't want to license the target/DRC environment.

In this case, the topology can be like the following;
                                     

The network between Prod and DRC is also important. A low latency will increase the recoverability.
As for the servers, here are some notes;

Production servers are preferred to be physical (For ex: HP servers, Sun Servers)
DRC server can be a VM (Oracle VM Server , ESX etc..)
Test environments can be placed on the passive production node, or they can be placed on Virtual server outside this configuration. For every TEST environment, 32GB Memory, 750 GB disk sapce and 8 core CPU is enough.
CPU Cores specified in this note are Intel. The latest generation intel cpus are prefferred.
There can be an additional storage in the Production Site, to increase the storage level fault tolerancy , as well.

Lastly, as for the "U" sizes, here are some notes;

ODA is a 4U Rack moutnable system.
The traditional servers supporting these sources can be between 1U and 4U sizes as well and if VMs are preffered, they dont even count for the U sizes, as they are virtual.Storage for this capacity can be handled with a 4U architecture and
With a quick calculation, 8U free space is okay for the production site. DR site can be somewhere between 1U-4U according to the configuration.

Important note:  The values used for sizing are completely dependent on the environment. So it may vary accordingly. All this sizing information is given for a new EBS 12.2 environment, which is expected to have the ability to support 70 users. %10 of these users are expexted to currently access the EBS applications. The environment is expected to grow %10 in a year. The configuration is sized to support 3 years. This is for an EBS environment, on which 60000 invoices are made out in a single month, and 40000 bills invoices are received in a single month.

Wednesday, May 18, 2016

EBS -- Upgrading EBS 11i(11.5.10.2) to 12.2, a compilation

Here is a enterance level document to be used to have a general picture before the EBS 11i to 12.2 upgrade.
It covers the navigation for the documents that are used in the upgrade process, as well as the general steps that should be performed for upgrading EBS 11i to EBS 12.2.
Note that: This document is just a compliation of Oracle documents, but  I believe it will save you time as reading it , you will not have to go back and forth among all the related documents for determining the required upgrade tasks.

Main Document :Oracle E-Business Suite Release Notes, Release 12.2 (Doc ID 1320300.1)
    ->Section 3: Instructions for Upgrade Customers :"Oracle E-Business Suite Upgrade Guide, Release 11i to 12.2 or Oracle E-Business Suite Upgrade Guide: Release 12.0 and 12.1 to 12.2 guides"

Direct Document:
https://docs.oracle.com/cd/E26401_01/doc.122/e48834/toc.htm (Oracle E-Business Suite Upgrade Guide Release 11i to 12.2 Part Number E48834-15)
----
Upgrade Path:
11.5.10 CU2 -> Minimum Patch Requirements for Sustaining Support for 11.5.10 -> Release 12.2
------

Steps:
Before starting the process, it is important to note that it is required to have a new application tier for placing the EBS 12.2's apps tier. This is required as EBS 11i 's Apps Tier is not supported to run on Linux x86-64 bit and EBS 12.2's Apps and DB tiers are delivered as 64 bit.
So at least for Linux installations, there is a need to have a supported Linux OS for placing the EBS 12.2 's Apps Tier.

1)
Upgrading to Oracle E-Business Suite Release 12.2 requires your database to be at the minimum version 11.2.0.4.
To complete the upgrade to Release 12.2, you must upgrade your database to 11.2.0.4 or higher.
Follow the instructions in Database Preparation Guidelines for an Oracle E-Business Suite Release 12.2 Upgrade (Doc ID: 1349240.1).

2)
INFO:Completing this upgrade brings your system to the 12.2.0 release.
You must apply the 12.2.5 Release Update Pack (RUP3) or later(12.2.5) to your existing Release 12.2 system for production use.

3)
Do the sizing CPU,Memory,Disk Space, IO subsystem.

4)
Know that IDO 10g and OSSO 10g will be desupported. Think about implementing OAM and Oracle Access Gate.

5)
Apply 11i.AD.I.7 (conditional) -- if you are not on AD.I.7

6)
Run TUMS utility and reviw: Download and apply AD TUMS patch

Download patch 18342870 and apply it to the administration server node on your Release 11i APPL_TOP.
It supplies you with the script (adtums.sql) you need to generate the TUMS report (tumsr12.html).
Generate the TUMS report->
$ cd $AD_TOP/patch/115/sql
$ sqlplus <APPS username> @adtums.sql <DIRECTORY>

7)
Backup the database and apps tier including customizations (cold backup preferred)
Note that: Customization is not handled during the upgrade, so the customization should be reapplied when the upgrade is done.

8)
Prepare yourself for the customizations.

You may have customized your system for business use. Take note of these important considerations before upgrading custom database objects:
The Oracle E-Business Suite Developer's Guide contains extensive instructions about naming standards and issues related to upgrading custom database objects. Familiarize yourself with this information before you begin the upgrade.
Run several test upgrades and track their impact on your custom database objects.
Rename any custom database objects with Applications prefixes that you have created so that they do not conflict with Oracle object names.

Note: Failure to test the impact on custom database objects before the upgrade can result in a loss of functionality.
At your discretion, and depending on the customizations in your system, you should also perform the following tasks:
Preserve the CUSTOM library by making a backup copy of CUSTOM.pll. You can use this copy later in the upgrade process to migrate your CUSTOM library to Release 12.2.
If you have customized forms with Oracle Forms 6i, then upgrade them to Oracle Forms 10g after the upgrade.

9)
Convert to Multiple Organizations architecture (required).
Converting to Multiple Organizations does not require the use of multiple operating units or sets of books - it does enable you to use this functionality at any time in the future.

If you are converting from a Single Organization architecture to a Multiple Organization architecture, complete the following steps:
Create an operating unit.
Assign the operating unit you created to the profile option MO:Operating Unit.
Note: See the following documentation references: Use of Multiple Organizations in Release 12 in Use of Multiple Organizations (Multi-Org) in Release 11i (Doc ID: 210193.1), and MOAC in Oracle Purchasing (Doc ID: 404800.1). HRMS users should also see Setting Up Multiple Organizations for Oracle HRMS (Doc ID: 259546.1).

10)
Drop event alert triggers in custom schemas (conditional).
To drop all event alert database triggers in custom schemas, run the alrdtrig.sql script, located in $ALR_TOP/admin/sql (in your Release 11i system). Re-create the triggers after the upgrade is complete.

11)
Review sizes of old and new tablespaces (required).
Make sure you allocate sufficient tablespace.
For guidelines based on an upgrade of the largest Oracle production system (oraprod), see Applications Release 12 Upgrade Sizing and Best Practices (Doc ID: 399362.1).

12)
Run AD preparation scripts (conditional).

Download and unzip the new patch 13435302. Follow the instructions in the readme for running these scripts:

adgncons.sql

The tablespace model for this release (OATM) is based on database object type rather than product affiliation. The adgncons.sql script prepares adcrtbsp.sql, configures the database to hold the new products to be added during the upgrade, and switches your system to use the new tablespace model.

adcrtbsp.sql

Generated by adgncons.sql, this script creates the new tablespaces, allocates unlimited tablespace to all APPS users, updates fnd_product_installation table with correct data and index tablespace information, assigns default tablespace to all APPS users, and sets the new_ts_mode flag in fnd_product_groups to Y. Be sure to review the locations of the dbf files for each new tablespace listed in adcrtbsp.sql, and change the locations in the script as necessary. Ensure that you have adequate disk space for all the changes and additions that adcrtbsp.sql is going to make.

Note: All the tablespaces created by adcrtbsp.sql are critical and required for 12.2 as they hold critical objects. Therefore, you should not drop any of these tablespaces at any point.
Note: Oracle has tested successfully an extent size of 128 K for small systems (for example, 100 GB database or the Vision database), and, for large multi-terabyte system, an extent size of 4-10 MB has been tested successfully.

adgrants.sql :(adgrants_nt.sql for Windows) Grants SYS privileges needed by Applications, and creates required views in SYS.

13)
Migrate Existing Objects to New Tablespace Model (recommended).
Running the preparation scripts creates tablespaces and prepares the database objects only for new products.
However, it does not automatically migrate your existing objects.
Oracle strongly recommends that you use the Tablespace Migration Utility to perform this migration now.
Note that this utility is not supported for use after you enable Online Patching, so you cannot perform the migration after your environment is upgraded to Release 12.2.
If you choose not to migrate to OATM now, then you must continue to manage your tablespaces separately.
Check  the Oracle E-Business Suite Setup Guide, Release 12.2 for mor information.

14)
Convert Oracle Alert E-mail Processing to the Workflow Notification Mailer (conditional)
If you use response processing alerts and you have not already converted Oracle Alert to the Workflow Notification Mailer, convert now by applying Oracle Applications Technology 11i.ATG_PF.H Rollup 6 (patch 5903765) to your Release 11i APPL_TOP. This roll up patch causes Oracle Alert to use the Workflow Notification Mailer for new alerts, but allows you to continue to run the Alert Response Processor for incoming responses sent before the conversion. Continue to run the Response Processor until you have no more outstanding responses of this kind.
Note: See About Oracle Applications Technology 11i.ATG_PF.H Rollup 6 (Doc ID: 444524.1) for more information.

15)
Oracle XML Gateway:  In release 12.2, Oracle XML Gateway Web services depend on Oracle E-Business Suite Integrated SOA Gateway which has product dependencies on Oracle SOA Suite and Oracle Fusion Middleware Adapter for Oracle Applications (also called Oracle E-Business Suite Adapter). See Installing Oracle E-Business Suite Integrated SOA Gateway, Release 12.2 (Doc ID: 1311068.1).


16)
Ensure that the GUEST account is valid and active and that the fnd_user USER_ID for the GUEST account is set to a value of '6'.

17)
Do Product specific upgrade tasks documented in : Preparing for the Upgrade secetion of "Oracle E-Business Suite Upgrade Guide Release 11i to 12.2 Part Number E48834-15" - https://docs.oracle.com/cd/E26401_01/doc.122/e48834/T527285T527359.htm
-Customer Relationship Management Tasks
-Financials and Procurement Tasks
-Human Resource Management (HRMS)
-Lease and Finance Management
-Projects tasks
-Public Sector Tasks
-Supply Chain Management Tasks

18)
Gather schema statistics (required)
FND_STATS.GATHER_SCHEMA_STATISTICS('ALL', 10, :parallel_degree, 'NOBACKUP', NULL, 'LASTRUN', 'GATHER AUTO', 10, 'N');
Where: parallel_degree is set to the value of the database initialization (init.ora) parameter parallel_max_servers for your instance.

19)
Install JRE on the database tier (conditional)
If you are planning to run Rapid Install in Upgrade Mode by using the Use Existing ORACLE HOME option, then you must install JRE in the Database ORACLE_HOME/appsutil as follows:
Download the latest JRE 7 Update. For optimum stability, performance, scalability, and OS vendor support, use the latest available update of JRE for the Oracle E-Business Suite database tier. The JRE download location is: http://www.oracle.com/technetwork/java/javase/downloads/index.html

20)
Reset init.ora parameters (required)
Follow the instructions in Database Initialization Parameters for Oracle E-Business Suite Release 12.2 and reset the init.ora parameters as needed.

21)
Run Rapid Install (required)
Use the Rapid Install wizard to lay down the file system and install the new technology stack for your Release 12.2 Oracle E-Business Suite system.

-Start Rapid Install by typing rapidwiz on the command line. The Welcome screen lists the components that are included in, or supported by, this release of Oracle E-Business Suite. Click Next.
-On the Wizard Operation screen, select Upgrade to Oracle E-Business Suite Release 12.2. Click Next.
-On the Select Upgrade Action screen, select Create Upgrade File System.
-In the associated screen flow, enter the parameters required to set up your new environment. Then, run Rapid Install.
-Note: Oracle E-Business Suite Installation Guide: Using Rapid Install contains complete instructions for running Rapid Install for both new installations and upgrades. Chapter 3 contains the information specific to running an upgrade.

22)
Synchronize values of APPLPTMP with UTL_FILE_DIR for PL/SQL based Concurrent Requests (required)

23)
Backup the database and apps tier including customizations (cold backup preferred)

24)
At this point, the database must be 11.2.0.4 or higher. So if not upgraded earlier, upgrade the database.
If you have not already done so, you can upgrade your production database to 11g Release 2 (11.2.0.4) or higher now, before the upgrade downtime. Follow the instructions in Database Preparation Guidelines for an Oracle E-Business Suite Release 12.2 Upgrade (Doc ID: 1349240.1).
Note: If you are upgrading to 11gR2 or 12cR1 from 10g or 9i, then you MUST set the parameter sec_case_sensitive_logon = False in your init.ora file to avoid login issues with sqlplus. For more information on this parameter and other database initialization parameters, refer to Database Initialization Parameters for Oracle Applications Release 12 (Doc ID: 396009.1).
If you are upgrading to a Release 12c database, then ensure that your sqlnet_ifile.ora contains the line: SQLNET.ALLOWED_LOGON_VERSION_SERVER = 8

25)
Disable AOL Audit Trail (conditional)

26)
Shut down application tier listeners and concurrent managers (required)

27)
Update init.ora with upgrade parameters (required)

28)
Set FAILED_LOGIN_ATTEMPTS to UNLIMITED for Oracle E-Business Suite schema

29)
Disable custom triggers, constraints, and indexes (conditional)

30)
Drop MRC schema (conditional) --it is no longer needed, it can be dropped.

31)
Back up the database (recommended)

32)
Enable maintanence mode

33)
Download and apply the AD Upgrade Patch for 12.2 (patch 10117518)(required) by following the patch read me.

34)
Apply all Consolidated Upgrade Patches (CUPs) (required)
--Apply the Consolidated Upgrade Patches (CUP) and all of the pre-install patches listed in the Oracle E-Business Suite Release Notes, Release 12.2 (Doc ID: 1320300.1)

35)
Run the American English upgrade patch driver (required)
AutoPatch to run the (American English) upgrade patch driver (u10124646.drv) merged with latest Consolidated Upgrade Patch for Oracle E-Businees Suite Release 12.2 . Release 12.2 upgrade patch driver (u10124646.drv) is located under $AU_TOP/patch/115/driver on Run File System.

36)
Disable Maintenance Mode (required)

37)
Follow the instructions in Database Initialization Parameters for Oracle Applications Release 12 (Doc ID: 396009.1) and reset the init.ora parameters as needed.

38)
Backup the database and apps tier

39)
Review the recommended security processes documented in Best Practices for Securing Oracle Applications Release 12 (Doc ID: 403537.1)

40)
Configure Release 12.2 E-Business Suite instance (required)
  -create appsutil using admkappsutil.pl
  -copy appsutil to database tier.
  -install jre on the database tier.
  -create EBS/autocofig related directories in the database tier such as $ORACLE_HOME/network/admin/context_name
  -create context file in the database tier using adbldxml.pl
  -exec fnd_conc_clone.setup_clean to clean the fnd_nodes
  -Synchronize values of APPLPTMP with UTL_FILE_DIR for PL/SQL based Concurrent Requests
  -run autoconfig on db tier
  -Run AutoConfig on the Apps Tier.
  -Run Rapid Install to configure Release 12.2 E-Business Suite instance. (second time Rapid install run)

41)
Upgrade Add-on Localization products using note:  Add-on Localizations - Upgrade Consideration (Doc ID: 1491965.1)

42)
Integrate custom objects and schemas (conditional).
follow the steps in Oracle E-Business Suite Developer's Guide to reintegrate these customizations with the APPS schema.

43)
Re-enable custom triggers, constraints, and indexes

44)
Enabling Online Patching

45)configure init.ora
The parameters described in this section apply to Oracle E-Business Suite Release 12.2. For details, refer to Database Initialization Parameters for Oracle E-Business Suite Release 12 (Doc ID: 396009.1).

46)
Apply the Latest AD and TXK for Release 12.2


47)
Apply the Latest Oracle E-Business Suite Release Update Pack for Release 12.2 --"12.2.5 at the moment"

48)
Reset ORACLE schema passwords (recommended)

49)
Reset ORACLE schema passwords (recommended)

50)
Verify completion of concurrent programs (recommended)

51)
Drop Obsoleted Product Schema (optional)

Obsolete Product Schemas that can be considered for dropping include:
ABM, AHM, AMF, AMW, BIL, BIV, BIX, BSC, CSS, CUE, CUF, CUI, CUN, CUP, CUS, DDD, EAA, EDWREP, EVM, FEM, FII, FPT,
FTP, GCS, HCC, IBA, IBP, IGF, IGS, IGW, IMT, IPD, ISC, ITA, JTS, ME, MST, OKB, OKI, OKO, OKR, OZP, OZS, PFT, POA,
PSB, RCM, RHX, RLA, VEH, XNC, XNI, XNM, XNS, ZFA, ZPB, ZSA.

52)
Drop dangling synonyms by using "sqlplus APPS/****@DB @$AD_TOP/sql/adzd_drop_synonyms.sql"

53)
Install online help (recommended)

54)
Apply latest product patches (required)
Determine the latest product-specific patches. Then, download the American English patches. You must apply the patches using AD Online Patching (adop).
Note: See Patch Wizard Main Page in Oracle E-Business Suite Maintenance Guide Release 12.2.

55)
Installing NLS upgrade patch driver and NLS Online Help (conditional) by requesting requesting a translation patch,
See Requesting Translation Synchronization Patches (Doc ID: 252422.1)

56)
Update and verify custom responsibilities (conditional)

57)
Grant flexfield value set access to specific users (required) by following:  Flexfield Value Set Security chapter in Oracle E-Business Suite Flexfields Guide, Release 12.2 (Part No. E22963-07).

58)
Migrate custom development to new technologies (recommended)
Check for general custom development :Preparing Custom Development for Next Oracle Applications Release (Doc ID: 374398.1)
Check for mod_plsql custom development:Application Framework Developer's Guide (Doc ID: 1315485.1) f

59)
Migrate the CUSTOM library
Place a copy of the new CUSTOM library (CUSTOM.pll) in a safe place. It is located in the $AU_TOP/resource directory (UNIX), or the %AU_TOP%\resource directory (Windows). Then, make a copy of the old Oracle Forms CUSTOM library and place it in the new directory. Upgrade to Oracle Forms Developer 10g by regenerating the library. Or, you can cut and paste the existing custom code into the new library, and then regenerate it.

60)
Copy and re-customize modified scripts or reports (conditional)
Copy custom shell scripts or reports to the custom application directories and re-customize the copy as necessary.

61)
Copy existing custom start scripts  (for example if you have customized concurrent manager start script in 11i earlier...)

62)
Review user responsibility assignments
Although user/responsibility assignments are preserved during the upgrade, the effective permissions granted by the seeded responsibilities, menus, functions, and report security groups may have change

63)
Configure applications client software for forms applet (required).
See:Deploying JRE (Native Plug-in) for Windows Clients in Oracle E-Business Suite Release 12 (Doc ID: 393931.1)

64)
Associate organization names with custom Alert definitions

65)
Set operating unit mode for customizations (conditional)

66)
Update Status Monitor URLs (required)

67)
Do Product specific post upgrade tasks documented in : Preparing for the Upgrade secetion of "Oracle E-Business Suite Upgrade Guide Release 11i to 12.2 Part Number E48834-15" - https://docs.oracle.com/cd/E26401_01/doc.122/e48834/T527285T527359.htm
-Customer Relationship Management Tasks
-Financials and Procurement Tasks
-Human Resource Management (HRMS)
-Lease and Finance Management
-Projects tasks
-Public Sector Tasks
-Supply Chain Management Tasks

68)
Perform System Maintanence Tasks documented in : "Post-Upgrade Tasks section" of "Oracle E-Business Suite Upgrade Guide Release 11i to 12.2 Part Number E48834-15" - https://docs.oracle.com/cd/E26401_01/doc.122/e48834/T527285T527359.htm

69)
Implement New Product and Country-specific Functionality

70)
Back Up Oracle E-Business Suite

Wednesday, May 11, 2016

EBS - DR - Dataguard - switchover tests & flashback

Normally EBS dataguard switch over tests including the application tier switch over and switch back operations can be done using the document:

"Business Continuity for Oracle E-Business Release 12.1 Using Oracle 11g Release 2 Physical Standby Database (Doc ID 1070033.1)" , this is for R12 (12.1)

The section named "Role Transitions" is easy to implement and the documentation is accurate.

The one thing to consider is the need for running the apps tier postclone/adcfgclone.pl in case all the required steps is not done in preparing the DR environment, but except that, the documentation is perfect.

Here in this post, I will share you an alternative disaster test method, which is based on the Oracle 's Flasback database technology.

In this method, we can test our DR environments, without doing a Dataguard switchover. So it is easier to implement and it may be needed in some cases.

Here is the action plan(more or less) for accomplishing that;

On Primary DB tier:

Stop the log shipping;

alter system set log_archive_dest_state_2=defer;

On Standby DB tier:

alter database recover managed standby database cancel;

alter database flashback on;

create restore point OPEN_STANDBY guarantee flashback database;

alter database activate standby database;

alter database open;

sqlplus apps/apps_password
exec fnd_conc_clone.setup_clean;
commit;

cd $ORACLE_HOME/appsutil/bin

perl adbldxml.pl tier=db appsuser=apps

sh adconfig.sh

On Standby Apps tier:

cd $ADMIN_SCRIPTS_HOME
perl adcfgclone.pl appsTier

Start the services.
--update conc request queues and wf mailer conf. if necessary

Fail/Switch Back:
On Standby DB tier:

sqlplus "/as sysdba"
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
FLASHBACK DATABASE TO RESTORE POINT OPEN_STANDBY;
DROP RESTORE POINT OPEN_STANDBY;
ALTER DATABASE CONVERT TO PHYSICAL STANDBY;
SHUTDOWN IMMEDIATE;
STARTUP NOMOUNT;
ALTER DATABASE MOUNT STANDBY DATABASE;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;

On Primary DB Tier:
alter system set log_archive_dest_state_2=enable scope=both;

This procedure reflects the actual work that is need to be done(more or lesss), in case of a DR test.
What we do in this procedure is, we stop the managed recovery, create guaranteed flashback restore point, then activate our standby database and then configure our application tier (standby app tier) to this newly activated database and let our functional teams to test the DR environment. After the test, we restore our restore point, start the managed recovery and get back where we start.
This routine may be changed/enhanced according to the needs of the environment, but the important thing is that you get the idea...

EBS 12.2.5 - adopmon

As a new feature of EBS 12.2.5 , we have adopmon(adop monitor, Online Patching Monitoring utility) utility for monitoring the online patching cycles. It provides a readable output, letting us monitor the events that occur in the online patching cycle phases.
The overall progress of an online patching cycle can be followed by running the new Online Patching Monitoring utility (adopmon), while seeing the various individual adop actions being taken (in a readable format).adopmon is executed after setting the run environment. It requires us to provide the apps password and that's it.
Here is an example run, captured during a cutover phase.

[root@ermanserver~]# su - applmgr

  E-Business Suite Environment Information
  ----------------------------------------
  RUN File System           : /app/oracle/PROD/fs1/EBSapps/appl
  PATCH File System         : /app/oracle/PROD/fs2/EBSapps/appl
  Non-Editioned File System : /app/oracle/PROD/fs_ne

  DB Host: ermanserver  Service/SID: PROD

Sourcing the RUN File System ...

[applmgr@ermanserver ~]$ adopmon

Running script. Press Ctrl-C to quit.

Enter the APPS password:

Validating credentials...
Printing the log statements starting from sequence #632872

Timestamp           Node name           Message Type Message Text
---------------------------------------------------------------------------------------------------------------------------
2016/05/11 15:43:55 ermanserver         EVENT        Log: /app/oracle/PROD/fs_ne/EBSapps/log/adop/13/adop_20160511_154324.l
                                                     og
2016/05/11 15:43:56 ermanserver         EVENT        Validating configuration on node: [ermanserver].
2016/05/11 15:43:56 ermanserver         EVENT        Log: /app/oracle/PROD/fs_ne/EBSapps/log/adop/13/cutover_20160511_15432
                                                     4/PROD_ermanserver
2016/05/11 15:44:17 ermanserver         EVENT        Checking if finalize required.
2016/05/11 15:44:17 ermanserver         EVENT        Finalize phase is not required
2016/05/11 15:44:18 ermanserver         EVENT        Submitting request for Internal Concurrent Manager shutdown.
2016/05/11 15:44:18 ermanserver         EVENT        Log file: /app/oracle/PROD/fs1/inst/apps/PROD_ermanserver/logs/appl/ad
                                                     min/log/adcmctl.txt
2016/05/11 15:44:18 ermanserver         EVENT        Cancelling ADZDPATCH concurrent request
2016/05/11 15:44:19 ermanserver         EVENT        Waiting for Internal Concurrent Manager to go down.

Monday, May 9, 2016

EBS 12.2 -- sync errors due to missing patch files/directory does not exists.

In Oracle documents, we see ;
"The directories where you extracted the patches applied in a given patching cycle must be retained, in the same location and with the same contents, until the next prepare phase completes. This is also a requirement for patches applied in a hotpatch session."

See what happened in the next prepare phase, after we applied a patch and then deleted it from PATCH_TOP.


You see, adop complained about the patch directory that was not exists.
This was actually an expected behaviour..
So  according to Oracle , what needed to be done is to download the missing patch from Oracle Support and restore it to the PATCH_TOP.

However, what we did in one of our test environment and want to share with you in this post is a little different.
That is, as we have fs_clone utility, we aborted the patch cycle in which the prepare phase was encountering this error and then executed fs_clone for the solution.
After the fs_clone, we saw the filesystem objects were synched and then we executed the prepare phase again. This time prepare phase completed with success.

Altough we don't suggest this approach, it might become handy one day.
executing fs_clone is actually useful some other cases as well.

As documented in  E-Business Suite - ADOP Basic Usage Training Videos [Video] ( Doc ID 2103131.1 ),  sometimes we can even execute fs_clone by our own decision to decrease the job of the next prepare phase.

Bytheway, this missing patch error, the oracle support solution and fs_clone solution are valid for both online and hot patches.

So the following scenario works; (not recommended..)

time 0: Suppose we apply a hotpatch using adop hotpatch=yes
time 1 : Suppose the patch (in hotpatch mode) applied successfully
time 2: Suppose we delete the patches (including the hotpatch at time0, that we have applied using hotpatch option) from PATCH_TOP
time 3: Development asks us to apply an online patch
-- now we need to start a new patching cycle to apply the online patch requested at time 3.
time 4: To apply the patch requested at time 3, we execute adop phase=fs_clone before executing adop phase=prepare for creating the new patching cycle , because " we deleted the patch files from PATCH_TOP, so adop prepare won't  able to sync the filesystems."

RDBMS -- enabling custom maintanence for the deployments

During the deployment cycles, we may want to have a normal database, running with all of its components including its listener. We may want to have a database in a form equal to its runtime environment and we want to be the only one that is able to connect to the database.
These kinds of desires may arise for having the deployment processes, which may require no one except the administrator to be connected on an Oracle Database, for which we need all the database functionality up&running. So we are talking about a custom maintanence mode here.

Let's see how we implement such a maintanence mode functionality

We first create a trigger(in this example, it is "LOGON_PREVENT_DURING_DEPLOY") to read the sys_context and prevent the login requests except the one coming from the administrator's machine. (in this example: it only lets the user who has "ermanarslan" as its OS user name, to connect to the database). Note that: This trigger may be modified according to our needs.

Trigger:

CREATE OR REPLACE TRIGGER SYSTEM.logon_prevent_during_deploy
   AFTER LOGON ON DATABASE
DECLARE
   v_osuser    v$session.osuser%TYPE;
   v_program   v$session.program%TYPE;
   CURSOR user_prog
   IS
      SELECT osuser, program
        FROM v$session
       WHERE audsid = SYS_CONTEXT ('USERENV', 'SESSIONID');
BEGIN
   OPEN user_prog;
   FETCH user_prog
    INTO v_osuser, v_program;
   CLOSE user_prog;

   IF LOWER (v_osuser) != ('ermanarslan')
   THEN
      raise_application_error
         (-20001,
          '<<<<< The System is in maintanence mode right now. Check again later  >>>>>'
         );
   END IF;
END;
/
ALTER TRIGGER LOGON_PREVENT_DURING_DEPLOY DISABLE;

Then we create 2 procedure for enabling and disabling the trigger. For enabling the trigger(actually enabling the maintanence mode) before we begin our deployment activities , we execute the enabling procedure and for disabling the trigger(actually the maintanence mode), we execute the disabling procedure.

Procedure for enabling the maintanance mode:

CREATE OR REPLACE procedure Enable_maintanence
 IS
 counter INTEGER := 1 ;
 count_son number;
 sqlStmt VARCHAR2(1000);
 cursor c1 is select a.sid,a.job,b.serial# from dba_jobs_running a, v$session b where a.sid=b.sid;
 jobs c1%ROWTYPE;
BEGIN
select count(*) into count_son from dba_jobs_running;
execute immediate 'alter trigger system.logon_prevent_during_deploy enable';
execute immediate 'revoke ADMINISTER DATABASE TRIGGER from dba';
execute immediate 'revoke ADMINISTER DATABASE TRIGGER from IMP_FULL_DATABASE';
loop
exit when count_son=0;
open c1 ;
fetch c1 into jobs;
DBMS_JOB.REMOVE(jobs.job);
sqlStmt := 'ALTER SYSTEM KILL SESSION ''' ||
                TO_CHAR(jobs.sid) ||
                ',' ||
                TO_CHAR(jobs.serial#) ||
                '''' ;

EXECUTE IMMEDIATE sqlStmt;

EXIT WHEN count_son=counter;
counter := counter+1;
close c1;
END LOOP;
execute immediate 'alter system set job_queue_processes=0';
END;
/

Procedure for disabling the maintanance mode:

CREATE OR REPLACE procedure Disable_maintanence
 IS
BEGIN
execute immediate 'alter trigger system.logon_prevent_during_deploy disable';
execute immediate 'grant ADMINISTER DATABASE TRIGGER to dba';
execute immediate 'grant ADMINISTER DATABASE TRIGGER to IMP_FULL_DATABASE';
execute immediate 'alter system set job_queue_processes=20';
END;

/

Of course, the enabling procedure can be modified for killing all current user sessions before performing any deployment/maintanence tasks. Lastly, you can always use the standard restricted mode of the Oracle Database (ALTER SYSTEM ENABLE RESTRICTED SESSION). This custom maintanence mode functioanlity just an alternative.