EBS 12.2 -- interesting question on Business Continuity for Oracle E-Business Suite Release 12.2

One of my collegues asked me the following question ->
"Before the postclone , there should be only the  Ebsapps directory(<APPL_TOP>, <common_top>,Oracle AS Tools 10.1.2 Oracle_Home) present in the host where the postclone script is wanted to be executed. This means $APPL_TOP_NE should be deleted beforehand.
If that so; why do we rsync the concurrent request log and out files from primary to standby application server? I m asking this question, because during a failover a switchover, I will execute a post clone on standby apps server, thus I will have to delete the APPL_TOP_NE before executing it and as the concurrent log and out files are stored in APPL_TOP_NE, why do we rsync them?"

Well, the answer is there in My Oracle Support.
The important point is that, according to the latest documentation, we execute the post clone in the first place when we are creating the standby application tier environment. This postclone will save us from executing future postclones when there will be a switchover or a failover operation required in the future. So as we execute the post clone in the configuration phase of the standby apps tier, we only need to execute the autoconfig in apps tier for switching over ort failing over to the standby site.
This means, our synched concurrent request log and out files will be there in the standby application tier after the switchover or failover operation.

I wanted to share this with you , because the question and its answer was interesting.
So once again, the importance of following the Oracle Support notes was proven.
read -> Business Continuity for Oracle E-Business Suite Release 12.2 Using Oracle 12c (12.1.0.2) Physical Standby Database (Doc ID 1963472.1)

EBS 12.2- Road map and High level action plan for OAM+OID integration

In these days, I have new OAM+OID and EBS 12.2 integration projects.
Altough I have done couple of ciritical SSO+OID and EBS integrations so far, I felt that; the new authenticatator named OAM(Oracle Access Manager) will introduce a new challange for me.

As the first thing to do, I have created a high level action plan which seems like a roadmap and wanted to share it with you. Note that, I also included the HA and DR related reference documents at the bottom of this document to make them to be within arms reach.

Let's take at the look our target, the new EBS login flow after the OAM enablement and the required EBS-OAM installation and integration process from the surface;

Our EBS environment is 12.2.4.
At the moment the latest version of OAM is 11.2.3.0 . OAM 11.2.3.0 is certified with EBS 12.2.5,12.2.4,12.2.3, 12.1.3, 12.1.2, 12.1.1, 12.0.6, 11.5.10.2
So we will install OAM 11.2.3.0.
We will use EBusiness Access Gate to integrate EBS 12.2 with OAM. Actually, we will use Webgate agent in conjunction with EBS Access Gate.
EBS can't go directory to Active Directory/3rd party ldap (not supported), so OID must be used in between EBS-OAM and 3rd party LDAP/Active directory.  We will use OID plugins to speak(ldapbind) with the 3rd party LDAP/Active directory, which is residing in the backend.

When integrated with OAM, EBS login flow works like below;

During login; the request is directed to EBS access gate.
OAM's Web Gate agent intercepts the traffic between the user and EBS Access gate.
Once the traffic is intercepted, OAM's Web Gate agent connects EBS user to EBS access gate to collect the user's credentials. At this point, the user credentials given by the user is submitted to OAM server.
Once the credentials are collected, OAM server decides if the user is required to be authenticated and authenticates the user.
After the authentication, OAM Server creates an SSO session and sends the user identifier to EBS Access Gate.
EBS Access Gate, then connects to the database and links the authenticated OID/LDAP user with and EBS user account. (from fnd_user guid column)
Lastly, user is redirected to original EBS url with an authenticated EBS session.

We will install the following components;

Oracle Access Manager 11.1.2.3
Oracle Identity Management 11.1.1.9.0
Oracle Access Manager WebGate 11.1.2.3.0 (will be installed on top of Oracle HTTP Server 11.1.1.9)
Oracle E-Business Suite AccessGate version: Patch 21522495
Oracle EBS 12.2.4 (already installed)

The detailed installation plan is as follows;
--Note that, this is the first draft, so the actual action/installation plan may be changed during the installation.

*Install an Oracle Database for OID and OAM (Oracle Database 12c Release 1 (12.1.0.1) or higher)-- oracle recommend Enteprise edition for production environments.
----------------------------------------------------------------------------------------------------------------------------
this step doesn't require any references, as it is a standard database installation :)

*Install OID 11.1.1.9 (Oracle Internet Directory 11gR1 Patch Set 7 (11.1.1.9.0) comes as both full installer and an upgrade)
----------------------------------------------------------------------------------------------------------------------------
Ref:Integrating Oracle E-Business Suite Release 12.2 with Oracle Internet Directory 11gR1 (Doc ID 1371932.1)
   --download weblogic 10.3.6
   --download Rcu 11.1.1.9.0 (for creating OAM + OID schemas in the database)
   --download  Oracle Identity Management 11g Patch Set 7 (11.1.1.9.0)
   --install OID + Oracle Directory Integration Platform
            -run rcu (only select 'Oracle Internet Directory')
            -Install Weblogic 10.3.6 and also apply mandatory patch for Weblogic.
            -Install OID +  Oracle Directory Integration Platform (11.1.1.9.0)
            -configure OID + Oracle Directory Integration Platform and Create a weblogic Domain+managed server (Follow Section 7.2.1 OID with ODIP and Fusion Middleware Control
                                                                                                                in a New WebLogic Domain of Oracle® Fusion Middleware Installation Guide for Oracle Identity Management)
            -Apply Required Patch  for Oracle Directory Integration Platform
            -Configure OID with EBS 12.2.
                    -start a new online patching cycle (adop prepare)
                    -set your patch env and run registration script (txkrun.pl)
                    -set required profile options in EBS 12.2(Applications SSO Enable OID Add Event, Link applications user with OID same username, Applications SSO type)
                    -run autoconfig from patch filesystem
                    -set run env
                    -execute adop cutover
                    -configure OID plugin to be able to connect to the Active Directory/3rd Ldap server

Configure Oracle Internet Directory to return operational attributes
Ref:Integrating Oracle E-Business Suite Release 12.2 with Oracle Access Manager 11gR2 (11.1.2) using Oracle E-Business Suite AccessGate (Doc ID 1576425.1)

*Install & Configure Oracle Access Manager 11g Release 2 (11.1.2.3.0)
----------------------------------------------------------------------------------------------------------------------------
Ref: Oracle Fusion Middleware Identity Management 11g Release 2 (11.1.2.3.0) Documentation Library -> Oracle Fusion Middleware Installation Guide for Oracle Identity and Access Management
--check the certification matrix also, for example for IE browswers; only IE 11+ is supported. Also for OEL 6, JDK 1.7.0_80 is required.
   -Install certified JDK
   -Review Database requirement (patches may be needed, some other actions may be needed , review: Oracle Fusion Middleware System Requirements and Specifications for Oracle Identity and Access Management
   11g Release 2 (11.1.2)
   -rcu is already downloaded in OID part. (rcu 11.1.1.9.0 will be used even if the Oracle Access Manager 11.1.2.3.0, because the Oracle instruction is "download and use the specific RCU version available as part of the Oracle Identity and Access Management 11g Release 2 (11.1.2) Media Pack on the Oracle Software Delivery Cloud." So when we  select the media pack we only see the rcu 11.1.1.9.0 there.
   -run rcu (select 'Identity Management - Oracle Access Manager' + Oracle Mobile Security Manager) Oracle Mobile Security is a must,
   because we must also manually select the Identity Management - Oracle Mobile Security Manager schema because when we install and
   configure Oracle Access Management in a WebLogic domain, the Oracle Mobile Security Manager server is installed and configured in the domain by default.
   -weblogic server is already downloaded in OID part.
   -Install Weblogic 10.3.6 and also apply mandatory patch for Weblogic.
   -Install the Oracle Identity and Access Management 11g software. (basically run "runInstaller")
   -Configure the Oracle Identity and Access Management 11g software
          -Configure Oracle Access Management - IAM_HOME/common/bin/config.sh
          -Configure the Database Security Store
          -Start Weblogic Admin Server and managed servers
          -Run the Environment Health Check Utility to Verify our Installation and Configuration (cd IAM_HOME/healthcheck/bin; idmhc.sh -manifest IAM_HOME/healthcheck/config/PostInstallChecks.xml )
   -Apply BP3 to OAM (Oracle strongly recommends applying Oracle Access Manager 11.1.2.3 Bundle Patch 3 (OAM 11.1.2.3.3) as this includes a fix for Patch 19438948. )
*Apply EBS prereqs for OAM
   -Apply the Latest AD and TXK Delta Release Update Packs (Document 1617461.1, Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2, and follow the instructions to apply the required code level of AD and TXK for your system.)
   -Apply patches for EBS 12.2 - OAM 11.1.2.3 interoperability:
        12.2 R12.TXK.C Patch 21523147
        12.2 R12.TXK.C Patch 20735848
        12.2 R12.TXK.C Patch 21229697
*Download and install Oracle Access Manager WebGates
        -Download Oracle Access Manager OHS 11g WebGates 11.1.2.3.0 from Identity & Access Management 11gR2 Downloads.
        -Upgrade EBS's HTTP Server to 11.1.1.9 if it is not already.
        -install Access Manager Web Gates to EBS's HTTP Server.(txkrun.pl -script=SetOAMReg -installWebgate=yes -webgatestagedir=<webgate stage directory>)
        -Apply bundle patch for Oracle Access Manager Webgates - BP01.
*Integrate Oracle E-Business Suite with Oracle Access Manager
       -Deploy Oracle E-Business Suite AccessGate (do it with perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-create-oaea_resources -contextfile=$CONTEXT_FILE -deployApps=accessgate )
       -Register Oracle E-Business Suite with Oracle Access Manager
       -txkrun.pl -script=SetOAMReg -registeroam=yes
*Test Single Sign-On with Oracle E-Business Suite (http://<ebshost>.<domain>:<port>/OA_HTML/AppsLogin)
*Perform fs_clone      
*Configure TLS (SSL) - optional ; Configure SSL/TLS between Webgate and Access Manager + EBS itself.
      -For Webgate and Access Manager:
         Ref:Oracle Fusion Middleware Administrator's Guide for Oracle Access Management: Securing Communication Between OAM Servers and WebGates  
         Securing Communication provides instructions on how to secure communications between Oracle Access Manager 11g and WebGates.
      -For EBS 12.2:
         Ref:Enabling TLS in Oracle E-Business Suite Release 12.2 (Doc ID 1367293.1) To BottomTo Bottom
       --EBS's SSL/TLS version should be the same as OAM's SSL/TLS version.

Additional References if DR and H/A need to be implemented: 

Reference Figure 8-1 shows a sample Oracle Fusion Middleware 11g Oracle Identity Management high availability architecture from the Fusion Middleware High Availability Guide at the URL:
https://docs.oracle.com/cd/E28280_01/core.1111/e10106/imha.htm#ASHIA804

8.2 Prerequisites for Oracle Identity Management High Availability Configuration
https://docs.oracle.com/cd/E28280_01/core.1111/e10106/imha.htm#ASHIA3047

8.3 Oracle Internet Directory High Availability
https://docs.oracle.com/cd/E28280_01/core.1111/e10106/imha.htm#ASHIA806

8.3.3.3 Configuring Oracle Internet Directory With a WebLogic Domain

8.3.3.3.1 Configuring Oracle Internet Directory on OIDHOST1

This section describes the steps to deploy Oracle Internet Directory in a high availability configuration as part of a WebLogic Server domain.

8.3.3.3.3 Configuring Oracle Internet Directory on OIDHOST2
Ensure that the Oracle Internet Directory repository is running and then follow these steps to configure the Oracle Internet Directory instance on OIDHOST2:
On the second node, you will be extending the domain using config.sh tool.

The following document is for Active-Passive solution:

12 Active-Passive Topologies for Oracle Fusion Middleware High Availability
Fusion Middleware High Availability Guide
http://docs.oracle.com/cd/E28280_01/core.1111/e10106/ap.htm#ASHIA3205

The following document is for DR solution:

Fusion Middleware Disaster Recovery Guide
1.2.1 Oracle Fusion Middleware Disaster Recovery Architecture Overview
http://docs.oracle.com/cd/E28280_01/doc.1111/e15250/intro.htm#ASDRG106
The Oracle Fusion Middleware Disaster Recovery solution supports these methods of providing data protection for Oracle Fusion Middleware data and database content:
- Oracle Fusion Middleware product binary files, configuration files, and metadata files
Use storage replication technologies.
- Database content
Use Oracle Data Guard for Oracle databases (and vendor-recommended solutions for thirdparty databases)

Also  note the following in "http://docs.oracle.com/cd/E28280_01/doc.1111/e15250/creating_sites.htm#ASDRG512" ->
4.5.6.1 Using rsync and Oracle Data Guard for Oracle Fusion Middleware Disaster Recovery Topologies

RDBMS -- restore/recover a dropped table column, ORA-01466, 12C rman recover table option

One day , you may accidentally drop a column from one of your critical column in your Production database. I hope you won't be in this situation, but here is the facts that you need to know foır restoring/recovering a dropped column in Oracle.

• Flashback techniques doesn't apply for dropped columns. That is, you can not use flasback query or flashback table to restore a dropped column. You will get "ORA-01466 : Unable to read data -table definitions has changed"  As for the Flashback Table, it cannot rewind tables through structural changes even after a truncate table operation.
• imp(import) or impdp can be used for restoring the dropped column, as we can import the whole table from a backup dump file and then import it to another schema and lastly get the data from that table to our target table. On the other hand, you  eed to have a backup procedure, that regularly takes logical backups of the database using exp or expdp. (you can even roll forward the imported column data by analyzing the archivelogs and extracting dml operations from there using log miner or something)
• Flashback database can be used, but with this option, you will flashback your whole database. So it is not an applicable method in most of the cases.
• If the datafiles are on storage, storage snapshots can be used to create a new luns from the snapshot of the lun that contains the production's database files.  After mounting the lun created from the snapshot, the database can be opened with a different name from that lun and the table(before alter drop column operation) can be gathered from there.
• Rman full restore can be used. That is , the database backup can be restored to a different server or different location and recovered just before the table drop operation. The column data can be done gathered from that restored database using db links. However, this is a heavy and time consuming process.
• Performing RMAN Tablespace Point-in-Time Recovery (TSPITR) is also an option, but this is also heavy and it makes all the objects of that tablespace rolled back/forward to a specified point in time.
• Opening standby read/write by creating a guaranteed restore point (for taking it back) can be used, if the dropped column is not dropped from standby database yet.
• Lastly, if you are on 12C, then you have the option to restore a table using rman from rman backups. This is a new feature introduced in 12C . as in 12c RMAN has an option for 'recover table'. See: RMAN RECOVER TABLE Feature New to Oracle Database 12c Note 1521524.1 (My favorite one :)

                Rman recover table feature works with the following;

• It finds the rman backup that includes the table to be recovered (based on the point in time recovery specified for the recovery)
• It create an auxiallary database and it recovers the table until the specified point in time, into this auxiliary database.
• It then creates a export dump file that contains the recovered table.
• Lastly, it imports this export dump file into the target instance. (optional) --By default, RMAN imports the recovered tables or table partitions, which are stored in the export dump file, into the target database. However, you can choose not to import the recovered tables or table partitions by using the NOTABLEIMPORT clause of the RESTORE command.

                Here are the limitations:

• Tables and table partitions belonging to SYS schema cannot be recovered.
• Tables and table partitions from SYSTEM and SYSAUX tablespaces cannot be recovered.
• Single table partitions can be recovered only if your Oracle Database version is Oracle Database 11g Release 1 or later.
• Tables and table partitions on standby databases cannot be recovered.
• Tables with named NOT NULL constraints cannot be recovered with the REMAP option. 

Before closing this topic, I want to shed a light on a few things and share my comments on them.

Why rman creates an auxilary database and then take an export from there, must be to isolate the mission critical target environments from the restore works. Also It seems like "rman recover table" option is just like an expansion pack for the rman, because still rman restores the backup datafiles in order to restore a table from a backup.

EBS - Why context file is updated from the database? Why any recent changes in the context file are taken back during autoconfig run?

The questions in the title are obvious, so the answers...
There is serial number stored in each context file. It is stored with the tag named oa_context_serial.
Example:

<oa_context_serial oa_var="s_contextserial">55</oa_context_serial>

This serial number is also stored in the FND_OAM_CONTEXT_FILES table. In that table, we have all the context files stored and ordered according to their serial numbers.

So if we have a context file with serial number of 55, and if we have a context file stored in the FND_OAM_CONTEXT_FILES table with a serial number 56 , then our changes done in our apps filesystem context file can be overwritten after an autoconfig run. This is because during autoconfig run, autoconfig will look at the FND_OAM_CONTEXT_FILES table and see there is a high versioned context file stored there. So what autoconfig does in such a situation is, it replaces the values stored in our apps filesystem context file with the values stored in the high versioned context file that is stored in the database.

So, this is an expected behaviour , but it sometimes creates problems.

Here in my forum, there was a discussion on this. -> http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/adcfgclone-pl-errors-td265.html#a483

Some of the values stored in the context file , which reside in the FND_OAM_CONTEXT_FILES were causing problems in a post clone run.
What I suggested was, truncating the FND_OAM_CONTEXT_FILES(actually, oa_context_serial value of the context file stored in the application tier filesystem can also be updated to a higher version value , as well) and that suggested action actually solved the problem.

"How a context file can be in a lower version than the context file stored in the database?" -> is the question that we should ask. My answer is , it sometimes can be lower, really :)

It can be lower in cases where we clone only the database tier of an EBS. In this scenario, database will have more recent context file, but the application tier filesystem will have the old versioned one.
Also look at the following note, even the standard things can make the version of Apps tier context file to be lower than the context file stored in the database -> adop phase=abort causes "Serial number in context file contains lower value than that of database copy." (Doc ID 1916658.1)

So at the end of the day; these things are happening... What we need to do is to take the lesson from them.
At least, with this in mind; when we see our context file values are overwritten after an autoconfig run, we know where to look at...

Linux -- auditing in Linux, auditd, setfacl - Linux ACLs

Recently, we were required to audit the file and directory accesses in Linux.
We had to audit the accesses the directory accesses and record them into files that can't be accessed except the root user and an OS account which is currently used by an auditing application .

What we have done is, we enabled auditd, which is the userspace component to the Linux Auditing System.

Manual of auditd:
http://man7.org/linux/man-pages/man8/auditd.8.html

We have configured the auditd to audit all the access types for some given directories and rotate the audit log files that it creates in a timely manner and keep them for 90 days. (we have done this using auditd configuration file called /etc/audit/auditd.conf  and the audit rules file called /etc/audit/audit.rules )

One of the requirements was as already mentioned; the audit log files which auditd was populating with the audit information, must not  be accessed except the root user and an OS account which was used by an auditing application.

auditd was creating the files in the /var/log/audit directory with the directory permissions 600. So root could read them but the  OS account which was used by the auditing application could not.

So in order to give a read permission for this directory and files, we used Linux ACLS.

We have used setfacl utility to give read permission to that specific user for the audit log directory and the files stored in it. (another unfavourable method could be modifying /etc/audit/auditd.conf and changing the log_group from root to the group of Os account that is used by the auditing application) One thing which was important was, in Linux to change into a directory or list its content or read a file in it,  the user needs to have execute permissions on that directory. 

So we had to give the execute permission in addition to the read permission to the that user for that directory . (command was something like this: setfacl -Rm u:B:rx /var/log/audit.

Note that, Linux ACLs can be used for defining the directory and file permissions automatically, when they are created. That is using ACLs we can make the files to be in any permission we want and it will be done on-the-fly.  (you can set it and then test it using a simple command like "touch examplefile") . We actually use setfacl's "-d" option for this.

-d, --defaultAll operations apply to the Default ACL. Regular ACL entries in the input set are promoted to Default ACL entries. Default ACL entries in the input set are discarded. (A warning is issued if that happens).

Manual of setfacl : http://linux.about.com/library/cmd/blcmdl1_setfacl.htm

ODA -- configuring memory & cpu for ODA Virtualized Environments

I want to shed a light on Cpu and memory configuration of ODA virtualized environments.
As the ODA machine is actually used mostly for providing a small considolation environment for both applications and databases, it is good to configure it in the appropriate way .

Well, as documented in "Solution-in-a-box: Best practices for deploying Oracle E-Business Suite Release 12.2.4 on Oracle Database Appliance 12.1.2.2" ( I think, this can be considered as appropriate even for the applications other than EBS),  it is good to give %50 of the ODa hardware resources to ODA_BASE and leave the rest to dom0 and other vm machines.

That is , it is a good start to configure the CPU and MEmory of ODA_BASE nodes to be %50 percent of the total resources.
oakcli configure oda_base command will help you on that. (run it from dom0 and run it once for every node.. That is, run it from node0 and then node1)
configure oda_base command will let you specify the cpu and memory resources for ODA_BASE nodes.

At this point, it is important to mention that, the cpu cores that we see in ODA_BASE nodes using cat /proc/cpuinfo command are actually the virtual cores. 1 physical core equals 2 virtual core, and in this sense, if we give 8 cores to an ODA_BASE node using oakcli configure, then we see 16 cores when we check it using the OS tools from the ODA_BASE.

So, after configuring the ODA_BASE, the remaining cores and memory, can be scattered among the necessary vm servers.
For the vm servers, what needs to be done is to use oackli configure vm command.

Here is an example;

oakcli configure vm ermanvm -vcpu 8 -maxcpu 8 -> these are virtual cpus. ( run it from oda_base)oakcli configure vm ermanvm -memory 8g -maxmemory 8g

As suggest in my earlier post, we recommend using the ODA_BASE domains for placing the databases. (even the TEST database environments should be placed there, as the performance of the VM domain will not be sufficient for handling a serious database IO activity) )

An example configuration can be giving 12 cores to ODA_BASE nodes and giving 48 GB memory for the ODA_BASE nodes.

It is up to you to decide how to distribute the remaining resources to the available vms in your ODA virtualized environment, but here is an example configuration that can be used in an EBS 11i environment , which is planned to be upgraded to 12.2

Each App VM machine can have 4 physical cores and 8 GB rams. (if PAE is enabled, 16GB ram is preferred)

Each new apps 12.2 servers, which are planned to be used in the upgrade process can have 4 physical cpus and 20GB rams each.

The distribution should be evenly, as it is not good to increase the load of one ODA node , while the other node is sleeping.

Also, as the standard EBS deployments are done in way that, the first node is a little more utilized than the second node (especially in database level); it is better to give a little more mercy to the first node  while evenly distributing the resources to the node.

An example distribution, which can be used for hosting an EBS farm that consist of 3 11i application vm machines, and 2 12.2 application vm machine is given as follows:

1st node:

1 11i app (prod app) --a high utilized apps env.

1 11i app (test app)  -- a semi utilized apps env.

-- Databases are on ODA_BASE...-- Databases are on ODA_BASE... (semi or high utilized db instances)

2nd node:

1 11i app (crp1 app) -- low utilized apps env.

1 12.2 app (crp1 app) -- low utilized apps env.

1 11i app(crp2 app) -- low utilized apps env.

1 12.2 app (crp2 app) -- low utilized apps env.

-- Databases are on ODA_BASE... (low utilized db instances)

EXADATA/RAC -- CRS-2683: It is locked by 'SYSTEM' & Do not have sufficient voting files, found 1 of 2 configured files, needed at least 2

It was just a week ago, that I have encountered this strange problem in one of my Customer site.
The system was an Exadata X3-2 , which was hosting a production EBS database on top of it.
A flash card had to be replaced , so we have shut down the machine properly for the replacement.
However, after the flash card replacement, databases could not be started again.
Problem: RAC services can not be started after rebooting Exadata compute and cell nodes.

Analysis: When the issue was escalated to me , I directly jump into the OHASD log file, as OHASD was the process that should have been the process starting the Oracle Clusterware processes.

Picture Ref : http://docs.oracle.com/cd/E14795_01/doc/rac.112/e10717/intro.htm

The flashing error was there;

2016-05-12 13:29:15.815: [ CRSPE][1145149760] {0:0:178} ICE has queued an operation. Details: Operation [START of [ora.crsd 1 1] on [osrvdb01] : local=0, unplanned=00x2aaab427de30] cannot run cause it needs W lock for: WO for Placement Path RI:[ora.ctssd 1 1] server [osrvdb01] target states [ONLINE ], locked by op [START of [ora.asm 1 1] on [osrvdb01] : local=0, unplanned=00x2aaab00bee30].
Owner: CRS-2683: It is locked by 'SYSTEM' for command 'Resource Autostart : osrvdb01' 

It was obvious that there was a cluster level locking problem...
Kept checking on the other logs and saw that  the CSSD daemon OCCSD was terminating abornormally.

Note that: OCR of this RAC was  in DBFS_DG diskgroup in ASM...

2016-05-12 14:54:02.453: [ CSSD][1141094720]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2016-05-12 14:54:02.453: [ CSSD][1141094720]clssnmCompleteVFDiscovery: Completing voting file discovery
2016-05-12 14:54:02.453: [ CSSD][1141094720]clssnmvDiskStateChange: state from discovered to pending disk o/192.168.10.3/DBFS_DG_CD_03_osrvcel01
2016-05-12 14:54:02.453: [ CSSD][1141094720]clssnmvDiskStateChange: state from pending to configured disk o/192.168.10.3/DBFS_DG_CD_03_osrvcel01
2016-05-12 14:54:02.453: [ CSSD][1141094720]clssnmvDiskStateChange: state from discovered to deconfigured disk o/192.168.10.4/DBFS_DG_CD_02_osrvcel02
2016-05-12 14:54:02.927: [ CLSF][478835504]Allocated CLSF context
2016-05-12 14:54:02.929: [ SKGFD][478835504]Handle 0x2aaaac668b60 from lib :OSS:: for disk :o/192.168.10.3/DBFS_DG_CD_03_osrvcel01:
2016-05-12 14:54:02.929: [ CLSF][478835504]Opened hdl:0x2aaaac5af9d0 for dev:o/192.168.10.3/DBFS_DG_CD_03_osrvcel01:
2016-05-12 14:54:02.929: [ CSSD][478835504]ASSERT clssnml.c 4532016-05-12 14:54:02.929: [ CSSD][478835504]clssnmlgetleasehdls: Do not have sufficient voting files, found 1 of 2 configured files, needed at least 2
2016-05-12 14:54:02.929: [ CSSD][478835504]###################################
2016-05-12 14:54:02.929: [ CSSD][478835504]clssscExit: CSSD aborting from thread Main
2016-05-12 14:54:02.929: [ CSSD][478835504]###################################2016-05-12 14:54:02.929: [ CSSD][478835504](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally

From this perspective, there was a Voting Disk problem.
Note that: Voting disks manage information about node membership. Each voting disk must be accessible by all nodes in the cluster for nodes to be members of the cluster.

As seen in the logs above, OCSSD found 2 disks, but then made one of them deconfigured .
Then it was saying "lssnmlgetleasehdls: Do not have sufficient voting files, found 1 of 2 configured files, needed at least 2"
There could be a problem with OCSSD, as  it did not like the voting disk DBFS_DG_CD_02_osrvcel02.(which was physically in a normal situation, no errors were on that disk physically, as we checked it using cellcli , celldisk, physicaldisk and griddisk, all were fine for that disk.)

So OCSSD discovered that disk but then deconfigured it and then it could not continue with only 1 voting disk and aborts.

Note that, In 11gR2, ocssd.bin discover voting disk with setting from GPnP profile, if not enough voting disks can be identified, ocssd.bin will abort itself.

ASM, on the other hand, was started and terminated due to ORA-600 errors, reported in its alert log as follows;

Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_24013.trc (incident=433951):
ORA-00600: internal error code, arguments: [kfdvfGetCurrent_baddsk], [], [], [], [], [], [], [], [], [], [], []

Explaination of the cause of this bug ORA-600:

1) CSS is not able to recognize the online of the disks and re-establish that the majority is met for the voting file count. Solution : Reboot  (however, I rebooted the nodes and as suggested I rebooted them one by one, but it didnt solve the problem)

2)Bug 12330287 , upgrade to 12.1

CSS really could not recognize the disks. It was reported in its logs as I mentioned earlier, so we are in correct way.
At this point, if we summarize what we have;

-> crsdb can not be started
-> cssdb can not see all the voting disks
-> ASM was crashing because of cssdb , as it can not see the voting disks properly.

Decision about the possible causes:

• A race condition which might be caused by an improper shutdown of the machine nodes.
• Bug 14619103  "Operation [START of xxx] ... has been replaced with [START of yyy]" chooses incorrect resource.

Decision: 

There is repository level locking issue between ASM and CRS, which make CRS not be able to be started and which in turn make abnormal chaing of event happen (such as CSSD can not be started because it could not see the voting disks properly and ASM crashed because of that. )

Solution:

The problem resolved by starting crs in exclusive mode. By starting crs in exclusive mode, I thought that the locking issue may be resolved. I though , in exclusive mode, crs might be started without problem and then start might solve the locking problem(actually kind of a deadlock) between CRS and ASM.

What I have done was;
I executed the following command to start crs in exclusive mode in node1.
$GRID_HOME/bin/crsctl start crs -excl -nocrs

Note that:

The -nocrs option ensures that the crsd process and OCR do not start with the rest of the Oracle Clusterware stack.
-excl option ensures that the Oracle Clusterware stack start in exclusive mode on one node.

After executing the command above in node 1, I immediately saw the ASM2 in node2 was started. It seemed the lock was released.
So, after that, I rebooted node1 and after the reboot of node 1, everyting was started to be working.

Both of the nodes was seeing 3 voting disks and ASM instance and databases was started without any problems.

Outputs:
[oracle@osrvdb02 bin]$ ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE ad944505cc0c4f4ebf29eb746962ab06 (o/192.168.10.3/DBFS_DG_CD_03_osrvcel01) [DBFS_DG]
2. ONLINE 1a8ea7860b0a4f86bf7cbd7423710dfd (o/192.168.10.5/DBFS_DG_CD_02_osrvcel03) [DBFS_DG]
3. ONLINE 2a57155433e64f78bfb3424215e8dc16 (o/192.168.10.4/DBFS_DG_CD_02_osrvcel02) [DBFS_DG]
Located 3 voting disk(s).


[oracle@osrvdb01 bin]$ ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE ad944505cc0c4f4ebf29eb746962ab06 (o/192.168.10.3/DBFS_DG_CD_03_osrvcel01) [DBFS_DG]
2. ONLINE 1a8ea7860b0a4f86bf7cbd7423710dfd (o/192.168.10.5/DBFS_DG_CD_02_osrvcel03) [DBFS_DG]
3. ONLINE 2a57155433e64f78bfb3424215e8dc16 (o/192.168.10.4/DBFS_DG_CD_02_osrvcel02) [DBFS_DG]
Located 3 voting disk(s).

Before finishing this blog post, I would like to underline 2 things;

This situation can be thought as kind of a one time issue, which can be considered as a generic CRS issue, which was actually caused by the design of Oracle RAC.

The suggested solution was upgrading GRID + DB to 11.2.0.4, as for 11.2.0.3 and versions below that are already desupported and no patches will be provided.

EBS -- EBS 12.2.5 - 12C DB installation on Linux

In this blog post, we will see an example EBS 12.2 installation , which is done on a Linux x86-64 system.

The aim of this installation is to obtain an EBS 12.2.5 instance running on Oracle Linux 6 64-bit.

We will use the latest startCD, so we will have an 12C database in our newly installed EBS 12.2.5 instance. This document involves some post upgrade steps, which may be required or changed according to your needs. The underlying server system used for this installation is based on a single node server environment. However, the steps are quite similar in a multi node system.

The general overview of the process is to install Oracle Linux, install EBS 12.2.0 and upgrade it to EBS 12.2.5.  This blog post is a little different than my earlier post, as it will be like a compilation, but still it will give the valuable filtered information required for EBS 12.2. installations.

Please do not take the configuration values, licensed products as well as the post upgrade steps used in this document as the orders :), because they are only there for giving you the general explanation of the installation process, as they may vary according to your environments and needs.

Let 's see, how we do it.

Main Document: Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.2) for Linux x86-64 (Doc ID 1330701.1)

1. Installation Linux 6.5

2. Installing rpms for EBS 12.2 and RDBMS 12C

[root@ermantest yum.repos.d]# wget http://public-yum.oracle.com/public-yum-ol6.repo

Note : use vi editor to open repo and change enable 0 to 1 for addon and latest repositery

yum install -y oracle-ebs-server-R12-preinstall

yum install –y oracle-rdbms-server-12cR1-preinstall

3. Check all rpms

rpm -qa --qf "%{n}-%{v}-%{r}.%{arch}\n" | grep openmotif21

4. Run ldconfig –v command to relink all symbolic libraries

5. Disable selinux and shutdown iptables,ip6tables services also on run levels

vi /etc/selinux/config ( change SELINUX=enforcing to SELINUX=disabled )

service iptables stop

service ip6tables stop

chkconfig --level 0123456 ip6tables off

chkconfig --level 0123456 iptables off

6. Check kernel settings for the parameters are correctly set

Note: If the current value for any parameter is higher than the value listed in the following table, then do not change the value of that parameter.

vi /etc/sysctl.conf

kernel.semmsl 256

kernel.semmns 32000

kernel.semopm 100

kernel.semmni 142

kernel.shmall 20971524

kernel.shmmax Half the size of the physical memory (in bytes), and at least 42949672952

kernel.shmmni 4096

kernel.msgmax 8192

kernel.msgmnb 65535

kernel.msgmni 2878

fs.file-max 6815744

fs.aio-max-nr 1048576

net.ipv4.ip_local_port_range 9000 65500

net.ipv4.tcp_tw_recycle 05

net.core.rmem_default 262144

net.core.rmem_max 4194304

net.core.wmem_default 262144

net.core.wmem_max 1048576

Note : Parameter kernel.sem = 250 32000 100 128 is changed to kernel.sem = 250 32000 100 142

7. Check Domain Name System (DNS) Resolver Parameters , it should be below

options attempts:5
options timeout:15

8. Verifying Host Names

· Verify that the /etc/hosts file is formatted as follows:

127.0.0.1 localhost.localdomain localhost
[ip_address] [node_name].[domain_name] [node_name]

· Verify that the /etc/sysconfig/network file is formatted as follows:
HOSTNAME=[node_name].[domain_name]
or as follows:

HOSTNAME=[node_name]
NISDOMAIN=[domain_name]

· If the /etc/sysconfig/networking/profiles/default/network file exists, remove it.

· If you changed any files in the previous steps, restart the system

9. Modify ulimit settings in limits.conf
vi /etc/security/limits.conf

# oracle-rdbms-server-11gR2-preinstall setting for nofile soft limit is 1024
oracle soft nofile 4096
# oracle-rdbms-server-11gR2-preinstall setting for nofile hard limit is 65536
oracle hard nofile 65536
# oracle-rdbms-server-11gR2-preinstall setting for nproc soft limit is 2047
oracle soft nproc 2047
# oracle-rdbms-server-11gR2-preinstall setting for nproc hard limit is 16384
oracle hard nproc 16384
# oracle-rdbms-server-11gR2-preinstall setting for stack soft limit is 10240KB
oracle soft stack 10240
# oracle-rdbms-server-11gR2-preinstall setting for stack hard limit is 32768KB
oracle hard stack 32768

10. Download EBS Installation Packages, Oracle E-Business Suite (12.2.5)

V100052-01_1of3.zipOracle E-Business Suite Release 12.2.0 Rapid Install Start Here 725.3 MB
V100052-01_2of3.zipOracle E-Business Suite Release 12.2.0 Rapid Install Start Here 81.6 MB
V100052-01_3of3.zipOracle E-Business Suite Release 12.2.0 Rapid Install Start Here 97.3 MB
V100059-01_1of2.zipOracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install Databases PROD - Disk 1 1007.0 MB
V100059-01_2of2.zipOracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install Databases PROD - Disk 1 1.6 GB
V100060-01_1of3.zipOracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install Databases PROD - Disk 2 1.6 GB
V100060-01_2of3.zipOracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install Databases PROD - Disk 2 1.1 GB
V100060-01_3of3.zipOracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install Databases PROD - Disk 2 1.0 GB
V100061-01.zipOracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install Technology one-off Patches 839.9 MB
V100078-01.zipOracle E-Business Suite Release 12.2 Supplemental Documentation - REVIEW FIRST 21.6 MB
V100079-01.zipOracle E-Business Suite Release 12.2 Supplemental Media 603.2 MB
100102-01.zipOracle Database 12c Release 1 (12.1.0.2.0) EXAMPLES for Linux x86-64 476.1 MB
V77840-01_1of9.zipOracle E-Business Suite Release 12.2.5 Update Pack for Linux x86-64 514.5 MB
V77840-01_2of9.zipOracle E-Business Suite Release 12.2.5 Update Pack for Linux x86-64 527.6 MB
V77840-01_3of9.zipOracle E-Business Suite Release 12.2.5 Update Pack for Linux x86-64 459.4 MB
V77840-01_4of9.zipOracle E-Business Suite Release 12.2.5 Update Pack for Linux x86-64 530.4 MB
V77840-01_5of9.zipOracle E-Business Suite Release 12.2.5 Update Pack for Linux x86-64 571.4 MB
V77840-01_6of9.zipOracle E-Business Suite Release 12.2.5 Update Pack for Linux x86-64 554.9 MB
V77840-01_7of9.zipOracle E-Business Suite Release 12.2.5 Update Pack for Linux x86-64 526.7 MB
V77840-01_8of9.zipOracle E-Business Suite Release 12.2.5 Update Pack for Linux x86-64 596.3 MB
V77840-01_9of9.zipOracle E-Business Suite Release 12.2.5 Update Pack for Linux x86-64 100.5 MB
V77972-01.zipOracle E-Business Suite Release 12.2.5 Documentation Library 730.2 MB

Oracle Database (12.1.0.2.0)

V46095-01_1of2.zipOracle Database 12c Release 1 (12.1.0.2.0) 1.6 GB
V46095-01_2of2.zipOracle Database 12c Release 1 (12.1.0.2.0) 967.5 MB

Oracle E-Business Suite (12.2.0)

V35802-01.zipOracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install AS10.1.2 711.6 MB
V35803-01_1of3.zipOracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 1 1.2 GB
V35803-01_2of3.zipOracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 1 1022.8 MB
V35803-01_3of3.zipOracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 1 1.3 GB
V35804-01_1of2.zipOracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 2 1.2 GB
V35804-01_2of2.zipOracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 2 1.6 GB

Oracle Web Tier (11.1.1.9.0)

V75792-01.zipOracle Fusion Middleware Web Tier Utilities 11g Patch Set 7 (11.1.1.9.0) for Linux x86-64 2.0 GB

Oracle WebLogic Server (10.3.6.0.0)
V29856-01.zipOracle WebLogic Server 11gR1 (10.3.6) Generic and Coherence 1018.5 MB

11. Create Stage

[root@ermantest EBS1225]# cd startCD/Disk1/rapidwiz/bin/

[root@ermantest bin]# ./buildStage.sh

Oracle E-Business Suite Rapid Install
Version 12.2.0

Press Enter to continue...
Build Stage Menu
------------------------------------------------------
1. Create new stage area
2. Copy patches to existing stage area
3. List files in TechPatches directory
4. Exit menu

Enter your choice [4]: 1

Rapid Install Platform Menu

1. Oracle Solaris SPARC (64-bit)
2. Linux x86 (64-bit)
3. IBM AIX on Power Systems (64-bit)
4. HP-UX Itanium
5. Exit Menu

Enter your choice [5]: 2

Running command:
……………..
Specify the directory containing the zipped installation media:
/ebssetupfiles/EBS1225/

File list:
/ebssetupfiles/EBS1225/startCD/Disk1/rapidwiz/bin/stageData/zipFiles.dat

The following files are missing from /ebssetupfiles/EBS1225/.
Return to the Oracle Software Delivery Cloud and download them:
====================================================

Part number: V100053-01

Description:
Oracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 1
File name: V100053-01_1of2.zip
====================================================
Part number: V100053-01
Description:
Oracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 1
File name: V100053-01_2of2.zip

====================================================

Part number: V100054-01
Description:
Oracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 2
File name: V100054-01.zip
====================================================

Part number: V100055-01

Description:
Oracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 3
File name: V100055-01.zip

====================================================

Part number: V100056-01

Description:
Oracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 4
File name: V100056-01.zip

====================================================

Part number: V100057-01

Description:
Oracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 5
File name: V100057-01.zip

====================================================

Part number: V100058-01

Description:
Oracle E-Business Suite Release 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 6
File name: V100058-01.zip

====================================================

WARNING: Some of the required zip files have not yet been downloaded. Do you want to continue, and obtain them later? (Y)/N

Y

Note that: During creation of stage , enter to continue couple of times

12. Start E-Business Installation / EBS 12.2.0 installation

[root@ermantest rapidwiz]# pwd

/ebssetupfiles/EBS1225/startCD/Disk1/rapidwiz
[root@ermantest rapidwiz]# ./rapidwiz

13. Now follow the screens

If we encounter some errors for database pre-install checks and/or for filesystems,  fix them as documented below ;

Open new terminal with root user and run xhost +

Database Pre-installchecks;

Create oraInst.loc file and point it to the inventory location, which can be ORACLE_BASE directory.
ORACLE_HOME==/u01/oracle/TEST/12.1.0
ORACLE_BASE==/u01/oracle/TEST

So the orainventory should be in ora base directory

[oracle@ermanrtest app]$ cat /etc/oraInst.loc 

inventory_loc=/u01/oracle/TEST/oraInventory
inst_group=dba
For filesystem check error ;
chmod –R 777 /u01   (or any appropriate file permissions)

After installation is completed we can connect to the EBS and check the version 12.2.0 and go for next step...

At this point, EBS 12.2.0 installation is complete, so we continue with the EBS 12.2.5 upgrade.

--PREPARING for 12.2.0 to 12.2.5 upgrade--

APPLYING TECHNOLOGY PATCHES FOR 12.2.0 Oracle E-Business Suite Release 12.2.5 Readme (Doc ID 1983050.1) Master Note

1. EBS Technology Codelevel Checker (ETCC) (Doc ID 1594274.1)

Note: This patch verifies the missing bugfixes which is not applied for db and middle tiers. You can apply the patches directly to the related component but I recommend you go for the next steps and crosscheck the output of the log with the table you will see in section 3.2

Download patch 17537119

Upload patch to DB and APP node

For DB Node :

mkdir –p $ORACLE_HOME/appsutil/etcc
copy all files in the patch to the directory “mkdir –p $ORACLE_HOME/appsutil/etcc”
cd $ORACLE_HOME/appsutil/etcc
./checkDBpatch.sh

For APP Node :

cd <patchdirectory>
./checkMTpatch.sh

2. Apply RDBMS (12.1.0.2) Patches

Note: Shutdown database before apply patches

Not included in any startCD	Linux x86-64
	21841318
	22098146
	21286665
	21967332	opatch rollback -id 18893947
	21387964

3. After applying database patches, ensure the following additional tasks are performed:

Copy $APPL_TOP/admin/adgrants.sql to db node
sqlplus "/ as sysdba" @adgrants.sql apps
sqlplus "/ as sysdba" @$ORACLE_HOME/rdbms/admin/utlrp.sql

4. Apply Consolidated Seed Table Upgrade Patch (Required)

Note: If the Release 12.2.0 Vision or PROD instance (New Installation) was created using startCD 12.2.0.51 (Patch 22066363), you must apply Consolidated Seed Table Upgrade Patch 17204589:12.2.0 by merging with Patch 21900871:12.2.0 on the run file system using adop hotpatch mode.

sh $ADMIN_SCRIPTS_HOME/adstpall.sh “ kill -9 `ps -ef|grep applmgr | awk '{print $2}'` “
sh $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start

Note : copy patches to the patch_top directory “/app/oracle/PROD/fs_ne/EBSapps/patch “

adop phase=apply patches=17204589,21900871 merge=yes hotpatch=yes

5.   3.1 Path A — Upgrade and New Installation Customers. Run ETCC (1617461.1 )

Note: Applying the AD and TXK Delta 7 release update packs checks for the existence of the ETCC tables. Even if you already have all the required database patches installed, you must execute the latest version of ETCC on your database before proceeding to Section 3.

6. Technology Inventory Utility (Optional )

The Technology Inventory Utility generates a report that lists the installed technology stack components, as well as patches that have been applied. Any patches listed in this section that are not shown in the generated report must be installed. Refer to Oracle E-Business Suite Maintenance Guide for instructions on generating the technology report.

perl $FND_TOP/patch/115/bin/TXKScript.pl -script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP -contextfile=$CONTEXT_FILE -appspass=apps321 -outfile=$APPLTMP/Report_Inventory.html

7. Apply Oracle Reports and Forms Patches ( 10.1.2.3.0 )

Note : Any configuration file changes suggested in post-install steps can be skipped, as those changes are controlled through technology stack templates initialized by AutoConfig. Any recommendation in any Forms patch readme to redeploy Forms ear file can be skipped as it is automatically deployed by Oracle E-Business Suite service control scripts before starting services. Please regenerate jars from adadmin after applying any of the forms patch, do not opt to force the regeneration of all JAR files. If the patch is being applied in the patch file system then ensure Online Patching prepare phase (adop phase=prepare) has completed successfully first before running adadmin. If the patch is being applied in the run file system, run adadmin after applying the patch

Note: The 10.1.2.3 distribution is 32-bit, even on 64-bit platforms. Download the available patch for your platform.

	Linux x86-64
Not included in any startCD	21539521

Default Oracle Home path is already set to 10.1.2 directory but if you need to set or to be make sure use below command

export ORACLE_HOME=/u01/oracle/TEST/fs1/EBSapps/10.1.2
export PATH=$ORACLE_HOME/OPatch:$PATH
opatch lsinventory

[applmgr@ermantest~]$ opatch lsinventory

Oracle Home : /u01/oracle/TEST/fs1/EBSapps/10.1.2
Oracle Home Inventory : /u01/oracle/TEST/fs1/EBSapps/10.1.2/inventory
Central Inventory : /u01/oracle/TEST/oraInventory
from : /etc/oraInst.loc
OUI location : /u01/oracle/TEST/fs1/EBSapps/10.1.2/oui
OUI shared library : /u01/oracle/TEST/fs1/EBSapps/10.1.2/oui/lib/linux/liboraInstaller.so
Java location : /u01/oracle/TEST/fs1/EBSapps/10.1.2/jdk/jre/bin/java

Opatch apply
After apply patch
Run adadmin
Generate jar files ( dont use force )

8. Apply Oracle Home Patch (Version 10.1.0.5.0)

There is no patch to apply if you installed EBS with the latest startCD. Check “Table 3 - Patches and bug numbers for 10.1.0.5.0” in the document.

9. Apply patch for Oracle Fusion Middleware (FMW) webtier 11.1.1.9

Note: When obtaining patches listed in these tables, download and install the 64-bit version of the patches.

	Linux x86-64
Not included in any startCD	22107540

Set the ORACLE_HOME environment variable to Webtier home and Opatch version should be 11.1.0.9.0 or higher

export ORACLE_HOME=/u01/oracle/TEST/fs1/FMW_Home/webtier

export PATH=$ORACLE_HOME/OPatch:$PATH
[applmgr@ermantest ContentsXML]$ opatch lsinventory

Oracle Interim Patch Installer version 11.1.0.12.9
Copyright (c) 2016, Oracle Corporation. All rights reserved.
Oracle Home : /u01/oracle/TEST/fs1/FMW_Home/webtier
Central Inventory : /u01/oracle/TEST/oraInventory
from : /u01/oracle/TEST/fs1/FMW_Home/webtier/oraInst.loc
OPatch version : 11.1.0.12.9
OUI version : 11.1.0.11.0
OPatch detects the Middleware Home as "/u01/oracle/TEST/fs1/FMW_Home"

Installed Top-level Products (1):
Oracle WebTier and Utilities CD 11.1.1.9.0
cd 22107540/
opatch apply -jre $ORACLE_HOME/jdk/jre

10. Apply patch for Oracle Fusion Middleware (FMW) oracle_common 11.1.1.9

Not included in any startCD	Linux x86-64
	21366277
	21628307

Set the ORACLE_HOME environment variable to oracle_common home and Opatch version should be 11.1.0.9.0 or higher

export ORACLE_HOME=/u01/oracle/TEST/fs1/FMW_Home/oracle_common
export PATH=$ORACLE_HOME/OPatch:$PATH
To be make sure for the oracle home is correctly set. Use opatch lsinventory command

cd <patchdirectory>
opatch apply

11. Apply patch for Oracle Weblogic Server 10.3.6.0

Footnote 1 - This is the 10.3.6.0.12 PSU. When applying a PSU all conflicting patches (including overlays and previous PSUs) would need to be removed. For a list of bug fixes included in this PSU, refer to My Oracle Support Knowledge Document 2030251.1, Oracle WebLogic Server Patch Set Update 10.3.6.0.12 Fixed Bugs List.

Footnote 2 - This overlay patch is specific to 10.3.6.0.12 PSU; apply the PSU Patch 20780171 then this overlay patch.

Footnote 3 - This patch is specific to the Windows Platform.

Footnote 4 - This Oracle WebLogic Server patch is required for Oracle Fusion Middleware 11.1.1.9.

Ensure you have applied R12.TXK.C.Delta.7 (Patch 20784380) and Patch 21846184.

Refer to My Oracle Support Knowledge Document 1617461.1, Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release.

Patch 21846184 includes TXK code that will enable Java Secure Socket Extension (JSSE) within Oracle E-Business Suite Release 12.2 when Oracle Fusion Middleware version is 11.1.1.9.

Follow the instructions in the patch readme, except for the last step (Step 6) in the post-install instructions.
Patch Installation Steps ;

· Copy content of this zip file with the exception of README file to your SmartUpdate cache directory ($FMW_HOME/utils/bsu/cache_dir by default)

· cd $FMW_HOME/utils/bsu

· you will see the oracle smart update interface and below of the screen displays the patches that needs to be apply

· Start to apply patches and follow their readme,footnotes.

· To apply patches, first you have to apply patch number 20780171 but you may have conflict warning with the previous patches which has older PSU patches like 10.3.6.0.7 because 20780171 patch number is PSU 10.3.6.0.12 patch

·we removed the below patches and then run ./checkMTpatch.sh

Y5AP,BLTG,1LRI,VKXF,2GYW,CW9T, FCX7

· According to the output we have to apply the below patches fort he PSU 10.3.6.0.12

12.2.0.51	13964737 (see Footnote 4)	YVDZ	13964737
Not included in any startCD	20780171 (see Footnote 1)	EJUW	20780171
	13729611 (see Footnote 2)	L34G	13729611
	17319481 (see Footnote 2)	N5FK	17319481
	19687084 (see Footnote 2)	8FBW	19687084
	22128205 (see Footnote 2)	GK5N	22128205,13337000, 21562338

12. Apply R12.TXK.C.Delta.7 patches

Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2 (Doc ID 1617461.1)

· Run ETCC according to the section “3.1.1 Prerequisite.
· Source the run edition environment file.

. /u01/oracle/TEST/fs1/EBSapps/appl/APPSTEST_ermtest.env

sh $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start ( It is already started for upgrade patch )

· Start up only Weblogic AdminServer service on the run edition application tier file system

· Apply below patches
Unzip patches and run adgrants
Patch 20745242 (R12.AD.C.Delta.7)
Patch 22123818:R12.AD.C
Patch 20784380 (R12.TXK.C.Delta.7)
Patch 22363475:R12.TXK.C
Patch 22495069:R12.TXK.C

Note : Compare adgrants.sql versions with in the patch directory and $APPL_TOP/admin

vi 20745242/admin/adgrants.sql
vi $APPL_TOP/admin/adgrants.sql

sqlplus / as sysdba @/app/oracle/PROD/fs_ne/EBSapps/patch/20745242/admin/adgrants.sql apps

· Copy all patches to the $PATCH_TOP (/u01/oracle/TEST/fs_ne/EBSapps/patch )

adop phase=apply patches=20745242 hotpatch=yes
sqlplus / as sysdba @/app/oracle/PROD/fs_ne/EBSapps/patch/22123818/admin/adgrants.sql apps

run etcc on db and app nodes

adop phase=apply patches=22123818 hotpatch=yes
adop phase=apply patches=20784380,22363475,22495069 hotpatch=yes merge=yes

After applied the patch with merge parameter. ( applied Delta Patches ) you will see the env file in your system base directory

/u01/oracle/TEST
-rw-r--r-- 1 applmgr oinstall 6467 Mar 10 15:06 EBSapps.env

· Migrate Latest Code to Database Tier
On the Application Tier (as the APPLMGR user):
Run AutoConfig on the run file system.
. /u01/oracle/TEST/EBSapps.env run
cd ADMIN_SCRIPTS_HOME
adautocfg.sh

Create the appsutil.zip file in <INST_TOP>/admin/out.
perl $AD_TOP/bin/admkappsutil.pl
Copy appsutil.zip file to database $ORACLE_HOME

On the Database Tier (as the ORACLE user):
Source the environment variables for RDBMS ORACLE_HOME.
Uncompress appsutil.zip, under <RDBMS ORACLE_HOME>.
cd <ORACLE_HOME>
backup appsutil folder
unzip -o appsutil.zip
cp –pr appsutil_backup/jre appsutil/
perl adbldxml.pl ( there is no Context File in new appsutil directory so regenerate it )
Run AutoConfig on <RDBMS ORACLE_HOME>

13.   Restart app and DB

--UPGRADING EBS FROM 12.2.0 TO 12.2.5--

Oracle E-Business Suite Release 12.2.5 Readme (Doc ID 1983050.1) Section 7 and 8
Path A — Upgrade and New Installation Customers upgrading to Oracle E-Business Suite 12.2.5 Release Update Pack

14.  Source the run edition applications environment.

It is already set in .bash_profile

15. Stop Weblogic AdminServer and Node Manager services.

Applying the patch in downtime mode requires all application tier services to be down. Therefore, Weblogic AdminServer and Node Manager that were started previously when applying the latest Release Update Packs for AD and TXK for Release 12.2 need to be shut down.

sh $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh stop
sh $ADMIN_SCRIPTS_HOME/adnodemgrctl.sh stop

16. Set the “optimizer_adaptive_features” database parameter to FALSE (required for Database version 12.1.0.2). 

sqlplus "/as sysdba"
alter system set optimizer_adaptive_features=false scope=both;

System altered.

show parameter optimizer_adaptive_features

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
optimizer_adaptive_features boolean FALSE

17. Apply Oracle E-Business Suite 12.2.5 Release Update Pack Patch 19676458 on the run edition application environment, using downtime mode

cd $ADMIN_SCRIPTS_HOME
adstpall.sh apps/password
adop phase=apply apply_mode=downtime patches=19676458

Note : Restarting 12.2.5 patch fails with "ERRORMSG: Cannot identify any valid application tier nodes/ORA-01400: Cannot Insert NULL Into ("APPLSYS"."FND_NODES"."NODE_NAME")" (Doc ID 2112122.1)

18. Set the “optimizer_adaptive_features” database parameter to TRUE again

sqlplus "/as sysdba"

alter system set optimizer_adaptive_features=true scope=both;

System altered.

19. Start all Application tier services on the run file system.

sh $ADMIN_SCRIPTS_HOME/adstrtal.sh

20. Perform adop cleanup action.

adop phase=cleanup

21. Synchronize file systems

Synchronize the file systems using the command shown below. This action will copy the new run edition code and configuration to the other file system, to ensure that both file systems are in sync before applying patches using the regular adop cycle on the other file system.

adop phase=fs_clone

--POST UPGRADE STEPS--

22. Apply Oracle E-Business Suite Release 12.2.5 Online Help Patch 19676460 using adop hotpatch mode on the run file system.

adop phase=apply patches=19676460 hotpatch=yes

        

23. To apply NLS ( Turkish ) Language patch follow the instructions in correct order

· Activate the new language using the Oracle Applications Manager (OAM) License Manager.

· Maintain snapshot information

· Run the AD Administration utility to maintain multi-lingual tables from the Maintain Applications Database Entities menu.

· Log on to your Applications' administration node as the APPLMGR user and source the APPL_TOP environment file. If you are on Release 12.2.0 or above, make sure you are connected to the RUN edition. Use the manifest generation tool to generate a manifest file by executing the following command:

perl $AD_TOP/bin/adgennls.pl

The manifest generated from the tool is located at :
$APPL_TOP/admin/$TWO_TASK/out/adgennls.txt 

Request a translation sync path using "https://updates.oracle.com/TransSync"
Upload and apply patch with adop through an online patching cycle (prepare,apply,finalize,cutover,cleanup)

24. Apply Localizations patches and implement localizations if necessary.

At this point, we reached the end. That is our EBS 12.2.5 is ready for use. But, before I put an end to this blog post, I want to share some useful documents with you.
Following is a list of  document that may be useful according to your needs.

For NLS- >Oracle E-Business Suite NLS Release Notes, Release 12.2 (Doc ID 1314621.1)
For HR Legislative: DATAINSTALL AND HRGLOBAL APPLICATION: 12.2 SPECIFICS (Doc ID 1469456.1)
For Localization : Creating CLE Product & Applying Mandatory Patches for R12.2 EMEA Add-on Localizations (Doc ID 1628719.1) --optional
EMEA Add-on Localizations - Upgrade steps to 12.2 (Doc ID 1593750.1) --optional
For Turkish Localizations -> R12 EMEA Add-on Localizations - Turkey (Doc ID 472686.1)