RDBMS/Linux -- Installing Oracle 11gR2 on CentOS 6

Altough we say CentOS is not certified with Oracle at all, sometimes customers insist on using CentOS as for the Operating System of their Oracle Database Server.

For Oracle RDBMS, here is the supported list of Linux Distributions:  Oracle RDBMS ile certified olan OS lerden; SLES, Asianux, Redhat and Oracle Linux .
Note that; CentOS, on the other hand can be considered as Redhat  (from Oracle perspective)

Inspite of this lack of certification, I m here to say that, Oracle Database can be installed and used with CentOS. We tested and saw it .

Here is the installation work that we did for installing an Oracle11gR2 on Centos 6 64 bit Operating System.

1. Download repository
cd /etc/yum.repos.d
wget https://public-yum.oracle.com/public-yum-ol6.repo

2. Install Pre-Install Database rpm
yum install --nogpgcheck oracle-rdbms-server-11gR2-preinstall

3. Configure the files below
/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=ermanserver.ermandomain.com
GATEWAY=89.106.24.88
# oracle-rdbms-server-11gR2-preinstall : Add NOZEROCONF=yes
NOZEROCONF=yes

/etc/hosts
127.0.0.1       localhost localhost.localdomain
89.106.24.139   ermanserver.ermandomain.com ermanserver

/etc/selinux/config
SELINUX=disabled
SELINUXTYPE=targeted

4. Create Oracle Home folders
mkdir -p /erm/app/oracle/product/11.2.0/dbhome_1
chown -R oracle:oinstall /erm/
chmod -R 775 /erm/

5. Download Oracle Software
http://download.oracle.com/otn/linux/oracle11g/R2/linux.x64_11gR2_database_1of2.zip
http://download.oracle.com/otn/linux/oracle11g/R2/linux.x64_11gR2_database_2of2.zip

6. Unzip Software
unzip linux.x64_11gR2_database_1of2.zip
unzip linux.x64_11gR2_database_2of2.zip
7. Set environments in the .bash_profile

[oracle@ermanserver ~]$ cat .bash_profile
PATH=$PATH:$HOME/bin
export PATH
export ORACLE_BASE=/erm/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1
export ORACLE_SID=ERMAN
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib;
export CLASSPATH=$ORACLE_HOME/jlib

8. Install Software
cd database
./runInstaller

9.    Follow the screens

10.    Run netca to setup the listener

     11.       Listener is created so It should be started automatically. (you can start it with “ lsnrctl start LISTENER_NAME”)

     

     12.       Create database by following the screens shown below; (this is a standard thing..)

      Actually all the things are standard , we are installing Oracle Database on CentOS, like we do it on an Oracle Linux..

      Well, that is it, no ORA-600 or 7445. Our database is ready to be used.

      At the end of this post, I want to mention one thing; that is giving this kind of standard installation procedure is actually not my style, but in this post, I did this to show that Oracle Database (at least 11G) can be installed to CentOS Operating System without problems.

      Hope this will help.

Linux/Scripting - drawing dialog boxes in shell/tui/terminal

This post is a cookie . I m writing this in my limited free time : ) but it is interesting , as I will explain how to draw dialog boxes in standard linux shells. (note that I m not talking about X windows)

Let's start;

In order to draw a dialog box, we install dialog utility using yum. (my demo enviroment is Oracle Linux)

[root@demoorcl ~]#yum install dialog
...
.....
........
Installed:
  dialog.x86_64 0:1.1-9.20080819.1.el6                            

Then we look at the definition: via "man dialog" command.

manual says " dialog - display dialog boxes from shell scripts" :)

Next, we write a shell script to use the features of this dialog utility. 

A shell script like the one below;

#!/bin/sh

DIALOG=${DIALOG=dialog}

tempfile=`tempfile 2>/dev/null` || tempfile=/tmp/test$$

$DIALOG  --ok-label "Do the selected!" --cancel-label Exit --clear --title "Admin Interface" \

        --menu "Choose the action to be taken:" 20 70 15 \

        "1"  "Check the integration" \

        "2"  "Shut down the integration Services" \

        "3"  "Startup the integration Services" \

        "4"  "Check the other services" \

        "5"  "Shut down the other services" \

        "6"  "Check the processes"  2> $tempfile

retval=$?

Next, we execute the script, and here our dialog is displayed... ->

Well, that is it.. We display a standard dialog. 

In this simple dialog, when we hit exit button, the dialog is closed, but naturally;  there is no event handler for "the selections"/"Do the selected" buttons, as our script did not include those events.

But if we want to go further and put all the event handlers, we can enhance our script and make it production ready .

An example script in this context is like the following;

Note that: This script is used in a production system, so I changed some paths and commented out some action lines. However,  when you analyze it you will get idea and use the info for your own benefit.

#!/bin/sh

script=/root

DIALOG=${DIALOG=dialog}

tempfile=`tempfile 2>/dev/null` || tempfile=/tmp/test$$

$DIALOG  --ok-label "Do the selected!" --cancel-label Exit --clear --title "Admin Interface" \

        --menu "Choose the action to be taken:" 20 70 15 \

        "1"  "Check the integration" \

        "2"  "Shut down the integration Services" \

        "3"  "Startup the integration Services" \

        "4"  "Check the other services" \

        "5"  "Shut down the other services" \

        "6"  "Check the processes"  2> $tempfile

retval=$?

choice=`cat $tempfile`

case $retval in

  0)

    if [ $choice == "1" ]

    then

    ps -aux | grep eventprocessor | grep -v grep | grep -v Warning > /tmp/kontrol1

    cat /tmp/kontrol1 | grep eventprocessor

    if [ $? = "0" ]

     then

     dialog --exit-label "Back to Main Menu" --title "  ENTEGRASYON KONTROL  " --textbox  $script/arayuz_durum1.txt 22 70

     sh $script/main

    else

     dialog --exit-label "Back to Main Menu" --title "  ENTEGRASYON KONTROL  " --textbox  $script/arayuz_durum2.txt 22 70

    sh $script/main

     fi

    elif [ $choice == "4" ]

    then

    ps -aux | grep tomcat | grep -v grep | grep -v Warning > /tmp/kontrol2

    cat /tmp/kontrol2 | grep tomcat

     if [ $? = "0" ]

     then

     dialog --exit-label "Back to Main Menu" --title "  ARAYUZ KONTROL  " --textbox  $script/tomcat_durum1.txt 22 70

     sh $script/main

    else

     dialog --exit-label "Back to Main Menu" --title "  ARAYUZ KONTROL  " --textbox  $script/tomcat_durum2.txt 22 70

    sh $script/main

     fi

    elif [ $choice == "2" ]

    then

    ps -aux | grep eventprocessor | grep -v grep | grep -v Warning| awk '{print $2}'  > /tmp/arayuz_pid

     arayuz_pid=`cat /tmp/arayuz_pid|grep -v Warning`

     sudo kill $arayuz_pid

    dialog --exit-label "Back to Main Menu" --title "  ENTEGRASYON KAPATILDI  " --textbox  $script/arayuz_kapatildi.txt 22 70

     sh $script/main

     elif [ $choice == "3" ]

    then

    sudo nohup sh /erm/eventprocessor/bin/run &

     dialog --exit-label "Back to Main Menu" --title "  ENTEGRASYON ACILDI  " --textbox  $script/arayuz_acildi.txt 22 70

     sh $script/main

     elif [ $choice == "5" ]

    then

   #sudo sh /erm/tomcat/apache-tomcat-5.5.17/bin/shutdown.sh &>/dev/null

 sh /erm/tomcat/apache-tomcat-5.5.17/bin/shutdown.sh &>/dev/null

dialog --exit-label "Back to Main Menu" --title "  ARAYUZ  KAPATILDI  " --textbox  $script/tomcat_kapatildi.txt 22 70

    sh $script/main

   elif [ $choice == "6" ]

    then

sudo nohup sh /erm/tomcat/apache-tomcat-5.5.17/bin/startup.sh

#sudo sh  /erm/tomcat/apache-tomcat-5.5.17/bin/startup.sh &>/dev/null

   # sudo sh /home/helpdesk/arayuz_script/tomcat_start.sh

dialog --exit-label "Back to Main Menu" --title "  ARAYUZ  ACILDI  " --textbox  $script/tomcat_acildi.txt 22 70

sh $script/main

     else

     echo ...

    fi

   ;;

   1)

    echo "Program Exited"

;;

  255)

    echo "ESC pressed."

;;

esac

Okay, here we go..

Let's execute this script (named Main)

sh Main

We press enter(equivalent to "Do the selected") while our selection line is on "Check the integration";

Well, the action is taken in the backround to check the integration control, and the status of the integration is reported in the dialog :) .. Actually, I modified the script to not to show what is done there, as it is a production code :)

Okay. Lets press enter while our selection line is on "Back to Main Menu"

Here we are in Main Menu again .. 

So that is it. You can go further and create your own dialogs, you can  even add control mechanisms and enhance your dialogs accordingly. By the help of this admin dialogs, you can ease the job of Linux and Oracle admins and even offload some works to the Operators.

That's enough for today.

See you in next blog posts :)

RDBMS -- how to rollforward a database using incremental backups after a failed duplicate/recovery session

In this post, I will give a quick tip about the incremental backup based recovery.
I will explain the concept by giving an example "rman duplicate" scenario.
Actually, this type of incremental backup based recovery is automatically done  by the rman duplicate command.  However, I will show you a failed automatic recovery that is supposed to be done by a rman duplicate and then, I will give you the command to roll forward the database using the incremental backups.

Suppose you started your rman duplicate work;

Rman output:

LEVEL0_DB

channel ORA_AUX_DISK_5: restore complete, elapsed time: 02:10:28

channel ORA_AUX_DISK_4: restored backup piece 12

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42248_12.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_4: reading from backup piece /BACKUP1/LEVEL_0/PROD_LEVEL_0_

42248_13.bck

channel ORA_AUX_DISK_7: restored backup piece 13

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42245_13.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_7: restore complete, elapsed time: 02:12:08

channel ORA_AUX_DISK_8: restored backup piece 14

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42247_14.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_8: reading from backup piece /BACKUP1/LEVEL_0/PROD_LEVEL_0_

42247_15.bck

channel ORA_AUX_DISK_4: restored backup piece 13

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42248_13.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_4: restore complete, elapsed time: 02:13:06

channel ORA_AUX_DISK_8: restored backup piece 15

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42247_15.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_8: reading from backup piece /BACKUP1/LEVEL_0/PROD_LEVEL_0_

42247_16.bck

channel ORA_AUX_DISK_8: restored backup piece 16

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42247_16.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_8: reading from backup piece /BACKUP1/LEVEL_0/PROD_LEVEL_0_

42247_17.bck

channel ORA_AUX_DISK_8: restored backup piece 17

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42247_17.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_8: reading from backup piece /BACKUP1/LEVEL_0/PROD_LEVEL_0_

42247_18.bck

channel ORA_AUX_DISK_8: restored backup piece 18

piece handle=/BACKUP1/LEVEL_0/PROD_LEVEL_0_42247_18.bck tag=LEVEL010JUL2016_2200

LEVEL0_DB

channel ORA_AUX_DISK_8: restore complete, elapsed time: 02:39:26

Finished restore at 15-JUL-16

..................

.........................

..............................

Now suppose: after restoring the level0 backups, "rman duplicate" started to restore the incremental backups and it encountered an error , so that your duplicate db could be successfully created.

Rman output:

channel ORA_AUX_DISK_8: reading from backup piece /BACKUP1/LEVEL_1/PROD_LEVEL_1_

42328_1.bck

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of Duplicate Db command at 07/15/2016 01:14:54

RMAN-03015: error occurred in stored script Memory Script

ORA-19870: error reading backup piece /BACKUP1/LEVEL_1/PROD_LEVEL_1_42327_1.bck

ORA-19505: failed to identify file "/BACKUP1/LEVEL_1/PROD_LEVEL_1_42327_1.bck"

ORA-27041: unable to open file

Linux-x86_64 Error: 13: Permission denied

Additional information: 2

The error in this example is so obvious. That is, it is a Linux permission denied error and what rman clearly says with this error is "I can't read the backup files(incremental in this case) , because I don't have the necessary filesystem permission"


At this point, if we summarize the situation we can say that:
we restored our database (level 0 backup), but it is not up-to-date.
So, we have 2 choices to roll forward the database to current time
1) archivelog apply
2) incremental backup + archivelog

In fact, if we have incremental backups for the level0 we have restored, it is more reasonable to recover our database using those incremental backup.

This is because; if the level0 was taken a long time ago and if, we choose applying all the archivelogs generated since the level0 was taken, we ll probably have to apply lots of archivelog..

So, in this case, the things that we need do for makin rman use the incremental backups for recovering our newly restored database, are as follows;

1) We fix the permission issues for the backup files (chmod or chown)
2) We connect our mountented duplicate database using rman target /
3) We register the incremental backups using catalog start with (catalog start with 'BACKUP1/LEVEL_1')
4) We execute "recover database" command.!! So here is the surprise :))) we just simple execute recover database command. So, in conclusion, Rman already knows that, it is better to recover a database using incremental backups if they are available :)
5)We apply the archivelogs, which could be created since the time that we have taken our level1 backup, to make our database consistent and up-to-date. (Remember all the backups -- "including rman backups", which are taken while the databases are open, are inconsistent)

EBS 12.2 -- interesting question on Business Continuity for Oracle E-Business Suite Release 12.2

One of my collegues asked me the following question ->
"Before the postclone , there should be only the  Ebsapps directory(<APPL_TOP>, <common_top>,Oracle AS Tools 10.1.2 Oracle_Home) present in the host where the postclone script is wanted to be executed. This means $APPL_TOP_NE should be deleted beforehand.
If that so; why do we rsync the concurrent request log and out files from primary to standby application server? I m asking this question, because during a failover a switchover, I will execute a post clone on standby apps server, thus I will have to delete the APPL_TOP_NE before executing it and as the concurrent log and out files are stored in APPL_TOP_NE, why do we rsync them?"

Well, the answer is there in My Oracle Support.
The important point is that, according to the latest documentation, we execute the post clone in the first place when we are creating the standby application tier environment. This postclone will save us from executing future postclones when there will be a switchover or a failover operation required in the future. So as we execute the post clone in the configuration phase of the standby apps tier, we only need to execute the autoconfig in apps tier for switching over ort failing over to the standby site.
This means, our synched concurrent request log and out files will be there in the standby application tier after the switchover or failover operation.

I wanted to share this with you , because the question and its answer was interesting.
So once again, the importance of following the Oracle Support notes was proven.
read -> Business Continuity for Oracle E-Business Suite Release 12.2 Using Oracle 12c (12.1.0.2) Physical Standby Database (Doc ID 1963472.1)

EBS 12.2- Road map and High level action plan for OAM+OID integration

In these days, I have new OAM+OID and EBS 12.2 integration projects.
Altough I have done couple of ciritical SSO+OID and EBS integrations so far, I felt that; the new authenticatator named OAM(Oracle Access Manager) will introduce a new challange for me.

As the first thing to do, I have created a high level action plan which seems like a roadmap and wanted to share it with you. Note that, I also included the HA and DR related reference documents at the bottom of this document to make them to be within arms reach.

Let's take at the look our target, the new EBS login flow after the OAM enablement and the required EBS-OAM installation and integration process from the surface;

Our EBS environment is 12.2.4.
At the moment the latest version of OAM is 11.2.3.0 . OAM 11.2.3.0 is certified with EBS 12.2.5,12.2.4,12.2.3, 12.1.3, 12.1.2, 12.1.1, 12.0.6, 11.5.10.2
So we will install OAM 11.2.3.0.
We will use EBusiness Access Gate to integrate EBS 12.2 with OAM. Actually, we will use Webgate agent in conjunction with EBS Access Gate.
EBS can't go directory to Active Directory/3rd party ldap (not supported), so OID must be used in between EBS-OAM and 3rd party LDAP/Active directory.  We will use OID plugins to speak(ldapbind) with the 3rd party LDAP/Active directory, which is residing in the backend.

When integrated with OAM, EBS login flow works like below;

During login; the request is directed to EBS access gate.
OAM's Web Gate agent intercepts the traffic between the user and EBS Access gate.
Once the traffic is intercepted, OAM's Web Gate agent connects EBS user to EBS access gate to collect the user's credentials. At this point, the user credentials given by the user is submitted to OAM server.
Once the credentials are collected, OAM server decides if the user is required to be authenticated and authenticates the user.
After the authentication, OAM Server creates an SSO session and sends the user identifier to EBS Access Gate.
EBS Access Gate, then connects to the database and links the authenticated OID/LDAP user with and EBS user account. (from fnd_user guid column)
Lastly, user is redirected to original EBS url with an authenticated EBS session.

We will install the following components;

Oracle Access Manager 11.1.2.3
Oracle Identity Management 11.1.1.9.0
Oracle Access Manager WebGate 11.1.2.3.0 (will be installed on top of Oracle HTTP Server 11.1.1.9)
Oracle E-Business Suite AccessGate version: Patch 21522495
Oracle EBS 12.2.4 (already installed)

The detailed installation plan is as follows;
--Note that, this is the first draft, so the actual action/installation plan may be changed during the installation.

*Install an Oracle Database for OID and OAM (Oracle Database 12c Release 1 (12.1.0.1) or higher)-- oracle recommend Enteprise edition for production environments.
----------------------------------------------------------------------------------------------------------------------------
this step doesn't require any references, as it is a standard database installation :)

*Install OID 11.1.1.9 (Oracle Internet Directory 11gR1 Patch Set 7 (11.1.1.9.0) comes as both full installer and an upgrade)
----------------------------------------------------------------------------------------------------------------------------
Ref:Integrating Oracle E-Business Suite Release 12.2 with Oracle Internet Directory 11gR1 (Doc ID 1371932.1)
   --download weblogic 10.3.6
   --download Rcu 11.1.1.9.0 (for creating OAM + OID schemas in the database)
   --download  Oracle Identity Management 11g Patch Set 7 (11.1.1.9.0)
   --install OID + Oracle Directory Integration Platform
            -run rcu (only select 'Oracle Internet Directory')
            -Install Weblogic 10.3.6 and also apply mandatory patch for Weblogic.
            -Install OID +  Oracle Directory Integration Platform (11.1.1.9.0)
            -configure OID + Oracle Directory Integration Platform and Create a weblogic Domain+managed server (Follow Section 7.2.1 OID with ODIP and Fusion Middleware Control
                                                                                                                in a New WebLogic Domain of Oracle® Fusion Middleware Installation Guide for Oracle Identity Management)
            -Apply Required Patch  for Oracle Directory Integration Platform
            -Configure OID with EBS 12.2.
                    -start a new online patching cycle (adop prepare)
                    -set your patch env and run registration script (txkrun.pl)
                    -set required profile options in EBS 12.2(Applications SSO Enable OID Add Event, Link applications user with OID same username, Applications SSO type)
                    -run autoconfig from patch filesystem
                    -set run env
                    -execute adop cutover
                    -configure OID plugin to be able to connect to the Active Directory/3rd Ldap server

Configure Oracle Internet Directory to return operational attributes
Ref:Integrating Oracle E-Business Suite Release 12.2 with Oracle Access Manager 11gR2 (11.1.2) using Oracle E-Business Suite AccessGate (Doc ID 1576425.1)

*Install & Configure Oracle Access Manager 11g Release 2 (11.1.2.3.0)
----------------------------------------------------------------------------------------------------------------------------
Ref: Oracle Fusion Middleware Identity Management 11g Release 2 (11.1.2.3.0) Documentation Library -> Oracle Fusion Middleware Installation Guide for Oracle Identity and Access Management
--check the certification matrix also, for example for IE browswers; only IE 11+ is supported. Also for OEL 6, JDK 1.7.0_80 is required.
   -Install certified JDK
   -Review Database requirement (patches may be needed, some other actions may be needed , review: Oracle Fusion Middleware System Requirements and Specifications for Oracle Identity and Access Management
   11g Release 2 (11.1.2)
   -rcu is already downloaded in OID part. (rcu 11.1.1.9.0 will be used even if the Oracle Access Manager 11.1.2.3.0, because the Oracle instruction is "download and use the specific RCU version available as part of the Oracle Identity and Access Management 11g Release 2 (11.1.2) Media Pack on the Oracle Software Delivery Cloud." So when we  select the media pack we only see the rcu 11.1.1.9.0 there.
   -run rcu (select 'Identity Management - Oracle Access Manager' + Oracle Mobile Security Manager) Oracle Mobile Security is a must,
   because we must also manually select the Identity Management - Oracle Mobile Security Manager schema because when we install and
   configure Oracle Access Management in a WebLogic domain, the Oracle Mobile Security Manager server is installed and configured in the domain by default.
   -weblogic server is already downloaded in OID part.
   -Install Weblogic 10.3.6 and also apply mandatory patch for Weblogic.
   -Install the Oracle Identity and Access Management 11g software. (basically run "runInstaller")
   -Configure the Oracle Identity and Access Management 11g software
          -Configure Oracle Access Management - IAM_HOME/common/bin/config.sh
          -Configure the Database Security Store
          -Start Weblogic Admin Server and managed servers
          -Run the Environment Health Check Utility to Verify our Installation and Configuration (cd IAM_HOME/healthcheck/bin; idmhc.sh -manifest IAM_HOME/healthcheck/config/PostInstallChecks.xml )
   -Apply BP3 to OAM (Oracle strongly recommends applying Oracle Access Manager 11.1.2.3 Bundle Patch 3 (OAM 11.1.2.3.3) as this includes a fix for Patch 19438948. )
*Apply EBS prereqs for OAM
   -Apply the Latest AD and TXK Delta Release Update Packs (Document 1617461.1, Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2, and follow the instructions to apply the required code level of AD and TXK for your system.)
   -Apply patches for EBS 12.2 - OAM 11.1.2.3 interoperability:
        12.2 R12.TXK.C Patch 21523147
        12.2 R12.TXK.C Patch 20735848
        12.2 R12.TXK.C Patch 21229697
*Download and install Oracle Access Manager WebGates
        -Download Oracle Access Manager OHS 11g WebGates 11.1.2.3.0 from Identity & Access Management 11gR2 Downloads.
        -Upgrade EBS's HTTP Server to 11.1.1.9 if it is not already.
        -install Access Manager Web Gates to EBS's HTTP Server.(txkrun.pl -script=SetOAMReg -installWebgate=yes -webgatestagedir=<webgate stage directory>)
        -Apply bundle patch for Oracle Access Manager Webgates - BP01.
*Integrate Oracle E-Business Suite with Oracle Access Manager
       -Deploy Oracle E-Business Suite AccessGate (do it with perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-create-oaea_resources -contextfile=$CONTEXT_FILE -deployApps=accessgate )
       -Register Oracle E-Business Suite with Oracle Access Manager
       -txkrun.pl -script=SetOAMReg -registeroam=yes
*Test Single Sign-On with Oracle E-Business Suite (http://<ebshost>.<domain>:<port>/OA_HTML/AppsLogin)
*Perform fs_clone      
*Configure TLS (SSL) - optional ; Configure SSL/TLS between Webgate and Access Manager + EBS itself.
      -For Webgate and Access Manager:
         Ref:Oracle Fusion Middleware Administrator's Guide for Oracle Access Management: Securing Communication Between OAM Servers and WebGates  
         Securing Communication provides instructions on how to secure communications between Oracle Access Manager 11g and WebGates.
      -For EBS 12.2:
         Ref:Enabling TLS in Oracle E-Business Suite Release 12.2 (Doc ID 1367293.1) To BottomTo Bottom
       --EBS's SSL/TLS version should be the same as OAM's SSL/TLS version.

Additional References if DR and H/A need to be implemented: 

Reference Figure 8-1 shows a sample Oracle Fusion Middleware 11g Oracle Identity Management high availability architecture from the Fusion Middleware High Availability Guide at the URL:
https://docs.oracle.com/cd/E28280_01/core.1111/e10106/imha.htm#ASHIA804

8.2 Prerequisites for Oracle Identity Management High Availability Configuration
https://docs.oracle.com/cd/E28280_01/core.1111/e10106/imha.htm#ASHIA3047

8.3 Oracle Internet Directory High Availability
https://docs.oracle.com/cd/E28280_01/core.1111/e10106/imha.htm#ASHIA806

8.3.3.3 Configuring Oracle Internet Directory With a WebLogic Domain

8.3.3.3.1 Configuring Oracle Internet Directory on OIDHOST1

This section describes the steps to deploy Oracle Internet Directory in a high availability configuration as part of a WebLogic Server domain.

8.3.3.3.3 Configuring Oracle Internet Directory on OIDHOST2
Ensure that the Oracle Internet Directory repository is running and then follow these steps to configure the Oracle Internet Directory instance on OIDHOST2:
On the second node, you will be extending the domain using config.sh tool.

The following document is for Active-Passive solution:

12 Active-Passive Topologies for Oracle Fusion Middleware High Availability
Fusion Middleware High Availability Guide
http://docs.oracle.com/cd/E28280_01/core.1111/e10106/ap.htm#ASHIA3205

The following document is for DR solution:

Fusion Middleware Disaster Recovery Guide
1.2.1 Oracle Fusion Middleware Disaster Recovery Architecture Overview
http://docs.oracle.com/cd/E28280_01/doc.1111/e15250/intro.htm#ASDRG106
The Oracle Fusion Middleware Disaster Recovery solution supports these methods of providing data protection for Oracle Fusion Middleware data and database content:
- Oracle Fusion Middleware product binary files, configuration files, and metadata files
Use storage replication technologies.
- Database content
Use Oracle Data Guard for Oracle databases (and vendor-recommended solutions for thirdparty databases)

Also  note the following in "http://docs.oracle.com/cd/E28280_01/doc.1111/e15250/creating_sites.htm#ASDRG512" ->
4.5.6.1 Using rsync and Oracle Data Guard for Oracle Fusion Middleware Disaster Recovery Topologies

RDBMS -- restore/recover a dropped table column, ORA-01466, 12C rman recover table option

One day , you may accidentally drop a column from one of your critical column in your Production database. I hope you won't be in this situation, but here is the facts that you need to know foır restoring/recovering a dropped column in Oracle.

• Flashback techniques doesn't apply for dropped columns. That is, you can not use flasback query or flashback table to restore a dropped column. You will get "ORA-01466 : Unable to read data -table definitions has changed"  As for the Flashback Table, it cannot rewind tables through structural changes even after a truncate table operation.
• imp(import) or impdp can be used for restoring the dropped column, as we can import the whole table from a backup dump file and then import it to another schema and lastly get the data from that table to our target table. On the other hand, you  eed to have a backup procedure, that regularly takes logical backups of the database using exp or expdp. (you can even roll forward the imported column data by analyzing the archivelogs and extracting dml operations from there using log miner or something)
• Flashback database can be used, but with this option, you will flashback your whole database. So it is not an applicable method in most of the cases.
• If the datafiles are on storage, storage snapshots can be used to create a new luns from the snapshot of the lun that contains the production's database files.  After mounting the lun created from the snapshot, the database can be opened with a different name from that lun and the table(before alter drop column operation) can be gathered from there.
• Rman full restore can be used. That is , the database backup can be restored to a different server or different location and recovered just before the table drop operation. The column data can be done gathered from that restored database using db links. However, this is a heavy and time consuming process.
• Performing RMAN Tablespace Point-in-Time Recovery (TSPITR) is also an option, but this is also heavy and it makes all the objects of that tablespace rolled back/forward to a specified point in time.
• Opening standby read/write by creating a guaranteed restore point (for taking it back) can be used, if the dropped column is not dropped from standby database yet.
• Lastly, if you are on 12C, then you have the option to restore a table using rman from rman backups. This is a new feature introduced in 12C . as in 12c RMAN has an option for 'recover table'. See: RMAN RECOVER TABLE Feature New to Oracle Database 12c Note 1521524.1 (My favorite one :)

                Rman recover table feature works with the following;

• It finds the rman backup that includes the table to be recovered (based on the point in time recovery specified for the recovery)
• It create an auxiallary database and it recovers the table until the specified point in time, into this auxiliary database.
• It then creates a export dump file that contains the recovered table.
• Lastly, it imports this export dump file into the target instance. (optional) --By default, RMAN imports the recovered tables or table partitions, which are stored in the export dump file, into the target database. However, you can choose not to import the recovered tables or table partitions by using the NOTABLEIMPORT clause of the RESTORE command.

                Here are the limitations:

• Tables and table partitions belonging to SYS schema cannot be recovered.
• Tables and table partitions from SYSTEM and SYSAUX tablespaces cannot be recovered.
• Single table partitions can be recovered only if your Oracle Database version is Oracle Database 11g Release 1 or later.
• Tables and table partitions on standby databases cannot be recovered.
• Tables with named NOT NULL constraints cannot be recovered with the REMAP option. 

Before closing this topic, I want to shed a light on a few things and share my comments on them.

Why rman creates an auxilary database and then take an export from there, must be to isolate the mission critical target environments from the restore works. Also It seems like "rman recover table" option is just like an expansion pack for the rman, because still rman restores the backup datafiles in order to restore a table from a backup.

EBS - Why context file is updated from the database? Why any recent changes in the context file are taken back during autoconfig run?

The questions in the title are obvious, so the answers...
There is serial number stored in each context file. It is stored with the tag named oa_context_serial.
Example:

<oa_context_serial oa_var="s_contextserial">55</oa_context_serial>

This serial number is also stored in the FND_OAM_CONTEXT_FILES table. In that table, we have all the context files stored and ordered according to their serial numbers.

So if we have a context file with serial number of 55, and if we have a context file stored in the FND_OAM_CONTEXT_FILES table with a serial number 56 , then our changes done in our apps filesystem context file can be overwritten after an autoconfig run. This is because during autoconfig run, autoconfig will look at the FND_OAM_CONTEXT_FILES table and see there is a high versioned context file stored there. So what autoconfig does in such a situation is, it replaces the values stored in our apps filesystem context file with the values stored in the high versioned context file that is stored in the database.

So, this is an expected behaviour , but it sometimes creates problems.

Here in my forum, there was a discussion on this. -> http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/adcfgclone-pl-errors-td265.html#a483

Some of the values stored in the context file , which reside in the FND_OAM_CONTEXT_FILES were causing problems in a post clone run.
What I suggested was, truncating the FND_OAM_CONTEXT_FILES(actually, oa_context_serial value of the context file stored in the application tier filesystem can also be updated to a higher version value , as well) and that suggested action actually solved the problem.

"How a context file can be in a lower version than the context file stored in the database?" -> is the question that we should ask. My answer is , it sometimes can be lower, really :)

It can be lower in cases where we clone only the database tier of an EBS. In this scenario, database will have more recent context file, but the application tier filesystem will have the old versioned one.
Also look at the following note, even the standard things can make the version of Apps tier context file to be lower than the context file stored in the database -> adop phase=abort causes "Serial number in context file contains lower value than that of database copy." (Doc ID 1916658.1)

So at the end of the day; these things are happening... What we need to do is to take the lesson from them.
At least, with this in mind; when we see our context file values are overwritten after an autoconfig run, we know where to look at...

Linux -- auditing in Linux, auditd, setfacl - Linux ACLs

Recently, we were required to audit the file and directory accesses in Linux.
We had to audit the accesses the directory accesses and record them into files that can't be accessed except the root user and an OS account which is currently used by an auditing application .

What we have done is, we enabled auditd, which is the userspace component to the Linux Auditing System.

Manual of auditd:
http://man7.org/linux/man-pages/man8/auditd.8.html

We have configured the auditd to audit all the access types for some given directories and rotate the audit log files that it creates in a timely manner and keep them for 90 days. (we have done this using auditd configuration file called /etc/audit/auditd.conf  and the audit rules file called /etc/audit/audit.rules )

One of the requirements was as already mentioned; the audit log files which auditd was populating with the audit information, must not  be accessed except the root user and an OS account which was used by an auditing application.

auditd was creating the files in the /var/log/audit directory with the directory permissions 600. So root could read them but the  OS account which was used by the auditing application could not.

So in order to give a read permission for this directory and files, we used Linux ACLS.

We have used setfacl utility to give read permission to that specific user for the audit log directory and the files stored in it. (another unfavourable method could be modifying /etc/audit/auditd.conf and changing the log_group from root to the group of Os account that is used by the auditing application) One thing which was important was, in Linux to change into a directory or list its content or read a file in it,  the user needs to have execute permissions on that directory. 

So we had to give the execute permission in addition to the read permission to the that user for that directory . (command was something like this: setfacl -Rm u:B:rx /var/log/audit.

Note that, Linux ACLs can be used for defining the directory and file permissions automatically, when they are created. That is using ACLs we can make the files to be in any permission we want and it will be done on-the-fly.  (you can set it and then test it using a simple command like "touch examplefile") . We actually use setfacl's "-d" option for this.

-d, --defaultAll operations apply to the Default ACL. Regular ACL entries in the input set are promoted to Default ACL entries. Default ACL entries in the input set are discarded. (A warning is issued if that happens).

Manual of setfacl : http://linux.about.com/library/cmd/blcmdl1_setfacl.htm

ODA -- configuring memory & cpu for ODA Virtualized Environments

I want to shed a light on Cpu and memory configuration of ODA virtualized environments.
As the ODA machine is actually used mostly for providing a small considolation environment for both applications and databases, it is good to configure it in the appropriate way .

Well, as documented in "Solution-in-a-box: Best practices for deploying Oracle E-Business Suite Release 12.2.4 on Oracle Database Appliance 12.1.2.2" ( I think, this can be considered as appropriate even for the applications other than EBS),  it is good to give %50 of the ODa hardware resources to ODA_BASE and leave the rest to dom0 and other vm machines.

That is , it is a good start to configure the CPU and MEmory of ODA_BASE nodes to be %50 percent of the total resources.
oakcli configure oda_base command will help you on that. (run it from dom0 and run it once for every node.. That is, run it from node0 and then node1)
configure oda_base command will let you specify the cpu and memory resources for ODA_BASE nodes.

At this point, it is important to mention that, the cpu cores that we see in ODA_BASE nodes using cat /proc/cpuinfo command are actually the virtual cores. 1 physical core equals 2 virtual core, and in this sense, if we give 8 cores to an ODA_BASE node using oakcli configure, then we see 16 cores when we check it using the OS tools from the ODA_BASE.

So, after configuring the ODA_BASE, the remaining cores and memory, can be scattered among the necessary vm servers.
For the vm servers, what needs to be done is to use oackli configure vm command.

Here is an example;

oakcli configure vm ermanvm -vcpu 8 -maxcpu 8 -> these are virtual cpus. ( run it from oda_base)oakcli configure vm ermanvm -memory 8g -maxmemory 8g

As suggest in my earlier post, we recommend using the ODA_BASE domains for placing the databases. (even the TEST database environments should be placed there, as the performance of the VM domain will not be sufficient for handling a serious database IO activity) )

An example configuration can be giving 12 cores to ODA_BASE nodes and giving 48 GB memory for the ODA_BASE nodes.

It is up to you to decide how to distribute the remaining resources to the available vms in your ODA virtualized environment, but here is an example configuration that can be used in an EBS 11i environment , which is planned to be upgraded to 12.2

Each App VM machine can have 4 physical cores and 8 GB rams. (if PAE is enabled, 16GB ram is preferred)

Each new apps 12.2 servers, which are planned to be used in the upgrade process can have 4 physical cpus and 20GB rams each.

The distribution should be evenly, as it is not good to increase the load of one ODA node , while the other node is sleeping.

Also, as the standard EBS deployments are done in way that, the first node is a little more utilized than the second node (especially in database level); it is better to give a little more mercy to the first node  while evenly distributing the resources to the node.

An example distribution, which can be used for hosting an EBS farm that consist of 3 11i application vm machines, and 2 12.2 application vm machine is given as follows:

1st node:

1 11i app (prod app) --a high utilized apps env.

1 11i app (test app)  -- a semi utilized apps env.

-- Databases are on ODA_BASE...-- Databases are on ODA_BASE... (semi or high utilized db instances)

2nd node:

1 11i app (crp1 app) -- low utilized apps env.

1 12.2 app (crp1 app) -- low utilized apps env.

1 11i app(crp2 app) -- low utilized apps env.

1 12.2 app (crp2 app) -- low utilized apps env.

-- Databases are on ODA_BASE... (low utilized db instances)

EXADATA/RAC -- CRS-2683: It is locked by 'SYSTEM' & Do not have sufficient voting files, found 1 of 2 configured files, needed at least 2

It was just a week ago, that I have encountered this strange problem in one of my Customer site.
The system was an Exadata X3-2 , which was hosting a production EBS database on top of it.
A flash card had to be replaced , so we have shut down the machine properly for the replacement.
However, after the flash card replacement, databases could not be started again.
Problem: RAC services can not be started after rebooting Exadata compute and cell nodes.

Analysis: When the issue was escalated to me , I directly jump into the OHASD log file, as OHASD was the process that should have been the process starting the Oracle Clusterware processes.

Picture Ref : http://docs.oracle.com/cd/E14795_01/doc/rac.112/e10717/intro.htm

The flashing error was there;

2016-05-12 13:29:15.815: [ CRSPE][1145149760] {0:0:178} ICE has queued an operation. Details: Operation [START of [ora.crsd 1 1] on [osrvdb01] : local=0, unplanned=00x2aaab427de30] cannot run cause it needs W lock for: WO for Placement Path RI:[ora.ctssd 1 1] server [osrvdb01] target states [ONLINE ], locked by op [START of [ora.asm 1 1] on [osrvdb01] : local=0, unplanned=00x2aaab00bee30].
Owner: CRS-2683: It is locked by 'SYSTEM' for command 'Resource Autostart : osrvdb01' 

It was obvious that there was a cluster level locking problem...
Kept checking on the other logs and saw that  the CSSD daemon OCCSD was terminating abornormally.

Note that: OCR of this RAC was  in DBFS_DG diskgroup in ASM...

2016-05-12 14:54:02.453: [ CSSD][1141094720]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2016-05-12 14:54:02.453: [ CSSD][1141094720]clssnmCompleteVFDiscovery: Completing voting file discovery
2016-05-12 14:54:02.453: [ CSSD][1141094720]clssnmvDiskStateChange: state from discovered to pending disk o/192.168.10.3/DBFS_DG_CD_03_osrvcel01
2016-05-12 14:54:02.453: [ CSSD][1141094720]clssnmvDiskStateChange: state from pending to configured disk o/192.168.10.3/DBFS_DG_CD_03_osrvcel01
2016-05-12 14:54:02.453: [ CSSD][1141094720]clssnmvDiskStateChange: state from discovered to deconfigured disk o/192.168.10.4/DBFS_DG_CD_02_osrvcel02
2016-05-12 14:54:02.927: [ CLSF][478835504]Allocated CLSF context
2016-05-12 14:54:02.929: [ SKGFD][478835504]Handle 0x2aaaac668b60 from lib :OSS:: for disk :o/192.168.10.3/DBFS_DG_CD_03_osrvcel01:
2016-05-12 14:54:02.929: [ CLSF][478835504]Opened hdl:0x2aaaac5af9d0 for dev:o/192.168.10.3/DBFS_DG_CD_03_osrvcel01:
2016-05-12 14:54:02.929: [ CSSD][478835504]ASSERT clssnml.c 4532016-05-12 14:54:02.929: [ CSSD][478835504]clssnmlgetleasehdls: Do not have sufficient voting files, found 1 of 2 configured files, needed at least 2
2016-05-12 14:54:02.929: [ CSSD][478835504]###################################
2016-05-12 14:54:02.929: [ CSSD][478835504]clssscExit: CSSD aborting from thread Main
2016-05-12 14:54:02.929: [ CSSD][478835504]###################################2016-05-12 14:54:02.929: [ CSSD][478835504](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally

From this perspective, there was a Voting Disk problem.
Note that: Voting disks manage information about node membership. Each voting disk must be accessible by all nodes in the cluster for nodes to be members of the cluster.

As seen in the logs above, OCSSD found 2 disks, but then made one of them deconfigured .
Then it was saying "lssnmlgetleasehdls: Do not have sufficient voting files, found 1 of 2 configured files, needed at least 2"
There could be a problem with OCSSD, as  it did not like the voting disk DBFS_DG_CD_02_osrvcel02.(which was physically in a normal situation, no errors were on that disk physically, as we checked it using cellcli , celldisk, physicaldisk and griddisk, all were fine for that disk.)

So OCSSD discovered that disk but then deconfigured it and then it could not continue with only 1 voting disk and aborts.

Note that, In 11gR2, ocssd.bin discover voting disk with setting from GPnP profile, if not enough voting disks can be identified, ocssd.bin will abort itself.

ASM, on the other hand, was started and terminated due to ORA-600 errors, reported in its alert log as follows;

Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_24013.trc (incident=433951):
ORA-00600: internal error code, arguments: [kfdvfGetCurrent_baddsk], [], [], [], [], [], [], [], [], [], [], []

Explaination of the cause of this bug ORA-600:

1) CSS is not able to recognize the online of the disks and re-establish that the majority is met for the voting file count. Solution : Reboot  (however, I rebooted the nodes and as suggested I rebooted them one by one, but it didnt solve the problem)

2)Bug 12330287 , upgrade to 12.1

CSS really could not recognize the disks. It was reported in its logs as I mentioned earlier, so we are in correct way.
At this point, if we summarize what we have;

-> crsdb can not be started
-> cssdb can not see all the voting disks
-> ASM was crashing because of cssdb , as it can not see the voting disks properly.

Decision about the possible causes:

• A race condition which might be caused by an improper shutdown of the machine nodes.
• Bug 14619103  "Operation [START of xxx] ... has been replaced with [START of yyy]" chooses incorrect resource.

Decision: 

There is repository level locking issue between ASM and CRS, which make CRS not be able to be started and which in turn make abnormal chaing of event happen (such as CSSD can not be started because it could not see the voting disks properly and ASM crashed because of that. )

Solution:

The problem resolved by starting crs in exclusive mode. By starting crs in exclusive mode, I thought that the locking issue may be resolved. I though , in exclusive mode, crs might be started without problem and then start might solve the locking problem(actually kind of a deadlock) between CRS and ASM.

What I have done was;
I executed the following command to start crs in exclusive mode in node1.
$GRID_HOME/bin/crsctl start crs -excl -nocrs

Note that:

The -nocrs option ensures that the crsd process and OCR do not start with the rest of the Oracle Clusterware stack.
-excl option ensures that the Oracle Clusterware stack start in exclusive mode on one node.

After executing the command above in node 1, I immediately saw the ASM2 in node2 was started. It seemed the lock was released.
So, after that, I rebooted node1 and after the reboot of node 1, everyting was started to be working.

Both of the nodes was seeing 3 voting disks and ASM instance and databases was started without any problems.

Outputs:
[oracle@osrvdb02 bin]$ ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE ad944505cc0c4f4ebf29eb746962ab06 (o/192.168.10.3/DBFS_DG_CD_03_osrvcel01) [DBFS_DG]
2. ONLINE 1a8ea7860b0a4f86bf7cbd7423710dfd (o/192.168.10.5/DBFS_DG_CD_02_osrvcel03) [DBFS_DG]
3. ONLINE 2a57155433e64f78bfb3424215e8dc16 (o/192.168.10.4/DBFS_DG_CD_02_osrvcel02) [DBFS_DG]
Located 3 voting disk(s).


[oracle@osrvdb01 bin]$ ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE ad944505cc0c4f4ebf29eb746962ab06 (o/192.168.10.3/DBFS_DG_CD_03_osrvcel01) [DBFS_DG]
2. ONLINE 1a8ea7860b0a4f86bf7cbd7423710dfd (o/192.168.10.5/DBFS_DG_CD_02_osrvcel03) [DBFS_DG]
3. ONLINE 2a57155433e64f78bfb3424215e8dc16 (o/192.168.10.4/DBFS_DG_CD_02_osrvcel02) [DBFS_DG]
Located 3 voting disk(s).

Before finishing this blog post, I would like to underline 2 things;

This situation can be thought as kind of a one time issue, which can be considered as a generic CRS issue, which was actually caused by the design of Oracle RAC.

The suggested solution was upgrading GRID + DB to 11.2.0.4, as for 11.2.0.3 and versions below that are already desupported and no patches will be provided.