Thursday, October 24, 2013

EBS R12-- and RAC database -- RAC to Single Cloning

In this post, I will try to explain how to clone an EBS running on a Rac database with 2 instances to another machine which will have a single instance database and application services.

Source environment:

Two instance Rac database (Exadata, Oracle Database Application , Ibm Servers or Ibm servers does not matter..)
One Application Node (conc+forms+web+reports)

Target environment:

One Single instance database
One Application Node (conc+forms+web+reports)

Here is the diagrammatic explanation of the process.

As this is directly an Apps Dba operation, Ad utilities are used for the cloning process.
Ad utilities generates the rman scripts for database cloning, and execute them automatically.
Of course, sometimes problems can be encountered.. So that's why, even if it s an Apps Dba operation, Core Dba knowledge is needed, too..

Things to consider;

  • All the rpm and OS packages should be installed to the servers belong to target environment.
  • Cron entries for backups should be commented out before backing up the source database using ad utilities. ( especially backups with delete input syntax..)

  • Os users should be created, and granted properly on the target environment. db owner (for ex:oratest) and application owner (for ex:appltest).

         Example (database user):
         uid=500(oracle) gid=500(oracle) groups=500(oracle),501(oinstall),502(dba)
         Example (application user):
         uid=501(oracleapp) gid=501(oinstall) groups=501(oinstall),502(dba)

  • Hostnames of the servers belong to the target environment should be maximum 30 character long..
  • Source Environment should be in Archivelog mode.
  • Source Environment should be checked with Rman crosscheck archive log all to validate the archivelogs..

So here are the steps to clone an EBS R12 environment with a Two node Rac Database to a EBS R12 environment with a Single node Database.

  • Prepare a preclone on the database tier in the source environment.

        cd $ORACLE_HOME/appsutil/scripts/[context_name]  --context_name in format => SID_ hostname
        perl dbTier   

  • Copy Oracle Home from one of the source servers to the target, and relink the binaries with $ORACLE_HOME/relink all command.. This action will create an usable Oracle Home on the target system.
  • Take a rman backup using on the source. This action will create the stage on the stage area.. The stage will contain everything necessary to clone a database with rman. Datafiles, archivelogs and etc.. After this step , copy the stage directory to the target node.. If you use a shared filesystem , you do not need to copy anything..
        perl \
        java=/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/jdk \     --> jdk path
        mode=stage \  
        stage=/u02/stage \      --> stage area to store the necessary backup file
        component=database \
        method=RMAN \
       dbctx=/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/SID_hostname.xml   --> context file of the db 

  • Create a pairs file on target system. 

  • Example path: /oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/clone
    The necessary inputs are below;
    s_db_oh=/oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1 --> target oracle home path.

  • Run on the target environment to create database context file.
           perl \
           contextfile=/oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/TEST_oratest.xml \
           template=/oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/template/adxdbctx.tmp \
           pairsfile=/oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/clone/pairsfile.txt \

    Example : Adclonectl Questions and Answer 

    Copyright (c) 2002 Oracle Corporation
                              Redwood Shores, California, USA

                              Oracle Applications Rapid Clone

                                       Version 12.0.0

                            adclonectx Version 120.23.12010000.1

      /oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/clone/bin/../jre/bin/java -Xmx600M -classpath /oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/clone/bin/../jlib/ojdbc5.jar:/oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/clone/bin/../jlib/xmlparserv2.jar:/oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/clone/bin/../jlib/java  -e /oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/TEST_oratest.xml -tmpl /oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/template/adxdbctx.tmp -pairsfile /oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/clone/pairsfile.txt -initialnode
      Enter the APPS password : 

      Log file located at /oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/clone/bin/CloneContext_0126133644.log

      Provide the values required for creation of the new Database Context file.

      Target System Hostname (virtual or normal) [oratest] : 

      It is recommended that your inputs are validated by the program.
      However you might choose not to validate them under following circumstances:

              -If cloning a context on source system for a remote system.
              -If cloning a context on a machine where the ports are taken and
               you do not want to shutdown the services at this point.
              -If cloning a context but the database it needs to connect is not available.

      Do you want the inputs to be validated (y/n) [n] ? : 

      Target Instance is RAC (y/n) [y] : n

      Target System Database SID : TEST

      Target System Base Directory : /oracle/u02/CLONE/u01/app/oracle

      Oracle OS User [oracle] : 

      Oracle OS Group [oracle] : dba

      Target System utl_file_dir Directory List : /tmp

      Number of DATA_TOP's on the Target System [3] : 1

      Target System DATA_TOP Directory 1 : /oracle/PR/data

      Target System Archive Log Directory [/oracle/PR/data/archive] : 

      Do you want to preserve the Display [0] (y/n)  : y

      Do you want the the target system to have the same port values as the source system (y/n) [y] ? : n

      Target System Port Pool [0-99] : 20
      Report file located at /oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/temp/portpool.lst
      Complete port information available at /oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/temp/portpool.lst

      New context path and file name [/oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/TEST_oraetest.xml] : 

      Creating the new Database Context file from :

      The new database context file has been created :

      Log file located at /oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/clone/bin/CloneContext_0126133644.log
      Check Clone Context logfile /oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/clone/bin/CloneContext_0126133644.log for details.

  • Open the xml context file created in previous step and make the following updates for downgrading the Instance count to 1(Rac to single).

           <cluster_database_instances oa_var="s_dbClusterInst">1</cluster_database_instances>
            <instance_name oa_var="s_instName">TEST</instance_name>
            <instance_number oa_var="s_instNumber">0</instance_number>
            <instance_thread oa_var="s_instThread">0</instance_thread>
               <ALTERNATE_SERVICE_INSTANCES oa_var="s_alt_service_instances"/>

  • Clone the database from source to target using adclone.. adclone will use the stage, generate the rman scripts and execute them to make the clone ready. Note that you have to use unzip version 5.x , so make the changes in your PATH before executing the command , if necessary. 
             perl \
             java=/oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/jdk \
             component=dbTier \ 
             mode=apply \
             stage=/oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/clone \
             method=CUSTOM \
             dbctxtg=/oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/TEST_oratest.xml \
             rmanstage=/stage/data/stage \
             rmantgtloc=/oracle/PR/data \
             srcdbname=TEST \
             pwd=apps \

  • After this step, the database should be recovered and opened.. On the other hand, there can be environment specific problems and adclone may not be able to open the database .. In this case, you need to solve the problems and continue the process manually.
    • For example: If database can not be opened because of a parameter file problem.. Lets say, your database is compatible to version, but in init.ora that adclone is using, the  compatible parameter is set to 11.2.0. In such a scenario, the database wont start.. So you need to edit the init.ora manually and start the database.. This is a constraint of adclone that I know and  have seen..
  • After the database is opened, put it in noarchivelog (optional)
    shutdownm immediate;
    startup mount;
    alter database Noarchivelog;
    alter database open;
  • Control the Global Name of the database
  • SQLPLUS>select * from global_name; 
    If the global_name is wrong, change it with the following;
    alter database rename global_name to TEST;
  • Disable thread 2
         alter database disable thread 2; 
         Note that you can not drop the redolog belong to thread 2 without disabling thread 2.

  • Drop the redolog files belong to thread 2
  • Drop the undo tablespace blong to 2nd node of the Rac.
  • Check the connectivity of the database through the listener.
  • Clean the environment tables and run autoconfig on db tier to populate them again.
  • sqlplus apps/apps_password
    SQL> exec fnd_conc_clone.setup_clean;
    cd $ORACLE_HOME/appsutil/bin
    sh contextfile=/oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/TEST_oratest.xml
After this point Database clone is finished, you can continue with cloning the application tier ...
  • Run the adpreclone on the apps tier of source system.
  • Copy the application filesystem from source to target..I will not give any details in this step as you can use scp to achieve this. You can choose not to copy log and out files of the concurrent managers and etc. It s up to you.
  • Run the adcfgclone (post clone) on the target node with application os user.. Answer the questions below according to your environment and needs..
              perl /oracle/PROD/apps/apps_st/comn/clone/bin/ appsTier

                     Copyright (c) 2002 Oracle Corporation

                        Redwood Shores, California, USA

                        Oracle Applications Rapid Clone

                                 Version 12.0.0

                      adcfgclone Version 120.31.12010000.8

Enter the APPS password : 

Provide the values required for creation of the new APPL_TOP Context file.

Target System Hostname (virtual or normal) [oratest] : 

Target System Database SID : TEST

Target System Database Server Node [oratest] : 

Target System Database Domain Name [] : 

Target System Base Directory : /oracle/PR

Target System Tools ORACLE_HOME Directory [/oracle/PR/apps/tech_st/10.1.2] : 

Target System Web ORACLE_HOME Directory [/oracle/PR/apps/tech_st/10.1.3] : 

Target System APPL_TOP Directory [/oracle/PR/apps/apps_st/appl] : 

Target System COMMON_TOP Directory [/oracle/PR/apps/apps_st/comn] : 

Target System Instance Home Directory [/oracle/PR/inst] : 

Target System Root Service [enabled] : 

Target System Web Entry Point Services [enabled] : 

Target System Web Application Services [enabled] : 

Target System Batch Processing Services [enabled] : 

Target System Other Services [enabled] : 

Do you want to preserve the Display [y] (y/n)  :  

Do you want the the target system to have the same port values as the source system (y/n) [y] ? : n

Target System Port Pool [0-99] : 20

Checking the port pool 20
done: Port Pool 20 is free
Report file located at /oracle/PR/inst/apps/TEST_oratest/admin/out/portpool.lst
Complete port information available at /oracle/PR/inst/apps/TEST_oratest/admin/out/portpool.lst

UTL_FILE_DIR on database tier consists of the following directories.

1. /usr/tmp
2. /tmp
3. /oracle/u02/CLONE/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/outbound/TEST_oratest
4. /usr/tmp
Choose a value which will be set as APPLPTMP value on the target node [1] : 

Creating the new APPL_TOP Context file from :
Completed Apply...

Do you want to startup the Application Services for TEST? (y/n) [y] : n

Do not startup the services immediately. Complete to post actions to guarantee the success..

  • Set the APPS environment using the env file generated by post clone.. It s located in $APPL_TOP with the name format APPS'SID'_'hostname'.env.. Source the environment file in your .bash_profile or .profile files to set it automatically in every session of yours..
  • Use adadmin to do the following,
    • generate JAR files (adadmin ekran navigation: 1>4 ) Do you wish to force regeneration of all jar files? [No] ? No
    • generate message files (adadmin ekran navigation: 1>1) Default answers for all the question , Press "Enter".
    • relink executables (adadmin ekran navigation: 2>1')
    • copy files to destination (adadmin ekran navigation: 2>2 ) Default answers for all the question , Press "Enter".
After these operations; the application is ready to be opened.. (unless you have to change the grants ,update some alerts, modify some packages or mask, delete, truncate, update some of the tables that have site specific sensitive information..)
  • Open the application services..
sh apps/apps_password

  • Log in as sysadmin and check the services through OAM, check the concurrent managers and notification mailer and etc..
If you have any questions, feel free to contact me..


  1. Hi Erman.

    Please kindly provide me with the steps to clone EBS R12 from a single note database to RAC(two node) database

    Thanking you in advance


    1. Altough there is no such supported scenario listed in EBS RAC certificatied cloning scenarios in Oracle Support, it seems it s possible, maybe even with rapid clone.
      I didnt test it yet but you can change this document a little bit and follow.

      I suggest you to read the Oracle Support document ;
      Using Oracle 11g Release 2 Real Application Clusters with Oracle E-Business Suite Release 12 (Doc ID 823587.1)
      It says:
      Oracle E-Business Suite Release 12 has numerous configuration options that can be chosen to suit particular business scenarios, uptime requirements,
      hardware capability, and availability requirements. This document describes how to migrate Oracle E-Business Suite Release 12 running
      on a single database instance to an Oracle Real Application Clusters (Oracle RAC) environment running Oracle Database 11g Release 2 (11gR2).

      On the other hand; as an alternative, you can always clone like it s single to single and then convert to RAC. I would prefer this method..

  2. Hi Sir,

    I am facing some as below, kindly help me in this :
    RMAN> startup nomount pfile=/oradata/app/oracle/11.2.0/dbs/initSTRESS.ora.tmp
    2> restore controlfile from '/stage/new_clone/backup_controlfile.ctl';
    3> alter database mount;
    4> exit
    Oracle instance started
    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-03002: failure of startup command at 05/21/2015 14:18:54
    RMAN-06403: could not obtain a fully authorized session
    ORA-00604: error occurred at recursive SQL level 1
    ORA-04031: unable to allocate 32 bytes of shared memory ("shared pool","unknown object","KGLH0^f47ce521","kglHeapInitialize:temp")

    1. When I executed below command it occurs :
      perl java=/usr/java component=dbTier mode=apply stage=/oradata/app/oracle/11.2.0/appsutil/clone method=CUSTOM

      dbctxtg=/oradata/app/oracle/11.2.0/appsutil/STRESS_stressdb.xml rmanstage=/stage/new_clone rmantgtloc=/oradata/app/oracle/db srcdbname=STRESS pwd=lelapps786 showProgress

  3. This seems to be a shared pool issue.. Increasing the Shared pool size can be a solution for this.But since it is controlled by , we need to make a necessary change in adclone's input files..
    /oradata/app/oracle/11.2.0/appsutil/STRESS_stressdb.xml can be used for this.

    Open it ..
    Increase the shared pool size value, as represented in the following tags

    Also modify the sga size

    Also modify shared_pool_reserved_size


    and retry your operation...

    1. Dear Arslan,

      I have re-executed after changes in Context file but again no luck.... :(

  4. Ali, Next time , use my forum to ask your questions..