Migrating 2 nodes (split configuration 1x32 bit app 1x64 bit db) EBS R12 (12.0.4) from "Windows" to 2 nodes (split configuration 1x64bit app 1x64bit db) "Oracle Linux"

Recently, migrated an EBS 12.0.4 environment from Windows to Linux.
Source system  was a split configuration ; app was on an 32 bit Windows Server and db was on an 64 bit Windows Server.
Target system was also a split configuration, app was on 64 bit Oracle Linux 5, and Db was also on an 64 bit Oracle Linux 5 .

Note tat: EBS 12.0.4 apps tier is not certified with Oracle Linux 6 at the moment. Db tier, however is certified with Oracle Linux 6 , but we wanted it to be the same OS as the apps tier so used Oracle Linux 5 for the db server' OS as well.

As you may noticed, we have automatically made a architectural migration as we have migrated EBS apps tier from 32 bit to 64 bit during this replatforming work.
In addition to the EBS, we have also upgraded Discoverer from 10g to 11g and created Sqlserver dblinks on the target platform as well.

The migration was done by using the following documents:

• root document-> R11i / R12: Oracle E-Business Suite Upgrades and Platform Migration (Doc ID 1377213.1)
• Database migration document-> R12 (12.0/12.1): 'Using Transportable Database to migrate Oracle E-Business Suite Release 12.0 and 12.1 using Oracle Database 10g Release 2 or 11g Enterprise Edition' - MOS Document 734763.1
• Application migration document -> Application Tier Platform Migration with Oracle E-Business Suite Release 12 (Doc ID 438086.1)

So , as the endians were not changing (both Windows and Linux are little endian Operating Systems), the migration was not a big deal for us. But it was still hard as there were lots of issues encountered and EBS migration experiences were required to quickly solve them. Still it was a pleasure to migrate an EBS environment to Linux :)

Well, before finishing this post, I want to share the issued steps in this migrations. They were written by us and they were written in a way that we could understand, but still you can get benefit from reading them:) -- I have no time for converting them in to a proper documentation as I have no time nowadays. (because of my book work)

Still, if you have any issues for these type of migrations, feel free to ask me your questions using my forum "http://ermanarslan.blogspot.com.tr/p/forum.html".

1) 

Target DB Linux Installation (Oracle Linux 5.9)

Configure hostname and IP 

Install all rpms in the document ( all rpms are in the comment )  on target

Configure Kernel Parameters

Database Software Installation on DB node

Install Database Examples ( 11.2.0.3 Installation files are include examples package )

"Create ORA_NLS10 Environment ( if you dont create NLS10 environment you will get error ORA-12701 during the conversion on target node )

run : $ORACLE_HOME/nls/data/old/cr9idata.pl"

2)

set oracle env manually try to conect sqlplus / as sysdba and check opatch lsinventory for example:

"Apply source rdbms patches to the target node. Windows patch is not applied

Patch  13366268

Patch  13258936

Patch  13001379

Patch  9858539

Patch  4247037"

"Apply patches in the document of the database installation

Patch 4247037 already applied 

Patch 9858539 already applied 

Patch 12834800 applied

Patch 12942119 applied

Patch 12951696 applied

Patch 12985184 applied

Patch 13001379 already applied 

Patch 13004894 applied

Patch 13258936 already applied 

Patch 13366268 already applied 

Patch 13477790 applied

Patch 13495307 applied

Patch 15955538 applied

Patch 16706896 applied"

3)

Target APP Linux Installation (Oracle Linux 5.9)

Set hostname and IP configuration (32 Bit Character issue for EBS )

Install all rpms in the document ( all rpms are in the comment )

After installing these rpms, run ldconfig -v

check packages installed or not with which command

4)

Preparing the Source Environtment (DB+APP )

Shutdown application and configure maintanance mode

"Copy Adgrants_nt.sql from patch 7305220  to database node ""$ORACLE_HOME/appsutil/admin"" ( create admin folder )  and run it

sqlplus / as sysdba

@adgrants_nt.sql applsys"

"Apply the latest AD patch ( on Source APP ) 

6767273 (Prereq of AD Delta 6 )

7305220 ( AD Delta 6 )"

Apply the latest AutoConfig Template patch ( 9386653 )

Run autoconfig on app and db node

"Error : %appsutil% in context file 

Solution : apply patch 6636108  and 4585869

after applying these patch create appsutil on app node and copy it to db node 

create context file on db node 

Set your environment again and run autoconfig"

Restart the database listener 

run autoconfig on app node

"Apply the latest Rapid Clone patches ( Check patches before apply )

5484000    ( Dont Apply )

9171651   ( Already Applied )

9833058   ( applied )

15969020 ( applied)

16958896 (applied)

16959080 (applied)

20408489 ( applied )"

"Apply Platform Migration Patches

6903505 ( already applied )

6510214 ( Prereq Patch of 6156498 ) Already applied

6329757 ( Prereq Patch of 6156498 ) Already applied

6145693 ( Prereq Patch of 6156498 ) Already applied

6156498  ( Applied )"

"After Applying patch 6156498.

Run autoconfig from %AD_TOP%

Execute admkappsutil.pl utility to create the file appsutil.zip

Copy or FTP the appsutil.zip file to the <RDBMS ORACLE_HOME>

unzip -o appsutil.zip

create context_file

configure environment again ( exit bash - relogin )

Run AutoConfig on the <RDBMS ORACLE_HOME>

"

Run adpreclone on the Source System (APP + DB )

"Application

cd [INST_TOP]/admin/scripts

perl adpreclone.pl appsTier

DB tier:

cd [RDBMS ORACLE_HOME]/appsutil/scripts/[CONTEXT_NAME]

perl adpreclone.pl dbTier"

Maintain Snapshot Information

"adadmin

Maintain Snapshot Information

Update Current View Snapshot 

Update Complete APPL_TOP"

Run adpreclone on the Source System (APP)

Prepare Source DB System

Upgrade AutoConfig to the latest version ( Already done during source APP preparation

Check prerequisites and external objects and create directories if neccessary

create the same files on target after conversion ( no action is taken )

Identify BFILE files that will need to be transferred ( create below sqls and run. Take action if necessary )

create tdb_get_bfile_dirs.sql  > No bfiles on source skip this step

create tdb_get_bfiles.sql > No bfiles on source skip this step

Export OLAP Analytic Workspaces ( Migrating OLAP From 32 Bits to 64 Bits or Across Platforms )

"Check Olap Usage and version with below sql. If the last usage day and the version of Aws delete AWS

select name ""FEATURE"", first_usage_date ""FROM"", last_usage_date ""TO"" from DBA_FEATURE_USAGE_STATISTICS where name like '%OLAP%'

select comp_id, comp_name, version, status from dba_registry where comp_name like '%OLAP%';

To Delete AWs 

exec dbms_aw.execute('aw delete OWNER.AW_NAME');"

5)

Backup source database by converting the backup files  to Linux

Start the database in READ ONLY mode

"shutdown immediate;

startup mount;

alter database open read only; "

Verify the database is ready for migration

"set serveroutput on

declare

retcode boolean;

begin

retcode := dbms_tdb.check_db('Linux x86 64-bit',dbms_tdb.skip_none);

if retcode

then

dbms_output.put_line('Yes,Database is compatible to convert');

else

dbms_output.put_line('No,Database is compatible to convert');

end if;

end;

/"

Run the RMAN CONVERT DATABASE Command

"rman target /

convert database

transport script 'R:\convertdb\transport_mydb.sql'

new database 'PROD'

to platform 'Linux x86 64-bit’

parallelism 4

format 'R:\convertdb\PROD\%U';"

6)

Identify Technology Stack Updates ( On Source APP node )

"Run below command. Check output of the command and apply patches if necessary ( after after the migration )

perl %FND_TOP%\patch\115\bin\TXKScript.pl -script=%FND_TOP%\patch\115\bin\txkInventory.pl 

-txktop=%APPLTMP% -contextfile=%CONTEXT_FILE% -appspass=erman-outfile=%APPLTMP%/Report_Inventory.html"

7)

Apply patches on target node

Generate and upload the manifest of customer-specific files

"cd [AD_TOP]/bin

perl adgenpsf.pl"

upload adgenpsf.txt file to https://updates.oracle.com/PlatformMigration

Download patch which you will receive via email

Apply patch which is sent from Oracle ( notebook C:\Oracle_Files\p22480111 )

8)

"Copy the Source System to the Target System ( APP )

Make sure the applmgr user owns the files copied to the Target System."

"[APPL_TOP]

[COMMON_TOP]/clone

[COMMON_TOP]/java

[COMMON_TOP]/_pages

[COMMON_TOP]/webapps

[COMMON_TOP]/util

Copy only the directories listed, not the full COMMON_TOP."

9)

Copy datafiles from ntfs disk to target local disk ( /u01/oracle/datafile/ )

copy init.ora ( INIT_00QQ1UBN_1_0.ORA ) to target $ORACLE_HOME/dbs ( rename the init.ora file as needed  initPROD.ora ) 

dos2unix initPROD.ora

modify initPROD.ora file ( compatible etc )

modify paths in transport.sql 

sqlplus / as sysdba @transport.sql

Check all datafiles are same on target node with source node

10)

configure hostname on new target (if required)

11)

"copy appsutil from source to the target oracle_home 

to copy updated appsutil create it again 

perl <AD_TOP>/bin/admkappsutil.pl"

Generate the Database Context File

perl <RDBMS_ORACLE_HOME>/appsutil/bin/adbldxml.pl

12)

run autoconfig on db node

set oracle  environment in bash_profile

restart database listener

startup db ( if it is working, db node is ok )

13)

create applmgr user and groups on target APP server

Install JDK and InfoZip utilities on the Target System

Check source node java version download java to target app node and install it  

14)

"Copy the Source System Context File to the Target System

cd %INST_TOP%/admin"

Clone the Applications Context File on the Target System

"Create a pairsfile (text file) with the following values:

s_systemname=[SID]

s_dbSid=[SID]

s_jdktop=[JDK_HOME]

s_jretop=[JDK_HOME]"

"Generate the Target System Context File:

cd [COMMON_TOP]/clone/bin

perl adclonectx.pl migrate java=[JDK_HOME] pairsfile=[pairsfile] contextfile=[Source Applications Context File]"

Output of Clone Context File

Install the Application Tier Technology Stack

"cd [Stage12]/startCD/Disk1/rapidwiz

./rapidwiz -techstack"

Check Log File

"Error : RW-50010: Error: - script has returned an error: 127  

Solution : 468695.1"

Run AutoConfig setup phase on the Target System

"cd [AD_TOP]/bin

./adconfig.sh run=INSTE8_SETUP contextfile=/u01/inst/apps/PROD_ermanserver/appl/admin/PROD_erman.xml

15)

Note: This command does not require the environment to be sourced."

Download and apply the customer-specific update with AutoPatch

Upload and Apply patch (notebook C:\Oracle_Files\p22480111 )

"To apply patch go to folder : Patch Directory/ad/patch/115/etc/

unzip adpatchR12_AD_A.zip

open README file ( for linux go to folder 46 )

cd /home/applmgr/22480111/ad/patch/115/etc/46

adpatch options=hotpatch,phtofile

for patch directory enter : /home/applmgr/22480111 "

Error : WICMLX failed during relink  ( no action has taken ) 

Review Component Versions and Technology Stack patch level

"Check txkInventory_Wed_Dec_30_11_07_48_2015_stdout file and apply patches which is not applied already"

16)

Apply Technology Stack Patches

Regenerate File System Objects

"cd [AD_TOP]/bin

./adgensgn.sh [Apps User]/[Apps Password]"

Run adadmin to generate messages, forms, reports and product jar files.

"Forms Compile Error : Document 1340049.1 Apply Patch 6400501 and follow the document ( Relink ) 

Relink Error : 

ra/10.1.2/lib//librw.a -lnsl

/usr/lib/libXtst.so.6: undefined reference to `__stack_chk_fail@GLIBC_2.4'

/usr/lib/libXtst.so.6: undefined reference to `__fprintf_chk@GLIBC_2.3.4'

/usr/lib/libXtst.so.6: undefined reference to `__sprintf_chk@GLIBC_2.3.4'

collect2: ld returned 1 exit status

make: *** [rwbuilder] Error 1

Solution : 1923486.1

root : unlink /usr/lib/libXtst.so.6

root: ln -s /usr/X11R6/lib/libXtst.so.6.1 /usr/lib/libXtst.so.6"

"Reports Compile Error: 

Solution:rpm package is missing ( xorg-x11-libs-compat-6.8.2-1.EL.33.0.1.i386 ) rpm package installed"

17)

Clean Nodes (Conditional) 

"Run afcpclean.sh on the Target Application Tier.

cd [INST_TOP]/admin/install

./afcpclean.sh apps password"

Run AutoConfig on both nodes

"Log into the target Database Tier and run Autoconfig 

cd $ORACLE_HOME/appsutil/bin

./adconfig.sh contextfile=[Target System Context File]

cd $AD_TOP/bin

./adconfig.sh contextfile=$CONTEXT_FILE"

Error : if autoconfig is completed with errors and if the error is 

" Apache Fails With Error ""failed to start a managed process after the maximum retry limit""

Solution : Document ID : 879522.1"

Update Oracle Workflow configuration settings

18)

Start Application Services and Test

"If you get http 404 not found error go to the document and try the command below

perl ojspCompile.pl --compile --flush -p 20 -log /tmp/ojspc_error.log  ( if you get error with this command as in the document, follow the solution )"

"PDF output problem : 

Solution:FNDWRR permission ( $FND_TOP/bin )"

"PDF Output Turkish character problem

Solution : uifont.ali"

Discoverer Installation

Missing Permissions manifest attribute for: disco5i.jarjar

Solution : 1596541.1

19)

Oracle to SQL Server DB Link Configuration(optional) using unix ODBC 2.3.0, Microsoft SQL Server Driver 11 for Linux and dg4ODBC

"install unixODBC 2.3.0 rpm

install Windows SQL Server Linux driver

Listener and Tns Configuration"

20) 

This step is completely conditional. (it depends on the environment, following actions were required in our case, they may not be related with your case)

update fnd_concurrent_processes set db_name='PROD', db_instance='PROD', sqlnet_string='PROD',node_name='ERMANPRODSERVER' where db_name='PROD' , if required..

Service Component Container is not Running' when Trying to Start Workflow Components (Doc ID 733335.1) --check this one.

Add  "export REPORTS_OUTPUTIMAGEFORMAT=gif" in $ORACLE_HOME/bin/reports.sh and add the relevant  NLS_LANG setting to reports.sh Increase reports cache