Tuesday, January 19, 2016

Migrating 2 nodes(split configuration 1x32 bit app 1x64 bit db) EBS R12 (12.0.4) from "Windows" to 2 nodes (split configuration 1x64bit app 1x64bit db) "Oracle Linux"

Recently, migrated an EBS 12.0.4 environment from Windows to Linux.
Source system  was a split configuration ; app was on an 32 bit Windows Server and db was on an 64 bit Windows Server.
Target system was also a split configuration, app was on 64 bit Oracle Linux 5, and Db was also on an 64 bit Oracle Linux 5 .

Note tat: EBS 12.0.4 apps tier is not certified with Oracle Linux 6 at the moment. Db tier, however is certified with Oracle Linux 6 , but we wanted it to be the same OS as the apps tier so used Oracle Linux 5 for the db server' OS as well.

As you may noticed, we have automatically made a architectural migration as we have migrated EBS apps tier from 32 bit to 64 bit during this replatforming work.
In addition to the EBS, we have also upgraded Discoverer from 10g to 11g and created Sqlserver dblinks on the target platform as well.

The migration was done by using the following documents:
  • root document-> R11i / R12: Oracle E-Business Suite Upgrades and Platform Migration (Doc ID 1377213.1)
  • Database migration document-> R12 (12.0/12.1): 'Using Transportable Database to migrate Oracle E-Business Suite Release 12.0 and 12.1 using Oracle Database 10g Release 2 or 11g Enterprise Edition' - MOS Document 734763.1
  • Application migration document -> Application Tier Platform Migration with Oracle E-Business Suite Release 12 (Doc ID 438086.1)
So , as the endians were not changing (both Windows and Linux are little endian Operating Systems), the migration was not a big deal for us. But it was still hard as there were lots of issues encountered and EBS migration experiences were required to quickly solve them. Still it was a pleasure to migrate an EBS environment to Linux :)

Well, before finishing this post, I want to share the issued steps in this migrations. They were written by us and they were written in a way that we could understand, but still you can get benefit from reading them:) -- I have no time for converting them in to a proper documentation as I have no time nowadays. (because of my book work)

Still, if you have any issues for these type of migrations, feel free to ask me your questions using my forum "http://ermanarslan.blogspot.com.tr/p/forum.html".

Target DB Linux Installation (Oracle Linux 5.9)
Configure hostname and IP 
Install all rpms in the document ( all rpms are in the comment )  on target
Configure Kernel Parameters
Database Software Installation on DB node
Install Database Examples ( Installation files are include examples package )
"Create ORA_NLS10 Environment ( if you dont create NLS10 environment you will get error ORA-12701 during the conversion on target node )
run : $ORACLE_HOME/nls/data/old/cr9idata.pl"

set oracle env manually try to conect sqlplus / as sysdba and check opatch lsinventory for example:
"Apply source rdbms patches to the target node. Windows patch is not applied
Patch  13366268
Patch  13258936
Patch  13001379
Patch  9858539
Patch  4247037"
"Apply patches in the document of the database installation
Patch 4247037 already applied 
Patch 9858539 already applied 
Patch 12834800 applied
Patch 12942119 applied
Patch 12951696 applied
Patch 12985184 applied
Patch 13001379 already applied 
Patch 13004894 applied
Patch 13258936 already applied 
Patch 13366268 already applied 
Patch 13477790 applied
Patch 13495307 applied
Patch 15955538 applied
Patch 16706896 applied"

Target APP Linux Installation (Oracle Linux 5.9)
Set hostname and IP configuration (32 Bit Character issue for EBS )
Install all rpms in the document ( all rpms are in the comment )
After installing these rpms, run ldconfig -v
check packages installed or not with which command

Preparing the Source Environtment (DB+APP )
Shutdown application and configure maintanance mode
"Copy Adgrants_nt.sql from patch 7305220  to database node ""$ORACLE_HOME/appsutil/admin"" ( create admin folder )  and run it
sqlplus / as sysdba
@adgrants_nt.sql applsys"
"Apply the latest AD patch ( on Source APP ) 
6767273 (Prereq of AD Delta 6 )
7305220 ( AD Delta 6 )"
Apply the latest AutoConfig Template patch ( 9386653 )
Run autoconfig on app and db node
"Error : %appsutil% in context file 
Solution : apply patch 6636108  and 4585869
after applying these patch create appsutil on app node and copy it to db node 
create context file on db node 
Set your environment again and run autoconfig"
Restart the database listener 
run autoconfig on app node
"Apply the latest Rapid Clone patches ( Check patches before apply )
5484000    ( Dont Apply )
9171651   ( Already Applied )
9833058   ( applied )
15969020 ( applied)
16958896 (applied)
16959080 (applied)
20408489 ( applied )"
"Apply Platform Migration Patches
6903505 ( already applied )
6510214 ( Prereq Patch of 6156498 ) Already applied
6329757 ( Prereq Patch of 6156498 ) Already applied
6145693 ( Prereq Patch of 6156498 ) Already applied
6156498  ( Applied )"
"After Applying patch 6156498.
Run autoconfig from %AD_TOP%
Execute admkappsutil.pl utility to create the file appsutil.zip
Copy or FTP the appsutil.zip file to the <RDBMS ORACLE_HOME>
unzip -o appsutil.zip
create context_file
configure environment again ( exit bash - relogin )
Run AutoConfig on the <RDBMS ORACLE_HOME>
Run adpreclone on the Source System (APP + DB )
cd [INST_TOP]/admin/scripts
perl adpreclone.pl appsTier

DB tier:
cd [RDBMS ORACLE_HOME]/appsutil/scripts/[CONTEXT_NAME]
perl adpreclone.pl dbTier"
Maintain Snapshot Information
Maintain Snapshot Information
Update Current View Snapshot 
Update Complete APPL_TOP"
Run adpreclone on the Source System (APP)
Prepare Source DB System
Upgrade AutoConfig to the latest version ( Already done during source APP preparation
Check prerequisites and external objects and create directories if neccessary
create the same files on target after conversion ( no action is taken )
Identify BFILE files that will need to be transferred ( create below sqls and run. Take action if necessary )
create tdb_get_bfile_dirs.sql  > No bfiles on source skip this step
create tdb_get_bfiles.sql > No bfiles on source skip this step
Export OLAP Analytic Workspaces ( Migrating OLAP From 32 Bits to 64 Bits or Across Platforms )
"Check Olap Usage and version with below sql. If the last usage day and the version of Aws delete AWS
select name ""FEATURE"", first_usage_date ""FROM"", last_usage_date ""TO"" from DBA_FEATURE_USAGE_STATISTICS where name like '%OLAP%'
select comp_id, comp_name, version, status from dba_registry where comp_name like '%OLAP%';

To Delete AWs 
exec dbms_aw.execute('aw delete OWNER.AW_NAME');"

Backup source database by converting the backup files  to Linux
Start the database in READ ONLY mode
"shutdown immediate;
startup mount;
alter database open read only; "
Verify the database is ready for migration
"set serveroutput on
retcode boolean;
retcode := dbms_tdb.check_db('Linux x86 64-bit',dbms_tdb.skip_none);
if retcode
dbms_output.put_line('Yes,Database is compatible to convert');
dbms_output.put_line('No,Database is compatible to convert');
end if;
"rman target /
convert database
transport script 'R:\convertdb\transport_mydb.sql'
new database 'PROD'
to platform 'Linux x86 64-bit’
parallelism 4
format 'R:\convertdb\PROD\%U';"

Identify Technology Stack Updates ( On Source APP node )
"Run below command. Check output of the command and apply patches if necessary ( after after the migration )
perl %FND_TOP%\patch\115\bin\TXKScript.pl -script=%FND_TOP%\patch\115\bin\txkInventory.pl 
-txktop=%APPLTMP% -contextfile=%CONTEXT_FILE% -appspass=erman-outfile=%APPLTMP%/Report_Inventory.html"

Apply patches on target node
Generate and upload the manifest of customer-specific files
"cd [AD_TOP]/bin
perl adgenpsf.pl"
upload adgenpsf.txt file to https://updates.oracle.com/PlatformMigration
Download patch which you will receive via email
Apply patch which is sent from Oracle ( notebook C:\Oracle_Files\p22480111 )

"Copy the Source System to the Target System ( APP )
Make sure the applmgr user owns the files copied to the Target System."
Copy only the directories listed, not the full COMMON_TOP."

Copy datafiles from ntfs disk to target local disk ( /u01/oracle/datafile/ )
copy init.ora ( INIT_00QQ1UBN_1_0.ORA ) to target $ORACLE_HOME/dbs ( rename the init.ora file as needed  initPROD.ora ) 
dos2unix initPROD.ora
modify initPROD.ora file ( compatible etc )
modify paths in transport.sql 
sqlplus / as sysdba @transport.sql
Check all datafiles are same on target node with source node

configure hostname on new target (if required)

"copy appsutil from source to the target oracle_home 
to copy updated appsutil create it again 
perl <AD_TOP>/bin/admkappsutil.pl"
Generate the Database Context File
perl <RDBMS_ORACLE_HOME>/appsutil/bin/adbldxml.pl

run autoconfig on db node
set oracle  environment in bash_profile
restart database listener
startup db ( if it is working, db node is ok )

create applmgr user and groups on target APP server
Install JDK and InfoZip utilities on the Target System
Check source node java version download java to target app node and install it  

"Copy the Source System Context File to the Target System
cd %INST_TOP%/admin"
Clone the Applications Context File on the Target System
"Create a pairsfile (text file) with the following values:
"Generate the Target System Context File:
cd [COMMON_TOP]/clone/bin
perl adclonectx.pl migrate java=[JDK_HOME] pairsfile=[pairsfile] contextfile=[Source Applications Context File]"
Output of Clone Context File
Install the Application Tier Technology Stack
"cd [Stage12]/startCD/Disk1/rapidwiz
./rapidwiz -techstack"
Check Log File
"Error : RW-50010: Error: - script has returned an error: 127  
Solution : 468695.1"
Run AutoConfig setup phase on the Target System
"cd [AD_TOP]/bin
./adconfig.sh run=INSTE8_SETUP contextfile=/u01/inst/apps/PROD_ermanserver/appl/admin/PROD_erman.xml

Note: This command does not require the environment to be sourced."
Download and apply the customer-specific update with AutoPatch
Upload and Apply patch (notebook C:\Oracle_Files\p22480111 )
"To apply patch go to folder : Patch Directory/ad/patch/115/etc/
unzip adpatchR12_AD_A.zip
open README file ( for linux go to folder 46 )
cd /home/applmgr/22480111/ad/patch/115/etc/46
adpatch options=hotpatch,phtofile
for patch directory enter : /home/applmgr/22480111 "
Error : WICMLX failed during relink  ( no action has taken ) 
Review Component Versions and Technology Stack patch level
"Check txkInventory_Wed_Dec_30_11_07_48_2015_stdout file and apply patches which is not applied already"

Apply Technology Stack Patches
Regenerate File System Objects
"cd [AD_TOP]/bin
./adgensgn.sh [Apps User]/[Apps Password]"
Run adadmin to generate messages, forms, reports and product jar files.
"Forms Compile Error : Document 1340049.1 Apply Patch 6400501 and follow the document ( Relink ) 
Relink Error : 
ra/10.1.2/lib//librw.a -lnsl
/usr/lib/libXtst.so.6: undefined reference to `__stack_chk_fail@GLIBC_2.4'
/usr/lib/libXtst.so.6: undefined reference to `__fprintf_chk@GLIBC_2.3.4'
/usr/lib/libXtst.so.6: undefined reference to `__sprintf_chk@GLIBC_2.3.4'
collect2: ld returned 1 exit status
make: *** [rwbuilder] Error 1

Solution : 1923486.1

root : unlink /usr/lib/libXtst.so.6
root: ln -s /usr/X11R6/lib/libXtst.so.6.1 /usr/lib/libXtst.so.6"
"Reports Compile Error: 
Solution:rpm package is missing ( xorg-x11-libs-compat-6.8.2-1.EL.33.0.1.i386 ) rpm package installed"

Clean Nodes (Conditional) 
"Run afcpclean.sh on the Target Application Tier.

cd [INST_TOP]/admin/install
./afcpclean.sh apps password"
Run AutoConfig on both nodes
"Log into the target Database Tier and run Autoconfig 

cd $ORACLE_HOME/appsutil/bin
./adconfig.sh contextfile=[Target System Context File]

cd $AD_TOP/bin
./adconfig.sh contextfile=$CONTEXT_FILE"
Error : if autoconfig is completed with errors and if the error is 
" Apache Fails With Error ""failed to start a managed process after the maximum retry limit""
Solution : Document ID : 879522.1"
Update Oracle Workflow configuration settings

Start Application Services and Test
"If you get http 404 not found error go to the document and try the command below
perl ojspCompile.pl --compile --flush -p 20 -log /tmp/ojspc_error.log  ( if you get error with this command as in the document, follow the solution )"
"PDF output problem : 
Solution:FNDWRR permission ( $FND_TOP/bin )"
"PDF Output Turkish character problem
Solution : uifont.ali"
Discoverer Installation
Missing Permissions manifest attribute for: disco5i.jarjar
Solution : 1596541.1

Oracle to SQL Server DB Link Configuration(optional) using unix ODBC 2.3.0, Microsoft SQL Server Driver 11 for Linux and dg4ODBC
"install unixODBC 2.3.0 rpm
install Windows SQL Server Linux driver
Listener and Tns Configuration"

This step is completely conditional. (it depends on the environment, following actions were required in our case, they may not be related with your case)
update fnd_concurrent_processes set db_name='PROD', db_instance='PROD', sqlnet_string='PROD',node_name='ERMANPRODSERVER' where db_name='PROD' , if required..
Service Component Container is not Running' when Trying to Start Workflow Components (Doc ID 733335.1) --check this one.
Add  "export REPORTS_OUTPUTIMAGEFORMAT=gif" in $ORACLE_HOME/bin/reports.sh
and add the relevant  NLS_LANG setting to reports.sh
Increase reports cache

No comments :

Post a Comment