Thursday, January 28, 2016

ODA X5, ACFS / 2TB ASM bug + ACFS bug

Recently I released a blog post for informing you the requirement to use ACFS in new ODA X5 platform.
ACFS is required because the disks delivered with ODA X5 are bigger than 2TB and ASM has a bug when using 2 TB disk sizes.

I mentioned about the corruption which may be caused by using disks sized more than 2 TB with ASM and in this post, I want to give you the details about it, because I discovered another Oracle support note which talks about similar bug in ACFS as well.

So , as documented below, a corruption may be caused by having disk devices larger than 2tb in ASM environment. Here is a bug record about that: Bug 19012119 : ORA-15196 WITH 5TB DISK SIZE IN 12.1

Using ACFS is a workaround for this problem and that's why it s a must in ODA X5, but here is a note introcuding another bug which is related directly with using Devices larger than 2 TB with ACFS in ODA X5..
Bug is affecting the ODA X5 environment , which have software versions or
ALERT Diskgroup Corruption Due to Invalid ASM Block Header [ENDIAN_KFBH] for Devices Larger Than 2TB with ADVM Volume on X5-2 ODA - and Only (Doc ID 2038152.1)

Just wanted to inform you ..

EBS R12 -- forwarded notification emails /Allow Forwarded Response/ Problem encountered when handling an unsolicited email

It is not supported to forward a notification email to any other employee and let him or her reply to this email either for approving or rejecting a request, thus making Workflow inbound processing take action in the database to make the requested thing to be approved or rejected.

Reference Oracle Support:

It is not supported to use the forward functionality in email to forward the approval notification whether it be the Approval Management Engine (AME) or standard approvals.
The act of using the email client forward functionality circumvents Oracle workflow and notification functionality.  By using the Forward function in email to forward the notification, the
ownership of the notification is not changed.   Whomever responds to the email will not be registered in the po_action_history table

The way for preventing the forwarded emails to be processed by the workflow mailer (inbound processor, imap processor), is to disable Allow Forwarded Responses by unchecking the related checkbox presented in the Workflow mailer configuration page, as shown in figure below;

By unchecking the Allow Forwarded Responses, we can make Notification mailer's Inbound processor not to process the forwarded emails.
But , the users can still forward their notification emails to other right? That is; we can't do anything in the client email tool.
So the users who receives these notification emails can still continue to respond these notification emails.

What about these emails then? The answer is ; our notification mailer just won't process them.
What will notification mailer do with these emails then? The expected behaviour : the workflow mailer should take these emails to the DISCARD folder.

Is it so in real life?
No it is not :)
There is a bug and that's why I write this blog post:)

The problem we say even in EBS 12.2 environment is depicted in the error below;]:Problem encountered when handling an unsolicited email -> Could not connect to SMTP host: someip, port: someport, response: -1

The questions can be ? IMAPResponsHandler doing SMTP? Isnt it inbound? So what is the relation with SMTP? It should be doing IMAP rather than smtp? The answer for these question is; Inbound can do Smtp too. Inbound does SMTP when sending unsolicited, second answer and invalid emails.

The action for fixing this in EBS 12.2 environment is applying patch 22198191:R12.OWF.C 1OFF:12.2.4:20137109:MAILER NOT USING CORRECT SMTP CONFIGURATION FOR UNSOLICITED MAIL

The fix is delivered with the note; 
Automatic Reply To Unsolicited Mail shows erros 530 5.7.1 Client was not authenticated (Doc ID 1991613.1)

Don't give attention to the above error message(it says Client was not authenticated there), as the things given in above document will fix the issue.

Sunday, January 24, 2016

ODA - Using ACFS for storing database is a must in ODA X5

Here is an important information for you, who purchased ODA X5 and planning to migrate EBS instances to the ODA environment.
Be aware that you will have to use at least an Oracle Database and you will have to use ACFS for storing the EBS database.

Reference Oracle:

Oracle Database Appliance introduces a fundamental change in the storage layer – the introduction of Oracle ACFS. Oracle ACFS is designed to deliver the best performance for Oracle databases. Oracle ACFS supports the creation of filesystem snapshots – a feature that vastly simplifies the creation of space efficient database clones. Going forward, new databases should only be created on ACFS filesystems in a new ODA deployment scenarios

Reference Oracle Support:

X5-2 does NOT support database versions (except ACFS DB created from ODA onwards) and Only (with ODA onwards), and databases are supported must run on ACFS. Reason: X5-2 contains 4TB disks which exceed the 11.2 ASM disk size limit of 2TB. Running 11.2 databases on ASM will lead to data corruption, regardless of the OAK version. While OAK will prevent creation of unsupported databases, OAK did not. If you have created any unsupported or ASM DBs on X5-2, you must immediately upgrade to, or and manually migrate from ASM to ACFS.

Saturday, January 23, 2016

EBS 12.2 -- downloading EBS 12.2.5 from

Here is a quick tip for you.
With the latest enhancements, it is a little different to download EBS from
That is; we don't have a link or selection for downloading EBS 12.2 directly in edelivery.
In order to download EBS 12.2.5, we choose a product that is delivered within the EBS and only after then edelivery brings us EBS 12.2.5 download urls.
Just like we did in the following example screens.
As seen, we choose Oracle Financial to reach the EBS 12.2.5 download page.
Note that : EBS 12.2.5 delivers an Oracle Database, which can be cosidered good in terms of RDBMS support policies.
It is even good for  ODA X5 customers, as the latest ODA (ODA X5) wants us to use ACFS for storing our database files, so having a ACFS-supported Database by default is an advantage.

Friday, January 22, 2016

Weblogic-- a note on applying PSU updates

Here is a note that is created by my collegue Hulya Yılmaz, while patching a 10.3.6 Weblogic instance with the WLS Patch Set Update 13.

The master note used in this upgrade was:
Master Note on WebLogic Server Patch Set Updates (PSUs)(Doc ID 1470197.1)

The Middleware and Weblogic Home were:


The steps :

1. Stop all WebLogic servers

2. unzip to {MW_HOME}/utils/bsu/cache_dir

3. . $WL_HOME/server/bin/

4. Navigate to the {MW_HOME}/utils/bsu directory

5. Execute -install -patch_download_dir={MW_HOME}/utils/bsu/cache_dir -patchlist={PATCH_ID} -prod_dir={MW_HOME}/{WL_HOME}

bash-3.2$ -install -patch_download_dir=/data/oracledev/Middleware/utils/bsu/cache_dir -patchlist=S8C2 - prod_dir=/data/oracledev/Middleware/wlserver_10.3 -log=/tmp/bsu.log
Checking for conflicts.......
No conflict(s) detected Installing Patch ID: S8C2..
Result: Success

Note: if bsu will hand, then we increase the heapsize by following the note below.

WebLogic Server: Smart Update Throwing OutOfMemoryError (Doc ID 1154089.1)

6. Verify psu:

bash-3.2$ java weblogic.version
WebLogic Server PSU Patch for BUG21984589 TUE NOV 26 15:54:42 IST 2015 WebLogic Server Tue Nov 15 08:52:36 PST 2011 1441050

7. Restart all WebLogic Servers

Thursday, January 21, 2016

RDBMS - sqlldr errors caused by CR and LF characters

Recently faced an issue in an EBS customer. It was related with sql loader program that was executing from EBS concurrent processing tier.
The sqlldr was trying to import the datafiles which were transffered from Windows to Linux, but it was failing because the data that it was trying to import, was not able to fit the columns residing in the relevant tables of the EBS database.
Actually, the solution was pretty easy for this. All that needs to be done was using an ascii mode transfer for copying these datafiles from Windows Linux. On the other hand, it was not possible in this customer site. (dont ask why :))
So we had to deal with it using a different solution.

What was the case?
EBS was executing sqlldr, so we can not manipulate it.
Datafiles transferred from Windows to Linux were in Windows format, having CR and LF characters.
Sqlloader was encountring errors because of the extra characters.

What I have did?
I have seen that, sqlldr in EBS is executed from $ORACLE_HOME (10.1.2 Oracle Home)
So I moved the sqlldr as sqlldr_original
mv sqlldr sqlldr.original

Then, recreated the sqlldr as a script that uses dos2unix to convert the datafiles to unix format and then executes the sqlldr with all the arguments supplied to it.

[applmgr@ermanserver bin]$ vi sqlldr
dos2unix /sqlldr_datafiles_location/*.txt
sqlldr_original $*

:) well, yes it is not supported and it is in danger to be overwritten by relinking the Oracle Home  :), but still it is unique and saves the day:)
Long time ago, we once have done it for ar60run executable as well. We have used this method(this wrapper method) to set the NLS_SORT parameter before executing the ar60run executable in EBS 11i production environment.

RDBMS-- Oracle to SQLSERVER dblink

When we talk about creating a Dblink from an Oracle Database (Oracle 11g or above) To MSSQL server, we actually talk about dg4odbc.
Well, creating a database link from an Oracle Database to a Microsoft Sql Server Database is based on the binary named DG4ODBC.(DAtabaes Gateway For ODBC) DG4ODBC is a new program that replaces hsodbc, which was used in earlier Oracle Database releases.
Actually, DG4ODBC is not the only thing required for this operation. There are linux drivers(I m talking about Oracle On Linux env bytheway, as %99 of the Oracle Environments that I have dealed with were  running on Linux), driver manager, oracle clients and driver managers that makes an Oracle to Sql Server dblink available to function.
The full path of reaching an Sql Server databae from an Oracle Instance is as follows:
Suppose we are using sqlplus to query the sql server;

"SQL*Plus -> Oracle Client -> DG4ODBC instance -> unixODBC -> ODBC driver -> Database"

So what we do here is ; we execute our query in a query tool such as sqlplus.
This tool used oracle client and when it sees the dblink, it executes the DG4ODBC program via the Oracle Database listener.
DG4ODBC passes the work to te Linux ODBC Driver Manager
Linux ODBC Driveer manager triggers the MS SQL Server ODBC driver to be taken to the stage and the MS Sql ODBC Driver of Linux connects to the Sql Server database and our query executed from Oracle Database gets data from Sql Server.

So as you may imagine , all of these components except the sqlplus and oracle client(there are already configured and nothing needs to be done for them) needs a proper configuration for this.

What we do to configure?

1) We first install a Microsoft Sql Server Driver to Linux OS where our Oracle database is running.
Then we add sql server related configuration to the odbc.ini as shown in the example below;
Note that: we specify the driver that comes with the ms sql server driver installation with the "Driver=" setting.

[root@ermanserver]# cat /etc/odbc.ini
Description=Microsoft ODBC Driver 11 for SQL Server

Then we add the sql server driver related information to the odbcinst.ini as shown in the example below;
Note that: the installation driver may do this automatically, but still it is good to be checked

[root@ermanserver]# cat /etc/odbcinst.ini
Description=ODBC for PostgreSQL
Description=ODBC for MySQL
Description=ODBC for PostgreSQL (64 bit)
Description=ODBC for MySQL (64 bit)
[ODBC Driver 11 for SQL Server]  ----> "here is our driver information"
Description=Microsoft ODBC Driver 11 for SQL Server

Next, we check the sqlserver connection and see if it can be done using our newly installed driver and its configuration.
isql utility can be used here.

[root@ermanserver ]# . /root/unixODBC-2.3.0/exe/isql -v SQLSERVER_ODBC_DSN ermanuser ermanpassword
| Connected!                            |
|                                       |
| sql-statement                         |
| help [tablename]                      |
| quit                                  |
|                                       |

After seeing the SQL> prompt, we can say that our work with Linux configuration files are finished.

We jump to Oracle Database home and create an .ora file in $ORACLE_HOME/hs/admin directory.
Note that: the file name should be in the following format init<DSN_NAME_WE_HAVE_CONFIGURED_IN_ODBCINSTFILE>.ora

[oracle@ermanserver ~]$ cd $ORACLE_HOME/hs/admin

[oracle@ermanserver admin]$ ls

The init file is important as we decleare lots of heterogenous parameters in this file.
The most important setting is the HS_FDS_SHAREABLE_NAME setting, as we set the Linux's ODBC driver manager related System object file(/usr/lib64/ here. Driver manager will choose our driver when it is needed.

[oracle@ermanserver admin]$ cat initSQLSERVER_ODBC_DSN.ora
set ODBCINI=/etc/odbc.ini

After configuring the init file, we configure our listener.ora and tnsnames.ora files
Note that : we have used the IFILE here because this example was done on an autoconfig enabled EBS environment and that's why we didnt want our newly added tns entry to be overwritten by a future autoconfig run.
Also note that: we set the hostname and port of our local Oracle Database environment, as we want the clients to reach the listener and then the listener to execute the DG4ODBC program to connect them internally to sqlserver.

[oracle@ermanserver admin]$ cat /u01/app/oracle/product/11.20/dbhome_1/network/admin/PROD_ermanserver/PROD_ermanserver.ora

       (ADDRESS =
          (PROTOCOL = tcp)
          (HOST = ermanserver)
          (PORT = 1523))
       (CONNECT_DATA =
          (SID = SQLSERVER_ODBC_DSN ))
       (HS = OK)

In listener.ora file, we also use the database host and port because of the same reason.
Note that: all the settings related to odbc are important here. (PROGRAM, ENVS=LD_LIBRARY_PATH etc..)
Bytheway, we have added our sqlserver related configuration directly to the listener.ora here, you can add it to the ifile of the listener :) to prevent it getting overwritten by autoconfig. :)

[oracle@ermanserver admin]$ cat listener.ora
      (ADDRESS = (PROTOCOL = TCP)(HOST = ermanserver)(PORT = 1523))
    (SID_DESC =
      (ORACLE_HOME= /u01/app/oracle/product/11.2.0/dbhome_1)
      (SID_NAME = PROD)
        (SID_DESC =
          (ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_1)
          (PROGRAM = /u01/app/oracle/product/11.2.0/dbhome_1/bin/dg4odbc)
LOG_DIRECTORY_LISTENER = /u01/app/oracle/product/11.2.0/dbhome_1/network/admin
TRACE_DIRECTORY_LISTENER = /u01/app/oracle/product/11.2.0/dbhome_1/network/admin


That's all :) We reload our listener and start using our dblink by  creating a dblink with the following syntax.

CREATE DATABASE LINK MYSQLSERVER CONNECT TO ermanuser identified by ermanpass

Example query select * from "dbo"."tablename"@SQLSERVER

Well... We have reached the end of this post. Note that: The configuration files used in this post in taken from a real system. So this is a working example . I hope you find it useful.

Tuesday, January 19, 2016

Migrating 2 nodes(split configuration 1x32 bit app 1x64 bit db) EBS R12 (12.0.4) from "Windows" to 2 nodes (split configuration 1x64bit app 1x64bit db) "Oracle Linux"

Recently, migrated an EBS 12.0.4 environment from Windows to Linux.
Source system  was a split configuration ; app was on an 32 bit Windows Server and db was on an 64 bit Windows Server.
Target system was also a split configuration, app was on 64 bit Oracle Linux 5, and Db was also on an 64 bit Oracle Linux 5 .

Note tat: EBS 12.0.4 apps tier is not certified with Oracle Linux 6 at the moment. Db tier, however is certified with Oracle Linux 6 , but we wanted it to be the same OS as the apps tier so used Oracle Linux 5 for the db server' OS as well.

As you may noticed, we have automatically made a architectural migration as we have migrated EBS apps tier from 32 bit to 64 bit during this replatforming work.
In addition to the EBS, we have also upgraded Discoverer from 10g to 11g and created Sqlserver dblinks on the target platform as well.

The migration was done by using the following documents:
  • root document-> R11i / R12: Oracle E-Business Suite Upgrades and Platform Migration (Doc ID 1377213.1)
  • Database migration document-> R12 (12.0/12.1): 'Using Transportable Database to migrate Oracle E-Business Suite Release 12.0 and 12.1 using Oracle Database 10g Release 2 or 11g Enterprise Edition' - MOS Document 734763.1
  • Application migration document -> Application Tier Platform Migration with Oracle E-Business Suite Release 12 (Doc ID 438086.1)
So , as the endians were not changing (both Windows and Linux are little endian Operating Systems), the migration was not a big deal for us. But it was still hard as there were lots of issues encountered and EBS migration experiences were required to quickly solve them. Still it was a pleasure to migrate an EBS environment to Linux :)

Well, before finishing this post, I want to share the issued steps in this migrations. They were written by us and they were written in a way that we could understand, but still you can get benefit from reading them:) -- I have no time for converting them in to a proper documentation as I have no time nowadays. (because of my book work)

Still, if you have any issues for these type of migrations, feel free to ask me your questions using my forum "".

Target DB Linux Installation (Oracle Linux 5.9)
Configure hostname and IP 
Install all rpms in the document ( all rpms are in the comment )  on target
Configure Kernel Parameters
Database Software Installation on DB node
Install Database Examples ( Installation files are include examples package )
"Create ORA_NLS10 Environment ( if you dont create NLS10 environment you will get error ORA-12701 during the conversion on target node )
run : $ORACLE_HOME/nls/data/old/"

set oracle env manually try to conect sqlplus / as sysdba and check opatch lsinventory for example:
"Apply source rdbms patches to the target node. Windows patch is not applied
Patch  13366268
Patch  13258936
Patch  13001379
Patch  9858539
Patch  4247037"
"Apply patches in the document of the database installation
Patch 4247037 already applied 
Patch 9858539 already applied 
Patch 12834800 applied
Patch 12942119 applied
Patch 12951696 applied
Patch 12985184 applied
Patch 13001379 already applied 
Patch 13004894 applied
Patch 13258936 already applied 
Patch 13366268 already applied 
Patch 13477790 applied
Patch 13495307 applied
Patch 15955538 applied
Patch 16706896 applied"

Target APP Linux Installation (Oracle Linux 5.9)
Set hostname and IP configuration (32 Bit Character issue for EBS )
Install all rpms in the document ( all rpms are in the comment )
After installing these rpms, run ldconfig -v
check packages installed or not with which command

Preparing the Source Environtment (DB+APP )
Shutdown application and configure maintanance mode
"Copy Adgrants_nt.sql from patch 7305220  to database node ""$ORACLE_HOME/appsutil/admin"" ( create admin folder )  and run it
sqlplus / as sysdba
@adgrants_nt.sql applsys"
"Apply the latest AD patch ( on Source APP ) 
6767273 (Prereq of AD Delta 6 )
7305220 ( AD Delta 6 )"
Apply the latest AutoConfig Template patch ( 9386653 )
Run autoconfig on app and db node
"Error : %appsutil% in context file 
Solution : apply patch 6636108  and 4585869
after applying these patch create appsutil on app node and copy it to db node 
create context file on db node 
Set your environment again and run autoconfig"
Restart the database listener 
run autoconfig on app node
"Apply the latest Rapid Clone patches ( Check patches before apply )
5484000    ( Dont Apply )
9171651   ( Already Applied )
9833058   ( applied )
15969020 ( applied)
16958896 (applied)
16959080 (applied)
20408489 ( applied )"
"Apply Platform Migration Patches
6903505 ( already applied )
6510214 ( Prereq Patch of 6156498 ) Already applied
6329757 ( Prereq Patch of 6156498 ) Already applied
6145693 ( Prereq Patch of 6156498 ) Already applied
6156498  ( Applied )"
"After Applying patch 6156498.
Run autoconfig from %AD_TOP%
Execute utility to create the file
Copy or FTP the file to the <RDBMS ORACLE_HOME>
unzip -o
create context_file
configure environment again ( exit bash - relogin )
Run AutoConfig on the <RDBMS ORACLE_HOME>
Run adpreclone on the Source System (APP + DB )
cd [INST_TOP]/admin/scripts
perl appsTier

DB tier:
cd [RDBMS ORACLE_HOME]/appsutil/scripts/[CONTEXT_NAME]
perl dbTier"
Maintain Snapshot Information
Maintain Snapshot Information
Update Current View Snapshot 
Update Complete APPL_TOP"
Run adpreclone on the Source System (APP)
Prepare Source DB System
Upgrade AutoConfig to the latest version ( Already done during source APP preparation
Check prerequisites and external objects and create directories if neccessary
create the same files on target after conversion ( no action is taken )
Identify BFILE files that will need to be transferred ( create below sqls and run. Take action if necessary )
create tdb_get_bfile_dirs.sql  > No bfiles on source skip this step
create tdb_get_bfiles.sql > No bfiles on source skip this step
Export OLAP Analytic Workspaces ( Migrating OLAP From 32 Bits to 64 Bits or Across Platforms )
"Check Olap Usage and version with below sql. If the last usage day and the version of Aws delete AWS
select name ""FEATURE"", first_usage_date ""FROM"", last_usage_date ""TO"" from DBA_FEATURE_USAGE_STATISTICS where name like '%OLAP%'
select comp_id, comp_name, version, status from dba_registry where comp_name like '%OLAP%';

To Delete AWs 
exec dbms_aw.execute('aw delete OWNER.AW_NAME');"

Backup source database by converting the backup files  to Linux
Start the database in READ ONLY mode
"shutdown immediate;
startup mount;
alter database open read only; "
Verify the database is ready for migration
"set serveroutput on
retcode boolean;
retcode := dbms_tdb.check_db('Linux x86 64-bit',dbms_tdb.skip_none);
if retcode
dbms_output.put_line('Yes,Database is compatible to convert');
dbms_output.put_line('No,Database is compatible to convert');
end if;
"rman target /
convert database
transport script 'R:\convertdb\transport_mydb.sql'
new database 'PROD'
to platform 'Linux x86 64-bit’
parallelism 4
format 'R:\convertdb\PROD\%U';"

Identify Technology Stack Updates ( On Source APP node )
"Run below command. Check output of the command and apply patches if necessary ( after after the migration )
perl %FND_TOP%\patch\115\bin\ -script=%FND_TOP%\patch\115\bin\ 
-txktop=%APPLTMP% -contextfile=%CONTEXT_FILE% -appspass=erman-outfile=%APPLTMP%/Report_Inventory.html"

Apply patches on target node
Generate and upload the manifest of customer-specific files
"cd [AD_TOP]/bin
upload adgenpsf.txt file to
Download patch which you will receive via email
Apply patch which is sent from Oracle ( notebook C:\Oracle_Files\p22480111 )

"Copy the Source System to the Target System ( APP )
Make sure the applmgr user owns the files copied to the Target System."
Copy only the directories listed, not the full COMMON_TOP."

Copy datafiles from ntfs disk to target local disk ( /u01/oracle/datafile/ )
copy init.ora ( INIT_00QQ1UBN_1_0.ORA ) to target $ORACLE_HOME/dbs ( rename the init.ora file as needed  initPROD.ora ) 
dos2unix initPROD.ora
modify initPROD.ora file ( compatible etc )
modify paths in transport.sql 
sqlplus / as sysdba @transport.sql
Check all datafiles are same on target node with source node

configure hostname on new target (if required)

"copy appsutil from source to the target oracle_home 
to copy updated appsutil create it again 
perl <AD_TOP>/bin/"
Generate the Database Context File
perl <RDBMS_ORACLE_HOME>/appsutil/bin/

run autoconfig on db node
set oracle  environment in bash_profile
restart database listener
startup db ( if it is working, db node is ok )

create applmgr user and groups on target APP server
Install JDK and InfoZip utilities on the Target System
Check source node java version download java to target app node and install it  

"Copy the Source System Context File to the Target System
cd %INST_TOP%/admin"
Clone the Applications Context File on the Target System
"Create a pairsfile (text file) with the following values:
"Generate the Target System Context File:
cd [COMMON_TOP]/clone/bin
perl migrate java=[JDK_HOME] pairsfile=[pairsfile] contextfile=[Source Applications Context File]"
Output of Clone Context File
Install the Application Tier Technology Stack
"cd [Stage12]/startCD/Disk1/rapidwiz
./rapidwiz -techstack"
Check Log File
"Error : RW-50010: Error: - script has returned an error: 127  
Solution : 468695.1"
Run AutoConfig setup phase on the Target System
"cd [AD_TOP]/bin
./ run=INSTE8_SETUP contextfile=/u01/inst/apps/PROD_ermanserver/appl/admin/PROD_erman.xml

Note: This command does not require the environment to be sourced."
Download and apply the customer-specific update with AutoPatch
Upload and Apply patch (notebook C:\Oracle_Files\p22480111 )
"To apply patch go to folder : Patch Directory/ad/patch/115/etc/
open README file ( for linux go to folder 46 )
cd /home/applmgr/22480111/ad/patch/115/etc/46
adpatch options=hotpatch,phtofile
for patch directory enter : /home/applmgr/22480111 "
Error : WICMLX failed during relink  ( no action has taken ) 
Review Component Versions and Technology Stack patch level
"Check txkInventory_Wed_Dec_30_11_07_48_2015_stdout file and apply patches which is not applied already"

Apply Technology Stack Patches
Regenerate File System Objects
"cd [AD_TOP]/bin
./ [Apps User]/[Apps Password]"
Run adadmin to generate messages, forms, reports and product jar files.
"Forms Compile Error : Document 1340049.1 Apply Patch 6400501 and follow the document ( Relink ) 
Relink Error : 
ra/10.1.2/lib//librw.a -lnsl
/usr/lib/ undefined reference to `__stack_chk_fail@GLIBC_2.4'
/usr/lib/ undefined reference to `__fprintf_chk@GLIBC_2.3.4'
/usr/lib/ undefined reference to `__sprintf_chk@GLIBC_2.3.4'
collect2: ld returned 1 exit status
make: *** [rwbuilder] Error 1

Solution : 1923486.1

root : unlink /usr/lib/
root: ln -s /usr/X11R6/lib/ /usr/lib/"
"Reports Compile Error: 
Solution:rpm package is missing ( xorg-x11-libs-compat-6.8.2-1.EL.33.0.1.i386 ) rpm package installed"

Clean Nodes (Conditional) 
"Run on the Target Application Tier.

cd [INST_TOP]/admin/install
./ apps password"
Run AutoConfig on both nodes
"Log into the target Database Tier and run Autoconfig 

cd $ORACLE_HOME/appsutil/bin
./ contextfile=[Target System Context File]

cd $AD_TOP/bin
./ contextfile=$CONTEXT_FILE"
Error : if autoconfig is completed with errors and if the error is 
" Apache Fails With Error ""failed to start a managed process after the maximum retry limit""
Solution : Document ID : 879522.1"
Update Oracle Workflow configuration settings

Start Application Services and Test
"If you get http 404 not found error go to the document and try the command below
perl --compile --flush -p 20 -log /tmp/ojspc_error.log  ( if you get error with this command as in the document, follow the solution )"
"PDF output problem : 
Solution:FNDWRR permission ( $FND_TOP/bin )"
"PDF Output Turkish character problem
Solution : uifont.ali"
Discoverer Installation
Missing Permissions manifest attribute for: disco5i.jarjar
Solution : 1596541.1

Oracle to SQL Server DB Link Configuration(optional) using unix ODBC 2.3.0, Microsoft SQL Server Driver 11 for Linux and dg4ODBC
"install unixODBC 2.3.0 rpm
install Windows SQL Server Linux driver
Listener and Tns Configuration"

This step is completely conditional. (it depends on the environment, following actions were required in our case, they may not be related with your case)
update fnd_concurrent_processes set db_name='PROD', db_instance='PROD', sqlnet_string='PROD',node_name='ERMANPRODSERVER' where db_name='PROD' , if required..
Service Component Container is not Running' when Trying to Start Workflow Components (Doc ID 733335.1) --check this one.
and add the relevant  NLS_LANG setting to
Increase reports cache

ODA -- Migrating EBS database from a Virtual Machine to ODA BASE (recommeded for optimum Performance)

As for the EBS on ODA implementations (any release 11i,R12 or 12.2), it is highly recommended to use ODA BASE machines for the EBS database tier.
In our practices, we have seen that non-ODA BASE virtual machines encounter I/O problems while dealing with the I/O operations created by the busy EBS 12.2 environments.
We have seen this in real life.
So if you are using Virtualized ODA for hosting your EBS 12.2 environment, or if you plan to use the virtualized ODA for your EBS environments in the future, your EBS database should be placed in ODA BASE virtual servers, as these virtual servers are optimized by Oracle and have the ability to directly reach the ODA hardware.

So even if it is a fresh installation, we recommend migrating EBS database tier to ODA BASE virtual machines, as most of the time we install EBS 12.2 on ODA using Oracle VM templates ,tus our database tier is placed in non-ODA Virtual machines.
Lastly, in order to migrate EBS database from non-ODA BASE vm machines to ODA BASE VM machines, we recommend you to follow the document "Solution-in-a-box: Best practices for deploying Oracle E-Business Suite Release 12.2.4 on Oracle Database Appliance" , which can be accessed by the url following:

The steps for migrating an EBS 12.2 database to ODA BASE is clearly documented in "Step 4:Migrate the Database to ODA_BASE"

Note that: if your EBS database version is lower than the Oracle RDBMS software version used in ODA BASE virtual machines, then we recommend you to upgrade your EBS database ersion to the same version as the ODA BASE virtual machines have, before beginning the migration process.
So, if that 's the case; you should follow the release specific documents :
  • If you are upgrading your EBS database to an 11G release such as, then -> 1623879.1, Interoperability Notes E-Business Suite Release 12.2 with Database 11g Release 
  • If you are upgrading your EBS database to an 12C release then ->: 1926201.1, Interoperability Notes Oracle EBS 12.2 with Oracle Database 12c Release 1

Thursday, January 14, 2016

EBS 12.0.4 , Discoverer 11g FRM-92101 no response from runtime process.

Recently upgraded Discoverer 10g Web to 11g in a EBS 12.0.4 customer environment.
After the upgrade, we integrated discoverer to EBS using EBS profiles and everything seemed fine till the end user tried to launch a form screen after closing a discoverer screen.
The problem was FRM-92101, and the message was no response from runtime process.
Customer was using Java 1.7 on the client machine.

Solution was setting "s_forms_tracking_cookies" to "enabled" in the applications context file and running autoconfig. When it is set to disabled, it just harms the session persistence it seems..

Tuesday, January 12, 2016

EBS 12.2 - Exalogic Installation Roadmap

There is no special installation guide for installing EBS 12.2 APPS Tier on Exalogic. (We say apps Tier, because it is recommended to have only the EBS Apps Tier in Exalogic)
By the way; the best practices is having an Exastack (Exalogic+Exadata) ( placing the Apps Tier in Exalogic, while placing the database tier in Exadata is recommended)

That 's why; we recommend following the Oracle EBS rapid install guide and Linux Specific Installation Guide..

Rapid Install Guide:
"Oracle E-Business Suite Installation Guide: Using Rapid Install Release 12.2 (12.2.0) Part No. E22950-22" (

Linux Specific Installation Guide:
Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.2) for Linux x86-64 ( Doc ID 1330701.1 )

But if you prefer installing EBS 12.2.4 in to the Virtual Machines residing in Exalogic, then the following document need to be followed instead;

Oracle Support Document: Deploying EBS 12.2 Oracle VM templates for Exalogic
Oracle VM Templates for Oracle E-Business Suite for Exalogic Deployment Guide, Release 12.2.4 (Doc ID 1954255.1)

ORA-27303 and ORA-27302 on instance startup (RAC to Single Clone)

If you need to clone a RAC database to a single node and if you copy an Oracle Home from one of the RAC nodes to the target single node for building an Oracle home in the target, then you may encounter the following error.

SQL> startup nomount;
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface failed bind. Check output from ifconfig command

Strange error right? We are on a single node, but still Oracle tries to bind to the interfaces like we are on a Oracle Cluster?
This error is actually an expected one. As we copy the Oracle Home Software from a RAC environment, the executables are linked with the RAC option.
In order to solve this, we relink the rdbms executable. Note that it is also good to check the environment, remove the unnecessary RAC related environments and relink the ioracle as well.

So the action plan is;

unset ORA_CRS_HOME and remove it from .bash_profile if it is there
unset cluster_interconnects db parameter

  • cd $ORACLE_HOME/rdbms/lib
  • make -f rac_off
  • make ioracle

Tuesday, January 5, 2016

EBS R12 - Apps Tier 32 bit or 64 bit?

When we talk about an 64 bit EBS, we actually talk about an 64 bit Oracle Database + an Application tier which have both 32 bit and 64 bit components.
So yes! Even if our EBS environments are an 64 bit we still have 32 bit components deployed and 32 bit code running our  EBS environments.
Except some of the 64 bit executables such as executables in the Advanced Planning product line, the EBS apps Tier is 32 bit. That's why we apply 32 bit version of patches in to the 10.1.2 and 10.1.3 Oracle Homes.
Even for application patches which we apply using adpatch, the situation is the same. We apply 64 bit version of these patches if they are available ..(these patches may deliver 64 bit files or may deliver 32 bit files, it may according to the situation, but we dont care), we just apply 64 bit adpatches if they are available, if tthere are not 64 bit patches, we apply the 32 bit patches.

What about 11i then? Why 11i apps tier is not certified and not supported in 64 bit Linux systems?
The answer is it is just not certified :) It is not optimized to run on 64 bit Linux Systems.
On the other hand; the difference between R12 apps tier and 11i apps tier is; EBS R12 apps tier, which also has 32 bit executables and components, is optimized to run on 64 bit Linux systems.

So keep that in mind, and 64 bit EBS R12 apps tier is not fully 64 bit, but it runs on 64 bit Linux servers stabily, because Oracle have made optimizations which let EBS R12 apps tier 32 bit code run on 64 bit servers.

Monday, January 4, 2016

EBS 12.2 - EXADATA installation roadmap

There is no official documentation released yet, but I see it is supported (OS and DB versions are supported anyways)
So here is the Road map for EBS 12.2 installation on Exadata.
  • Prepare the database nodes in Exadata
  • Download EBS 12.2 installation files, unzip them and create a stage 
  • Follow the “Oracle E-Business Suite Installation Guide: Using Rapid Install Release 12.2 (12.2.0), Part No. E22950-18” available on and “Using Oracle 11g Release 2 Real Application Clusters and Automatic storage management with Oracle E-Business Suite Release 12.2 (Doc ID 1453213.1)” to install EBS 12.2 on RAC. 
  • Follow the Document: 1617458.1 (Oracle E-Business Suite Release 12.2.4 Readme) to upgrade  EBS 12.2.0 to EBS 12.2.4.
  • Patch the EBS Oacle Homes deployed to Exadata(to CPU july 2015) or upgrade them( or 12c upgrade)
  • Follow Determine the documents: Database Patches Required by Oracle E-Business Suite on Exadata Database Machines ( Doc ID 1963786.1 ) , Database Patches Required by Oracle E-Business Suite on Oracle Engineered Systems: Exadata Database Machines and SuperClusters ( Doc ID 1392527.1) to determine and apply EBS 12.2 patches required for Exadata Database Machines.

EBS R12(12.0 and 12.1) - Roadmap for Windows to Linux Migration

In this post, I will give a roadmap for migrating an EBS R12 from Windows 32 bit to Linux 64 bit (Oracle Linux 5). Actually, this roadmap is applicable for any supported EBS migrations regardless of the source and Target Operating Systems.

Also a Real Life story which will give you the detailed patches, command and approaches used in this kind of a migration, will be released in this blog in the following days. So just be contented with this post, while I m preparing the other one.

Note that : These documents include directions to other documents as well.
If the target and source endians are the same, we use transportable database for the db migration and 
we install only the techstack and apply the patch that is supplied by Oracle in reply to our uploaded manifest of customer-specific files
  • root Document to follow -> R11i / R12: Oracle E-Business Suite Upgrades and Platform Migration (Doc ID 1377213.1)
  • Database Migration Document-> R12 (12.0/12.1): 'Using Transportable Database to migrate Oracle E-Business Suite Release 12.0 and 12.1 using Oracle Database 10g Release 2 or 11g Enterprise Edition' - MOS Document 734763.1
  • Application Migration Document -> Application Tier Platform Migration with Oracle E-Business Suite Release 12 (Doc ID 438086.1)