EBS 12.2 -- adop hangs -- adop problem-- case sensitive hostnames

Adop may hang /will not respond / will wait for input / will wait in read,  if your hostname does not match the hostname in $CONTEXT_FILE.

I have seen this even with the following scenario;

Machine's hostname is set to be "VisionR12", but the hostname in $CONTEXT_FILE is "visionr12"
adop hangs/waits on read system call..
Even a difference in case sensitivity may trigger this problem..
We have to pay attention for that.
Dont use Big letters in your hostnames at all. No need.

Linux-- Setting the hostname FQDN or Short? --a detailed approach , + a look from EBS perspective

EBS Application Tier processes running on Linux may encounter problems because of a wrong hostname setting of the Operating System. Thus the hostname we set for Linux must be appropriate.

Why appropriate? Because FNDLIBR uses the hostname it gathers from the kernel.

Lets use the strace utility to see what the process is doing when our start script executes FNDLIBR;
strace FNDLIBR FND FNDCPBWV apps/apps SYSADMIN 'System Administrator' SYSADMIN.
Okay.. I will not copy&paste the entire trace here, but the obvious thing is that FNDBLIR uses uname calls and gets the hostname..

uname({sys="Linux", node="ermanhost.domain.com", ...}) = 0

Note that, uname command also gets the hostname using uname system call. On success, zero is returned.

The hostname comes from the below strucute ;

struct utsname {
char sysname[]; /* Operating system name (e.g., "Linux") */
char nodename[]; /* Name within "some implementation-defined
network" */
char release[]; /* Operating system release (e.g., "2.6.28") */
char version[]; /* Operating system version */
char machine[]; /* Hardware identifier */
#ifdef _GNU_SOURCE
char domainname[]; /* NIS or YP domain name */
#endif
};

So using uname , FNDLIBR obtains the hostname from the kernel.

To demonstrate, I'll write a little C program and execute it while tracing with strace;

Our program to get and display the uname using struct data;

#include <stdio.h>

#include <sys/utsname.h>

int main ()

{

struct utsname u;

uname (&u);

printf (“%s release %s (version %s) on %s\n”, u.sysname, u.release, u.version, u.machine);

return 0;

}

We compile it;

gcc /tmp/ourprogram.c

We run it
It display the following.

./a.out
Linux release 2.6.32-100.26.2.el5 (version #1 SMP Tue Jan 18 20:11:49 EST 2011) on x86_64

When we trace it using strace;

./a.out 

execve("./a.out", ["./a.out"], [/* 22 vars */]) = 0

brk(0)                                  = 0xd2d000

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb55dc02000

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb55dc01000

access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)

open("/etc/ld.so.cache", O_RDONLY)      = 3

fstat(3, {st_mode=S_IFREG|0644, st_size=107884, ...}) = 0

mmap(NULL, 107884, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb55dbe6000

close(3)                                = 0

open("/lib64/libc.so.6", O_RDONLY)      = 3

read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\332a\0335\0\0\0"..., 832) = 832

fstat(3, {st_mode=S_IFREG|0755, st_size=1722304, ...}) = 0

mmap(0x351b600000, 3502424, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x351b600000

mprotect(0x351b74e000, 2097152, PROT_NONE) = 0

mmap(0x351b94e000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14e000) = 0x351b94e000

mmap(0x351b953000, 16728, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x351b953000

close(3)                                = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb55dbe5000

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb55dbe4000

arch_prctl(ARCH_SET_FS, 0x7fb55dbe46e0) = 0

mprotect(0x351b94e000, 16384, PROT_READ) = 0

mprotect(0x351b41b000, 4096, PROT_READ) = 0

munmap(0x7fb55dbe6000, 107884)          = 0

uname({sys="Linux", node="ermanhost.domain.com", ...}) = 0

fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 4), ...}) = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb55dc00000

write(1, "Linux release 2.6.32-100.26.2.el"..., 90Linux release 2.6.32-100.26.2.el5 (version #1 SMP Tue Jan 18 20:11:49 EST 2011) on x86_64

) = 90

exit_group(0)  

We see this program calls uname syscall and that returns the same thing as FNDLIBR's syscall..

uname({sys="Linux", node="ermanhost.domain.com", ...}) = 0

Then, we can say  that at the lowest level , FNDBLIR does the same thing to retrieve the hostname , same as our little program does.

Note that : hostname information is available in the proc file system too.

cat /proc/sys/kernel/hostname

ermanhost.domain.com

Also , hostname command can be used to display the hostname too.

hostname -f

ermanhost.domain.com

hostname

ermanhost.domain.com

But wait... You see, hostname -f and hostname returns the same thing in this system. "-f argument" is used to display FQDN ,but what about the output of hostname with no arguments??*

Actually they shouldnt return the same thing because;

hostname will print the name of the system as returned by the  gethost-name function.

The FQDN is the name gethostbyname returns for the host name returned by gethostname.

So in this system gethostname and gethostbyname return the same thing - > FQDN..

In other words; in this system hostname returns FQDN from everywhere..

Lets see what these gethostname and gethostbyname functions are...

int gethostname(char *name, size_t len);

gethostname() returns the null-terminated hostname in the character array name, which has a length of len bytes. If the null-terminated hostname is too large to fit, then the name is truncated, and no error is returned (but see NOTES below). POSIX.1-2001 says that if such truncation occurs, then it is unspecified whether the returned buffer includes a terminating null byte.

struct hostent *gethostbyname(const char *name);

The gethostbyname() function returns a structure of type hostent for the given host name. Here name is either a hostname, or an IPv4 address in standard dot notation If name is an IPv4 or IPv6 address, no lookup is performed and gethostbyname() simply copies name into the h_name field and its struct in_addrequivalent into the h_addr_list[0] field of the returnedhostent structure...
Basically; gethostbyname returns FQDN for the host name returned by gethostname.

When we trace the commands hostname and hostname -f , we see that "hostname -f" reaches nsswitch.conf and host.conf file. So it is network aware..

read(3, "#\n# /etc/nsswitch.conf\n#\n# An ex"..., 4096) = 1698
open("/etc/host.conf", O_RDONLY) ;

The nsswitch.conf file(The Name Service Switch (NSS) configuration file), /etc/nsswitch.conf, is used by the GNU C Library to determine the sources from which to obtain name-service information in a range of categories, and in what order.

When we open the /etc/nsswitch.conf;

we see the following line;

hosts:      files  dns

So this nsswitch.conf says that the system first attempts to resolve host names and IP addresses by querying files and if that fails, it tries querying a DNS server.

So the gethostbyname which is used by the hostname -f command reads /etc/nsswitch.conf and /etc/host.conf to decide whether to read information in /etc/sysconfig/network or /etc/hosts.

Note that :

We have fully qualified hostname defined in /etc/sysconfig/network.

cat /etc/sysconfig/network

NETWORKING=yes

NETWORKING_IPV6=no

HOSTNAME=ermanhost.domain.com

We have a lot of lines in start scripts which uses this file;

cd /etc/rc.d

grep -R /etc/sysconfig/network *|wc -l

317

When we open /etc/host.conf 

we see the following line;

order hosts,bind

This means  -> "first , use /etc/hosts to retrive the hostname , if you cant find it then try dns query"

So nsswitch.conf and host.conf say pretty much the same thing here.  So why do we have both of them?

It seems because the older Linux standard library, libc, used /etc/host.conf as its master configuration file, but new GNU standard library, glibc, uses /etc/nsswitch.conf.

Insteresting thing is that;

hostname -s command , which is used for returning the shortname of the servers uses /etc/nsswitch.conf and /etc/host.conf files, returns the short hostname as expected.

So what do we have so far;

hostname -s retruns ermanhost   (GOOD)
hostname -f returns ermanhost.domain.com (GOOD)
hostname returns ermanhost.domain.com  (BAD) ..

hostname command should not return ermanhost.domain.com(FQDN) when it is called without any arguments..

It uses gethostname as expected, but it should not return the FQDN..

As I mentioned above this might be related with the HOSTNAME defined as FQDN in /etc/sysconfig/network. 

Lets explore this ;

When we execute hostname command 

hostname is derived from -> uname directly(without going nsswitch.conf or host.conf -> uname derives this info from the kernel structrure -- > So we need to know ;

What does set this hostname as FQDN in the structure.

When is it set?

What is the configuration file that is used by the cede that sets the hostname as FQDN in the kernel structure?  ( I suspect this is /etc/sysconfig/network bytheway)

So we need to have a look to the boot process of Redhat Based Linux and here is it ;

I m not gonna startup from the Bios :)

Here is the info we need  :

When the init command starts, it becomes the parent or grandparent of all of the processes that start up automatically on the system. First, it runs the /etc/rc.d/rc.sysinit script....

When we look at the startup scripts , we see the following line in /etc/rc.sysinit file;

if [ -f /etc/sysconfig/network ]; then

    . /etc/sysconfig/network

We also see the following lines;

# Set the hostname.

update_boot_stage RChostname

action $"Setting hostname ${HOSTNAME}: " hostname ${HOSTNAME}

So , we have found the command that sets the hostname..

It basically gets the hostname from the /etc/sysconfig/network file at sets the hostname accordingly..

So far so good.. We know what we need to know about setting & getting the hostname in Linux.

Lets summarize the gathered info, make our comments and describe the best practice for setting hostnames in Linux :

• HOSTNAME in /etc/sysconfig/network should be the machine name- not the FQDN. 'hostname' should ideally simply return the actual hostname.
• /etc/resolv.conf must be properly configured for searching the domain.
• /etc/hosts mut be properly configured to contain both FQDN and machine name.

DEMO:

False setting: 

[root@ermanhost ~]# cat /etc/sysconfig/network

NETWORKING=yes

HOSTNAME=ermanhost.ermandomain.com

[root@ermanhost ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain

10.34.50.104 ermanhost.ermandomain.com ermanhost

[root@ermanhost ~]# hostname -s

ermanhost

[root@ermanhost ~]# hostname -f

ermanhost.ermandomain.com

[root@ermanhost ~]# hostname

ermanhost.ermandomain.com --> this shouldnt be FQDN

Good setting:

[root@ermanhost ~]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=ermanhost
[root@ermanhost ~]# hostname
ermanhost                    -->>   GOOD
[root@ermanhost ~]# hostname -f
ermanhost.ermandomain.com
[root@ermanhost ~]# hostname -s
ermanhost

Note that: if you change the /etc/sysconfig/network without rebooting the server, your hostname will still use the old hostname.. To change hostname in linux you need to issue hostname "newname" command , and then you must change the /etc/sysconfig/network file with the new hostname.. We change the /etc/sysconfig/network for making the change permenant after reboot.

Now lets see the effect of a wrong hostname setting in EBS ;

Note that : this is applicable for 11i, R12 and 12.2

Here it is documented as follows;

Concurrent Managers Fail To Start After New Install of Release 12 (Doc ID 413164.1)   

Basically, when we try to start concurrent managers in a machine with a long hostname; we end up with the following and concurrent managers cant be able to start.

ERROR
APP-FND-01564: ORACLE error 12899 in insert_icm_record
Cause: insert_icm_record failed due to ORA-12899: value too large for column "APPLSYS"."FND_CONCURRENT_PROCESSES"."NODE_NAME" (actual: 31, maximum: 30)

This is because the column in FND_CONCURRENT_PROCESSES table is VARCHAR2(30).

So we need to use a hostname which must be maximum 30 chars long.

But what if we need to have long fully qualified DN?

Then, we need to apply the approach as I wrote above.(The Good setting)

That's all what I need to say about this topic.

Okay.. In this article , we have learned the logic behind the hostname setting of linux. We have worked with strace , written a c program to get the hostname from the kernel structure, reviewed the boot process of linux  , stated the proper setting for hostname and lastly seen this information in action on  a EOracle BS problem.

I hope you 'll find it useful.

Linux -- a handy bash script for time syncronization, ntpdate

This one is a handy script for time syncronization.. It basically does a ntpdate..
If you dont want to use ntp daemon or do not have a ntp daemon, put this script on crontab and sync your time accordingly.

ntpdate sets the local date and time by polling the Network Time Protocol (NTP) server(s) given as the server arguments to determine the correct time. It must be run as root on the local host. A number of samples are obtained from each of the servers specified and a subset of the NTP clock filter and selection algorithms are applied to select the best of these. Note that the accuracy and reliability of ntpdate depends on the number of servers, the number of polls each time it is run and the interval between runs.

#!/bin/sh
NTPDATE=/usr/sbin/ntpdate
SERVER="YOUR NTP SERVERS IP ADDRESS HERE"

if ! test -t 0; then
  MYRAND=$RANDOM
  MYRAND=${MYRAND:=$$}

  if [ $MYRAND -gt 9 ]; then
    sleep `echo $MYRAND | sed 's/.*\(..\)$/\1/' | sed 's/^0//'`
  fi
fi

$NTPDATE -su $SERVER
if [ -f /sbin/hwclock ]; then
  /sbin/hwclock --systohc
fi

EBS 12.2 -- ADOP /Online Patching Common Errors --script timed out,resource unavailable,outofMemory and more..

Logic behind the Adop utility and Online Patching is complex, even if we cant feel it from the surface..

A lot of actions are taken by the patching utility and there seem to be a lot of dependencies.
Because of these complexity; sometimes applying a little patch becomes a throuble.. 

I have seen this in adop's prepare or apply phases mostly.. 

I have seen a lot of resource related errors .. Especially in Virtual environments..
Improperly added Custom Tops may also trigger errors during the patching cycle.

For more details about adop; check the following document for the infomation about adop..

http://ermanarslan.blogspot.com.tr/2014/03/ebs-122-applications-dba-online.html

These kind of little problems can make you spend more effort while performing patching operations in EBS 12.2.
Anyways; I m bringing these kind of error together.. 

These errors will make this article to be a good thing to check before creating a patch cycle.

"Like Bruce Lee said: The less effort, the faster and more powerful you will be." :)

Error : Script timed out.. Script Executed in 7200113 milliseconds, returning status -1

This error may be reported during a fs_clone or prepare.. During the pasteConfig.sh run.

Note that: The script may take some time depending on the server resources (CPU and RAM).

May be encountered in Low Cpu and Low Memory environments..

Running /u01/oracle/TEST/fs2/FMW_Home/oracle_common/bin/pasteConfig.sh -javaHome /u01/oracle/TEST/fs1/EBSapps/comn/adopclone_erptest/FMW/t2pjdk -al /u01/oracle/TEST/fs1/EBSapps/comn/adopclone_erptest/FMW/WLS/EBSdomain.jar -tdl /u01/oracle/TEST/fs2/FMW_Home/user_projects/domains/EBS_domain_TEST -tmw /u01/oracle/TEST/fs2/FMW_Home -mpl /u01/oracle/TEST/fs1/EBSapps/comn/adopclone_erptest/FMW/WLS/plan/moveplan.xml -ldl /u01/oracle/TEST/fs1/inst/apps/TEST_erptest/admin/log/clone/wlsT2PApply -silent true -domainAdminPassword /u01/oracle/TEST/fs1/EBSapps/comn/adopclone_erptest/FMW/tempinfo.txt
Script Executed in 7200113 milliseconds, returning status -1
Script timed out.

Solution: Increasing the timeout will fix this.

1)Open a new terminal.
2) Execute :
export TIMEDPROCESS_TIMEOUT=-1
3. Execute :
adop phase=fs_clone or adop_phase=prepare  -- according to your requirements..

Error : ERRORMSG: Adsplice action did not go through successfully. 

Inside evalADPATCHStatus()...
message_status: ERROR
Adsplice action did not go through successfully.
*******FATAL ERROR*******
PROGRAM : (/oracle/product/prp_apps/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME : Fri Sep 19 16:37:58 2014
FUNCTION: main::execADSPLICE [ Level 1 ]
ERRORMSG: Adsplice action did not go through successfully.

SQL Command: SELECT status||',' FROM ad_adop_session_patches WHERE node_name = 'erptest' AND applied_file_system_base = '/u01/oracle/TEST/fs1' AND patch_file_system_base IS NULL AND bug_number = 'ADSPLICE_YOURCUSTOMTOP' ORDER BY TO_CHAR(end_date,'YYYY.MM.DD:HH24:MI:SS') DESC
patch_status = Y
Updated patch_status = Y
EXIT STATUS: 1

Solution: This is because of an erroneous Custom top adding operation .. There is an improper Custom top definition in the System .

1)Remove the following entries from the adxmlctx.tmp file:
<oa_customized>
<c_fss oa_var="c_fss" scope="CUSTOM" oa_type="PROD_TOP" oa_enabled="FALSE" default="/u01/sim2/fs1/EBSapps/appl">%c_fss%</c_fss>
</oa_customized>
</oa_context>
2. Run autoconfig .
3. Re-run fs_clone.

If this does not fix the error, add your custom again/the same custom top, but this time properly.

For adding the custom top: http://ermanarslan.blogspot.com.tr/2014/05/ebs-122-add-custom-top.html

Error:  Exception in thread "Thread-1" java.lang.OutOfMemoryError: Java heap space

Exception in thread "Thread-1" java.lang.OutOfMemoryError: Java heap spaceat java.util.Arrays.copyOfRange(Arrays.java:2694)

at java.lang.String.(String.java:203)
at java.io.BufferedReader.readLine(BufferedReader.java:349)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at com.oracle.cie.common.util.CRLF.readData(CRLF.java:129)
at com.oracle.cie.common.util.CRLF.processFile(CRLF.java:67)
at com.oracle.cie.common.util.CRLF.process(CRLF.java:57)
at com.oracle.cie.domain.util.stringsub.SubsScriptHelper.processCRLF(SubsScriptHelper.java:167)
at com.oracle.cie.domain.DomainGenerator.generate(DomainGenerator.java:478)
at com.oracle.cie.domain.script.ScriptExecutor$2.run(ScriptExecutor.java:2992)

Solution: The Xms and Xmx parameters of java are not enough for the adop..

1. Run the following command before executing adop .. In the same terminal... export CONFIG_JVM_ARGS="-Xms1024m -Xmx2048m"
2. Run adop again.

If this does not fix the problem; increase the parameters a little more.

1. Run the following command before executing adop .. In the same terminal...
export CONFIG_JVM_ARGS="-Xms1500m -Xmx3000m"
2. Run adop again.

Error:  AutoPatch error: Missing file format id in file {YOUR _APPL_TOP LOCATION}/admin/applcust.txt

This version of AutoPatch requires a file format id on the first line of the file.
Error reading customized files file
Freeing includes hash table
Freeing fixes hash table
Freeing basedons hash table
Freeing entities hash table

Solution: Rename the applcust.txt file : mv applcust.txt applcust_BAK.txt , and retry the patching operation.
If renaming the file wont fix the error; check out following Support note: Doc ID 1227113.1

Error: [ERROR][thread ] Could not start thread ExecuteThread: '4' for queue: 'default'. Resource temporarily unavailable

Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
Connecting to t3://yourhost:7002 with userid weblogic ...
[ERROR][thread ] Could not start thread weblogic.timers.TimerThread. Resource temporarily unavailable
This Exception occurred at Wed Nov 05 21:54:26 EET 2014.
java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1096).
Attempting to allocate 4G bytes
There is insufficient native memory for the Java
Runtime Environment to continue.
Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit

Solution: Restart your system to be a clean enviroment.. We need free memory here , or clear your env to have more free memory. Also you may decrease your Xmx using CONFIG_JVM_ARGS="-Xms1024m -Xmx2048m" before starting adop.

Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Disable compressed references (-XXcompressedRefs=false)

EBS 12.2 -- adop apply-mode=downtime -- Patching downtime is back!

After a while, after we have face with a lot of throubles -- especially in Virtual Machines and Machines with weak hardware; today we are happy to find out that downtime for the EBS patches have come back :)
Actually it is optional to apply a patch with downtime, but it is more fast and it requires less system resources  as it seems.. On the other hand; with downtime mode; you 'll have an increased system downtime..
As you know, in EBS 12.2; we dont have a maintanence mode anymore .. That's why; to apply a patch with downtime , we only need to stop our application tier services..

What we need to do is ;

1)Source our run environment.
2)Stop our application services
3)adop phase=apply apply_mode=downtime patches=PATCH_NUMBER

With apply_mode=downtime ; adop directly applies the patch .. No patching cycles...

But we need to face the facts;
As Oracle states;

• Release 12.2 patches are not normally tested in downtime mode.
• Downtime mode is only supported for production use where explicitly documented, or when directed by Oracle Support or Development.

So unless Oracle supports say that "This patch can be applied using apply_mode=downtime";  you need to solve the problems you may face during the adop downtime patching , by yourself..

There are some examples for these kind of situations;

• Is it supported to apply 17050005 R12.HR_PF.C.delta.4 in downtime mode ? (Doc ID 1916385.1)   ->  If customer is using AD-TXK Delta 5 or above, then it is supported to apply the patch R12.HR_PF.C.delta.4 (17050005) in downtime mode.
• Is It Possible To Apply ALL Patches During An Upgrade From 11i To 12.2.4 With Apply_mode=downtime (Doc ID 1918842.1) -> Can all post-12.2.4 patches can be applied in downtime mode? Yes, all post-12.2.4 patches can be applied in downtime mode as long as the Applications Tier processes have not been started (first time after upgrade).  Can downtime mode be used to apply patches once the upgrade is complete?  No.  Once the system is open for users, all subsequent patches must be applied on-line unless otherwise stated in the Readme or corresponding Note
• Here another important doc: Oracle E-Business Suite Release 12.2.4 Readme (Doc ID 1617458.1) --> adop phase=apply apply_mode=downtime patches=17919161

Okay.. In my opinion;  Oracle suggests and supports this downtime apply mode for the fresh installations, upgrades and  for some specific product upgrades at the moment;
When the system is started to be used by the users ; in other words when we put data into it/ when we start the business processes,  we need to apply our patches online unless otherwise started in the Readme or corresponding note.

To be able to use dowtime mode; you need to upgrade AD-TXK Delta 5, or you need to upgrade to 12.2.4.. Note that you need  to upgrade your AD-TXK DELTA 5 prior to 12.2.4 upgrade anyways :) When you apply AD TXK Delta 5 as a prereq for 12.2.4, you may even use downtime mode during the application of your 12.2.4  upgrade patch.

For  AD-TXK Delta 5 upgrade >

Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2 (Doc ID 1617461.1)

For 12.2.4 Upgrade >
Oracle E-Business Suite Release 12.2.4 Readme (Doc ID 1617458.1)

EBS 12.2 -- Adop Online Patching Cycle -- Restricted Functionalities

While Apps Dbas are patching the EBS online using adop tool, an online patching cycle is created.. As the name of it is "online patching" , we APPS DBAs may start to system without taking any approvals from the business side.

We may think; that the patching activity is online and only a 2 minutes downtime is required when the time comes for the switch over operation where our middle tier services are restarted..

Some may even think that; keeping the system always prepared(adop phase=prepare) for the future patch applications is a good thing..
I mean, as adop phase=prepare may take a long time, we may think that it is logical to complete a prepare operation in advance without a need to apply any patches -- to just to be prepared.. or we may think; that  it is like a preclone :) But it is not!

Following is the product functionalities which will be restricted during an online patching cycle.. (until it is completed by abort , or usual way.)
That's why, even it is online ,we still need to  take approvals before starting a patching activity..

• Payroll
  
  • Users will not be able to define Fast Formulas or use the Fast Formula Assistant.
  • Users will not be able to perform dynamic trigger maintenance.
  • Users will not be able to create, update, or delete US Cities.
  • Data Pump meta-mapper generator will be disabled.
  • The Japanese Balance Dimensions concurrent program will be deferred to after the cutover phase is complete.
  • Pension Calculation Setup cannot be used.
  • US localization earnings and deduction setup cannot be used.
  • Tax Withholding Rules Setup cannot be used.
  • Wage Attachment Earnings Rules Setup cannot be used.
  • Garnishment Rules Setup cannot be used.
  • Quick Paint Reports cannot be used.
  • Quantum Program Update Installer execution is unavailable.
• Order Management:
  
  • Creation of a new Defaulting Condition in the Attribute Defaulting Rules form is disabled, unless the same seeded condition already exists for a given attribute.
• Warehouse Management:
  
  • WMS Rule creation is restricted.
• Inventory:
  
  • Concurrent program “Generate Stock Locator Flexfield Definition for Mobile Transactions” will be disabled.
• Public Sector Financials International:
  
  • Users will not be able to run the following concurrent programs:
    
    • Subledger Security: Apply Security
    • Subledger Security: Import/Export Data Fix
• Subledger Accounting:
  
  • Users will not be able to Validate the Application Accounting definitions.
• Accounts Receivable:
  
  • Users will not be able to create new Transaction Sources.
• Incentive Compensation:
  
  • Transaction collection process for new mappings will not be available and any changed mapping will continue to use previous mapping rules.
  • Users will not be able to run the “Synchronize Classification Rulesets” program.
  • Users will not be able to use the “Formula Generation” feature.
  • Users will not be able to specify new formulas or changes to compensation rules.
• Oracle Demand Planning:
• Demand plans will not be available for users.

Finally, If your users try to do a restricted operation (like validating Application Account definitions) during the patching cycle; they will see an error like the following;

Note that ; this error is taken from a turkish customer , so the error message will differ according to your language. The error says; Patch application process is continuing .. Retry Validating Account Definition after the patch the patch application.

Exadata/RAC -- Investigating TNS-12514/Vip failover/Tns failover/crs_relocate -- a detailed approach

Last week, a TNS-12154 error was reported for one of the Exadata nodes.During the problem hours, as you may guess,  Clients / Apps services could not establish their connections to the back end database server..

A workaround would be to use the second node( by changing the dns) to reach the database, as In Exadata , we have at least 2 db nodes working in the RAC infrastructure.

Using a load balance-or-failover based tns would also help , but the issue was critic and needed to be fixed immediately.

In the problematic environment, EBS 11i was used as an Enterprise ERP application.. Number of Clients, which need to connect to the database was less than 10 and the clients and EBS tecstack were using the Tns entries based on virtual ip addresses of Exadata Database Machine Nodes.

As 11i could not use Scan listener, I was lucky :) This meant a decrease in things to check :)
Note that ; for scan listeners, youc an check the following http://ermanarslan.blogspot.com.tr/2014/02/rac-listener-configuration-in-oracle.html

Okay , I directly jumped in to the first node, because the error TNS-12514 was reported for the connections towards the instance 1. 

Before going further, I want to explain the TNS-12514 /ORA-12514 error typically.

When you this error, you can think that your connection request is transferred to the listener, but the service name or sid that you have provided is not listened by the listener.  In other words, the database service that you want to connect and specified in your connection request is not registered with the listener.

This may be a local_listener parameter problem, or it may be a problem directly caused by the listener process.. Okay. We will see the details in the next paragraph..

So, when I connect to the first node; I saw that listener was running. Actualy, It was normal, because the error was not an "No Listener" error.  So, I directly restarted it.. I wanted to have a clean environment..

While starting the listener , I saw the following in my terminal.

<msg time='2014-10-26T10:54:16.302+02:00' org_id='oracle' comp_id='tnslsnr'

 type='UNKNOWN' level='16' host_id='osrvdb01.somexampleerman.net'

 host_addr='10.10.10.100'>

 <txt>Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=exa01-vip)(PORT=1529)(IP=FIRST)))

 </txt>

</msg>

<msg time='2014-10-26T10:54:16.302+02:00' org_id='oracle' comp_id='tnslsnr'

 type='UNKNOWN' level='16' host_id='exa01.blabla.net'

 host_addr='10.10.10.100'>

 <txt>TNS-12545: Connect failed because target host or object does not exist

 TNS-12560: TNS:protocol adapter error

  TNS-00515: Connect failed because target host or object does not exist

   Linux Error: 99: Cannot assign requested address

Okay, the problem was there .. Especially the line ->

Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=exa01-vip)

On the other hand this was not the reason behind the problem, it was the result of the problem. 

The real causes were the following lines:

 TNS-00515: Connect failed because target host or object does not exist

   Linux Error: 99: Cannot assign requested address

So , this was the error stack, and as it is a stack,we can understand that the first error was Linux Error 99.

Linux Error 99 which came from the Exadata Compute Node's Operating System (it is Oracle Linux) means ; 

errno.h:
99 EADDRNOTAVAIL Cannot assign requested address

Wow what a surprise! :)

I guess  the Listener uses bind here ->

int bind (int socket, struct sockaddr *addr, socklen_t length)

And , it failed because , the specified address exa01-vip,  was not available on this database node.

The code could be something like the following;

#include <sys/socket.h>

⋮

int rc;

int s;

struct sockaddr_in myname;

⋮

memset(&myname, 0, sizeof(myname));

myname.sin_family      = AF_INET;

myname.sin_port        = 1529;

myname.sin_addr.s_addr = inetaddr("192.168.0.91");

rc = bind(s, (struct sockaddr *) &myname, sizeof(myname));  /*virtual ip address of Node1*/ --> I guess at this point, the code should be broken with EADDRNOTAVAIL)

"TNS-00515: Connect failed because target host or object does not exist" was also saying the same think , but in Oracle's Language..

Okay..

After analyzing the above, I used ifconfig command to see whether the vip was there or not..

Yes.. The ifconfig command did not listed the vip..

Then I try to ping the vip .. The ping was okay. The ip was up , but not in its original node (node1 in this case).. So, it was normal to get an error while starting the associated listener ..
When I login to the 2nd node and run ifconfig, I saw that the vip interface of the 1st node is up in the 2nd node.
So my thoughts were true.. The vip of node 1 was not present in 1st node.. On the other hand; it was present in the 2nd node and this was the cause of this problem..

But why and how this vip interface was migrated to 2nd node?

To answer these questions; I first checked the messages of Linux running in Node1, and saw the following;

Oct 26 09:10:33 osrvdb01 kernel: igb: eth2 NIC Link is Down
Oct 26 09:10:33 osrvdb01 kernel: igb: eth1 NIC Link is Down
Oct 26 09:10:33 osrvdb01 kernel: bonding: bondeth0: link status down for idle  interface eth1, disabling it in 5000 ms.
Oct 26 09:10:33 osrvdb01 kernel: bonding: bondeth0: link status down for idle  interface eth2, disabling it in 5000 ms.
Oct 26 09:10:38 osrvdb01 kernel: bonding: bondeth0: link status definitely down for interface eth1, disabling it
Oct 26 09:10:38 osrvdb01 kernel: bonding: bondeth0: link status definitely down for interface eth2, disabling it

So the problem was obvious , there was  a failure with the network links, which were related with the virtual interface..(Note that : the client was made a system operation and changed the switches :) , this was a result of that.. It was not Exadata's fault :))
So Linux in node 1 seems detected and disabled the bondeth0 interface because of this Link errors.
It was normal that disabling bondeth0 made the public ip and virtual ips of Node 1 to become unavailable on node1.. And as a result, they were migrated to 2nd nodes.. (This is Rac :))

Now, we came to a point to know the "thing" that migrated this vip interface from node 1 to node2.

To find this thing, I checked the RAC logs..

In listener log , which was located in Grid Home, listener was saying that "I m no longer listening from exa01-vip) This was true because,  when I first checked the server, I saw the listener was up , but could not listen to the vip address ..

Then I check the crsd.log ;

In crs log, I saw the following;

Received state change for ora.net1.network exadb01 1 [old state = ONLINE, new state = OFFLINE]

So , it seems the crsd understood that the network related to the problematic interface became down..

I saw the restart attempts , too..

CRS-2672: Attempting to start 'ora.net1.network' on 'exadb01'

The attempts were failing..

CRS-2674: Start of 'ora.net1.network' on 'exadb01' failed
quencer for [ora.net1.network exadb01 1] has completed with error: CRS-0215: Could not start resource 'ora.net1.network'.


Then I saw the vip was failed over.. It was migrated from node 1 to node 2 as follows;

2014-10-26 10:21:56.371: [   CRSPE][1178179904] {0:1:578} RI [ora.exadb01.vip 1 1] new external state [INTERMEDIATE] old value: [OFFLINE] on ecadb02 label = [FAILED OVER]
2014-10-26 10:21:56.371: [   CRSPE][1178179904] {0:1:578} Set LAST_SERVER to exadb02 for [ora.exadb01.vip 1 1]
2014-10-26 10:21:56.371: [   CRSPE][1178179904] {0:1:578} Set State Details to [FAILED OVER] from [ ] for [ora.exadb01.vip 1 1]
2014-10-26 10:21:56.371: [   CRSPE][1178179904] {0:1:578} CRS-2676: Start of 'ora.exadb01.vip' on 'exadb02' succeeded

So, te failover was done by the Clusterware...

Okay.. But what was the advantage or benefit of this failover?, This question comes to minds as the clients still were not able to connect to PROD1 even if the corresponding vip was failed over & up on node2.

The purpose of this failover is to make the vip of node1 to be available on node2 . Thus, the connection attempts by the clients towards the node1's listener/vip encounter "Tns no listener" errors without waiting the TIME (TCP TIMEOUT)..

Ofcourse the clients should use a tns that supports this kind of failover; like the following;

PROD =
(DESCRIPTION =
(ADDRESS=(PROTOCOL=TCP)(HOST=exavip1)(PORT=1521))
(ADDRESS=(PROTOCOL=TCP)(HOST=exavip2)(PORT=1521))
(CONNECT_DATA =
(SERVICE_NAME = PROD)
)
)

So by using a tns like above, clients will first go  to the exavip , they will reach the vip and encounter errors immediately.. Then they will go to the second vip and connect to the database.. 

This is te logic of using Vips in Oracle Rac.

Okay. So far so good. We analyzed the problem , found the causes and saw the mechanism that have made the failover.. Now we will see the solution..

The solution I applied was as follows;

[root@exa02 bin]# crs_relocate ora.exadb01.vip
Attempting to stop `ora.exadb01.vip` on member `exadb02`
Stop of `ora.exadb01.vip` on member `exadb02` succeeded.
Attempting to start `ora.exadb01.vip` on member `exadb01`
Start of `ora.exa01.vip` on member `exadb01` succeeded.
Attempting to start `ora.LISTENER.lsnr` on member `exadb01`
Attempting to start `ora.LISTENER_PROD.lsnr` on member `exadb01`

Start of `ora.LISTENER_PROD.lsnr` on member `exadb01` succeeded.

So , I basically used the crs_relocate utility...

Here is the general definition of the crs_relocate utility.

The crs_relocate command relocates applications and application resources as specified by the command options that you use and the entries in your application profile. The specified application or application resource must be registered and running under Oracle Clusterware in the cluster environment before you can relocate it.

That is it.
In this article, we have seen a detailed approach for diagnosing network errors in Exadata (actually in RAC)..
We have seen the Vip failover, and the logic of using Vips in RAC.
Lastly we have seen the crs_relocate utility to migrate the vip to its original location/node.

I hope you will find it useful.. Feel free to comment.

RDBMS, Java -- Working with the Java inside the Oracle Database

It all started with a Java Source compilation error. The problematic code was a Java Source, which was tried to be compiled inside the Oracle Database..
The code was written to make a web service call remotely using the Oracle Database using Java. The creation script was starting as follows;

CREATE OR REPLACE AND RESOLVE JAVA SOURCE NAMED "BLABLA"
AS import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.Reader;
import java.nio.charset.Charset;
import java.sql.Clob;
import java.sql.SQLException;
import javax.xml.soap.MessageFactory;
import javax.xml.soap.MimeHeaders;
import javax.xml.soap.SOAPConnection;
import javax.xml.soap.SOAPConnectionFactory;

import javax.xml.soap.SOAPException;
import javax.xml.soap.SOAPMessage;

public class WebServiceCall {
private static SOAPMessage getSoapMessageFromString(String xml) throws SOAPException, IOException {
MessageFactory factory = MessageFactory.newInstance();
SOAPMessage message = factory.createMessage(new MimeHeaders(), new ByteArrayInputStream(xml.getBytes(Charset.forName("UTF-8"))));
return message;
}

......

And it was continuing, without any syntax errors. It was actually compiled in an 11.2.0.4 Oracle database without any problems, but when it have come compiling it in an 11.2.0.3 Oracle Database , the customer had throubles, which made me writing this post ..

As you may expect, the Java Souce could not be compiled on 11.2.0.3..

The key errors were as follows;

09:54:01 AS import java.io.BufferedReader;
09:54:01 ...
09:54:02 ORA-24344: compilation error
09:56:01 Start Compiling 1 object(s) ...
09:56:01 Executing ALTER JAVA SOURCE blabla COMPILE ...
09:56:02 [0:0] blabla:10: cannot find symbol
09:56:02 [0:0] symbol : class MessageFactory
09:56:02 [0:0] location: package javax.xml.soap
09:56:02 [0:0] import javax.xml.soap.MessageFactory;
09:56:02 [0:0] blabla:11: cannot find symbol
09:56:02 [0:0] symbol : class MimeHeaders
09:56:02 [0:0] location: package javax.xml.soap
09:56:02 [0:0] import javax.xml.soap.MimeHeaders;
09:56:02 [0:0] blabla:12: cannot find symbol
09:56:02 [0:0] symbol : class SOAPConnection
09:56:02 [0:0] location: package javax.xml.soap

..

...

.....

And a lot more ....

Okay, the cause of these errors were the JDK embedded / inside the Oracle Database..  It was clear that some packages could not be found , especially javax.xml.soap..

The developers were thinking that at least Oracle 11.2.0.4 was needed to compile such a Java Source, as it was clear that 11.2.0.3 didnt have the necessary java packages.. That's why an 11.2.0.4 upgrade was requested immediately..

On the other hand; the customer's Dba didnt approve this request because it was only about a single Java object.
At this point , I step in, made the following analysis and  solved the problem without a need to upgrade the entire database.

It was a productive day, which made me do practices in Java.

First of all, there is no document or a whitepaper saying that 11.2.0.4 is needed for compiling these kind of Java Souce objects..

The only statement I could find in Oracle, was  "Release 11.2.0.4 provides an enterprise class platform, Oracle JVM, for developing and deploying server-based Java applications."

So, this could mean that the Entprise Java packages(like soap) in 11.2.0.4 were coming by default.

That seemed to be true, because the Java source was compiling without any errors in 11.2.0.4.
However; we had 11.2.0.3 , so we needed to find a solution in place.

First, checked the component status in the reqisty using the following query;

I was interested to see the Java Virtual Machine Status and Jasva Database Java Packages status.

Select comp_name, status, version
from dba_registry ;

COMP_NAME	STATUS	VERSION
OWB	VALID	11.2.0.3.0
Oracle Application Express	VALID	3.2.1.00.12
Oracle Enterprise Manager	VALID	11.2.0.3.0
OLAP Catalog	VALID	11.2.0.3.0
Spatial	VALID	11.2.0.3.0
Oracle Multimedia	VALID	11.2.0.3.0
Oracle XML Database	VALID	11.2.0.3.0
Oracle Text	VALID	11.2.0.3.0
Oracle Expression Filter	VALID	11.2.0.3.0
Oracle Rules Manager	VALID	11.2.0.3.0
Oracle Workspace Manager	VALID	11.2.0.3.0
Oracle Database Catalog Views	VALID	11.2.0.3.0
Oracle Database Packages and Types	VALID	11.2.0.3.0
JServer JAVA Virtual Machine	VALID	11.2.0.3.0
Oracle XDK	VALID	11.2.0.3.0
Oracle Database Java Packages	VALID	11.2.0.3.0
OLAP Analytic Workspace	VALID	11.2.0.3.0
Oracle OLAP API	VALID	11.2.0.3.0
Oracle Real Application Clusters	VALID	11.2.0.3.0

Everything seemed okay..

Then I checked to see the soap package .. I was interested to see its presence..

select *
from dba_objects
where object_type like '%JAVA%'
and owner = 'SYS'
and object_name like '%soap%'

Okay, the problem was that as expected.. The soap package was missing.. That is, it was not coming in 11.2.0.3 by default.. So it seemed; we had to install/load these kind of missing packages manually using jars...

Okay good, but there were no references for this kind of operations , especially for soap..
Maybe that was the reason that made the Developers think 11.2.0.4 as one an only solution.

Only the following documents were making sense, but they weren't point shots, and unfortuneatly they were for older releases.

How to Load Soap.jar into Oracle Database (Doc ID 344799.1)

 http://docs.oracle.com/cd/B28359_01/java.111/b31225/cheleven.htm#CACCHAIE

They were not answering the question likes which soap package to use; where to download what are the dependencies and etc..

On the other hand; there was a document in Oracle Support for making these kind of web services operations using PL/SQL rahter than Java.

Using UTL_DBWS to Make a Database 11g Callout to a Document Style Web Service (Doc ID 841183.1)

Altough, this document seemed unrelated , it had an excellent reference to a jar file..

In te 4th Step of this  documents; it was saying that : Load the necessary core web services callout jar files into the database. This step is to load the core Java components and is a completely separate action from loading the PL/SQL package as described in the previous steps. This means that the considerations for completing this step are entirely different from the loading of the PL/SQL components.

The load that was mentioned here was by using the dbwsclientws.jar and dbwsclientdb11.jar files.

These jar files were located in te UTL_DBWS utiliy , which can be downloaded from Oracle Support. Download the LATEST copy of the UTL_DBWS utility zip file from the Oracle Technology Network (OTN).  This file, for an 11G database, is named dbws-callout-utility-10131.zip and can be obtained from here. 

By using the following sequence of commands loading of soap package could be done;

cd $ORACLE_HOME/sqlj/lib (replacing $ORACLE_HOME with the proper directory structure)
loadjava -u username/password -r -v -f -s -grant public -genmissing dbwsclientws.jar dbwsclientdb11.jar

The customer loaded those jars; and as expected it seemed loading these jars would bring the soap java package in to the database , and these actions actually did bring the soap java package :)

So far so good. No effort for manual load was needed, but this time another compilation errors were encountered..

14:31:13 Executing ALTER JAVA SOURCE blabla COMPILE ...
14:31:13 [0:0] blabla:21: cannot find symbol
14:31:13 [0:0] symbol : method getBytes(java.nio.charset.Charset)
14:31:13 [0:0] location: class java.lang.String
14:31:13 [0:0] SOAPMessage message = factory.createMessage(new MimeHeaders(), new ByteArrayInputStream(xml.getBytes(Charset.forName("UTF-8"))));
14:31:13 [0:0] ^
14:31:13 [0:0] blabla:31: cannot find symbol
14:31:13 [0:0] symbol : method isEmpty()
14:31:13 [0:0] location: class java.lang.String
14:31:13 [0:0] if(outputParameterText.isEmpty() == false){
14:31:13 [0:0] ^
14:31:13 [0:0] 2 errors
14:31:13 Compilation complete - 11 error(s) found
14:31:13 End Compiling 1 object(s)

This time, the compiler was complaing about the the java.nio.charset.Charset.. It seemed it had no method named forName..

Checked the java package using the following;

select *
from dba_objects
where object_type like '%JAVA%'
and owner = 'SYS'
and object_name like '%java/nio%'

It was there.. Oracle Database 11.2.0.3 had java.nio.charset, so what was the problem?

The problem should be the method "forName" .. I guessed that it was missing because I knew that this method comes with JDK 1.6.. So the jdk in this release may be below 1.6..

According to Oracle Support JDKs or lets say JVMs in Oracle Database were as follows;

Executing the same java stored procedure on Oracle 10.1, 10.2, 11.1, 11.2 and 12.1 databases will show following results for JVM version:

Oracle 10.1 runs java.vm.version=1.4.1
Oracle 10.2 runs java.vm.version=1.4.2
Oracle 11.1 runs java.vm.version=1.5.0_01
Oracle 11.2 runs java.vm.version=1.6.0_43
Oracle 12.1 runs java.vm.version=1.6.0_43 or higher

But , I had doubts about this info, as it might be referring to Oracle Database 11.2.0.4 when saying Oracle 11.2 runs java.vm.version=1.6.0_43 

Then I checked the JDK version of Oracle Database using the following support doc..

How To Determine The JDK Version Used by the Oracle JVM in the Database (Doc ID 131872.1)

The result was as I expected;

The JDK in 11.2.0.4 Oracle Database was 1.6, but the JDK in 11.2.0.3 Oracle Database was 1.5...

So this error was normal ..

The idea to upgrade JDK, which resided in Oracle Database 1.5 to 1.6 seemed utopic, and it was also not supported.

Thus, I had to find another solution in place. 

In this manner; I suggested to change a line in the problematic java source script as follows;
I recommended to use UTF8 directly..  Sourcing a script would do the job..


java.lang
Class String
getBytespublic byte[] getBytes( String charsetName) throws UnsupportedEncodingException
Encodes this String into a sequence of bytes using the named charset, storing the result into a new byte array.

The behavior of this method when this string cannot be encoded in the given charset is unspecified. The CharsetEncoder class should be used when more control over the encoding process is required.

Parameters:charsetName - the name of a supported charsetReturns:The resultant byte arrayThrows:UnsupportedEncodingException - If the named charset is not supportedSince:JDK1.1

Modify this line;

....................ByteArrayInputStream(xml.getBytesCharset.forName("UTF-8")));

To be like the following;

.....................................ByteArrayInputStream(xml.getBytes("UTF8")));

This was the needed action to compile this Java Source with a 1.5 JDK, and this action solved the remaining compilation problems.. It saved the day :)

In this incident; I realized that again..I realized that being a Senior/Principle Oracle Dba/Apps Dba consultant, requires we to have a good developer perspective, too..

No need to say that; using Oracle Support efficiently is a must for being successful.
One last thing about the dependency in Oracle Database; upgrading the Jdk seems unsupported.
Alternatively, you can use OS tier to develop your Java code if it satisfies your needs.. You can have several Jdks in Os tier and you can upgrade them if needed..
One other alternative is to use PLSQL for web service operations, like mentioned in the following doc:
Using UTL_DBWS to Make a Database 11g Callout to a Document Style Web Service (Doc ID 841183.1) .. UTL_DBWS will make you use java indirectly :)

Okay that 's all for now.. Hope you 'll find this useful.

Linux bash-- primitive Incremental backup script

Following is a little script that can take incremental backups in a way.. It might come handy.

 find SOURCE_DIR -type f -mtime -2 -exec cp -rfp --parents {} TARGET_DIR\;

What this script does is;

it finds the files modified at least 2 days ago, and copies this files to the backup location with the same directory structure..

An example:

[root@ermanhost/]# tree /test2
/test2   --> Backup location.

0 directories, 0 files --> empty.

[root@ermanhost /]# tree /test
/test --> Our Source directory ,which has sub directories and files in it.
`-- dir1
    |-- dir2
    |   |-- dir3
    |   |   `-- testfile1
    |   `-- testfile2
    `-- testfile3

3 directories, 3 files
[root@ermanhost /]# find /test -type f -mtime -2 -exec cp -rfp --parents {} /test2/ \;
[root@tegvoracle /]# tree /test2
/test2  --> That's it. Our Backup is taken with the same directory structure as you see below
`-- test
    `-- dir1
        |-- dir2
        |   |-- dir3
        |   |   `-- testfile1
        |   `-- testfile2
        `-- testfile3

4 directories, 3 files

To test further, I can add a new file and run the script again..

[root@ermanhost /]# touch /test/dir1/testfile4
[root@ermanhost /]# find /test -type f -mtime -2 -exec cp -rfp --parents {} /test2/ \;
[root@ermanhost /]# tree /test2
/test2
`-- test
    `-- dir1
        |-- dir2
        |   |-- dir3
        |   |   `-- testfile1
        |   `-- testfile2
        |-- testfile3
        `-- testfile4  -->Here, the new file is copied with the same directory structure as it source..

OID 11g-- Analysis , http-500 internal server error in ODSM, http://hostname:port/odsm

In one of our customer 's OID environment which was integrated to EBS and SSO, the clients were encountering HTTP-500 Internal Server Error while trying to reach Oracle Directory Services Manager(ODSM) url..

The problematic url was used for managing the OID configuration , like checking the attributes etc.
The error was reported as follows;

OID system is working properly , but we cant open its management interface to control the configuration or its attributes.. In the past, when we see the error, we were just refreshing the web page and were able to continue our work, but nowadays refreshing the webpage does not fix the problem anymore..
Also, when we restart the OID services, the problem dissapears for a while.. On the otherhand, this is not an applicable solution for us.. In Oracle Support, it is said that the error may be related wit Browser Certifications, but in our case it is not relevant.

Okay..  

When I check Oracle Support , I saw that one of the workaround was restarting the managed server ..

Case 3 in the document "CheckList For OID 11g ODSM Page Launching / Loading / Displaying 

Problems Or Errors (Doc ID 972416.1)" was mentioning the restart as a workaround..

But this workaround could not be applied as a solution in this case, because the error was encountring periodically and repeatedly.

So , I requested the OID 's Managed Server log for analysis.

The managed server name was wls_ods1 ,which was a default one ...

After setting the domain environment; the log file could be reached by the following directory path;

$MW_HOME/user_projects/domains/domain_name/servers/server_name/logs

Analyzing the log file of a Weblogic server is like debugging a java program to me.

I have basically, searched for the word Exception that led me to the following..

<[ServletContext@1709139423[app:odsm module:/odsm path:/odsm spec-version:2.5 version:11.1.1.2.0]] Servlet failed with Exception
java.lang.RuntimeException: java.lang.Exception: MDSLockedSessionManager already registered. Can't register more than one.
.....
.....
Caused By: java.lang.Exception: MDSLockedSessionManager already registered. Can't register more than one.
at oracle.adf.share.mds.MDSTransManager.registerMDSLockedSessionManagerInst(MDSTransManager.java:132)
at oracle.adf.share.mds.MDSTransManager.registerMDSLockedSessionManager(MDSTransManager.java:124)

So it was obvious that the Exception "java.lang.RuntimeException: java.lang.Exception: MDSLockedSessionManager already registered. Can't register more than one"  was the cause that I was looking for.. Because the error was saying that Servlet was failed , and the path was the reflecting the odsm.

After finding the low level cause of the problem, I jumped in to the Oracle Support, and found the following document : Accessing ODSM 11g 11.1.1.7 Intermittently Fails with: java.lang.Exception: MDSLockedSessionManager already registered. Can't register more than one. (Doc ID 1586149.1)

Finally, this document brought me to the following bug : bug 17997221

To get the patch for this bug , the customer needed to open an SR to the Oracle Support.

It is planned that; once obtained, the patch was going be applied to the ORACLE_COMMON home of OID .. 

This Oracle home was hosting the binary,library and JRF files which are used for controlling the Fusion..

I will write the conclusion of this story when the patch will be applied, but I m sure that this action plan will fix the problem.