Rdbms-- Standby ORA-00600: internal error code, arguments: [krastrso_rectype]

When implementing Dataguard for a 11gR2 Oracle Database, you can encounter and ORA-00600[krastrso_rectype] on Standby Database.
This error prevents archived logs to be transferred to the Standby Site.
Actually , the RFS process triggers this error..
What 's happening in here is, the archived log file is transffered from the network, and written to the location pointed by Standby Database's log_archive_dest_1 parameter.

Following picture describes the flow; (the red circle is the problematic point)

RFS writes the blocks, but encounters and error while closing the file.. ORA-00600 [krastrso_rectype]
Unfortuneatly, there aint no information in Google or Oracle Support(Lookup tool or knowledge base) about this error.

Even when you enable maximum trace for RFS , LGWR or ARCH processes, you can not find any extra information about the error. You can only say that there is an error produced on the standby site, while closing the transferred archived log..

Actually, I have faced with this problem in one of my clients. I was called to analyze the problem, as this environment was not created by me or by my company.

In other words, I was involved the project subsequently..,

When I was there, I analyzed the situation, gathered process traces, checked Linux OS , searched google and Support , but couldnt find any info.

I knew that there is some problem closing the transffered archived log file on standby site, but couldnt find a reason directly, because I supposed the installation was proper..

After digging for several hours, I tried to change the log_archive_dest_1 parameter in standby database. It was pointing the db_recovery_file_dest, so I changed it to be pointed to a normal directory..(/u1/erman)

Suddenly, standby database stopped producing the ORA-600 and started working properly..

That was  a clue for me, and that was also a workaround, too..

Then I decided the find the problem lying behind... Because it was abnormal.. use_db_recovery_dest could be used in standby database normally without any issue.

I checked the parameter file on standby database and saw that db_file_name_convert and log_file_name_convert parameters were set..

Lets identify those parameters; 

DB_FILE_NAME_CONVERT converts the filename of a new datafile on the primary database to a filename on the standby database. If you add a datafile to the primary database, you must add a corresponding file to the standby database. When the standby database is updated, this parameter converts the datafile name on the primary database to the datafile name on the standby database. 

LOG_FILE_NAME_CONVERT converts the filename of a new log file on the primary database to the filename of a log file on the standby database. If you add a log file to the primary database, you must add a corresponding file to the standby database.

Actually; this is the important information : 

The initialization parameters DB_FILE_NAME_CONVERT and LOG_FILE_NAME_CONVERT cannot be used to rename OMF (non-ASM) file names for the auxiliary instance because this method generates invalid OMF file names. If you must control the generation of new OMF file names that do not use ASM storage, you must rename them using one of the following alternate techniques. The various naming options are listed in order from most recommended to least recommended.

Also , lets look at the limitations of db_file_name_convert and log_file_name_convert.
Usage and Limitation of db_file_name_convert and log_file_name_convert (Doc ID 1367014.1)

In this document , Oracle says;

It cannot be used for ;

Logical Standby Databases - only log_file_name_convert is possible for Logical Standby Databases
RMAN Restore - RMAN Duplicate/TSPITR only
If Oracle Managed Files (OMF) is used
To rename existing Datafiles, eg. when you move a Database. The Conversion only takes Place on File Creation on the Database where it is set.


And in OMF documents; we see that db_recovery_file_dest parameter is enabling the OMF. 

Enabling the Creation and Use of Oracle Managed Files

The following table lists the initialization parameters that enable the use of Oracle Managed Files.

Initialization Parameter	Description
DB_CREATE_FILE_DEST	Defines the location of the default file system directory or Oracle ASM disk group where the database creates data files or temp files when no file specification is given in the create operation. Also used as the default location for redo log and control files ifDB_CREATE_ONLINE_LOG_DEST_n are not specified.
DB_CREATE_ONLINE_LOG_DEST_n	Defines the location of the default file system directory or Oracle ASM disk group for redo log files and control file creation when no file specification is given in the create operation. By changing n, you can use this initialization parameter multiple times, where n specifies a multiplexed copy of the redo log or control file. You can specify up to five multiplexed copies.
DB_RECOVERY_FILE_DEST	Defines the location of the Fast Recovery Area, which is the default file system directory or Oracle ASM disk group where the database creates RMAN backups when no format option is used, archived logs when no other local destination is configured, and flashback logs. Also used as the default location for redo log and control files or multiplexed copies of redo log and control files if DB_CREATE_ONLINE_LOG_DEST_n are not specified.

So, in conclusion, 

the issue was fixed by deleting log_file_name_convert and db_file_name_convert parameters in standby's parameter file, as those parameters make Oracle to behave unexpectedly in closing transffered acrhivelog files on the remote file system..

This seems to be an expected behaviour but ORA-00600 scared us :)

After removing those parameters, we could use db_recovery_file_dest as our log_archive_dest in standby database..

Linux -- Oracle announces Oracle Linux 7 beta 1

Ref: Oracle's Linux Blog

By Michele Casey on Apr 10, 2014

We are pleased to announce the availability of the first beta build for Oracle Linux 7.  The beta is available to download from the Oracle Linux download page on Oracle Technology Network.  The Oracle Linux 7 beta includes the latest beta build of Unbreakable Enterprise Kernel (UEK) Release 3 Update 2, as well the Red Hat Compatible Kernel (RHCK).  We are very interested in your feedback and have setup a new mailing list for users to post their questions and comments.  For more information please review the Oracle Linux 7 Beta Release Notes.

Linux -- boot hangs "Mounting Cifs Filesystems"

While booting the Linux, you can encounter a hang problem if you have cifs or nfs mounts defined in your /etc/fstab file..
Like this;

The problem can be an unreachable network drive or a cifs share that requires a password.
If it is a password problem, you can try pressing Enter multiple times and wait. Maybe pressing the Enter key can make the linux to skip the cifs mount , as the cifs host will reject the multiple failed login attempts..
Anyways, if you cant boot your linux and want to edit your fstab file even if you cant boot properly..
Follow the steps below;

While booting, press ESC in the grub menu

choose the desired boot line and press 'e’'to edit that entry

Add "single" (with a space in front of it)kernel parameter to the end of the kernel line ; kernel /vmlinuz-blabla ro root=/blabla blabla single

Press enter to apply the changes and press b to boot in single user mode.

After booting in single user , check the / root filesystem , if it s not mounted rw, then execute the following to remount it read-write; mount -n -o remount,rw /

Now , you can edit/fix your fstab file , and reboot your machine when it s ready.

Note that; you can alternatively use rw init=/bin/bash kernel argument for doing the same job...

Linux -- Oracle/Redhat Linux 6.2 Network Manager problems, device state change: 8 -> 3

This week, in one of the Oracle Linux 6.2 servers, we have faced with a strange problem.
There was a problem in Linux network layer, as our ping request were cut off time to time.
/var/log/messages displays the following messages;

Apr  7 11:35:55 otmprodoic NetworkManager[2044]: <info> Activation (eth0) successful, device activated.
Apr  7 11:35:55 otmprodoic NetworkManager[2044]: <info> Activation (eth0) Stage 5 of 5 (IP Configure Commit) complete.
Apr  7 11:35:56 otmprodoic dnsmasq[2782]: reading /etc/resolv.conf
Apr  7 11:35:56 otmprodoic dnsmasq[2782]: using nameserver 10.10.10.18#53
Apr  7 11:35:56 otmprodoic dnsmasq[2782]: using nameserver 10.10.10.7#53

Apr  7 11:35:53 ermansrv NetworkManager[2044]: <info> (eth0): device state change: 8 -> 3 (reason 39)

Apr  7 11:35:53 ermansrv NetworkManager[2044]: <info> (eth0): deactivating device (reason: 39).

We directly disabled IPV6, as we have suspected from it, but didnt help.

As for the analysis, I have gathered following information and jumped in to google to look for the bug records..

Device State 3 means "disconnected".
Reason 39 seems to mean ; a deactivation requested , maybe from a GUI or from ifup/ifdown command or from similar thing..

So why does Network Manager take such an action , I mean why it deactivates the device..

I habe looked at the last action that Network Manager did before deactivating the device and saw that; before deactivating the device, it just reads resolv.conf , which is the file that contains the dns server information in Linux. So the 3 lines above the errors seemed to be unrelated with the error.

Anyways, our eth0 device was become disconnected , and as a result ,our tcp packets were dropping from time to time..
After analyzing and searching a little bit, I have found the following bug reported in Redhat;
Bug 590704 - intermittent connection caused by Networkmanager
It was for Fedora actually, but thought that would apply in our case, too, because the problematic component was Network Manager.

About Network Manager :  Red Hat initiated a NetworkManager project in 2004 with the goal of enabling Linux users to deal more easily with modern networking needs, particularly wireless networking. NetworkManager takes an opportunistic approach to network selection, attempting to use the best available connection as outages occur, or as the user roams between wireless networks. It prefers Ethernet connections over “known” wireless networks, which are preferred over wireless networks with SSIDs to which the user has never connected.  

So , because there is a similar bug report for the Network Manager in Fedora and also because the messages are coming from the network manager and lastly because the meaning of reason 39 reported in the error message; I disabled the Network Manager , manually configured eth devices from ifcgh-eth files and manage them using ifup/ifdown commands..

These actions saved the day. The problem doesn't appear anymore.

Disabling Network Manager;
chkconfig NetworkManager off
service NetworkManager stop
service network restart

Of course, the disadvantage of disabling the network manager is losing the ability to use a GUI for configuring the network interfaces. 

EBS -- 12.2.3 supported on Internet Explorer 8,9,10

E-Business Suite 12.2.3 is supported on IE 8,9 and 10.
I m writing this post because there is missing certification information in the certification tab of Oracle Support Website(MOS).. That is;  if you will search for the browser certifications of EBS 12.2.3 , you will see only Safari ,there.

On the other hand, if you will search for the browser certification of EBS 12.2.2, you will see IE, Safari and Firefox listed there. (which is expected) In this point, the question comes to my mind is -> Do Oracle certify IE with EBS 12.2.3 or not? This question comes to my mind because there is IE certification information in 12.2.2 , but there is no info for 12.2.3..  

In addition, in 12.2.3 , we have a lot of changes in GUI. Those changes also make us search for the browser certifications, and if we use certifications tab in Oracle Support we end up with Safari certifications only..

Of couse, if we analyze the document and make our comments by reding the sentences, we see that 12.2.3 is certified with IE.. Because in that document, we see certifications with the statements like Oracle E-Business Suite Release 12.2 or higher.

So the short answer is "12.2.3 is certified with Internet Explorer 8,9 and 10. "

"Note:389422.1-Recommended Browsers for Oracle E-Business Suite Release 12", is the long answer..

Note that , I already opened a Service Request just to be sure.. They approved IE 8,9,10 certification for EBS 12.2.3. 

EBS 12.2.3 -- Upgrade Paths

EBS and Java 7 Security-- missing permissions manifest attribute in main jar, block unsigned

I have faced an incident recently.

It was an EBS 12.2 instance, and the clients were congifured to use Java 7update51.. As known, EBS uses server-side Java  in the interfaces coming from oacore..  Forms interfaces in EBS, on the other hand; are provided throught the client-side Java.. The incident was encountered using client-side Java,  in other words on Forms sessions.

The problem was that; when the user tried to open a new form (For Ex: System Administrator > Concurrent Manager  > Administer)  while another form was already open in the same client, following error was produced;

connectMode=HTTP, native.
Forms Applet version is : 10.1.2.3
Missing Permissions manifest attribute in main jar: http://erman:8000/OA_JAVA/oracle/apps/fnd/jar/fndaol.jar
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at oracle.forms.engine.Heartbeat.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
-- DialogThread.showDialog(): Interrupted Exception.
Exception in thread "thread applet-oracle.forms.engine.Main-2" java.lang.NullPointerException
at java.awt.EventQueue.isDispatchThread(Unknown Source)
at com.sun.deploy.uitoolkit.impl.awt.AWTAppletAdapter.runOnEDT(Unknown Source)
at com.sun.deploy.uitoolkit.impl.awt.AWTAppletAdapter.doClearAppletArea(Unknown Source)
at sun.plugin2.applet.Plugin2Manager$AppletExecutionRunnable.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Missing Permissions manifest attribute in main jar: http://erpdemo.takasdom.takasbank.com.tr:8000/OA_JAVA/oracle/apps/fnd/jar/fndforms.jar
proxyHost=null
proxyPort=0
connectMode=HTTP, native.
oracle.forms.net.ConnectionException: Connection from new browser window not supported 
at oracle.forms.net.HTTPNStream.getInfoFromServlet(Unknown Source)
at oracle.forms.net.HTTPNStream.<init>(Unknown Source)
at oracle.forms.net.HTTPConnection.connect(Unknown Source)
at oracle.forms.engine.FormsDispatcher.initConnection(Unknown Source)
at oracle.forms.engine.FormsDispatcher.init(Unknown Source)
at oracle.forms.engine.Runform.initConnection(Unknown Source)
at oracle.forms.engine.Runform.startRunform(Unknown Source)
at oracle.forms.engine.Main.createRunform(Unknown Source)
at oracle.forms.engine.Main.start(Unknown Source)
at com.sun.deploy.uitoolkit.impl.awt.AWTAppletAdapter.start(Unknown Source)
at sun.plugin2.applet.Plugin2Manager$AppletExecutionRunnable.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

I suspected from the following message, and focused on that;

"Missing Permissions manifest attribute in main jar.."

Acutally, Java was trying to say, "you didnt give permission for this Main jar."
Also it got my attention that, while opening the forms, a java security warning was appearing.
The following message was displayed on the warning box: Running unsigned applications like this will be blocked in a future release because it is potentially unsafe and a security risk.

On the other hand, I could continue using Forms by clicking the OK button on that message box.. By clicking OK, I actually give permission for that running usigned java code in my client machine.

The important thing in here was that, this message box didnt appear when I tried to open a new form (For Ex: System Administrator > Concurrent Manager  > Administer)  while another form was already open in the my client and this caused those missing permissions errors... "Missing Permissions manifest attribute in main jar.."

Note that , when you install EBS, you get unsigned java code or let say:  JAR files that are downloaded to the desktop client from Oracle E-Business Suite have been signed using a self-signed certificate with a 1024-bit key size ,  Using a self-signed certificate is no longer deemed secure when using the latest Java security standards. So the produced warning message by Java is an expected warning message, and also we can push OK button to continue working on Forms..

The problem is, we cannot say "OK for always" or "Remember my answer" any more, because with the Java version 7u40 and above, there is an higher security introduced.
Unfortunately, this creates a problem when opening a new form if there is an already opened form in our clients, because Java doesnt want us  to allow permissions to unsigned code again. (as far as I see). This create the permission problem..

To workaround this, you can use a Java with version 7u39 or lower in your client machine. (I didnt test 7u39.. But according to the document it should be useful..)

To fix this, I mean if you want to continue using 7u51 for instance; then you need to sign the EBS jar files. For signing the jar files , I suggest you to take a look at the following MOS Doc;
Enhanced Jar Signing for Oracle E-Business Suite (Doc ID 1591073.1)

Lastly;
I share the information below to support my findings.

The information provided below is from Oracle Support and Java documentation. ..

Signing Certificate	JRE Security Setting	JRE 7u21 and 7u25	JRE 7u40 and 7u45	JRE 7u51
Self-Signed Certificate	Very High	Forms won't launch	Forms won't launch	Forms won't launch
	High (Default Setting)	Forms launch normally	Forms won't launch1	Forms won't launch
	Medium	Forms launch normally	Forms launch with warning 2	Forms launch with warning 2
Code signing certificate from a Trusted CA 3	Very High	Forms launch normally	Forms launch normally	Forms launch normally
	High (Default Setting)	Forms launch normally	Forms launch normally	Forms launch normally

¹ JRE 7u40 and JRE 7u45 are now both past their expiration date, forms will no longer launch on a High security setting when using a self-signed certificate.

² The security Warning: Running unsigned applications like this will be blocked in a future release because it is potentially unsafe and a security risk will pop when jars using a self-signed certificate are detected. The user can click through this message to launch the Java content. Unlike earlier JRE releases there is no longer the option to remember this decision for future logins. When running JRE 7u40 and later this message will therefore appear every time you start a new session, it can no longer be suppressed unless your jars are signed with a Trusted CA.

³ Unless previously installed into the Java certificate store you will still be asked to trust the publisher when launching an EBS environment for the first time from the desktop. See Step 6.3. Running an Environment for the First Time for further details.

Java documentation
Changes to Security Slider:
The following changes to Security Slider were included in this release(7u51):

Block Self-Signed and Unsigned applets on High Security Setting
Require Permissions Attribute for High Security Setting
Warn users of missing Permissions Attributes for Medium Security Setting
For more information, see Java Control Panel documentation.

NTP and Timezone Data -- Why do we need DST patches

This year , Turkey 's government delayed changing over Daylight Saving Time by one day.
Normally, the swtiching would be at March 30, 2014 03:00 AM, but this year it was at March 31, 2014 03:00 AM.. Ofcourse, this have brought some questions into our minds, as our application and database servers should be configured to delay this DST switching by one day, too.

During my researches , I have seen that there is a misconception between Timezones and NTP. Some customer have trusted their Ntp servers, as they will be get the correct time from the Ntp server pools, and they decided not to do anything..  On the other hand, Oracle, Microsoft and other big vendors released new patches especially for this DST rule change in Turkey.. This misconception created the idea that " we dont need to patch our environments because we already use NTP". Anyways, this is not correct. Eventhough we use Ntp server for time syncronization , we still need the DST patches..

To understand why we need those DST patches even if we use NTP, we need to understand what NTP does.
NTP operates in GMT or lets say UTC. UTC is not related with DST rules.
It is unversal, not local time. So if we need to calculate the local time from universal time , we need to add or subtract our local time zone.
It is the job of the local system to convert this UTC to Local Time. Operating systems do this conversion using their timezone files.
For Oracle Linux , you can display your current DST rules with zdump as follows;
zdump -v Europe/Istanbul | grep 2014

Europe/Istanbul Mon Mar 30 00:59:59 2014 UTC = Mon Mar 30 02:59:59 2014 EET isdst=0 gmtoff=7200
Europe/Istanbul Mon Mar 30 01:00:00 2014 UTC = Mon Mar 30 04:00:00 2014 EEST isdst=1 gmtoff=10800
Europe/Istanbul Sun Oct 26 00:59:59 2014 UTC = Sun Oct 26 03:59:59 2014 EEST isdst=1 gmtoff=10800
Europe/Istanbul Sun Oct 26 01:00:00 2014 UTC = Sun Oct 26 03:00:00 2014 EET isdst=0 gmtoff=7200
As you see above, first two lines indicate that this Linux OS will switch the time at Mar 30 , which is not appropriate for 2014, as this year Turkey decided to changing over DST time at Mar 31..
So if we will leave this system as is, we will end up with a DST time switch at Mar 30 ..
That 's why we need those OS patches..

EBS-- 12.2 Applications Dba Online Patching (ADOP)

Online patching is our new toy in EBS 12.2.
Adop is the utility for patching EBS 12.2 . Adop replaces adpatch, altough it uses adpatch in background :)
Oracle provides this online patching system by using two Apps tier filesystems named fs1 and fs2 in Application tier , by using edition based redefinition feature of 11gR2 Rdbms in Database Tier.
Normally, we have 5 phases when applying a EBS patch. These phases are prepare,apply,finalize,cutover and cleanup. (adop phase=..)

We start with prepare for syncronizing the patch filesystem for Applications and the patch version for db tier.
Then we continue with apply phase. This phase is actual phase that the patch operations are done.. adop uses patch filesystem to upload new apps tier files and uses the patch edition of the database to take db actions.  After apply we continue with adop=finalize. In this phase, application and database specific actions such as compilation takes place.
So those 3 phases require no downtime , but the next phase which is called cutover need a little downtime. It is automatic though.. In cutover phase, adop switches the patched filesystem and db edition with the current/run filesystem and db edition... That's why Application Tier components are restarted in this phase.. Downtime is coming from restarting the components..
The last phase is cleanup phase. It is self explanatory..  In this phase ADOP starts cleaning up the old editions.

One last thing, if you are using 12.2.0 VISION, be careful if you need to restart the apps tier process after a patch application, because in 12.2 VISION we dont have an environment file for choosing and setting the correct environment.. I mean EBSapps.env..
On the other hand, you can use the following script maybe;

# Script to set the current RUN Edition APPL_TOP environment automatically.
# Available through My Oracle Support Note 1545584.1
# Requirements: Set the XMLfile variable as specified below. This needs
# to be set only once and not again after a file system cutover in adop.
# Save the script as fsauto<SID>.env in a non-EBS directory, eg.:
# /home/oracle/fsautoPROD.env
# Usage: . <directory>/fsauto<SID>.env
# Eg.: . /home/oracle/fsautoPROD.env
# note the space ' ' character between the leading period '.' and first
# forward-slash '/' characters
# Alternatively use: source <directory>/fsauto<SID>.env
# If needed make the script runnable using command: chmod u+x fsautoPROD.env
# Carlo den Otter, 02-APR-2013
# Note this script is for educational purposes only, and as such is not supported.
# It was created on Linux and not tested on other platforms.
# Use at own risk.

Lastly, I suggest you to take a look to the following note for the environment setting :
How To Automatically Set the Current Run or Patch Edition / File System for EBS 12.2 (Doc ID 1545584.1)


One last thing.. I share a drawing of mine , that explains this online patching process.. Maybe it become useful for understanding..

Exadata -- Learning Path

To start learning Exadata , you can follow one of the learning paths in Exadata- Learning Path below.
I dont want to say that you need to attend the related courses to get those skills, but  somehow you need to learn the pieces composing this Engineered System.. Otherwise, your knowledge about Exadata will be shallow.

In many cases, we use single admin for Exadata, on the other hand; following Learning Path describes different roles(which is the desired method I think) for managing Exadata.

In my opinion following skills should be enough for administrating Exadata Machine as a whole ;
Database Administration
Rac Administration
Oracle Linux Administration + Oracle Solaris Administration (if your compunte nodes are Solaris)
Exadata Database Machine Administration

Any extra skill to the above skill will be easy to learn and a bonus I think :)

Here is the Oracle 's Learning Path for Exadata;