Linux -- IO Error on boot read failed, No such File or directory /dev/, Dropping you to a shell, error occurred during the file system check

Filesystem check errors can prevent Linux from booting.

When the Linux system boots, the boot process checks the root filesystem and mounts it read/write.

Note it was mounted by the kernel as read-only).. It basically checks the filesystems with fsck command.

If these filesystem check is failed for root filesystem or any other filesystems presented in fstab, following prompt appears and Linux start process drops the user into a shell to correct the errors..

***An error occurred during the file system check.
***Dropping you to a shell; the system will reboot
***when you leave the shell.
Give root password for maintenance
(or type Control-D for normal startup):

This is a troubling event, particularly because it might well appear if you have other problems with the system—for example, a lockup (leading you to press the Reset button) or a spontaneous reboot.

You can use the following command to check filesystems...

fsck -A -V ; echo == $? ==

Note that: The purpose of the echo == $? == command is to unambiguously interpret the outcome of the fsck operation.

The outcome from this command will be one of the following.. Every outcome which is smaller than 4 is good..

0 No errors
1 Filesystem errors corrected
2 System should be rebooted
4 Filesystem errors left uncorrected
8 Operational error
16 Usage or syntax error
128 Shared library error

On the other hand; sometimes, the real problem is caused by a hardware failure. -- altough it seems like a file system error.. 

if you watch the process closely, you can catch it from server screen,  

If that is the case (San cable broken, controller error, Storage problems.. etc.) and if the root filesystem is not in the faulty disk/fault path, you can use a workaround by deleting the relevant line in the fstab file..

But in order to do that, you have to mount the root filesystem in the read write mount. This can be accomplished by the following command;

mount -o remount,rw / 

After mounting the root filesystem read/write, edit the /etc/fstab file and reboot. You will able to boot the Linux..

EBS-- change OAF session limit, timeout

FOR EBS R12
In Oracle EBS R12 session limit for OAF pages, can be configured through the s_oc4j_sesstimeout value in CONTEXT_FILE .. After modifying the context file, you should run autoconfig for your change to take effect.

To change the limit directly; you can change session-timeout valuein orion-web.xml file which resides under the following directory $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/application-deployments/oacore/html.

FOR EBS 11i
In Oracle EBS 11i These settings are located at: ../*ora/iAS/Apache/Jserv/etc

JServ Timeout is specified by the value of the property session.timeout in the JServ configuration file zone.properties, and represents the number of milliseconds to wait before ending an idle JServ session (the default is 30 minutes). This timeout is used by products based on Oracle Applications Framework (OAF).

Standby Database- MRP restart script // WARN: ARCH: Termination pid hung on an I/O operation

To workaround RFS hung problems (probably due to network problems), following script can be used..

The known issues in Oracle Support does not help, as they declare this problem is fixed in 11.2.0.3.

But I m writing this post because I faced this problem in 11.2.0.3..

I will try to make a brief explanation about the problem..

Following is the method/script that I believe, can be used temporarily, for getting rid of the problem when the usual or planned method isn't working.( for example: if you encounter this error on 11.2.0.3)

STANDBY --> 

WARN: ARCH: Terminating pid 7993 hung on an I/O operation (7993 is the pid of RFS)

Note that ARCH is killing RFS processes,again and again.. 

PROD --> 

ORA-03113: end-of-file on communication channel

WARN: ARC1: Terminating pid 8147 hung on an I/O operation..

RELATED BUGS:

-------------

1. Bug 12737862

- Recommendations there were:

a) Apply the patch for 12776906. This is a merge of 10222719 and BP7 and we 

already have that fix.

b) Set _redo_transport_max_stall_time=3600. Not sure if this is relevant.

2. Bug  11853815

- The problem here was that RFS process kept getting killed.

- In our case it's ARCn processes that get killed.

- Need your advice if this bug is relevant

3. Bug 13595410

- This is more like our issue. Logged recently and still under investigation.

# This ksh automates restarting the MRP process by periodically monitoring
# the V$ARCHIVE_GAP for redo log gaps. If any rows are returned (i.e. a
# gap has been detected) it stops the MRP process and starts a new one.
#
if [ $# -eq 0 ]; then
$0 300 /tmp
exit 0
elif [ $# -ne 2 ]; then
echo "Usage: $0 <monitor interval (s)> <log file directory>"
exit 1
fi
#
monInterval=$1
logDir=$2
#
while [ 1 ]
do
sqlplus -s "/ as sysdba" << eof >> ${logDir}/monGap.log
set echo off feedback off serveroutput on pages 2000 linesize 132 pages 20000
declare
v_isGap number;
begin
select count(*) into v_isGap from v\$archive_gap;
if ( v_isGap > 0 ) then -- We have a gap
dbms_output.put_line(to_char(sysdate,'HH24:MM:SS DD-MON-YYYY')||': Gap detected, restarting MRP');
execute immediate 'alter database recover managed standby database cancel';
execute immediate 'alter database recover managed standby database disconnect using current logfile';
end if;
end;
/
eof
sleep ${monInterval}
done

Aurora - Oracle's Java Virtual Machine

A Java virtual machine (JVM) is a virtual machine that can execute Java bytecode. It is the code execution component of the Java platform.Oracle includes a JVM in its database. The JVM is called Aurora.The Aurora JVM runs in the same process space and address space as the RDBMS kernel, sharing its memory heaps and directly accessing its relational data.

Oracle includes a JVM in its database. The JVM is called Aurora.The Aurora JVM runs in the same process space and address space as the RDBMS kernel, sharing its memory heaps and directly accessing its relational data

In this post, I will briefly describe the Aurora towards oracle documents and give an example approach for using the sqlj option.

Main Components of Aurora JVM:

Library Manager: Loads java source, class files or resource files into the database. Library manager is invoked bye CREATE JAVA {SOURCE | CLASS |RESOURCE} statement.

Memory Manager:Automated storage management is one of Java's key features. In particular, the Java run-time system requires automatic garbage collection (deallocation of memory held by unused objects). The memory manager uses memory allocation techniques tuned to object lifetimes. Objects that survive beyond call boundaries are migrated to appropriate memory areas. Also, the memory manager minimizes the footprint per session by sharing immutable object state such as class definitions and final static variables.

Compiler: It translates Java source files into architecture-neutral instructions called bytecodes. Invoked by the "CREATE JAVA SOURCE" statement.

Interpreter: The interpreter and associated Java run-time system execute standard Java class files.

Class Loader: 

Locates, loads, and initializes Java classes stored in the RDBMS. 

resolve external references.

Invokes the Java compiler automatically when Java class files must be recompiled   

Verifier: prevents the inadvertent use of "spoofed" Java class files

Server-Side JDBC Driver: a specially tuned JDBC driver runs directly inside the RDBMS. It supports Oracle-specific datatypes, NLS character sets, and stored procedure.

Server-Side SQLJ Translator: SQLJ enables you to embed SQL statements in Java programs. A highly optimized SQLJ Translator runs directly inside the RDBMS, where it provides run-time access to Oracle data via the server-side JDBC drive. SQLJ forms can include queries, DML, DDL, transaction control statements, and calls to stored procedures

JServer Accelerator: It translates standard Java class files into specialized C source files that are processed by a platform-dependent C compiler into native libraries, which the Aurora JVM can load dynamically.

----------------------------------------------------------------------------------------------------------

SQLJ Translator Example:

Following can be considered an example for the SQLJ usage;

Here we create a java source that executes a sql. Note that we use sqlj to embed sql statement into this java program. SQLJ declarations are preceded by the #sql token.

CREATE OR REPLACE AND COMPILE JAVA SOURCE NAMED "TestDba2"

   AS import java.sql.*;

public class TestDba2                                 

{

  public static String DualeYaz ()

    throws SQLException

  {

    String addr;                                    

    #sql { SELECT dummy INTO :addr FROM dual };

    return "TRUE";

  }

};

We create a function to run the java.

create or replace function Duale_Yaz return varchar2

AS LANGUAGE JAVA

NAME 'TestDba2.DualeYaz() return java.lang.String';

We execute the function..

select Duale_Yaz from dual;

EBS-- Notification mailer Inbound/Imap java.lang.NullPointerException

Environment: 

Exchange Server 2010 servicing IMAP for notification mailer ( EBS R12)

Symptoms:

• Notification mails can not be approved..
• Following errors seen in Notification mailer's logfile ->Thread[inboundThreadGroup1,5,inboundThreadGroup]:217.78.110.53:81254:1373535220200:19:ERROR:[SVC-GSM-WFMLRSVC-48532-10006 : oracle.apps.fnd.wf.mailer.IMAPInboundProcessor.readNewMessages()]:Lost connection to INBOX when performing {Obtaining UID}. Try again later -> java.lang.NullPointerException

Control from backend:

telnet imapserver imapport(143) 1 login <IMAP Account> <IMAP Account password> 2 LIST "" INBOX 3 SELECT INBOX 4 SEARCH UNSEEN ALL 5 FETCH 1 (ENVELOPE INTERNALDATE RFC822.SIZE UID) 6 FETCH 1 (FLAGS) 7 FETCH 1 (BODY.PEEK[HEADER.FIELDS (X-UID)]) --> Error: "BAD Command Argument Error. 11" 8 FETCH 1 (BODY.PEEK[HEADER.FiELDS (X-UID)]) 9 LOGOUT Solution: The problem is due to regional settings of the Imap Server. It should be English.  Otherwise, If you are using MS Exchange as IMAP Server, you will probably encounter this error.. This problem was figured out while analyzing an environment where Turkish language was set for the Regional Language. "Change Regional Language, welcome screen ,unicode and user settings of the IMAP server(Exchange Server) to English.."

EBS-- Common Client Problems & Solutions

As is known ; Oracle E-Business Suite's client interface is provided through HTML for HTML-based applications, and via a Java applet in a Web browser for the traditional Forms-based applications.

However; it s undeniable that Oracle E-Business Suite users encounter browser and java problems time-to-time..

Although, these problems seems causeless from their perspectives, there is always a reason that makes the browser not to perform as expected.

In addition to that, most of the times Apps Dba's have to deal with this client problems, and this process annoys the Apps Dba's, as they have a lot of real admin stuff to do..

So, I decided to write this post to throw a light on this problems. I tried to sum up the experiences, known issues and browser/java/OS certifications in one post.

• Make sure hostname of the environment that you are trying to connect, is defined in your dns or /hosts file correctly.
• Ensure proper version of Java installed on your client. For example: If environment uses Java 1.6.0.7, then your client should have at least Java 1.6.0.7 installed.
• Ensure your browser's pop up blocker is disabled.
• Make sure, unnecessary add-ons are disabled.
• Make sure, java plugin is detected and enabled by your browser..
• If your browser is Google Chrome, then ensure your java plugin is 32 bit. Dont forget to add Oracle EBS R12&11i Enablement to Chrome.
• IF your registry is corrupt/inconsistent and if this problem prevent you to install or delete java plugins; use "Fix it" program in Microsoft Support and clean out the inconsistency.
• Even if you use a certified version of Internet Explorer, and even if you set all the things correctly, Oracle Forms can still have problems (like not opening without any error or warning). If this is the scenario, Use IE options> advanced > reset to reset the IE.. 3rd party software can damage the IE configuration..
• If you use Internet Explorer 10, and if the menus in the EBS pages display “undefined” responsibility names and buttons garbled, use compatability settings in tools menu .
• Ensure you have correct proxy settings.
• Oracle recommends that customers running Oracle E-Business Suite through Internet Explorer (IE) should use a 'Medium' security setting through the 'Trusted Sites'zone in the browser at runtime. This setting does not automatically download the oaj2se.exe file from the middle tier if the user does not already have the appropriate version installed on their desktop.
• IE9 users running in 'IE9 Browser Mode' will see misalignment of the buttons and text as well as changes in the font within certain FWK pop ups, in this case use compatability settings in tools menu .
• If you receive an 'Unable to authenticate session' error in your browser, use  Tools -> Internet Options -> General (tab) -> Browsing History -> Delete (button).
• If you encounter cross-site scripting error, a server side operation will be needed.. That is, Forms redeploy will be needed.. Contact Apps dba for this.
• Check support note 389422.1 for the updates, and environment specific known issues..
• If you cant open forms, and see jar signing exception in your java console while opening forms;
java.lang.SecurityException: com.sun.deploy.net.JARSigningException: Found unsigned entry in resource #name of the applet#
You should do the following;
control panel
java
General Tab
Temporary Internet File
Settings button
List item
New Dialog : Temporary Files Settings dialog
disable the option : keep temporary files on my computer.

Users who cant open forms in Google Chrome, even with the EBS Enablement, should check their Chrome version. A recent Chrome update may break the code. If so, they can uninstall the current version and install an older version of Chrome using the following url:  http://www.oldapps.com/google_chrome.php

• Use a certified client environment; certification table for IE, Safari and Firefox is like below;

OID-- OID 11g installation on Two Node IBM Cluster/Hacmp

Assuming all the minimum requirements are met, and installation platform is certified;

OID 11g installation process consists of the following steps;

• Installation of Oracle Database.
• Creating OID Application schemas using Rcu Utility.
• Installation of Weblogic Server.
• Installation of OID.

The version of OID that will be explained in this post; is "11.1.1.7 ". Anyways, Installation is pretty much the same for all OID 11.* versions..

This post was written in under the Cluster concept, that is why it is important to install and configure the stack based on the Cluster requirements..

Without any other clustering software like Weblogic clustering or Oracle Rac, Active/passive OID cluster can be build on top of  IBM HACMP Cluster..
The key point in here is the service ip used by the HACMP cluster nodes.. By using the service ip, applications and databases can run on all the nodes of the cluster in case of failover, without any problem , without any manual intervention..

Click here to read full document

DATABASE-- the Oracle Database 12c MULTITENANT ARCHITECTURE DIAGRAM

Database-- Oracle Database Installation/runInstaller -- using virtual hostname

If there is a need for using a virtual hostname during Oracle installations, it can be accomplished by the following;

export ORACLE_HOSTNAME=virtualhost.domain.com
./runInstaller OUI_HOSTNAME=virtualhost.domain.com

Oracle Database installation on a IBM HACMP cluster can be an example for using virtual hostname. That is ; after a failover, the secondary server should able to listen on the same ip/hostname as the primary server. This is accomplished by using the service ip.. So, Oracle Database installation on a HACMP cluster should use the service name as hostname..

EBS-- Autoconfig Reading the Context File contents directly from the database

Context Files are stored in the database.. The table that store the context files( historically) is "FND_OAM_CONTEXT_FILES"
Context files in this table is stored in a huge clob column named TEXT..
In TEXT column the context file is stored as is, that is in xml format ..

fnd_oam_context_files
----------------------------------------- -------- ----------------------------
NAME NOT NULL VARCHAR2(512)
VERSION NOT NULL VARCHAR2(30)
PATH NOT NULL VARCHAR2(1024)
LAST_SYNCHRONIZED NOT NULL DATE
TEXT NOT NULL CLOB
LAST_UPDATE_DATE NOT NULL DATE
LAST_UPDATED_BY NOT NULL NUMBER(15)
CREATION_DATE NOT NULL DATE
CREATED_BY NOT NULL NUMBER(15)
LAST_UPDATE_LOGIN NOT NULL NUMBER(15)
NODE_NAME NOT NULL VARCHAR2(30)
STATUS VARCHAR2(1)
SERIAL_NUMBER NUMBER
EDIT_COMMENTS VARCHAR2(2000)
CTX_TYPE NOT NULL VARCHAR2(1)
So by quering the table and using the proper xml functions , the values in context file can be read.

For example, to gather the web port information of the E-Busines Suite system, following query can be used;

SELECT extractvalue(xmltype(text),'//web_port')
FROM fnd_oam_context_files where status='S';


Note that, the query should be modified in order to read the xml tags with childs.

Database-- Redo Transport Compression Custom Method

As we know, Redologs/Archivelogs produced from Production environments are transported and applied to the Standby environments..  This is Oracle 's standard Disaster Solution, known as Standby Database, and managed through the Data Guard..

In Oracle Data Guard Disaster Solution, all the redo produced in Production database are transferred to the Standby database server, and then Standby database server is updated by the transferred redo entries..
Transfering phase of this disaster solution sometimes become a bottleneck, due to slow networks, or excessive redo generation..

At this point, the compression is included to solve this network bottlenecks. Compressed redo become smaller in size, and require less network throughput.

With 11g Oracle can make compressed redo transportation.. But in order to use Oracle 's standard compression technique, Advanced Compression option should be licensed.. ( you need to buy Advanced Compression license)..
Oracle Support Doc : 729551.1
Redo transport compression is a feature of the Oracle Advanced Compression option. You must purchase a license for this option before using the redo transport compression feature.  Compression of redo transport for versions prior to Oracle Database 11g Release 1 can be accomplished using third party WAN accelerators, such as those available from Cisco, Riverbed, F5, and other vendors.

On the other hand, compressed redo transportation can be accomplished by a custom method, too..
Basically, this method is based on a ssh tunnel, and ssh compression..

Here are the steps :

1) Create ssh equivalence between oracle software owner os accounts in production and standby servers..

IN PROD and STANDBY with oracle user;

mkdir ~/.ssh
chmod 755 ~/.ssh
/usr/bin/ssh-keygen -t rsa
/usr/bin/ssh-keygen -t dsa

IN PROD with oracle user;

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh oracle@standbyserver cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keysssh oracle@ds2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
chmod 644 ~/.ssh/authorized_keys

IN STANDBY with oracle user;

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh oracle@ds1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh oracle@ds1 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
chmod 644 ~/.ssh/authorized_keys

2)Create ssh tunnel from production to standby server. -f option will make it run in the background and -C will make it to run in compression mode.

/usr/bin/ssh -f -C -N -L LOCALPORT:REMOTESERVER:REMOTEPORT

3)Stop redo transportation in Production.

sqlplus "/as sysdba"
SQL> alter system set log_archive_dest_state_2=defer;
alter system switch logfile;

4)Change Standby tns entry in Production to use the ssh tunnel.

STDBY.EXAMPLE.COM.TR =
(DESCRIPTION=
(SDU=32768)
(ADDRESS=(PROTOCOL=tcp)(HOST=localhost)(PORT=LOCALPORT))
(ADDRESS=(PROTOCOL=tcp)(HOST=REMOTESERVER)(PORT=REMOTEPORT))
(CONNECT_DATA=
(SID=STANDBYSID)
)
)

5)Start the redo transportation in Production.

sqlplus "/as sysdba"
SQL> alter system set log_archive_dest_state_2=enable;
alter system switch logfile;

6)Write a script to monitor and to recreate the tunnel, if it s broken.. Schedule the script to run in cron or daemonize the script..

#!/bin/sh

if test `ps -ef|grep -v grep|grep -c "LOCALPORT:REMOTESERVER:REMOTEPORT"` -eq 0
then
echo "ssh tunnel broken, starting it again"
date
/usr/bin/ssh -f -C -N -L LOCALPORT:REMOTESERVER:REMOTEPORT
fi

RAC -- Important fundamentals about listeners in Oracle RAC 11g Release 2

• With Oracle RAC 11g Release 2 using SCAN is the default.
• SCAN is a combination of an Oracle managed VIP and a listener.
• The SCAN listener represents a standard Oracle listener used in a certain way.
• As with other listeners, there is no direct communication between the node and the SCAN listeners.
• The listeners are only aware of the instances and services served, since the instances (PMON) register themselves and the services they host with the listeners.
• The instances use the LOCAL and REMOTE Listener parameters to know which listeners to register with.
• Any node listener is recommended to be run out of the Oracle Grid Infrastructure home, although the home that a listener uses can be specified.
• Listeners used for a client connection to Oracle RAC should be managed by Oracle Clusterware and should be listening on an Oracle managed VIP.
  
  Ref Doc: RAC: Frequently Asked Questions [ID 220970.1]

EBS-- adadmin>Generate Jar Files -- Do you wish to force generation of all jar files? question

After upgrading Application tier technology files(for example after developers6i upgrade), generating jar files may be necessary...

Using adadmin utility, one can generate jar files...

Before generating the jar files, adadmin asks the question  Do you wish to force generation of all jar files?

The possible answers and explanations of the answers are below;

Do you wish to force generation of all jar files?
No -> Only files that are missing or out of date will be generated.
Yes -> All files are regenerated. This option updates all jar files. Jinitiator then downloads required JAR files to each client again, which can cause performance degregation..