RAC-- 11.2.0.3 RAC Installation on 2 node Linux 6.5 64 bit, a comprehensive installation from Storage to RAC

In this post, I will try to explain 11.2.0.3 RAC Installation on an environment consisting of 2 Linux nodes.

Oracle Grid and RDBMS versions mentioned in this document are 11.2.0.3 64 bit.

Lets take a quick look of the system components used in this paper;

OS: Oracle Linux 6.5 Enterprise Edition 64bit ( booted with UEK) are used for the OS layer.
Machine: Server are HP, There are 4 Local Disks on each node. These disks are used for the OS installation (root). The raid configuration used for these 4 local disks is Raid 1+0
Storage : used in the environment is Netapp. Storage connects to the servers through fiber(FCP) from 4 different paths.
A separete lun created for each node. These luns are mapped to the nodes for storing the Grid and Rdbms softwares.
6 luns for ASM are mapped on each node. These luns are used for ASM DATA and FRA Diskgroups.
Network: Both virtual and private networks are redundant.(ethernet devices and network switches are redundant.)

As can be understood from the title, we do a RAC installation but we do all the work from the raid configuration to OS installation, from the multipath the ASM disk creation and so on..

So; I will explain the installation process screen by screen (not step by step :)) in the following paragraphs ..

For now, lets take a look at the installation steps;

1)  Raid configuration of Local Disks
2)  Mounting OS Installation ISO from the ILOM and installing Oracle Linux 6.5 64 bit to our rac nodes.
--Note: Firmware bug error displayed in first boot may be ignored.The functionality and performance of the operating system and the server are not affected.
3)  Disabling selinux , iptables and ip6tables once our OS Installation is finished.
4)  Configuring public and private interconnect as bond devices.(active-backup mode)
5)  Configuring sysctl and limits.conf on our rac nodes. (Note that these configurations come configured by default if you install preinstall rpm during the OS installation)
6)  Checking the rpms for Oracle installation , and installing the missing ones.
7)  Resetting oracle OS user's password and adding the needed groups and users.
8)  Preparing the hosts files.
9)  Syncing the dates initially.Configuring the ntp daemons.
10) Installing telnet on each node.(optional)
11) Checking the /dev/shm size on each node and increasing it if needed.(consider memtarget..)

12) Checking the update level of oracleasm on each node., increasing it if necessary.13) Checking oracleasm kernel driver.. (it comes built in with the Oracle Linux 6 installation)
14) Checking the oracleasm-support ..(If you choose it ; it can be installed during the Oracle Linux 6 installation)
15).Installation of oracleasmlib.
16) Uploading the installation files to node 1, and unzipping them.
17) Installation of libHBAAPI rpm (prereq for host utils) on each node. (rpm can be found in the installation media)
18) Installation of Netapp host utils on each node.
19) After the luns are mapped, configuring the multipaths on each node.(adding some disks to blacklist if needed)
20) Aligning the SAN disks planned to be used for the storing Grid and Rdbms software on each node.. Formatting them ext3/or ext4 and mounting them on each node..  Also adding the needed lines to te fstab file on each node.
21) Aligning the SAN disk planned to be used for ASM
22) Configuring dns on each node.(resolv.conf) (Dns admin should add the public, vip and scan ip addresses in to the customer's dns server. We should be able to resolve them before the installation.)23) Creating the installation/oracle directories(optional)
24) Configuring the .bash_profile of the root user to make the root user be able to execute crs command(optional)
25) Initializing oracleasm on each node.

26) Creating oracle ASM disks from node 1.
27) Installing Grid Infrastructure using oracle Os user in node 1.. (executing root.sh scripts when requested by the installer)
28) Adding the Diskgroup for FRA using asmca
29) Installing Rdbms software  using oracle OS user. (executing root.sh scripts when requested by the installer)

Before going into the installation screenshots, lets identify the technologies implemented during this work;

Ilom: 
Actual name of the technology is ILO, which server management technology that comes embedded with the HP servers. By using ILOM, we connect to the servers console using ILOM's own ethernet port and do the management using our browser (without a need for going in to the System Room)
ILOM can be found most Proliant servers of the 300 and above series.

So, we do our Linux 6.5 installation remotely using ILOM.

Raid Configuration 1+0:

These raid configuration can also be called as "stripe of mirrors”.  It requires minimum of 4 disks.

The idea in this configuration is "data in the same groups are mirrored and data accross the groups are striped." It is something like displayed in the following picture.

So for 4 disks -> we can say that: 

we mirror the data in Disk 1 and Disk 2, also we mirror the data in Disk 3 and Disk 4. Then we concatenate disk two disk groups together in a single structure. This single structure is named as Raid 1+0.

The fault tolerancy is high..

Given 4 disks; , even if 2 disks fails (one in each group), the RAID 10 is still functional.

So we build a Raid 1+0 group on each node to place our OS files.

Oracle Linux 6.5 :

Oracle Linux is an open-source operating system available under the GNU General Public License (GPLv2). Suitable for general purpose or Oracle workload.
With Oracle Linux 6.5, the latest Unbreakable Enterprise Kernel  Release 3 is installed as the default kernel, enabling customers and partners to test, validate and certify against the most up-to-date Oracle Linux distribution.

So, we install Oracle Linux 6.5 64 bit to our Rac servers.

Uek 3:

The Unbreakable Enterprise Kernel Release 3 is Oracle's third major release of its heavily tested and optimized operating system kernel for Oracle Linux 6, x86_64 platforms..

The Unbreakable Enterprise Kernel Release 3 is based on the mainline Linux kernel version 3.8.13 and boasts a wide range of new features and improvements relevant for enterprise workloads.

Testing of the Oracle Unbreakable Enterprise Kernel includes both the operating system and the full Oracle application stack running on top of it, verifying stability and correct behavior in all situations. The test suite includes installation, functional, stress and destructive tests, which are executed under varying workloads (e.g. I/O or CPU intensive), with various database sizes. Several parameters are varied during the full run of the test suites, which also include crash scenarios in Oracle Real Application Cluster configurations.

So , we choose to install Oracle Linux 6 and boot it with Uek 3.. It is a verified against all of the application stack and Rac ..

Selinux:

Selinux provides a fine grained level of access control in Linux Operating System.

In our installation, we 'll disable selinux ,because it may block the network traffic as follows;

PRVF-7617 : Node connectivity between "racnode01 : 10.0.0.1" and "racnode02 : 10.0.0.2" failed Result: TCP connectivity check failed for subnet "10.0.0.0"

We dont want to face these kind of security error, and that's why disable selinux before the installation.

On the other hand; it seems running Selinux in enforcing mode on OL6 seems okay(see below), but as I said; I prefer to disable it unless it is really needed.

 By default, RHEL 6 x86_64 Linux is installed with SELinux as "enforcing". This is fine for the 11gR2 installation process. However, to subsequently run "sqlplus", switch SELinux to the "Permissive" mode. See  NOTE 454196.1 "./sqlplus: error on libnnz11.so: cannot restore segment prot after reloc" for more details. 

UPDATE: Internal testing suggests that there is no problem running "sqlplus" with SELinux in "enforcing" mode on RHEL6/OL6. The problem only affects RHEL5/OL5.

Iptables:

It is a firewall on Linux. It is used to set up, maintain, and inspect the tables of IP packet filter rules in the Linux kernel.
We disable it before the installation for making a clean installation.
Later, it may be configured if needed. But if we need to configure it, we should configure it with the rules which do not affect the cluster interconnect interfaces, protocols and network address space.

Ip6tables:

IPv6 version of Iptables. It is a firewall.Ip6tables is used to set up, maintain, and inspect the tables of IPv6 packet filter rules in the Linux kernel.We just disable it before the installation. We dont use IPV6 anyways.

Bond devices/bonding:

/dev/shm:

Recent 2.6 Linux kernel builds have started to offer /dev/shm as shared memory in the form of a ramdisk, more specifically as a world-writable directory that is stored in memory with a defined limit in /etc/default/tmpfs. /dev/shm support is completely optional within the kernel config file. It is included by default in both Fedora and Ubuntu distributions, where it is most extensively used by the Pulseaudio application.Oracle uses /dev/shm to pass segments from SGA(shared memory) to PGA (process memory).. I think it just gives back the shared memory back to OS ,and OS gives it to the processes in case they needed. Giving back the unused memory should be enough for managing the PGA,as is not a limit and it s is not a fixed memory area resides in memory..So far so good.. On the other hand, 10g Oracle does not use memory_target , naturally it does not use a /dev/shm based memory access..

As we deal with 11gR2 Rac , we configure this /dev/shm according to desired our memory_target value.

So far so good.. On the other hand, 10g Oracle does not use memory_target , naturally it does not use a /dev/shm based memory access..

sysctl.conf:

sysctl is an interface that allows you configure some aspects of a running Linux kernel. We canfigure the needed settings for Oracle(like shared memory size, semaphore counts etc..) using /etc/sysctl.conf , and activate this change using sysctl -p command.

limits.conf:

The pam_limits.so module applies ulimit limits, nice priority and number of simultaneous login sessions limit to user login sessions. Using /etc/security/limits.conf file we configure the limits(such as number of open files, process count etc..) for our oracle user.

oracleasm kernel driver:

oracleasm is a Linux kernel driver also known as the Oracle ASMLib kernel driver.
This is an open-source (GPL) kernel driver and is available from Oracle as source and binary
RPMs. Note that although this driver is provided under an open source license, it has not been accepted into the mainline Linux kernel.

oracleasm kernel driver is included in Oracle Linux 6.

oracleasmlib:

ASMLib is an optional support library for the Automatic Storage Management feature of the Oracle Database. Automatic Storage Management (ASM) simplifies database administration and greatly reduces kernel resource usage (e.g. the number of open file descriptors). It eliminates the need for the DBA to directly manage potentially thousands of Oracle database files, requiring only the management of groups of disks allocated to the Oracle Database. ASMLib allows an Oracle Database using ASM more efficient and capable access to the disk groups it is using.

multipath/device mapper/dm-n devices:

Using multipath (Device mapper multipathing) we can access a device from multiple I/O paths. When I say I/O paths, I mean accessing the same disk/IO resource from different ways .

For example : Suppose we have 2 hba in our host and we connected these hbas to 2 SAN controller which accessing one single storage disk. Then we can say that have 2 ways to access that particular disk.

Way1 ) HBA1 -> Storage Controller 1 > Disk(shelf)
Way2 ) HBA1 -> Storage Controller 2 > Disk(shelf)

These paths can be used in active-active for increasing the performance or they can be used as active-passive for providing failover.
Also, we can even increase these paths by putting an SAN switch in front of the Storage Controllers and etc..
So , multipath appears in this stage. Once configured, we see underlying devices as a single multipath device, and make the IO using it.  So multipath provides a single device on top of the underlying devices.

For example: In the example above;  Linux will see the same disk 2 times. Lets say /dev/sdb and /dev/sdc..

These devices are actually representing the same thing. 

Multipath creates a new device such as /dev/mapper/mpathn, /dev/mpath/mpathn, and /dev/dm-n  for representing these devices as a single device and reroutes the IO to the underlying devices..

The devices in /dev/mapper are created early in the boot process. Use these devices to access the multipathed devices, for example when creating logical volumes , or creating ASM diskgroups in our case..

The devices in /dev/mpath are provided as a convenience so that all multipathed devices can be seen in one directory. These devices are created by the udev device manager and may not be available on startup when the system needs to access them. We dont  use these devices for creating logical volumes or filesystems.

/dev/dm-n are for internal use only and should never be used. These dm-n are created by some udev rules and they may be changed according to the sequence of device activation. These dm-n device names seems like kernel names for corresponding devices. Maybe they are there for protecting the standards, as their root directory is /dev..

Anyways we dont use these dm-n devices, we use /dev/mapper devices instead.

/etc/multipath.conf is the configuration file of multipath.Supported devices can be found in the multipath.conf.defaults file. If your storage array supports DM-Multipath and is not configured by default in this file, you may need to add them to the multipath.conf.

Multipath consists of a kernel module(dm-multipath), a command line program(multipath), a daemon(multipathd) and another command line program (kpartx)

Okay lets introduce these components;

So, if we want to have a fault tolerant IO path and if we have the multi physical paths, then device mapper is what we need to implement.

In this RAC installation, we configured and used multipath devices for both ASM and ext4 drives.

libHBAAPI:

The libhbaapi library is the Host Bus Adapter (HBA) API library for Fibre
Channel and Storage Area Network (SAN) resources. It contains a unified API
that programmers can use to access, query, observe and modify SAN and Fibre
Channel services.

We install it , because it is a prereq for Netapp Host Utils package.

Netapp Host utils:

The Host Utilities include a set of software programs to help you use your Linux host with storage systems.

We use this utils to manage our San luns, see their status and more.

For example "sanlun" command comes with NetApp host utils.

Disk alignment:

Host file system need to be aligned with a Storage Lun, in order to achieve best performance.
This can be accomplished specifying the correct starting sector or logical block address (LBA) for the partition, and filesystem should be configured to use the same or multiple of the Storage block size..

On Linux , you can configure a partition and filesystem using fdisk (x option)
Alignment is important, and should be considered for Storage IO performance.Otherwise, you can end up with the following;

It is almost equal -> doing 2 Storage IO for 1 OS IO..

Oracle Grid:

The Oracle Grid Infrastructure for a standalone server is the Oracle software that provides system support for an Oracle database including volume management, file system, and automatic restart capabilities. If you plan to use Oracle Restart or Oracle Automatic Storage Management (Oracle ASM), you must install Oracle Grid Infrastructure before installing your database. Oracle Grid Infrastructure for a standalone server is the software that includes Oracle Restart and Oracle ASM. Oracle combined the two infrastructure products into a single set of binaries that is installed as the Oracle Grid Infrastructure home. Oracle Grid Infrastructure should be installed before installing Oracle Database 11g Release 2.

Oracle ASM is a volume manager and a file system for Oracle database files that supports single-instance Oracle Database and Oracle Real Application Clusters (Oracle RAC) configurations. 

Oracle ASM also supports a general purpose file system for your application needs including Oracle Database binaries. Oracle ASM is Oracle's recommended storage management solution that provides an alternative to conventional volume managers, file systems, and raw devices.

So, in order to use RAC and ASM we first install the Grid Infrastructure to our nodes, and install Oracle Rdbms afterwards.

Oracle RDBMS:

It is the Oracle Database. In this post ,we  will see the instructions related to Oracle Database 11GR2.

So far so good...
At this point ,we know what we are dealing with.. We have the idea about  the components which need to be configured/implemented/used in the pipeline and we have the method for installing our RAC environment as well.

Lets proceed with installation screens by briefly explaining them..

We start with Raid configuration.. We configure Raid 1+0 for our 4xlocal disks on our HP nodes.

Click here to read more/ for the installation screenshots