In this post, I will cover an ODA(Oracle Database Appliance) X4-2 installation to support virtualization of EBS 12.2 environments.
As you know, Oracle Database Appliance is a point of entry for the Oracle Engineered Systems. It is known as Baby Exadata. It has 2 compute nodes and 1 storage server for building a 4U consolidated environment. It supports Oracle RAC and Oracle Virtual Server environments..
So to install or configure ODA, we have 2 options.
One option is to use ODA hardware as Bare metal and the other one is to use ODA hardware with the virtualization technology of Oracle. (Oracle VM server)
In this arcticle, you will find an overview of using Oracle Database Appliance as a virtualized plaftorm.
I will go on by giving the explanation from a real life example , we just implemented 1 week ago..
So, we actually implemented 5X EBS 12.2.3(including VISION) environments on a virtualized ODA x4-2 platform, and we have seen the difficulties and advantages of using ODA for the virtualization, and implementing virtual EBS servers..
Lets start by taking a look to the ODA X4-2 installation&configuration;
In order to have a virtualized ODA, we need supply Ip addresses for following interfaces;
2x HOST IP' s. (Oracle Virtual Db Servers)
2x Private IP's (If you will use RAC)
2xDom0 IP's (For Oracle VM Servers)
2xSCAN IP's(IF you will use RAC)
2x Oracle ILOM IP'S( For connecting to the Compunte nodes through NETMGT)
Also, at least 1 ip address for Each Virtual Machine..
There is also 2 method for using Virutalized ODA. We can use RAC (if we have license), or we can choose not to use RAC.
I'll explain Virtualized ODA installation with a non-RAC configuration..
Note that: the best practice is using a RAC based ODA virtualization, but I will not explain it in here, because it seems it is easy to handle, thus less effort neeeded for the deployment.(Oracle Doc: Best practices for deploying Oracle E-Business Suite Release 12.2.x on Oracle Database Appliance)
The installation of ODA X4-2 was made in Linkplus Partner Hub.
Mr Timur Yalcin from Linkplus prepared a similar network environment for us to deploy and work with the new purchased ODA in Linkplus PartnerHub. We worked with Mr Timur throughout the deployment process .. So after the configuration and deployment , we transfferred ODA x4-2 to the Data Center of our Customer, and everyting have worked properly without a need to change even an Ip address..
The initial installation of ODA was done from the serial port. (SERMGT)
Firstly, we connected to ODA from serial port(root/changeme) and found the ip address of the nodes.
Then we connected to the both nodes through ILOM interfaces(NETMGT) using ip information we just gathered, and opened consoles.
Note that, you can find related information in Support document :888888.1(ODA related document) Also, you can download patches and installation packages from the document above..
As mentioned earlier, there are two download options out there. 1) Virtualized 2)Bare :Metal.. We have implemeted the 1st one.
Okay, after downloading the ISO files, we used our clients to install ISO files.. (virtual cd rom)
I mean, We mount the ISO to the ODA through network..
The installation had 2 steps .. First the OVM servers was installed to the Compute nodes of ODA, and then ODA Base was installed on top of the OVM servers..
Since the customer had no RAC license, we couldnt apply the Best practice , as we shouldnt use RAC in ODA x4-2. So we use the database servers coming with the ODA Base ,as our ASM isntances.. Therefore, we only started ASM instances on them..
Our plan was to use the ASM diskgroups though transperent ACFS for placing our vm repositories. So, all the virtual disk were created on these ACFS filesystems and it all went well..
Lets explore this process a little furhter;
We needed to use ASM because ODA has no Raid controller on its Storage server.. A raid configuration was needed.. The Storage disks could be seen from the ODA Base virtual machines , and those virtual machines had ASM configured by default. So we have used ASM for our shared disk/repository storage..
So the architecture was like above..
We use ASM in order to build a fault tolerant IO Subsystem, and we created our virtual machine repositories using ASM disgroups (For example: oakcli create repo ERMAN -dg DATA -size 700(Gig by default)
Note that : Another repository should be created on top of ASM Diskgroup that includes SSD disks. That is , Virtual disks for storing redologs should be created on the Vm disks which reside on this diskgroup.
We created two repositories actually.. One for the templates which we would import and use as virtual machines, and other one for actual virtual machines..
Note that: you can resize your Repositories created on ACFS filesystems..
Altough , I couldnt see an oakcli argument for that, it can be done by resizing the underlying IO device.. In case of an ACFS filesystem, resizing OVM respository means resizing the underlying ACFS filesystem, which can be done using asmca or acfsutil tools.
Note that: I have done this, and didnt encounter any problems
but I couldnt find an Oracle Support Document to support this. That'why, the risk is yours.
Anyways, We imported our templated into our repository dedicated for templates and we created virtual machines from these templates in the repository created for our virtual machines.. (oakcli import vm template..)
Our goal was to build a Virtualized little EBS Farm, so we created our repository based on EBS sizes..
Following table lists the minimum requirements for the EBS 12.2.3 templates..
Virtual Appliance
|
Size of Virtual Disk
(in GB)
|
Actual Disk Space Used on File System (in GB)
|
RAM
|
VCPU
|
Domain Type
|
|---|
| Oracle E-Business Suite Release 12.2.3 Vision Demo Database Tier |
300
|
189
|
2
|
1
|
HVM
|
| Oracle E-Business Suite Release 12.2.3 Production Database Tier |
300
|
93
|
2
|
1
|
HVM
|
| Oracle E-Business Suite Release 12.2.3 Application Tier |
300
|
67
|
6
|
1
|
HVM
|
| Oracle E-Business Suite Release 12.2.3 Single Node Vision Install |
500
|
256
|
6
|
1
|
HVM
|
| Oracle E-Business Suite Release 12.2.3 Sparse Tier/OS Install |
300
|
2
|
6
|
1
|
HVM
|
Okay lets have a closer look to the template operations..
Once we set up our VM repository environments , we were ready to import our EBS templates.
For using EBS 12.2 , we downloaded 12.2.3 templates from Oracle's website edelivery.oracle.com/oraclevm..
We unzip and concatenate them to have a one single ova file for each template and we imported them using oakcli..
After the import, we cloned virtual machines from these templates.. Once the cloning ass process done, we were ready to configure and start our virtual machines. Configuration is needed after the cloning, because we needed to configure the Memory and Cpu limits of these virutal machines.
Anyways, we used oakcli configure and oakcli modify commands to configure our requirements, and that's it ..
Examples of using oakcli commands are below;
To import a Template:
oakcli import vmtemplate EBS_12_2_3_PROD_DB -assembly /OVS/EBS/Oracle-E-Business-Suite-PROD-12.2.3.ova -repo vmtemp2 -node 0
To list the available Templates:
oakcli show vmtemplate
To list the Virtual Machines:
oakcli show vm
To list the repositories:
oakcli show repo
To start a virtual machine:
oakcli start vm EBS_12_2_3_VISION
To create a Repository (size in gb, by default):
oakcli create repo vmrepo1 -dg data -size 2048
To configure a Virutal Machine: (cpu, memory etc..)
oakcli configure vm EBS_12_2_3_PROD_APP -vcpu 16 -maxvcpu 16
oakcli configure vm EBS_12_2_3_PROD_APP -memory 32768M -maxmemory 32768M
To open a console for a virtual machine (Vnc required)
oakcli show vmconsole EBS_12_2_3_VISION
To create a virtual machine from a template:
oakcli clone vm EBS_12_2_3_PROD_APP -vmtemplate EBS_12_2_3_PROD_APP -repo vmrepo1 -node 1
After this point; we could start our virtual EBS servers..
Note that : All the oakcli operations was done from the ODA base machines. I mean the virtual machines which host ASM instances.. (not from the DOM 0 machines)
We first started with EBS 12.2.3 vision, which was as a single server in our case.. When we booted EBS 12.2.3 Vision, it didnt not make us configure itself automatically. Therefore, we used following scripts to configure it ..
Note that: In vision 12.2.3 templates, OS account "oracle" is the owner of both Application tier and database tier..
We first configured the ip addresses and hostnames using script named configstatic.sh.. (there was a script named configdhcp for dhcp configuration, but in our case we needed to use static ip configuration)
The configstatic.sh path is as follows;
/u01/install/scripts/configstatic.sh
Note that : There is a bug in configdhcp.sh "unexpected fi" at line 62.
Therefore, you need to modify the script and add "then" to the if..then..fi statement, before using it..
Okay after configuring our network, we used visiondbconfig.sh script to configure our database environment.. (for setting SID etc...)
/u01/install/VISION/scripts/visiondbconfig.sh
After configuring the db tier, we used visionappsconfig.sh to configure apps tier.. I mean to connect our application services to the database we just configured. We always used root for running these scripts...
/u01/install/VISION/scripts/visionappsconfig.sh
Note that : All these scripts are mentioned in note: Oracle VM Virtual Appliances for Oracle E-Business Suite Deployment Guide, Release 12.2.3 (Doc ID 1620448.1)
We have done these configuration for our PROD , TEST , DEV and similar environments ..
In generally , the configurations were made similar, but the script names were different, as they were different templates.. ( Fresh &Vision)
You can find the configuration scripts for EBS 12.2.3 Fresh/PROD Templates below;
The scripts to manage the Oracle E-Business Suite PROD instance are:
SCRIPTS BASE_DIR : /u01/install/PROD/scripts/
START SCRIPT : /u01/install/PROD/scripts/startproddb.sh
STOP SCRIPT : /u01/install/PROD/scripts/stopproddb.sh
DB VM RECONFIG SCRIPT : /u01/install/PROD/scripts/prodbconfig.sh
DB VM CLEANUP SCRIPT : /u01/install/PROD/scripts/proddbcleanup.sh
The scripts to manage the Oracle E-Business Suite application tier instance are:
SCRIPTS BASE_DIR : /u01/install/APPS/scripts/
START SCRIPT : /u01/install/APPS/scripts/startapps.sh
STOP SCRIPT : /u01/install/APPS/scripts/stopapps.sh
APPS VM RECONFIG SCRIPT : /u01/install/APPS/scripts/appsconfig.sh
APPS VM CLEANUP SCRIPT : /u01/install/APPS/scripts/appscleanup.sh
CONFIGURE A NEW WEB ENTRY POINT : /u01/install/scripts/configwebentry.sh
Note that;
While importing EBS 12.2 PROD database template in to a Virtualized Oracle Database Appliance(ODA) platform, you may get OAKERR:7044 error.
Example command:
oakcli import vmtemplate EBS_12_2_3_PROD_DB -assembly /OVS/EBS/Oracle-E-Business-Suite-PROD-12.2.3.ova -repo vmtemp2 -node 0
Error:
OAKERR:7044 Error encountered during importing assembly - Cannot find OVF descriptor file.
This error is caused by the files in the ova files.. That is, oakcl cant handle the situation , if the name of the file packaged in the ova, contains a space character..
As follows;
tar -tvf Oracle-E-Business-Suite-PROD-12.2.3.ova.problem
-rw------- someone/someone 13485 2014-02-07 09:16:36 Oracle-E-Business Suite-PROD-12.2.3.ovf
-rw------- someone/someone 17032424448 2014-02-07 10:22:42 Oracle-E-Business Suite-PROD-12.2.3-disk1.vmdk
As you see above, the names for both ovf and vmdk contains empty spaces.. That's why, oakcli cant handle and throws "Cannot find OVF descriptor file" error.
To fix this, you may use tar command in Linux, or a similar tool depending on your platform.
Here is what I did to resolve the problem;
Solution:
1) First, I extracted the files to a directory called erman_ova_1
tar xvf Oracle-E-Business-Suite-PROD-12.2.3.ova -C erman_ova_1/
2)Then I changed the file names, and make them not to contain any empty spaces
cd erman1/
mv Oracle-E-Business\ Suite-PROD-12.2.3.ovf Oracle-E-Business-Suite-PROD-12.2.3.ovf
mv Oracle-E-Business\ Suite-PROD-12.2.3-disk1.vmdk Oracle-E-Business-Suite-PROD-12.2.3-disk1.vmdk
3)Also , opened ovf file and changed all the file names containing space characters.. Note that: the file name to be imported is also written in the ovf file..
4)Then I moved the original ova file to not to overwrite it.
mv Oracle-E-Business-Suite-PROD-12.2.3.ova Oracle-E-Business-Suite-PROD-12.2.3.ova.problems
5)Lastly I recreated the ova file from the directory where I have modified the filenames..
cd erman1/
tar -cvf /OVS/EBS/Oracle-E-Business-Suite-PROD-12.2.3.ova Oracle-E-Business-Suite-PROD-12.2.3.ovf Oracle-E-Business-Suite-PROD-12.2.3-disk1.vmdk
Note that: ovf file should be packaged first..
So, after importing templates and creating virtual machines we had a Virtualized ODA environments as follows;
As you see above, all of our EBS Virtual machines are using ASM to communicate with storage..
So VM machines which were deployed with ASM are present on each ODA node..
ASM instances manage the shared repo that reside on the Storage server of ODA ..
This also have brought an opportunity to start a virtual machine in another ODA node, if its current node fails.. On the other hand; there was a dependency created .. So our vm machines have become dependent to the vm machines which host ASM instances.. So we need to start them first in order to start our EBS virtual machines..
In ODA X4, the pci cards that are connected to the disk storage, are bypassed. That is, the main machines are bypassed, so the ASM VM nodes communicate with the disks almost directly.
This seems to be an improvement in IO peformance.. Also the disks are SAS based. So we have 4 x fault tolerant SAS connections to the Storage device. (2 from each node)
Also note that: Fiber is used for interconnect ..
ODA computes nodes are 24 cores each. With hyperthreading it makes 48 logical cores each.
You can find the information as it seen from the OS..
Note that: If we know the power of the underlying hardware, we can confiure our nodes better to achive the best performance and to have a better utilization..Here is the specs of a ODA x4-2 node.. Note that, in a node, we have 2 cpus , 12 cores and 2 threads per core. So Hyper Threading is enabled. This makes 48 virtual cores..
Also, we have 256 gb memory per node and We use 2.6.39-4000 uek smp kernel with xen.
Below is the hardware stat gather from an ODA node;
release : 2.6.39-400.126.1.el5uek
version : #1 SMP Fri Sep 20 10:54:38 PDT 2013
machine : x86_64
nr_cpus : 48
nr_nodes : 2
cores_per_socket : 12
threads_per_core : 2
cpu_mhz : 2693
hw_caps :
virt_caps : hvm hvm_directio
total_memory : 262086
free_memory : 192913
free_cpus : 0
xen_major : 4
xen_minor : 1
xen_extra : .3OVM
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
xen_commandline : dom0_mem=4096M crashkernel=256M@64M
cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)
cc_compile_by : mockbuild
cc_compile_domain : us.oracle.com
cc_compile_date : Mon Feb 4 16:40:15 PST 2013
xend_config_format : 4
In conclusion, even with ASM as a repository, I can say that: after configuring and starting our EBS servers, we havent seen any anomalies. The performance is acceptable, and the enrironments are stable. Also the management of the virtualized platform is so easy, even without Oracle VM manager..
Okay that's I have to say for now..
Ooo I almost forgot.. One more thing , which is a little important :)
EBS 12.2.3 Fresh templates (PROD) come with US7ASCII character set.. That is , if you are in a country which has special characters in its alphabet, you are in throuble :) You will even not able to licese your language with that character set.. (Note that: There is no such a problem in VISION template because it comes with utf8)
So, if he installation is already done the best option is to change the characterset following below document..
Migrating an Applications Installation to a New Character Set (Doc ID 124721.1)
Oracle suggests you to perform 2 iterations before doing the same in production.
