ODA X6-2M -- virtualization with KVM -- a real life example and my first thoughts

Recently, created a virtualized environment in ODA X6-2M.
I used Kernel Based Virtual Machine for virtualizing this new ODA Medium Model, as instructed by Oracle.

The machine that I worked was like the following ->

[root@odax6 ~]# odacli describe-component
System Version  
---------------
12.1.2.11.0

Component                            Installed Version    Available Version   
---------------------------------------- -------------------- --------------------
OAK                                      12.1.2.11.0               up-to-date          
GI                                         12.1.0.2.170418       up-to-date          
DB                                        11.2.0.4.170418       up-to-date          
ILOM                                   3.2.7.26.a.r112632   3.2.9.23.r116695    
BIOS                                    38050100                 38070200            
OS                                        6.8                           up-to-date  


[root@odaX6 ~]# odacli describe-appliance

Appliance Information                                           
---------------------------------------------------------------- 
                     ID: xxxxxxxxxxxxxxxxxxxx
               Platform: OdaliteM
        Data Disk Count: 2
         CPU Core Count: 20

                Created: August 22, 2017 1:19:18 PM EET

The OS of this ODA machine was Oracle Linux 6.8.
I want to call it as the new ODA, but ODA X7-2 is just released :) It is hard to keep up with this ODA family :)

Anyways, ODA X6-2M is not configured with KVM out of the box.

So, I needed to make the KVM enablement of this environment.

I must admit that, it was pretty easy to enable KVM on this machine.

I just started the libvirtd and installed the virt-manager, which is the GUI of ODA.

[root@odax6 ~]# service libvirtd start

Starting libvirtd daemon: 

[root@odax6 ~]# service libvirtd status

libvirtd (pid  8943) is running...

[root@odax6 yum.repos.d]# wget http://yum.oracle.com/public-yum-ol6.repo

[root@odax6 yum.repos.d]# yum install virt-manager

That was it, the KVM enablement was done!.

After this point, I continued with the storage pool and KVM network configurations.

In order to configure/create the storage pool; I first created an ACFS volume using asmca ->

[grid@odax6 asmca]$ asmca -silent -createVolume -volumeName kvm_repo1 -volumeDiskGroup DATA -volumeSizeGB 300 -sysAsmPassword welcome1
[grid@odax6 asmca]$ asmcmd volinfo -G DATA kvm_repo1 | grep -oE '/dev/asm/.*'
/dev/asm/kvm_repo1-33

Then, I created the ACFS filesystem on top of it and mounted it using a single command; --again using asmca silently ->

[grid@odax6 asmca]$ asmca -silent -createACFS -acfsVolumeDevice /dev/asm/kvm_repo1-33 -acfsMountPoint /kvm_repos/kvm_repo1

ASM Cluster File System created on /dev/asm/kvm_repo1-33 successfully. Run the generated ACFS registration script /u01/app/grid/cfgtoollogs/asmca/scripts/acfs_script.sh as privileged user to register the ACFS with Grid Infrastructure and to mount the ACFS. The ACFS registration script needs to be run only on this node: odax6.

-- needed to run acfs_script.sh using root as a part of this ACFS creation.

[root@odax6 ~]# sh /u01/app/grid/cfgtoollogs/asmca/scripts/acfs_script.sh

ACFS file system /kvm_repos/kvm_repo1 is mounted on nodes odax6

Later on, I checked my mounts and saw that the new ACFS is there.

root@odax6 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot
                       30G   21G  7.7G  73% /
tmpfs                 126G  631M  126G   1% /dev/shm
/dev/sda1             477M   46M  406M  11% /boot
/dev/mapper/VolGroupSys-LogVolOpt
                       59G   11G   46G  18% /opt
/dev/mapper/VolGroupSys-LogVolU01
                       99G   21G   73G  23% /u01
/dev/asm/dattest-33   100G  1.7G   99G   2% /u02/app/oracle/oradata/test
/dev/asm/reco-481     149G  5.2G  144G   4% /u03/app/oracle
/dev/asm/commonstore-33
                      5.0G   49M  5.0G   1% /opt/oracle/dcs/commonstore
/dev/asm/kvm_repo1-33
                      300G  648M  300G   1% /kvm_repos/kvm_repo1

After the creation of the repo, I started the repo and also made it autostart.

[root@odax6 ~]# virsh pool-start kvm_repo1

Pool kvm_repo1 started

[root@odax6 ~]# virsh pool-autostart kvm_repo1

Pool kvm_repo1 marked as autostarted

Then checked it to see whether it is there and whether its size and everything were configured properly.

Note that virsh is a command line management tool for KVM.

[root@odax6 ~]# virsh pool-info kvm_repo1

Name:           kvm_repo1

UUID:           97dda9d1-ca6b-c9e2-bbfc-901cc2274898

State:          running

Persistent:     yes

Autostart:      yes

Capacity:       300.00 GiB

Allocation:     647.62 MiB

Available:      299.37 GiB

[root@odax6 ~]# virsh vol-list --pool kvm_repo1

Name                 Path                                    

-----------------------------------------

lost+found           /kvm_repos/kvm_repo1/lost+found  

At this point , my storage pool and volumes were configured properly.

Continued with the network stack..

I needed to arrange a network, a virtual interface for the virtual machines that would reside on this ODA environment. (The virtual machines in my case, were EBS Application Tier nodes. )

I had 2 option. (actually 3 , if we count networking with MacVTap..)

Anyways, the first option was Nat forwarding. Nat forwarding could not be used in my case, because the network of ODA X6-2M and the network of the virtual Application tier nodes(that would reside on ODA X6-2M) were the same. Their IPs were from the same block , so I had to use the other option, which was the Bridged networking ("shared physical device").

This method was actually the full bridging, which could let the guest (EBS Application tier nodes in my case) to be able to connect directly to the LAN.

In order to configure these network things , I used the virt-manager. 

However, virt-manager had some fonts problems, so I needed to fix them first.

Here is a little info, and the fix for it:

virt-manager is management interface that eases the administration of the KVM environment (in ODA or in anywhere else) It is called  Virtual Machine Manager and it is executed using the command virt-manager (using root).
As it is a GUI, it needs a X environment to run it.
In Oracle Linux world, as you may also agree, we mostly use vncserver for displaying the X screens remotely.
So, we connect to the vncserver (or we can use ILOM remote connection or anything that does the same thing) and execute the virt-manager to start the Virtual Machine Manager for KVM.
The issue starts here.
After the deployment of ODA and enabling the KVM, we run the virt-manager command and we see the garbage characters.
We actually see little squares rather than the characters and fonts.
So, in order to fix this, we basically need to install the fonts that Virtual Machine Manager needs.
A simply yum command can do this work and this little piece of information may save you time :)
Fix: yum install dejavu-lgc-sans-fonts

Well.. After the fix, I could use the virt-manager without any problems.

So in order to configure the vm network; I did the following;

Opened virt-manager.

Connected to the KVM environment.

Created a bridge named br1 on btbond1 and activated it directly. (using the network interface tab)

--I used this bridge for multiple machines. (I had 2 apps Vm machines on ODA, so their virtual NICs are based on this bridge called br1)

All done from GUI (virt-manager) and that was it..

My KVM network was configured.

The last thing to do was, creating my virtual machines for my EBS Apps nodes and installing the Operating Systems (in my case, Linux) on them.

The virtual machine creation and OS installation was extremely straight forward.

Again , I used the virt-manager.

In order to create a virtual machine and configure it to be booted with OS installation media, I did the following ->

I clicked the "Create a new virtual machine" button to open the new vm wizard

Specified the installation type "Local install media (ISO image)" -- clicked next:)

Located the ISO image, Configure OS Type and Version (linux, Redhat 6 in my case, I already downloaded the OS installation ISO and placed it into ODA X6-2M earlier.) -- clicked next:)

Configured CPU and memory -- clicked next:)

Configured the VM's local disks and their sizes. (on the ACFS volume created previously) --clicked next:)

Lastly, selected the network device : (br1 - the bridge in my case) -- this case clicked the Finish button:)

I did these things 2 times, because I had to have 2 apps virtual machines on ODA.

After creating the virtual machines, I started them using virt-manager and they were booted with Oracle Linux 6 installation media. I used the console that comes with the virt-manager to install the OS and then directly started using the Apps Nodes without any problems. (after configuring the network, their IPs, ofcourse)

At the end of the day, I got myself a virtualized ODA X6-2M.
This virtualization was a little different than the Oracle VM Server virtualization that we had in the earlier releases of ODA.

In ODA X6-2M, we use KVM ... So, there is no ODA_BASE, we just place our databases directly on ODA nodes and create our guest machines for the Apps Tier nodes.

In short, apps nodes are running on VMs on top of KVM, and databases are running directly on ODA nodes. (so they are running on Bare Metal) .. ("as instructed by Oracle")

We still have capacity on demand, both for databases and virtual machines.

This was an interesting work for me.. After all these years dealing with Oracle VM Server, I configured a new virtualized ODA with a different virtualization technology. Anyways, I liked it and found it as a good and an easy virtualization solution.

We will also see its performance in the couple of days...