ODA X4 / virtualized, Adding ACFS mount point and make use of SSD disks manually

Before we start let's conclude :) ;

In Oracle Database Appliance, ODA BASE has direct access to the ODA storage.
That 's why, Grid Infrastructure , ASM and Oracle VMs are managed from ODA BASE -- oackli. The things called shared repositories are actually ACFS Volumes, which are exported to the hypervisor (Oracle VM Server's hyperviosor) using NFS and normally Virtual machines are stored in repositories.

Well, to use SSD s from the Virtual Machines in DOMU*, we need to create an ACFS disk on the SSD Diskgroup (REDO diskgroup), export it to hypervisor using nfs and create a disk on it using "dd" and edit the vm.cfg file which belongs to the vm from which we want to reach the SSD disks, manually.

Why manually?

Because oakcli was not designed to create a vm repository on REDO diskgroup.

Let's start.

(this post will be a quick one.. I will not explain every command and output one by one, so feel free to ask questions..)

[root@ermansrv1 ~]# oakcli create repo ermanrepo -size 10g -dg REDO

ERROR: Invalid Memory Unit : 10g

ERROR: Invalid Disk Group value : REDO

Usage:

oakcli create repo <repo_name> -size <size> -dg <diskgroup>

where:

         repo                  -  shared repo name

         -size                 -  size of shared repo to be created

                               -  Minimum Size : 500M or 1G

                               -  Default unit is G

                               -  size must be a whole number.

         -dg                   -  Disk Group of shared repo

                               -  [DATA | RECO]

Well, maybe it is not supported but could not stop me..

Here we begin...

In ODA BASE ;

ASMCMD> volcreate -G REDO -s 100G ssdvolume1
ASMCMD> volinfo -G REDO -a

Diskgroup Name: REDO
Volume Name: SSDVOLUME1
Volume Device: /dev/asm/ssdvolume1-266
State: ENABLED
Size (MB): 102400
Resize Unit (MB): 32
Redundancy: HIGH
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:

[root@ermansrv1~]# /sbin/mkfs -t acfs /dev/asm/ssdvolume1-266
mkfs.acfs: version = 11.2.0.4.0
mkfs.acfs: on-disk version = 39.0
mkfs.acfs: volume = /dev/asm/ssdvolume1-266
mkfs.acfs: volume size = 107374182400
mkfs.acfs: Format complete.


[root@ermansrv1~]# /bin/mount -t acfs /dev/asm/ssdvolume1-266 /u01/app/acfsmounts/ssdrepo1/

[root@ermansrv1~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda2 57192344 52868852 1418220 98% /
/dev/xvda1 470844 24135 422398 6% /boot
/dev/xvdb1 96120004 87525692 3711604 96% /u01
tmpfs 7970240 193300 7776940 3% /dev/shm
/dev/asm/vmtemp2-209 734003200 317151292 416851908 44% /u01/app/sharedrepo/vmtemp2
/dev/asm/vmrepo1-209 4194304000 3615656168 578647832 87% /u01/app/sharedrepo/vmrepo1
/dev/asm/vmtemp1-209 2097152000 843172536 1253979464 41% /u01/app/sharedrepo/vmtemp1
/dev/asm/ssdvolume1-266 104857600 247160 104610440 1% /u01/app/acfsmounts/ssdrepo1


[root@ermansrv1 ssdrepo1]# dd if=/dev/zero of=redodisk1.img bs=1M count=20000

20000+0 records in
20000+0 records ouy

20971520000 bytes (21 GB) copied, 153.403 seconds, 137 MB/s


[root@ermansrv1 ssdrepo1]#vi vm.cfg
vif = ['']
name = 'EBS_12_2_3_ERMAN_DB'
extra = 'NODENAME=EBS_12_2_3_ERMAN_DB'
builder = 'hvm'
cpus = '4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47'
vcpus = 16
memory = 65536
vnc = 1
serial = 'pty'
disk = [u'file:/OVS/Repositories/vmrepo1/VirtualMachines/EBS_12_2_3_ERMAN_DB/128111a52def477b81a1c9758e250ef2.img,xvda,w','file:/u01/app/acfsmounts/ssdrepo1/redodisk1.img,xvdb,w']
maxvcpus = 16
maxmem = 65536

[root@ermansrv1~]# /sbin/acfsutil registry -a /dev/asm/ssdvolume1-266 /u01/app/acfsmounts/ssdrepo1

acfsutil registry: mount point /u01/app/acfsmounts/ssdrepo1 successfully added to Oracle Registry

(this is like adding an entry to fstab. :)

AGAIN IN ODA BASE ;

 [root@ermansrv1 ~]#vi /var/lib/nfs/etab

/u01/app/acfsmounts/ssdrepo1    192.168.16.14(rw,sync,no_wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=65534,anongid=65534)

/u01/app/sharedrepo/vmtemp2     192.168.16.14(rw,sync,no_wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=65534,anongid=65534)

/u01/app/sharedrepo/vmrepo1     192.168.16.14(rw,sync,no_wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=65534,anongid=65534)

/u01/app/sharedrepo/vmtemp1     192.168.16.14(rw,sync,no_wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=65534,anongid=65534)  (ADDED an export for the Hypervisors private network ip)

[root@ermansrv1 ~]# exportfs

In HYPERVISOR;

[root@ermanhype ~]#mount -t nfs 192.168.16.17:/u01/app/acfsmounts/ssdrepo1 /OVS/ssdrepo1   (The ip address is the private network address of ODA BASE machine 1)

[root@ermanhype~]# df

Filesystem           1K-blocks      Used Available Use% Mounted on

/dev/md1              19840804   1684132  17132536   9% /

tmpfs                  1758728         0   1758728   0% /dev/shm

/dev/md3             543397992 271347912 244001800  53% /OVS

/dev/md0                497765     41610    430456   9% /boot

none                   1758728       240   1758488   1% /var/lib/xenstored

192.168.16.17:/u01/app/sharedrepo/vmtemp1

                     2097152000 843172512 1253979488  41% /OVS/Repositories/vmtemp1

192.168.16.17:/u01/app/sharedrepo/vmrepo1

                     4194304000 3615655360 578648640  87% /OVS/Repositories/vmrepo1

192.168.16.17:/u01/app/sharedrepo/vmtemp2

                     734003200 317151264 416851936  44% /OVS/Repositories/vmtemp2

192.168.16.17:/u01/app/acfsmounts/ssdrepo1

                     104857600  20781056  84076544  20% /OVS/ssdrepo1

AGAIN IN ODA BASE ;

[root@ermansrv1~]# df

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/xvda2 57192344 52866888 1420184 98% /

/dev/xvda1 470844 24135 422398 6% /boot

/dev/xvdb1 96120004 87520388 3716908 96% /u01

tmpfs 7970240 193300 7776940 3% /dev/shm

/dev/asm/vmtemp2-209 734003200 317151292 416851908 44% /u01/app/sharedrepo/vmtemp2

/dev/asm/vmrepo1-209 4194304000 3615655372 578648628 87% /u01/app/sharedrepo/vmrepo1

/dev/asm/vmtemp1-209 2097152000 843172536 1253979464 41% /u01/app/sharedrepo/vmtemp1

/dev/asm/ssdvolume1-266 104857600 20781456 84076144 20% /u01/app/acfsmounts/ssdrepo1

[root@ermansrv1 ~]# cd /u01/app/sharedrepo/vmrepo1

[root@ermansrv1 vmrepo1]# cd VirtualMachines/

[root@ermansrv1VirtualMachines]# cd EBS_12_2_3_ERMAN_DB/

[root@ermansrv1 EBS_12_2_3_ERMAN_DB]# vi vm.cfg

vif = ['']

name = 'EBS_12_2_3_ERMAN_DB'

extra = 'NODENAME=EBS_12_2_3_ERMAN_DB'

builder = 'hvm'

cpus = '4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47'

vcpus = 16

memory = 65536

vnc = 1

serial = 'pty'

disk = [u'file:/OVS/Repositories/vmrepo1/VirtualMachines/EBS_12_2_3_ERMAN_DB/128111a52def477b81a1c9758e250ef2.img,xvda,w', 'file:/OVS/ssdrepo1/redodisk1.img,xvdb,w']

maxvcpus = 16

maxmem = 65536

At this point; reboot the oda base node and you will see your the new disk /dev/xvdb  which is built on top of SSD disks. Format it, mount it and start to use it :)

[root@ermansrv1~]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/xvda2             55G   51G  1.4G  98% /

/dev/xvda1            460M   24M  413M   6% /boot

/dev/xvdb1             92G   84G  3.6G  96% /u01

tmpfs                 7.7G  189M  7.5G   3% /dev/shm

/dev/asm/vmtemp2-209  700G  303G  398G  44% /u01/app/sharedrepo/vmtemp2

/dev/asm/vmrepo1-209  4.0T  3.4T  552G  87% /u01/app/sharedrepo/vmrepo1

/dev/asm/vmtemp1-209  2.0T  805G  1.2T  41% /u01/app/sharedrepo/vmtemp1

/dev/asm/ssdvolume1-266

                      100G   20G   81G  20% /u01/app/acfsmounts/ssdrepo1