ODA X4 / virtualized, Adding ACFS mount point and make use of SSD disks manually

Before we start let's conclude :) ;

In Oracle Database Appliance, ODA BASE has direct access to the ODA storage.
That 's why, Grid Infrastructure , ASM and Oracle VMs are managed from ODA BASE -- oackli. The things called shared repositories are actually ACFS Volumes, which are exported to the hypervisor (Oracle VM Server's hyperviosor) using NFS and normally Virtual machines are stored in repositories.

Well, to use SSD s from the Virtual Machines in DOMU*, we need to create an ACFS disk on the SSD Diskgroup (REDO diskgroup), export it to hypervisor using nfs and create a disk on it using "dd" and edit the vm.cfg file which belongs to the vm from which we want to reach the SSD disks, manually.

Why manually?

Because oakcli was not designed to create a vm repository on REDO diskgroup.

Let's start.

(this post will be a quick one.. I will not explain every command and output one by one, so feel free to ask questions..)

[root@ermansrv1 ~]# oakcli create repo ermanrepo -size 10g -dg REDO

ERROR: Invalid Memory Unit : 10g

ERROR: Invalid Disk Group value : REDO

Usage:

oakcli create repo <repo_name> -size <size> -dg <diskgroup>

where:

         repo                  -  shared repo name

         -size                 -  size of shared repo to be created

                               -  Minimum Size : 500M or 1G

                               -  Default unit is G

                               -  size must be a whole number.

         -dg                   -  Disk Group of shared repo

                               -  [DATA | RECO]

Well, maybe it is not supported but could not stop me..

Here we begin...

In ODA BASE ;

ASMCMD> volcreate -G REDO -s 100G ssdvolume1
ASMCMD> volinfo -G REDO -a

Diskgroup Name: REDO
Volume Name: SSDVOLUME1
Volume Device: /dev/asm/ssdvolume1-266
State: ENABLED
Size (MB): 102400
Resize Unit (MB): 32
Redundancy: HIGH
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:

[root@ermansrv1~]# /sbin/mkfs -t acfs /dev/asm/ssdvolume1-266
mkfs.acfs: version = 11.2.0.4.0
mkfs.acfs: on-disk version = 39.0
mkfs.acfs: volume = /dev/asm/ssdvolume1-266
mkfs.acfs: volume size = 107374182400
mkfs.acfs: Format complete.


[root@ermansrv1~]# /bin/mount -t acfs /dev/asm/ssdvolume1-266 /u01/app/acfsmounts/ssdrepo1/

[root@ermansrv1~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda2 57192344 52868852 1418220 98% /
/dev/xvda1 470844 24135 422398 6% /boot
/dev/xvdb1 96120004 87525692 3711604 96% /u01
tmpfs 7970240 193300 7776940 3% /dev/shm
/dev/asm/vmtemp2-209 734003200 317151292 416851908 44% /u01/app/sharedrepo/vmtemp2
/dev/asm/vmrepo1-209 4194304000 3615656168 578647832 87% /u01/app/sharedrepo/vmrepo1
/dev/asm/vmtemp1-209 2097152000 843172536 1253979464 41% /u01/app/sharedrepo/vmtemp1
/dev/asm/ssdvolume1-266 104857600 247160 104610440 1% /u01/app/acfsmounts/ssdrepo1


[root@ermansrv1 ssdrepo1]# dd if=/dev/zero of=redodisk1.img bs=1M count=20000

20000+0 records in
20000+0 records ouy

20971520000 bytes (21 GB) copied, 153.403 seconds, 137 MB/s


[root@ermansrv1 ssdrepo1]#vi vm.cfg
vif = ['']
name = 'EBS_12_2_3_ERMAN_DB'
extra = 'NODENAME=EBS_12_2_3_ERMAN_DB'
builder = 'hvm'
cpus = '4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47'
vcpus = 16
memory = 65536
vnc = 1
serial = 'pty'
disk = [u'file:/OVS/Repositories/vmrepo1/VirtualMachines/EBS_12_2_3_ERMAN_DB/128111a52def477b81a1c9758e250ef2.img,xvda,w','file:/u01/app/acfsmounts/ssdrepo1/redodisk1.img,xvdb,w']
maxvcpus = 16
maxmem = 65536

[root@ermansrv1~]# /sbin/acfsutil registry -a /dev/asm/ssdvolume1-266 /u01/app/acfsmounts/ssdrepo1

acfsutil registry: mount point /u01/app/acfsmounts/ssdrepo1 successfully added to Oracle Registry

(this is like adding an entry to fstab. :)

AGAIN IN ODA BASE ;

 [root@ermansrv1 ~]#vi /var/lib/nfs/etab

/u01/app/acfsmounts/ssdrepo1    192.168.16.14(rw,sync,no_wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=65534,anongid=65534)

/u01/app/sharedrepo/vmtemp2     192.168.16.14(rw,sync,no_wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=65534,anongid=65534)

/u01/app/sharedrepo/vmrepo1     192.168.16.14(rw,sync,no_wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=65534,anongid=65534)

/u01/app/sharedrepo/vmtemp1     192.168.16.14(rw,sync,no_wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=65534,anongid=65534)  (ADDED an export for the Hypervisors private network ip)

[root@ermansrv1 ~]# exportfs

In HYPERVISOR;

[root@ermanhype ~]#mount -t nfs 192.168.16.17:/u01/app/acfsmounts/ssdrepo1 /OVS/ssdrepo1   (The ip address is the private network address of ODA BASE machine 1)

[root@ermanhype~]# df

Filesystem           1K-blocks      Used Available Use% Mounted on

/dev/md1              19840804   1684132  17132536   9% /

tmpfs                  1758728         0   1758728   0% /dev/shm

/dev/md3             543397992 271347912 244001800  53% /OVS

/dev/md0                497765     41610    430456   9% /boot

none                   1758728       240   1758488   1% /var/lib/xenstored

192.168.16.17:/u01/app/sharedrepo/vmtemp1

                     2097152000 843172512 1253979488  41% /OVS/Repositories/vmtemp1

192.168.16.17:/u01/app/sharedrepo/vmrepo1

                     4194304000 3615655360 578648640  87% /OVS/Repositories/vmrepo1

192.168.16.17:/u01/app/sharedrepo/vmtemp2

                     734003200 317151264 416851936  44% /OVS/Repositories/vmtemp2

192.168.16.17:/u01/app/acfsmounts/ssdrepo1

                     104857600  20781056  84076544  20% /OVS/ssdrepo1

AGAIN IN ODA BASE ;

[root@ermansrv1~]# df

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/xvda2 57192344 52866888 1420184 98% /

/dev/xvda1 470844 24135 422398 6% /boot

/dev/xvdb1 96120004 87520388 3716908 96% /u01

tmpfs 7970240 193300 7776940 3% /dev/shm

/dev/asm/vmtemp2-209 734003200 317151292 416851908 44% /u01/app/sharedrepo/vmtemp2

/dev/asm/vmrepo1-209 4194304000 3615655372 578648628 87% /u01/app/sharedrepo/vmrepo1

/dev/asm/vmtemp1-209 2097152000 843172536 1253979464 41% /u01/app/sharedrepo/vmtemp1

/dev/asm/ssdvolume1-266 104857600 20781456 84076144 20% /u01/app/acfsmounts/ssdrepo1

[root@ermansrv1 ~]# cd /u01/app/sharedrepo/vmrepo1

[root@ermansrv1 vmrepo1]# cd VirtualMachines/

[root@ermansrv1VirtualMachines]# cd EBS_12_2_3_ERMAN_DB/

[root@ermansrv1 EBS_12_2_3_ERMAN_DB]# vi vm.cfg

vif = ['']

name = 'EBS_12_2_3_ERMAN_DB'

extra = 'NODENAME=EBS_12_2_3_ERMAN_DB'

builder = 'hvm'

cpus = '4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47'

vcpus = 16

memory = 65536

vnc = 1

serial = 'pty'

disk = [u'file:/OVS/Repositories/vmrepo1/VirtualMachines/EBS_12_2_3_ERMAN_DB/128111a52def477b81a1c9758e250ef2.img,xvda,w', 'file:/OVS/ssdrepo1/redodisk1.img,xvdb,w']

maxvcpus = 16

maxmem = 65536

At this point; reboot the oda base node and you will see your the new disk /dev/xvdb  which is built on top of SSD disks. Format it, mount it and start to use it :)

[root@ermansrv1~]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/xvda2             55G   51G  1.4G  98% /

/dev/xvda1            460M   24M  413M   6% /boot

/dev/xvdb1             92G   84G  3.6G  96% /u01

tmpfs                 7.7G  189M  7.5G   3% /dev/shm

/dev/asm/vmtemp2-209  700G  303G  398G  44% /u01/app/sharedrepo/vmtemp2

/dev/asm/vmrepo1-209  4.0T  3.4T  552G  87% /u01/app/sharedrepo/vmrepo1

/dev/asm/vmtemp1-209  2.0T  805G  1.2T  41% /u01/app/sharedrepo/vmtemp1

/dev/asm/ssdvolume1-266

                      100G   20G   81G  20% /u01/app/acfsmounts/ssdrepo1

EBS, Discoverer 11g libclntsh.so.11.1: cannot restore segment prot after reloc: Permission denied

You may encounter "libclntsh.so.11.1: cannot restore segment prot after reloc: Permission denied" error while executing applypreferences.sh.

In order to solve this, just disable selinux..
Disable it in standard selinux configuration file and disable it for the current situation using "setenforce 0 command"

EBS-- Workflow Mailer and Gmail certificates , unable to find valid certification path to requested target

One day, you may decide using gmail IMAP and SMTP services with your Oracle Workflow Mailer inbound and outbound processing. So if that day comes, here is a key information for you.

As you may know ; gmail operates in ssl. That both gmail smtp and imap services are operating in SSL and if you want to configure Oracle Workflow Mailer with gmail 's imap and smtp services, you need to configure Oracle Workflow Mailer with ssl.
Configuring Oracle Workflow MAiler with ssl , is not a big thing and it is documented already.

Generally, what we do for configuring SSL in imap and smtp processes of Oracle Workflow mailer is;  gathering the mail server's certificates from the mail server and making the Oracle Workflow mailer server use them.
We usually gather the certificates using openssl s_client .

For example:

openssl s_client -connect imap.gmail.com:993
openssl s_client -connect smtp.gmail.com:465


Then we copy and paste the ouputs of openssl to cer files and import these cer files to the cacert keystore.

For example:
keytool -import -trustcacerts -keystore $AF_JRE_TOP/lib/security/cacerts -storepass changeit -alias smtpimap -file gmailimap.cer
keytool -import -trustcacerts -keystore $AF_JRE_TOP/lib/security/cacerts -storepass changeit -alias smtpgmail -file gmailsmtp.cer

With this approach, you may download gmail's certificated using openssl and import them in to EBS to make Oracle Workflow Mailer be able to use them.
This method will work normally, but for gmail it will not work stably.

That is; if you are using gmail with your Oracle Workflow Mailer and even if you configure your Workflow mailer with SSL properly using the certificates gathered by accessing imap.gmail.com and smtp.gmail.com, you may encounter the following error randomly.

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Actually, this is a certificate problem , which is caused by Google, as Google has many servers spread out to hold the load from everyone connecting to them they would all use different SSL.

This problem can be fixed by using the gmail certificates located in "https://pki.google.com/"  ; with the title of“ Google's Issuing CA certificate” .

Note that: this certificates are like wildcard certificates(*.) and works with all the gmail servers.

This is direct link :  https://pki.google.com/GIAG2.crt 

After downloading the cer file from the link above;  we follow the standard Workflow Mailer SSL configuration documents and issue the following commands to import the cer file in to the cacerts file and make the workflow mailer use them. 

keytool -import -trustcacerts -keystore $AF_JRE_TOP/lib/security/cacerts -storepass changeit -alias GIAG2 -file GIAG2.cer

Also, both "Outbound SSL Enabled" and "Inbound SSL Enabled" checkboxes  located in Workflow Mailer's configuration page  should be checked.

Laslty, be sure that your SSL trust store is pointing the cacerts file that you import the gmail certificates.

[applmgr@ermanserver log]$ sqlplus apps/<pass> @$FND_TOP/sql/afsvcpup.sql

SQL*Plus: Release 10.1.0.5.0 - Production on Fri Nov 13 09:35:30 2015

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

Component Id Component Name                 Component Status Type            Containe

------------ ------------------------------ ---------------- --------------- --------

       10000 ECX Inbound Agent Listener     STOPPED          WF_AGENT_LISTEN GSM

       10001 ECX Transaction Agent Listener STOPPED          WF_AGENT_LISTEN GSM

       10002 Workflow Deferred Agent Listen RUNNING          WF_AGENT_LISTEN GSM

       10003 Workflow Deferred Notification RUNNING          WF_AGENT_LISTEN GSM

       10004 Workflow Error Agent Listener  RUNNING          WF_AGENT_LISTEN GSM

       10005 Workflow Inbound Notifications RUNNING          WF_AGENT_LISTEN GSM

       10006 Workflow Notification Mailer   RUNNING          WF_MAILER       GSM

       10020 Web Services IN Agent          STOPPED          WF_JAVA_AGENT_L GSM

       10021 Web Services OUT Agent         STOPPED          WF_DOCUMENT_WEB GSM

       10022 Workflow Java Deferred Agent L RUNNING          WF_JAVA_AGENT_L GSM

       10023 Workflow Java Error Agent List RUNNING          WF_JAVA_AGENT_L GSM

       10040 WF_JMS_IN Listener(M4U)        RUNNING          WF_JAVA_AGENT_L GSM

       10041 Workflow Inbound JMS Agent Lis STOPPED          WF_AGENT_LISTEN GSM

Enter Component Id: 10006

Example path:  " /u01/oracle/TEST/fs1/EBSapps/comn/util/jdk32/jre/lib/security/cacerts"

Note that : Dont use environment variables when specifying this path.

EBS -- Java plugin runtime parameter for a faster Java client

When we open a Forms based EBS application, a Java applet gets started  and Forms interfaces starts running on our clients.
So, when we want to use Forms based EBS application, we just wait for that applet to be opened and Forms jar files (if not located in client cache) to be downloaded by Java .
In order to increase the speed of this process, we can use the Runtime Parameters and increase the default heapsizes of our client's JRE environment as depicted in the picture below. (Windows Start Menut > Control Panel > Java)

As shown in picture above and just like we do in server side Java , we increase the heap sizes and determine a more stable and faster client side java. Note that: it is acceptable to set Xmx as 1024MB and Xms as 512MB, considering your client machines have sufficient memory.

Well, this is the tip of the day. Nowadays, I just give some little but important tips. That is I cant write long stories in here, because I m writing a book and as it is reaching the deadline. Soon(after 15th of January,), my posts will be like the way they were.. I hope :)

EBS -- Workflow Mailer IMAP -- Unable to connect Mail Store due to Exchange Configuration

You may encounter "Unable to connect Mail Store" and can even start your IMAP(Exchange IMAP) enabled Workflow Mailer.. If that's is the case; I suggest you to make a quick telnet test and try to login to the imap account as follows.

[appprod@ermanhost~]$ telnet 10.10.10.123 143   (change ip and port according to your imap server and imap port)

Trying 10.10.10.123...

Connected to 10.10.10.123.

Escape character is '^]'.

* OK The Microsoft Exchange IMAP4 service is ready.

a1 LOGIN erman_imap_user erman_imap_password   (change imap user and password according to your environment)

a1 BAD Command received in Invalid state.

So,  if you encounter "BAD command received in Invalid State" error as shown in the example output above; you should  open your Exchange Management Console then reach to the Imap 4 Configuration by navigating Server Configuration>Client Access > Pop3 & Imap 4 Configuration tab. Choose Plain Text Logon and restart Microsoft Exchange IMAP 4 service..