ODA X4 / virtualized, Adding ACFS mount point and make use of SSD disks manually

Before we start let's conclude :) ;

In Oracle Database Appliance, ODA BASE has direct access to the ODA storage.
That 's why, Grid Infrastructure , ASM and Oracle VMs are managed from ODA BASE -- oackli. The things called shared repositories are actually ACFS Volumes, which are exported to the hypervisor (Oracle VM Server's hyperviosor) using NFS and normally Virtual machines are stored in repositories.

Well, to use SSD s from the Virtual Machines in DOMU*, we need to create an ACFS disk on the SSD Diskgroup (REDO diskgroup), export it to hypervisor using nfs and create a disk on it using "dd" and edit the vm.cfg file which belongs to the vm from which we want to reach the SSD disks, manually.

Why manually?

Because oakcli was not designed to create a vm repository on REDO diskgroup.

Let's start.

(this post will be a quick one.. I will not explain every command and output one by one, so feel free to ask questions..)

[root@ermansrv1 ~]# oakcli create repo ermanrepo -size 10g -dg REDO

ERROR: Invalid Memory Unit : 10g

ERROR: Invalid Disk Group value : REDO

Usage:

oakcli create repo <repo_name> -size <size> -dg <diskgroup>

where:

         repo                  -  shared repo name

         -size                 -  size of shared repo to be created

                               -  Minimum Size : 500M or 1G

                               -  Default unit is G

                               -  size must be a whole number.

         -dg                   -  Disk Group of shared repo

                               -  [DATA | RECO]

Well, maybe it is not supported but could not stop me..

Here we begin...

In ODA BASE ;

ASMCMD> volcreate -G REDO -s 100G ssdvolume1
ASMCMD> volinfo -G REDO -a

Diskgroup Name: REDO
Volume Name: SSDVOLUME1
Volume Device: /dev/asm/ssdvolume1-266
State: ENABLED
Size (MB): 102400
Resize Unit (MB): 32
Redundancy: HIGH
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:

[root@ermansrv1~]# /sbin/mkfs -t acfs /dev/asm/ssdvolume1-266
mkfs.acfs: version = 11.2.0.4.0
mkfs.acfs: on-disk version = 39.0
mkfs.acfs: volume = /dev/asm/ssdvolume1-266
mkfs.acfs: volume size = 107374182400
mkfs.acfs: Format complete.


[root@ermansrv1~]# /bin/mount -t acfs /dev/asm/ssdvolume1-266 /u01/app/acfsmounts/ssdrepo1/

[root@ermansrv1~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda2 57192344 52868852 1418220 98% /
/dev/xvda1 470844 24135 422398 6% /boot
/dev/xvdb1 96120004 87525692 3711604 96% /u01
tmpfs 7970240 193300 7776940 3% /dev/shm
/dev/asm/vmtemp2-209 734003200 317151292 416851908 44% /u01/app/sharedrepo/vmtemp2
/dev/asm/vmrepo1-209 4194304000 3615656168 578647832 87% /u01/app/sharedrepo/vmrepo1
/dev/asm/vmtemp1-209 2097152000 843172536 1253979464 41% /u01/app/sharedrepo/vmtemp1
/dev/asm/ssdvolume1-266 104857600 247160 104610440 1% /u01/app/acfsmounts/ssdrepo1


[root@ermansrv1 ssdrepo1]# dd if=/dev/zero of=redodisk1.img bs=1M count=20000

20000+0 records in
20000+0 records ouy

20971520000 bytes (21 GB) copied, 153.403 seconds, 137 MB/s


[root@ermansrv1 ssdrepo1]#vi vm.cfg
vif = ['']
name = 'EBS_12_2_3_ERMAN_DB'
extra = 'NODENAME=EBS_12_2_3_ERMAN_DB'
builder = 'hvm'
cpus = '4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47'
vcpus = 16
memory = 65536
vnc = 1
serial = 'pty'
disk = [u'file:/OVS/Repositories/vmrepo1/VirtualMachines/EBS_12_2_3_ERMAN_DB/128111a52def477b81a1c9758e250ef2.img,xvda,w','file:/u01/app/acfsmounts/ssdrepo1/redodisk1.img,xvdb,w']
maxvcpus = 16
maxmem = 65536

[root@ermansrv1~]# /sbin/acfsutil registry -a /dev/asm/ssdvolume1-266 /u01/app/acfsmounts/ssdrepo1

acfsutil registry: mount point /u01/app/acfsmounts/ssdrepo1 successfully added to Oracle Registry

(this is like adding an entry to fstab. :)

AGAIN IN ODA BASE ;

 [root@ermansrv1 ~]#vi /var/lib/nfs/etab

/u01/app/acfsmounts/ssdrepo1    192.168.16.14(rw,sync,no_wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=65534,anongid=65534)

/u01/app/sharedrepo/vmtemp2     192.168.16.14(rw,sync,no_wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=65534,anongid=65534)

/u01/app/sharedrepo/vmrepo1     192.168.16.14(rw,sync,no_wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=65534,anongid=65534)

/u01/app/sharedrepo/vmtemp1     192.168.16.14(rw,sync,no_wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,mapping=identity,anonuid=65534,anongid=65534)  (ADDED an export for the Hypervisors private network ip)

[root@ermansrv1 ~]# exportfs

In HYPERVISOR;

[root@ermanhype ~]#mount -t nfs 192.168.16.17:/u01/app/acfsmounts/ssdrepo1 /OVS/ssdrepo1   (The ip address is the private network address of ODA BASE machine 1)

[root@ermanhype~]# df

Filesystem           1K-blocks      Used Available Use% Mounted on

/dev/md1              19840804   1684132  17132536   9% /

tmpfs                  1758728         0   1758728   0% /dev/shm

/dev/md3             543397992 271347912 244001800  53% /OVS

/dev/md0                497765     41610    430456   9% /boot

none                   1758728       240   1758488   1% /var/lib/xenstored

192.168.16.17:/u01/app/sharedrepo/vmtemp1

                     2097152000 843172512 1253979488  41% /OVS/Repositories/vmtemp1

192.168.16.17:/u01/app/sharedrepo/vmrepo1

                     4194304000 3615655360 578648640  87% /OVS/Repositories/vmrepo1

192.168.16.17:/u01/app/sharedrepo/vmtemp2

                     734003200 317151264 416851936  44% /OVS/Repositories/vmtemp2

192.168.16.17:/u01/app/acfsmounts/ssdrepo1

                     104857600  20781056  84076544  20% /OVS/ssdrepo1

AGAIN IN ODA BASE ;

[root@ermansrv1~]# df

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/xvda2 57192344 52866888 1420184 98% /

/dev/xvda1 470844 24135 422398 6% /boot

/dev/xvdb1 96120004 87520388 3716908 96% /u01

tmpfs 7970240 193300 7776940 3% /dev/shm

/dev/asm/vmtemp2-209 734003200 317151292 416851908 44% /u01/app/sharedrepo/vmtemp2

/dev/asm/vmrepo1-209 4194304000 3615655372 578648628 87% /u01/app/sharedrepo/vmrepo1

/dev/asm/vmtemp1-209 2097152000 843172536 1253979464 41% /u01/app/sharedrepo/vmtemp1

/dev/asm/ssdvolume1-266 104857600 20781456 84076144 20% /u01/app/acfsmounts/ssdrepo1

[root@ermansrv1 ~]# cd /u01/app/sharedrepo/vmrepo1

[root@ermansrv1 vmrepo1]# cd VirtualMachines/

[root@ermansrv1VirtualMachines]# cd EBS_12_2_3_ERMAN_DB/

[root@ermansrv1 EBS_12_2_3_ERMAN_DB]# vi vm.cfg

vif = ['']

name = 'EBS_12_2_3_ERMAN_DB'

extra = 'NODENAME=EBS_12_2_3_ERMAN_DB'

builder = 'hvm'

cpus = '4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47'

vcpus = 16

memory = 65536

vnc = 1

serial = 'pty'

disk = [u'file:/OVS/Repositories/vmrepo1/VirtualMachines/EBS_12_2_3_ERMAN_DB/128111a52def477b81a1c9758e250ef2.img,xvda,w', 'file:/OVS/ssdrepo1/redodisk1.img,xvdb,w']

maxvcpus = 16

maxmem = 65536

At this point; reboot the oda base node and you will see your the new disk /dev/xvdb  which is built on top of SSD disks. Format it, mount it and start to use it :)

[root@ermansrv1~]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/xvda2             55G   51G  1.4G  98% /

/dev/xvda1            460M   24M  413M   6% /boot

/dev/xvdb1             92G   84G  3.6G  96% /u01

tmpfs                 7.7G  189M  7.5G   3% /dev/shm

/dev/asm/vmtemp2-209  700G  303G  398G  44% /u01/app/sharedrepo/vmtemp2

/dev/asm/vmrepo1-209  4.0T  3.4T  552G  87% /u01/app/sharedrepo/vmrepo1

/dev/asm/vmtemp1-209  2.0T  805G  1.2T  41% /u01/app/sharedrepo/vmtemp1

/dev/asm/ssdvolume1-266

                      100G   20G   81G  20% /u01/app/acfsmounts/ssdrepo1

EBS, Discoverer 11g libclntsh.so.11.1: cannot restore segment prot after reloc: Permission denied

You may encounter "libclntsh.so.11.1: cannot restore segment prot after reloc: Permission denied" error while executing applypreferences.sh.

In order to solve this, just disable selinux..
Disable it in standard selinux configuration file and disable it for the current situation using "setenforce 0 command"

EBS-- Workflow Mailer and Gmail certificates , unable to find valid certification path to requested target

One day, you may decide using gmail IMAP and SMTP services with your Oracle Workflow Mailer inbound and outbound processing. So if that day comes, here is a key information for you.

As you may know ; gmail operates in ssl. That both gmail smtp and imap services are operating in SSL and if you want to configure Oracle Workflow Mailer with gmail 's imap and smtp services, you need to configure Oracle Workflow Mailer with ssl.
Configuring Oracle Workflow MAiler with ssl , is not a big thing and it is documented already.

Generally, what we do for configuring SSL in imap and smtp processes of Oracle Workflow mailer is;  gathering the mail server's certificates from the mail server and making the Oracle Workflow mailer server use them.
We usually gather the certificates using openssl s_client .

For example:

openssl s_client -connect imap.gmail.com:993
openssl s_client -connect smtp.gmail.com:465


Then we copy and paste the ouputs of openssl to cer files and import these cer files to the cacert keystore.

For example:
keytool -import -trustcacerts -keystore $AF_JRE_TOP/lib/security/cacerts -storepass changeit -alias smtpimap -file gmailimap.cer
keytool -import -trustcacerts -keystore $AF_JRE_TOP/lib/security/cacerts -storepass changeit -alias smtpgmail -file gmailsmtp.cer

With this approach, you may download gmail's certificated using openssl and import them in to EBS to make Oracle Workflow Mailer be able to use them.
This method will work normally, but for gmail it will not work stably.

That is; if you are using gmail with your Oracle Workflow Mailer and even if you configure your Workflow mailer with SSL properly using the certificates gathered by accessing imap.gmail.com and smtp.gmail.com, you may encounter the following error randomly.

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Actually, this is a certificate problem , which is caused by Google, as Google has many servers spread out to hold the load from everyone connecting to them they would all use different SSL.

This problem can be fixed by using the gmail certificates located in "https://pki.google.com/"  ; with the title of“ Google's Issuing CA certificate” .

Note that: this certificates are like wildcard certificates(*.) and works with all the gmail servers.

This is direct link :  https://pki.google.com/GIAG2.crt 

After downloading the cer file from the link above;  we follow the standard Workflow Mailer SSL configuration documents and issue the following commands to import the cer file in to the cacerts file and make the workflow mailer use them. 

keytool -import -trustcacerts -keystore $AF_JRE_TOP/lib/security/cacerts -storepass changeit -alias GIAG2 -file GIAG2.cer

Also, both "Outbound SSL Enabled" and "Inbound SSL Enabled" checkboxes  located in Workflow Mailer's configuration page  should be checked.

Laslty, be sure that your SSL trust store is pointing the cacerts file that you import the gmail certificates.

[applmgr@ermanserver log]$ sqlplus apps/<pass> @$FND_TOP/sql/afsvcpup.sql

SQL*Plus: Release 10.1.0.5.0 - Production on Fri Nov 13 09:35:30 2015

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

Component Id Component Name                 Component Status Type            Containe

------------ ------------------------------ ---------------- --------------- --------

       10000 ECX Inbound Agent Listener     STOPPED          WF_AGENT_LISTEN GSM

       10001 ECX Transaction Agent Listener STOPPED          WF_AGENT_LISTEN GSM

       10002 Workflow Deferred Agent Listen RUNNING          WF_AGENT_LISTEN GSM

       10003 Workflow Deferred Notification RUNNING          WF_AGENT_LISTEN GSM

       10004 Workflow Error Agent Listener  RUNNING          WF_AGENT_LISTEN GSM

       10005 Workflow Inbound Notifications RUNNING          WF_AGENT_LISTEN GSM

       10006 Workflow Notification Mailer   RUNNING          WF_MAILER       GSM

       10020 Web Services IN Agent          STOPPED          WF_JAVA_AGENT_L GSM

       10021 Web Services OUT Agent         STOPPED          WF_DOCUMENT_WEB GSM

       10022 Workflow Java Deferred Agent L RUNNING          WF_JAVA_AGENT_L GSM

       10023 Workflow Java Error Agent List RUNNING          WF_JAVA_AGENT_L GSM

       10040 WF_JMS_IN Listener(M4U)        RUNNING          WF_JAVA_AGENT_L GSM

       10041 Workflow Inbound JMS Agent Lis STOPPED          WF_AGENT_LISTEN GSM

Enter Component Id: 10006

Example path:  " /u01/oracle/TEST/fs1/EBSapps/comn/util/jdk32/jre/lib/security/cacerts"

Note that : Dont use environment variables when specifying this path.

EBS -- Java plugin runtime parameter for a faster Java client

When we open a Forms based EBS application, a Java applet gets started  and Forms interfaces starts running on our clients.
So, when we want to use Forms based EBS application, we just wait for that applet to be opened and Forms jar files (if not located in client cache) to be downloaded by Java .
In order to increase the speed of this process, we can use the Runtime Parameters and increase the default heapsizes of our client's JRE environment as depicted in the picture below. (Windows Start Menut > Control Panel > Java)

As shown in picture above and just like we do in server side Java , we increase the heap sizes and determine a more stable and faster client side java. Note that: it is acceptable to set Xmx as 1024MB and Xms as 512MB, considering your client machines have sufficient memory.

Well, this is the tip of the day. Nowadays, I just give some little but important tips. That is I cant write long stories in here, because I m writing a book and as it is reaching the deadline. Soon(after 15th of January,), my posts will be like the way they were.. I hope :)

EBS -- Workflow Mailer IMAP -- Unable to connect Mail Store due to Exchange Configuration

You may encounter "Unable to connect Mail Store" and can even start your IMAP(Exchange IMAP) enabled Workflow Mailer.. If that's is the case; I suggest you to make a quick telnet test and try to login to the imap account as follows.

[appprod@ermanhost~]$ telnet 10.10.10.123 143   (change ip and port according to your imap server and imap port)

Trying 10.10.10.123...

Connected to 10.10.10.123.

Escape character is '^]'.

* OK The Microsoft Exchange IMAP4 service is ready.

a1 LOGIN erman_imap_user erman_imap_password   (change imap user and password according to your environment)

a1 BAD Command received in Invalid state.

So,  if you encounter "BAD command received in Invalid State" error as shown in the example output above; you should  open your Exchange Management Console then reach to the Imap 4 Configuration by navigating Server Configuration>Client Access > Pop3 & Imap 4 Configuration tab. Choose Plain Text Logon and restart Microsoft Exchange IMAP 4 service..

Linux / Oracle -- About DST patches / Turkey DST changes

This article is written for Turkey's upcoming DST change, that is caused by the elections in Turkey. The elections will be done on 1.Nov.2015, so the government has deviced delay the DST change for two weeks.

So, normally, the summer time in Turkey would end on 25.Oct.2015, but this year, it will end on 08.Nov.2015.

The problem that made me writing this article is that our Operating Systems, on which our databases and applications are running, does not know anything about this change. Even we, as citizens, have learned it recently..

Well.. Somehow we need to tell this change to our Operating Systems, so that the Operating Systems can adjust the time when the 8 Nov 2015 will come.

In other words, up-to-date Linux Operating Systems (Windows also) still think that Turkey's summer time will end on 25 Oct 2015, and if we can't do anything about it, Linux Operating Systems, which are configured for Turkey timezone, will take back the time when 25 Oct 2015 comes.

So, in order to prevent it, we patch the Operating System.

These type of patches are called as DST patches and they are available publically.

Here is the links for Oracle Linux 5 and 6;

OL5 :

http://public-yum.oracle.com/repo/OracleLinux/OL5/latest/x86_64/getPackage/tzdata-java-2015g-1.el5.x86_64.rpm

http://public-yum.oracle.com/repo/OracleLinux/OL5/latest/x86_64/getPackage/tzdata-2015g-1.el5.x86_64.rpm


OL6 :

http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/getPackage/tzdata-java-2015g-2.el6.noarch.rpm

http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/ getPackage/tzdata-2015g-2.el6.noarch.rpm

So, the question "Do I need those patches even if I have a Ntp server running on my environment"? is already answered in this blog: See: http://ermanarslan.blogspot.com.tr/2014/04/ntp-and-timezone-data-why-do-we-need.html

the answer is YES!

Well, I ll keep it short and give you some hints about this patching work.
We just apply these patches online using the command "rpm -Uvh <rpm_name>.rpm"
After applying the patches, a planned restart can be done later (Restart apps + db..)

If our system is an Exadata , we may encounter the following:

Error: Failed dependencies:

tzdata-java = 2015d-1.el6 is needed by (installed) exadata-sun-computenode-exact-12.1.2.1.2.150617.1-1.x86_64

error: Failed dependencies:

tzdata = 2015d-1.el6 is needed by (installed) exadata-sun-cellnode-exact-12.1.2.1.2.150617.1-1.x86_64

In order to prevent these errors, we remove the package named "exact" and install the packages "tzdata" and "tzdata-java" after that.

For example:

rpm -e exadata-sun-computenode-exact-12.1.2.1.2.150617.1-1.x86_64 --nodeps

rpm -Uvh tzdata-2015g-2.el6.noarch.rpm

rpm -Uvh tzdata-java-2015g-2.el6.noarch.rpm

If we are on Exadata, it is recommended to reboot the compute nodes.

So, after applying the patches, there is one more thing that we need to be sure of.

That is, we need to check that our current date should be true.

For Turkey, the command "date" should produce an output that has EEST inside of it. EEST means the summer time. The reason for this is that timezone data is configured for EEST. Linux will wait till 04:00:00 in EEST and then take back the time. So for example, if our date is in EET, Linux will wait till 05:00:00 as 05:00:00 in EET = 04:00:00 EEST and we don't want that.

So, if we execute the date command and see the "EET" rather than "EEST", we need to update our time to be EEST using "date -s" command.

Lastly, DST rules are automatically considered by Linux, nothing needed to be restarted or nothing should be set (like hwclock and etc..)

the only needed things are:

apply the dst patches & be sure that your system is in the relevant timezone and that's it.

EBS 12.2.5 released!

Oracle announced EBS 12.2.5 on Oct 16 2015.

The new EBS 12.2.5 is delivered via a RUP patch 19676458 (Patch 12.2.5: ORACLE E-BUSINESS SUITE 12.2.5 RELEASE UPDATE PACK).
The root of the EBS 12.2.5 documentation is "Oracle E-Business Suite Product Specific Release Notes, Release 12.2.5 (Doc ID 2049015.1)"

The prerequisites of the patch 19676458 are R12.AD.C.delta.7 and R12.TXK.C.delta.7.
The upgrade process, in other words, applying the RUP patch 19676458 is explained in "Oracle E-Business Suite Release 12.2.5 Readme (Doc ID 1983050.1)" with all the details. So the document to follow for applying the 12.2.5 patch is 1983050.1.

Just like 12.2.4, applying 12.2.5 RUP patch with the downtime option is supported for the newly installed EBS 12.2 systems.
Lastly, it is important to mention that if you are on 12.2 you can directly upgrade to 12.2.5, but if you are on earlier releases like 11i or 12.0, you need to upgrade to release 12.2 first.
It seems, we will install EBS 12.2.5, as it will be considered as the latest and most stable EBS release in the upcoming days.

EBS 12.2 -- OPM Pre Processor Signal 6 and Signal 11

You may encounter Signal 11 and Signal 6 while running OPM Pre Processor in EBS 12.2.
Especially, if you are in Oracle VM environment and if you have imported Oracle VM Machine templates to build you EBS environment, then you will likely encounter this problem.
Signal 11 is a memory pointer error. That is, if an application process try to reach a memory region that is not in its memory list, in other words if an application process try to reach a memory which is not permitted , then it will get a Signal 11 in Linux.
Such problems may be caused by the code itself or by the data or even by the Os limits.

On the other hand; EBS 12.2.4 is the latest of EBS and when you are in EBS 12.2.4, it means you are recommended code level. This takes away the probability of the cause being the code. Even if it is in the code, the issue is unknown in Oracle Side, as all the signal 11-related Oracle Documents states that they are waiting for the first Signal 11 issue to be occured in EBS 12.2. So it is unknown.

Normally, In order to diagnose such an signal error, we should be following the action below.

• Enable Debugging. (Using Debug mode (GMF_CONC_DEBUG) to analyze OPM Financial Processes (Doc ID 230743.1))
• Increase the limits (ulimit) (Signal 11 and Memory-Related Errors in OPM Financials Executables (Doc ID 1396727.1))
• Check the patch level of the related products (GMF ve OPM) . (Signal 11and Memory-Related Errors in OPM Financials Executables (Doc ID 1396727.1)) If the products are not at the latest patch level, then upgrade them.
• Open an SR, Oracle Support will quickly respond in such situations. (Signal 11 and Memory-Related Errors in OPM Financials Executables (Doc ID 1396727.1))
• Make the relevant process deal with portion of data, rather than dealing the whole of the data. As, there are issues resolved in community using this method. (processing one by one ( Inventory, Production , Costin , Purchasing etc )

Signal 6 means abort, in order to get rid of this, we just need to ulimit increase the ulimits.
The memory, lock memory and stack memory limits should be increased in this manner.

Of course, after increasing these limits, we should relogin to OS and restart the application accordingly.
After restarting it is good to check an application process and  see if it takes te appropriate limits. (we can use /proc filesystem for this.)..

So , after setting the limits and getting rid of the Signal 6 or Signal 11 error, you may encounter java heap errors in the related concurrent processes. The solution for this is to increase the java heap values. Using the concurrent program define screen, we can set these limits by writing  -mx2048 -ms 2048 (or bigger values) into the textbox labeled as Options.

Note that: We have seen in this in Real life, and the solution we have applied was increasing the ulimit values and java heap spaces. 

But there is important point though, it is not easy to increase the ulimits in OEL 6 and Redhat 6.. It is just not anymore.. Be careful, there are scripts that overwrites your settings.. 

Tip: Check the scripts in /etc/security/limits.d :)

In Oracle Linux --> oracle-ebs-server-R12-preinstall.conf .. Make your ulimits setting in this file..

Well, that's enough for today .Have a good weekend.

EBS 12.2 / Discoverer 11g --- problem running EBS and Discoverver on the same server by the same OS user

Using Discoverer 11g and EBS 12.2 from the same OS user is something we do not recommend.
As both of these two applications has running on top of their built-in Weblogic Application Servers, the environment may be a problem.
Of course we may have seperate environment using seperate environment scripts prepared according to the application we want to administer. (actually such a script is delivered by EBS 12.2, -- EBSapps.env script)
On the other hand, there is still no need to have only one user for 2 seperate products.

Following is a result of a conflicted environment, where Weblogic Application Server of the Discoverer 11g can not find its standard classes.
The error is clear and shown in the browser , when trying to reach the Discoverer login page.
As depicted in figure below, it is obvious that Weblogic application server of the Discoverer 11g can not find its classes. That is, the error in the top of the error stack "NoClassDefFoundError",  is so clear.

Note that, this error may also be arised even if we set the Weblogic environment using setDomain.env script before starting the Weblogic Application Server of the Discoverer 11g.
That s because the environment variables that EBS sets may harm te Discoverer ' s nature.
Also some of the environment variables are used both by EBS and Discoverer. Moreover, these environment variables may be set by appending their current values such as CLASSPATH=$CLASSPATH:/blfbbvbv/lbvb ..
So, contention is the result.

Well. You get the point. Also I want to keep it short :)
The fix in such a scenarios, is to have a proper environment by unsetting all the environment variables and setting it for the application we want to administer accordingly.

It is already documented in Support Note:
Unable to Access Discoverer Plus/Viewer 11g Installed On Server Having EBS 11i / R12 (Doc ID 1338244.1)

I just saw it in action..

Oracle Linux 6.6 - "kernel 3.8.13-98.1.1.el6uek.x86_64" "not using all available CPU cores"

We have recently seen a problem in Oracle Linux 6.6, as it was not using all the cpu cores available on that server. It was running on a VMWare, but the problem was not in VM actually.

The configuration was like below;

socket 1 = cpu0,cpu1,cpu2,cpu3 socket2= cpu4,cpu5,cpu6,cpu7

The problem was in the utilization.
That is, when using 3.8.13-98.1.1.el6uek.x86_64, Oracle Linux 6.6 was using only 4 cpu cores. We have analyzed cpu utilization properly and it didnt just not allocate the last 4 cpu cores.
Oracle Linux 6.6 was seeing all the 8 cpus on the other hand..

We have used taskset executable to force a process to run on a specific cpu core which Oracle Linux normally did not not utilize and seen that the process have started running on that cpu core without any problems and we could also see that cpu utilization of that cpu core have become %100, as expected.

[root@somehost opt]# taskset -c -p 6 2313
pid 2313's current affinity list: 0-7
pid 2313's new affinity list: 6

[root@somehost~]# top
top - 19:10:14 up 4 days, 6:02, 6 users, load average: 1.06, 0.63, 0.32
Tasks: 432 total, 2 running, 430 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.3%us, 0.7%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.7%us, 0.7%sy, 0.0%ni, 98.0%id, 0.7%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.3%us, 0.7%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.7%us, 0.3%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
..
..
Cpu6 : 99.7%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st


So, when forced, Oracle Linux 6.6 with 3.8.13-98.1.1.el6uek.x86_64 kernel was using all the cores, but normally the scheduler automatically did not utilize the 4 cores coming from the second cpu socket, even under a very loaded situation as seen below. (cpu4,5,6,7 is not used.. not utilized..)

op - 12:51:32 up 3 days, 23:43, 3 users, load average: 16.74, 9.82, 5.30
Tasks: 454 total, 18 running, 436 sleeping, 0 stopped, 0 zombie
Cpu0 : 92.2%us, 6.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.2%si, 0.0%st
Cpu1 : 94.2%us, 4.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.1%si, 0.0%st
Cpu2 : 93.4%us, 4.8%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.8%si, 0.0%st
Cpu3 : 92.7%us, 5.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.9%si, 0.0%st
Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32687204k total, 32521340k used, 165864k free, 104284k buffers
Swap: 33554428k total, 32992k used, 33521436k free, 21020212k cached


The strange thing was , the issue could not be reproduced in 3.8.13-44 el6uek kernel.
When booted with 3.8.13-44 el6uek kernel, Oracle Linux 6.6 have seen and utilized all the cpu cores without any problems, perfectly in balance.

So, the problem basically was   "Oracle Linux 6.6 with 3.8.13-98.1.1.el6uek.x86_64 kernel.

The problem was looking like the same that was discussed in the discussion that I have created in Oracle Community. Avi Miller from Oracle replied to that similar problem and stated that this is a known issue 3.8.13-98.2.1 (tracked by internal bug 21662) So , the workaround was downgrading to the previous UEK3 release or using the redhat compatible kernel for the time being.

Actually, a similar problem was there in 3.8.13-98.1.1, as well.

So, we are for now continuing with the older kernel 3.8.13-44 el6uek  and probably upgrade after the internal bug 21662 will be resolved.