Linux / Oracle -- About DST patches / Turkey DST changes

This article is written for Turkey's upcoming DST change, that is caused by the elections in Turkey. The elections will be done on 1.Nov.2015, so the government has deviced delay the DST change for two weeks.

So, normally, the summer time in Turkey would end on 25.Oct.2015, but this year, it will end on 08.Nov.2015.

The problem that made me writing this article is that our Operating Systems, on which our databases and applications are running, does not know anything about this change. Even we, as citizens, have learned it recently..

Well.. Somehow we need to tell this change to our Operating Systems, so that the Operating Systems can adjust the time when the 8 Nov 2015 will come.

In other words, up-to-date Linux Operating Systems (Windows also) still think that Turkey's summer time will end on 25 Oct 2015, and if we can't do anything about it, Linux Operating Systems, which are configured for Turkey timezone, will take back the time when 25 Oct 2015 comes.

So, in order to prevent it, we patch the Operating System.

These type of patches are called as DST patches and they are available publically.

Here is the links for Oracle Linux 5 and 6;

OL5 :

http://public-yum.oracle.com/repo/OracleLinux/OL5/latest/x86_64/getPackage/tzdata-java-2015g-1.el5.x86_64.rpm

http://public-yum.oracle.com/repo/OracleLinux/OL5/latest/x86_64/getPackage/tzdata-2015g-1.el5.x86_64.rpm


OL6 :

http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/getPackage/tzdata-java-2015g-2.el6.noarch.rpm

http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/ getPackage/tzdata-2015g-2.el6.noarch.rpm

So, the question "Do I need those patches even if I have a Ntp server running on my environment"? is already answered in this blog: See: http://ermanarslan.blogspot.com.tr/2014/04/ntp-and-timezone-data-why-do-we-need.html

the answer is YES!

Well, I ll keep it short and give you some hints about this patching work.
We just apply these patches online using the command "rpm -Uvh <rpm_name>.rpm"
After applying the patches, a planned restart can be done later (Restart apps + db..)

If our system is an Exadata , we may encounter the following:

Error: Failed dependencies:

tzdata-java = 2015d-1.el6 is needed by (installed) exadata-sun-computenode-exact-12.1.2.1.2.150617.1-1.x86_64

error: Failed dependencies:

tzdata = 2015d-1.el6 is needed by (installed) exadata-sun-cellnode-exact-12.1.2.1.2.150617.1-1.x86_64

In order to prevent these errors, we remove the package named "exact" and install the packages "tzdata" and "tzdata-java" after that.

For example:

rpm -e exadata-sun-computenode-exact-12.1.2.1.2.150617.1-1.x86_64 --nodeps

rpm -Uvh tzdata-2015g-2.el6.noarch.rpm

rpm -Uvh tzdata-java-2015g-2.el6.noarch.rpm

If we are on Exadata, it is recommended to reboot the compute nodes.

So, after applying the patches, there is one more thing that we need to be sure of.

That is, we need to check that our current date should be true.

For Turkey, the command "date" should produce an output that has EEST inside of it. EEST means the summer time. The reason for this is that timezone data is configured for EEST. Linux will wait till 04:00:00 in EEST and then take back the time. So for example, if our date is in EET, Linux will wait till 05:00:00 as 05:00:00 in EET = 04:00:00 EEST and we don't want that.

So, if we execute the date command and see the "EET" rather than "EEST", we need to update our time to be EEST using "date -s" command.

Lastly, DST rules are automatically considered by Linux, nothing needed to be restarted or nothing should be set (like hwclock and etc..)

the only needed things are:

apply the dst patches & be sure that your system is in the relevant timezone and that's it.

EBS 12.2.5 released!

Oracle announced EBS 12.2.5 on Oct 16 2015.

The new EBS 12.2.5 is delivered via a RUP patch 19676458 (Patch 12.2.5: ORACLE E-BUSINESS SUITE 12.2.5 RELEASE UPDATE PACK).
The root of the EBS 12.2.5 documentation is "Oracle E-Business Suite Product Specific Release Notes, Release 12.2.5 (Doc ID 2049015.1)"

The prerequisites of the patch 19676458 are R12.AD.C.delta.7 and R12.TXK.C.delta.7.
The upgrade process, in other words, applying the RUP patch 19676458 is explained in "Oracle E-Business Suite Release 12.2.5 Readme (Doc ID 1983050.1)" with all the details. So the document to follow for applying the 12.2.5 patch is 1983050.1.

Just like 12.2.4, applying 12.2.5 RUP patch with the downtime option is supported for the newly installed EBS 12.2 systems.
Lastly, it is important to mention that if you are on 12.2 you can directly upgrade to 12.2.5, but if you are on earlier releases like 11i or 12.0, you need to upgrade to release 12.2 first.
It seems, we will install EBS 12.2.5, as it will be considered as the latest and most stable EBS release in the upcoming days.

EBS 12.2 -- OPM Pre Processor Signal 6 and Signal 11

You may encounter Signal 11 and Signal 6 while running OPM Pre Processor in EBS 12.2.
Especially, if you are in Oracle VM environment and if you have imported Oracle VM Machine templates to build you EBS environment, then you will likely encounter this problem.
Signal 11 is a memory pointer error. That is, if an application process try to reach a memory region that is not in its memory list, in other words if an application process try to reach a memory which is not permitted , then it will get a Signal 11 in Linux.
Such problems may be caused by the code itself or by the data or even by the Os limits.

On the other hand; EBS 12.2.4 is the latest of EBS and when you are in EBS 12.2.4, it means you are recommended code level. This takes away the probability of the cause being the code. Even if it is in the code, the issue is unknown in Oracle Side, as all the signal 11-related Oracle Documents states that they are waiting for the first Signal 11 issue to be occured in EBS 12.2. So it is unknown.

Normally, In order to diagnose such an signal error, we should be following the action below.

• Enable Debugging. (Using Debug mode (GMF_CONC_DEBUG) to analyze OPM Financial Processes (Doc ID 230743.1))
• Increase the limits (ulimit) (Signal 11 and Memory-Related Errors in OPM Financials Executables (Doc ID 1396727.1))
• Check the patch level of the related products (GMF ve OPM) . (Signal 11and Memory-Related Errors in OPM Financials Executables (Doc ID 1396727.1)) If the products are not at the latest patch level, then upgrade them.
• Open an SR, Oracle Support will quickly respond in such situations. (Signal 11 and Memory-Related Errors in OPM Financials Executables (Doc ID 1396727.1))
• Make the relevant process deal with portion of data, rather than dealing the whole of the data. As, there are issues resolved in community using this method. (processing one by one ( Inventory, Production , Costin , Purchasing etc )

Signal 6 means abort, in order to get rid of this, we just need to ulimit increase the ulimits.
The memory, lock memory and stack memory limits should be increased in this manner.

Of course, after increasing these limits, we should relogin to OS and restart the application accordingly.
After restarting it is good to check an application process and  see if it takes te appropriate limits. (we can use /proc filesystem for this.)..

So , after setting the limits and getting rid of the Signal 6 or Signal 11 error, you may encounter java heap errors in the related concurrent processes. The solution for this is to increase the java heap values. Using the concurrent program define screen, we can set these limits by writing  -mx2048 -ms 2048 (or bigger values) into the textbox labeled as Options.

Note that: We have seen in this in Real life, and the solution we have applied was increasing the ulimit values and java heap spaces. 

But there is important point though, it is not easy to increase the ulimits in OEL 6 and Redhat 6.. It is just not anymore.. Be careful, there are scripts that overwrites your settings.. 

Tip: Check the scripts in /etc/security/limits.d :)

In Oracle Linux --> oracle-ebs-server-R12-preinstall.conf .. Make your ulimits setting in this file..

Well, that's enough for today .Have a good weekend.

EBS 12.2 / Discoverer 11g --- problem running EBS and Discoverver on the same server by the same OS user

Using Discoverer 11g and EBS 12.2 from the same OS user is something we do not recommend.
As both of these two applications has running on top of their built-in Weblogic Application Servers, the environment may be a problem.
Of course we may have seperate environment using seperate environment scripts prepared according to the application we want to administer. (actually such a script is delivered by EBS 12.2, -- EBSapps.env script)
On the other hand, there is still no need to have only one user for 2 seperate products.

Following is a result of a conflicted environment, where Weblogic Application Server of the Discoverer 11g can not find its standard classes.
The error is clear and shown in the browser , when trying to reach the Discoverer login page.
As depicted in figure below, it is obvious that Weblogic application server of the Discoverer 11g can not find its classes. That is, the error in the top of the error stack "NoClassDefFoundError",  is so clear.

Note that, this error may also be arised even if we set the Weblogic environment using setDomain.env script before starting the Weblogic Application Server of the Discoverer 11g.
That s because the environment variables that EBS sets may harm te Discoverer ' s nature.
Also some of the environment variables are used both by EBS and Discoverer. Moreover, these environment variables may be set by appending their current values such as CLASSPATH=$CLASSPATH:/blfbbvbv/lbvb ..
So, contention is the result.

Well. You get the point. Also I want to keep it short :)
The fix in such a scenarios, is to have a proper environment by unsetting all the environment variables and setting it for the application we want to administer accordingly.

It is already documented in Support Note:
Unable to Access Discoverer Plus/Viewer 11g Installed On Server Having EBS 11i / R12 (Doc ID 1338244.1)

I just saw it in action..

Oracle Linux 6.6 - "kernel 3.8.13-98.1.1.el6uek.x86_64" "not using all available CPU cores"

We have recently seen a problem in Oracle Linux 6.6, as it was not using all the cpu cores available on that server. It was running on a VMWare, but the problem was not in VM actually.

The configuration was like below;

socket 1 = cpu0,cpu1,cpu2,cpu3 socket2= cpu4,cpu5,cpu6,cpu7

The problem was in the utilization.
That is, when using 3.8.13-98.1.1.el6uek.x86_64, Oracle Linux 6.6 was using only 4 cpu cores. We have analyzed cpu utilization properly and it didnt just not allocate the last 4 cpu cores.
Oracle Linux 6.6 was seeing all the 8 cpus on the other hand..

We have used taskset executable to force a process to run on a specific cpu core which Oracle Linux normally did not not utilize and seen that the process have started running on that cpu core without any problems and we could also see that cpu utilization of that cpu core have become %100, as expected.

[root@somehost opt]# taskset -c -p 6 2313
pid 2313's current affinity list: 0-7
pid 2313's new affinity list: 6

[root@somehost~]# top
top - 19:10:14 up 4 days, 6:02, 6 users, load average: 1.06, 0.63, 0.32
Tasks: 432 total, 2 running, 430 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.3%us, 0.7%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.7%us, 0.7%sy, 0.0%ni, 98.0%id, 0.7%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.3%us, 0.7%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.7%us, 0.3%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
..
..
Cpu6 : 99.7%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st


So, when forced, Oracle Linux 6.6 with 3.8.13-98.1.1.el6uek.x86_64 kernel was using all the cores, but normally the scheduler automatically did not utilize the 4 cores coming from the second cpu socket, even under a very loaded situation as seen below. (cpu4,5,6,7 is not used.. not utilized..)

op - 12:51:32 up 3 days, 23:43, 3 users, load average: 16.74, 9.82, 5.30
Tasks: 454 total, 18 running, 436 sleeping, 0 stopped, 0 zombie
Cpu0 : 92.2%us, 6.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.2%si, 0.0%st
Cpu1 : 94.2%us, 4.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.1%si, 0.0%st
Cpu2 : 93.4%us, 4.8%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.8%si, 0.0%st
Cpu3 : 92.7%us, 5.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.9%si, 0.0%st
Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32687204k total, 32521340k used, 165864k free, 104284k buffers
Swap: 33554428k total, 32992k used, 33521436k free, 21020212k cached


The strange thing was , the issue could not be reproduced in 3.8.13-44 el6uek kernel.
When booted with 3.8.13-44 el6uek kernel, Oracle Linux 6.6 have seen and utilized all the cpu cores without any problems, perfectly in balance.

So, the problem basically was   "Oracle Linux 6.6 with 3.8.13-98.1.1.el6uek.x86_64 kernel.

The problem was looking like the same that was discussed in the discussion that I have created in Oracle Community. Avi Miller from Oracle replied to that similar problem and stated that this is a known issue 3.8.13-98.2.1 (tracked by internal bug 21662) So , the workaround was downgrading to the previous UEK3 release or using the redhat compatible kernel for the time being.

Actually, a similar problem was there in 3.8.13-98.1.1, as well.

So, we are for now continuing with the older kernel 3.8.13-44 el6uek  and probably upgrade after the internal bug 21662 will be resolved.

RDBMS -- ORA-28750: unkown error in SSL web service call

If you are trying to do someting like the following and end up with ORA-28750 error, then the server that you are trying to connect, is probably using an SHA2 ssl certificate.

SQL> DECLARE
2 lo_req UTL_HTTP.req;
3 lo_resp UTL_HTTP.resp;
4 BEGIN
5 UTL_HTTP.SET_WALLET ('file:/wallet/','welcome1');
6 lo_req := UTL_HTTP.begin_request('https://ip_address/erman?wsdl');
7 lo_resp := UTL_HTTP.get_response(lo_req);
8 dbms_output.put_line(lo_resp.status_code);
9 UTL_HTTP.end_response(lo_resp);
10 END;
11 /

DECLARE
*
ERROR at line 1:

ORA-29273: HTTP request failed
ORA-06512: at "SYS.UTL_HTTP", line 1029
ORA-28750: unknown error
ORA-06512: at line

You may think that you wallet is problematic but it is not..
If that 's the case, then you are probably using an older version of Oracle Database :)
Something like 11.1.0.7 maybe... SHA2 is certified in Oracle Database 11.2.0.3 and above.
(To check the certificate you can just save it to your laptop, rename it to .cer then double click on it..)

So if that 's case, then you are two options:

1) upgrade your db ..
2) If the server that gives that web services is on your control, change the server side certificate to be 'GeoTrustSSLCA-G3'

We choose option 1 :) , as it is time.. it is even the time for an 12c upgrade..

EBS 12.2 -- what if we apply a patch with hotpatch option, if that patch is not suitable to be applied as hotpatch?

If not stated by an Oracle Support document or by any Oracle Document including Patch readme's, we just create online patching cycle and apply EBS 12.2 applicaiton patches online.
So what if we apply a patch with hotpatch option, if that patch was not suitable for this?

First of all, these patches are considered as unsafe patches. They are not tested by Oracle and probably not suitable for applying directly in to the run filesystem while the application services are running.

The answer for the question, on the other hand; is simple. We just take the risk and need to be prepared for the followings ;

• invalid objects, missing code dependencies 
• mismatches between code level of file system and database 
• missing column data, or other data integrity problems 
• out-of-date indexes and materialized view definitions 
• invalid data in runtime caches

:)

So, hotpatches was maybe more stable in EBS 12.1 or we thought that they were stable as maybe we didn't know the problems that users was expriencing while we were patching the system with an hopatch. 

But in EBS 12.2, the effects are on system-wide as you see in the above list.

Well..At the bottom line, that's why I wan to remind that it is not safe to apply a patch with hopatch option in EBS 12.2 -- unless it is stated by Oracle Support Documents or patch read me files.

EBS 12.2 -- why a downtime patch is faster than hotpatch?

Recently wondering, how can a downtime patch be faster than a hotpatch..
It was stated in all the Oracle Documents, but the reasons was not there..
I was trying to find a logical reason for this, eventough i couldnt see a change in the adop's behaviour while applying the patches with downtime option. adpatch was used again in the background, the online patch cycle was not used and it was just like applying a hotpatch from those point views.
Thus, I concluded that applying a patch with downtime option is considered faster, just because the server resources was not used by EBS users or application services.
Anyways, today I got an answer from an SR that I recently created for this question. The return that Oracle support did was just approved my idea.
So at the bottom line, applying a patch in downtime patch is faster than applying a patch with hotpatch option and it just because the system resources was not used by EBS users or application services when applying a patch with downtime option.

EBS 12.2 -- The new Simplified Home Page

EBS 12.2 has a UI feature that looks really nice. Discovered it recently. Don't ask why so late :) It is probably becausea we are too busy in the backend, so can't find time to navigate in the Application Screens :)

Anyways, the simplified Home page introduced in EBS 12.2 make you feel like you are in an Ipad application..

Here is how the home page looks after enabling the simplified home page in EBS 12.2:

Pretty cool right:) ? It just does not look like an EBS webpage :)

Well here is the key for enabling it:

That is;
In order to use this feature, The "Self Service Personal Home Page Mode" profile must be set to
"Framework Simplified"
After setting this profile option , you can relogin and start the use this simplified Homepage.

Note that : the profile option can be set at multiple levels by the system administrator, including at the individual user level.

ASM / Grid 11g -- Resync vs Rebalance

I wrote an article about Asm Resilvering vs Asm Rebalance in one of my previous post
http://ermanarslan.blogspot.com.tr/2015/05/exadata-asm-resilvering-vs-asm-rebalance.html

In this post, I will explain the difference between Asm Resync vs Asm Rebalance shortly.
Nowadays, I can't post several blog posts like I have done in the previous month, because I m working on a Book which will be released in Jan 2016 .
So that's why I can write to several blog articles and, when I m able to write them, I need to keep them short in these days:)

Anyways, "the difference between Asm resyncing and rebalancing" was a question posted by one of the followers in Exadata Facebook group; and here is the answer for that:

Their usages are different and can be compared by examining the fast disk resync and fast rebalance operations introduced in 11g
Resync is something like syncronizing the disks with the data that should reside on them. (can be used on transient failure ) (ASM 11g New Features - How ASM Disk Resync Works. (Doc ID 466326.1))
Rebalancing is something like spreading the data evenly across all  thedisks in a disk group. (can be used on a disk replace operation)