Linux / Oracle -- About DST patches / Turkey DST changes

This article is written for Turkey's upcoming DST change, that is caused by the elections in Turkey. The elections will be done on 1.Nov.2015, so the government has deviced delay the DST change for two weeks.

So, normally, the summer time in Turkey would end on 25.Oct.2015, but this year, it will end on 08.Nov.2015.

The problem that made me writing this article is that our Operating Systems, on which our databases and applications are running, does not know anything about this change. Even we, as citizens, have learned it recently..

Well.. Somehow we need to tell this change to our Operating Systems, so that the Operating Systems can adjust the time when the 8 Nov 2015 will come.

In other words, up-to-date Linux Operating Systems (Windows also) still think that Turkey's summer time will end on 25 Oct 2015, and if we can't do anything about it, Linux Operating Systems, which are configured for Turkey timezone, will take back the time when 25 Oct 2015 comes.

So, in order to prevent it, we patch the Operating System.

These type of patches are called as DST patches and they are available publically.

Here is the links for Oracle Linux 5 and 6;

OL5 :

http://public-yum.oracle.com/repo/OracleLinux/OL5/latest/x86_64/getPackage/tzdata-java-2015g-1.el5.x86_64.rpm

http://public-yum.oracle.com/repo/OracleLinux/OL5/latest/x86_64/getPackage/tzdata-2015g-1.el5.x86_64.rpm


OL6 :

http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/getPackage/tzdata-java-2015g-2.el6.noarch.rpm

http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/ getPackage/tzdata-2015g-2.el6.noarch.rpm

So, the question "Do I need those patches even if I have a Ntp server running on my environment"? is already answered in this blog: See: http://ermanarslan.blogspot.com.tr/2014/04/ntp-and-timezone-data-why-do-we-need.html

the answer is YES!

Well, I ll keep it short and give you some hints about this patching work.
We just apply these patches online using the command "rpm -Uvh <rpm_name>.rpm"
After applying the patches, a planned restart can be done later (Restart apps + db..)

If our system is an Exadata , we may encounter the following:

Error: Failed dependencies:

tzdata-java = 2015d-1.el6 is needed by (installed) exadata-sun-computenode-exact-12.1.2.1.2.150617.1-1.x86_64

error: Failed dependencies:

tzdata = 2015d-1.el6 is needed by (installed) exadata-sun-cellnode-exact-12.1.2.1.2.150617.1-1.x86_64

In order to prevent these errors, we remove the package named "exact" and install the packages "tzdata" and "tzdata-java" after that.

For example:

rpm -e exadata-sun-computenode-exact-12.1.2.1.2.150617.1-1.x86_64 --nodeps

rpm -Uvh tzdata-2015g-2.el6.noarch.rpm

rpm -Uvh tzdata-java-2015g-2.el6.noarch.rpm

If we are on Exadata, it is recommended to reboot the compute nodes.

So, after applying the patches, there is one more thing that we need to be sure of.

That is, we need to check that our current date should be true.

For Turkey, the command "date" should produce an output that has EEST inside of it. EEST means the summer time. The reason for this is that timezone data is configured for EEST. Linux will wait till 04:00:00 in EEST and then take back the time. So for example, if our date is in EET, Linux will wait till 05:00:00 as 05:00:00 in EET = 04:00:00 EEST and we don't want that.

So, if we execute the date command and see the "EET" rather than "EEST", we need to update our time to be EEST using "date -s" command.

Lastly, DST rules are automatically considered by Linux, nothing needed to be restarted or nothing should be set (like hwclock and etc..)

the only needed things are:

apply the dst patches & be sure that your system is in the relevant timezone and that's it.

EBS 12.2.5 released!

Oracle announced EBS 12.2.5 on Oct 16 2015.

The new EBS 12.2.5 is delivered via a RUP patch 19676458 (Patch 12.2.5: ORACLE E-BUSINESS SUITE 12.2.5 RELEASE UPDATE PACK).
The root of the EBS 12.2.5 documentation is "Oracle E-Business Suite Product Specific Release Notes, Release 12.2.5 (Doc ID 2049015.1)"

The prerequisites of the patch 19676458 are R12.AD.C.delta.7 and R12.TXK.C.delta.7.
The upgrade process, in other words, applying the RUP patch 19676458 is explained in "Oracle E-Business Suite Release 12.2.5 Readme (Doc ID 1983050.1)" with all the details. So the document to follow for applying the 12.2.5 patch is 1983050.1.

Just like 12.2.4, applying 12.2.5 RUP patch with the downtime option is supported for the newly installed EBS 12.2 systems.
Lastly, it is important to mention that if you are on 12.2 you can directly upgrade to 12.2.5, but if you are on earlier releases like 11i or 12.0, you need to upgrade to release 12.2 first.
It seems, we will install EBS 12.2.5, as it will be considered as the latest and most stable EBS release in the upcoming days.

EBS 12.2 -- OPM Pre Processor Signal 6 and Signal 11

You may encounter Signal 11 and Signal 6 while running OPM Pre Processor in EBS 12.2.
Especially, if you are in Oracle VM environment and if you have imported Oracle VM Machine templates to build you EBS environment, then you will likely encounter this problem.
Signal 11 is a memory pointer error. That is, if an application process try to reach a memory region that is not in its memory list, in other words if an application process try to reach a memory which is not permitted , then it will get a Signal 11 in Linux.
Such problems may be caused by the code itself or by the data or even by the Os limits.

On the other hand; EBS 12.2.4 is the latest of EBS and when you are in EBS 12.2.4, it means you are recommended code level. This takes away the probability of the cause being the code. Even if it is in the code, the issue is unknown in Oracle Side, as all the signal 11-related Oracle Documents states that they are waiting for the first Signal 11 issue to be occured in EBS 12.2. So it is unknown.

Normally, In order to diagnose such an signal error, we should be following the action below.

• Enable Debugging. (Using Debug mode (GMF_CONC_DEBUG) to analyze OPM Financial Processes (Doc ID 230743.1))
• Increase the limits (ulimit) (Signal 11 and Memory-Related Errors in OPM Financials Executables (Doc ID 1396727.1))
• Check the patch level of the related products (GMF ve OPM) . (Signal 11and Memory-Related Errors in OPM Financials Executables (Doc ID 1396727.1)) If the products are not at the latest patch level, then upgrade them.
• Open an SR, Oracle Support will quickly respond in such situations. (Signal 11 and Memory-Related Errors in OPM Financials Executables (Doc ID 1396727.1))
• Make the relevant process deal with portion of data, rather than dealing the whole of the data. As, there are issues resolved in community using this method. (processing one by one ( Inventory, Production , Costin , Purchasing etc )

Signal 6 means abort, in order to get rid of this, we just need to ulimit increase the ulimits.
The memory, lock memory and stack memory limits should be increased in this manner.

Of course, after increasing these limits, we should relogin to OS and restart the application accordingly.
After restarting it is good to check an application process and  see if it takes te appropriate limits. (we can use /proc filesystem for this.)..

So , after setting the limits and getting rid of the Signal 6 or Signal 11 error, you may encounter java heap errors in the related concurrent processes. The solution for this is to increase the java heap values. Using the concurrent program define screen, we can set these limits by writing  -mx2048 -ms 2048 (or bigger values) into the textbox labeled as Options.

Note that: We have seen in this in Real life, and the solution we have applied was increasing the ulimit values and java heap spaces. 

But there is important point though, it is not easy to increase the ulimits in OEL 6 and Redhat 6.. It is just not anymore.. Be careful, there are scripts that overwrites your settings.. 

Tip: Check the scripts in /etc/security/limits.d :)

In Oracle Linux --> oracle-ebs-server-R12-preinstall.conf .. Make your ulimits setting in this file..

Well, that's enough for today .Have a good weekend.

EBS 12.2 / Discoverer 11g --- problem running EBS and Discoverver on the same server by the same OS user

Using Discoverer 11g and EBS 12.2 from the same OS user is something we do not recommend.
As both of these two applications has running on top of their built-in Weblogic Application Servers, the environment may be a problem.
Of course we may have seperate environment using seperate environment scripts prepared according to the application we want to administer. (actually such a script is delivered by EBS 12.2, -- EBSapps.env script)
On the other hand, there is still no need to have only one user for 2 seperate products.

Following is a result of a conflicted environment, where Weblogic Application Server of the Discoverer 11g can not find its standard classes.
The error is clear and shown in the browser , when trying to reach the Discoverer login page.
As depicted in figure below, it is obvious that Weblogic application server of the Discoverer 11g can not find its classes. That is, the error in the top of the error stack "NoClassDefFoundError",  is so clear.

Note that, this error may also be arised even if we set the Weblogic environment using setDomain.env script before starting the Weblogic Application Server of the Discoverer 11g.
That s because the environment variables that EBS sets may harm te Discoverer ' s nature.
Also some of the environment variables are used both by EBS and Discoverer. Moreover, these environment variables may be set by appending their current values such as CLASSPATH=$CLASSPATH:/blfbbvbv/lbvb ..
So, contention is the result.

Well. You get the point. Also I want to keep it short :)
The fix in such a scenarios, is to have a proper environment by unsetting all the environment variables and setting it for the application we want to administer accordingly.

It is already documented in Support Note:
Unable to Access Discoverer Plus/Viewer 11g Installed On Server Having EBS 11i / R12 (Doc ID 1338244.1)

I just saw it in action..