Sunday, December 21, 2014

EBS R12/LINUX -- 32 bit Application tier with large memory -- OMM-KILLER problem

Another 32 bit problem, this time OMM-killer comes into play and kills processes without knowing the importance of them..  Sometimes , OMM-killer kills an application processes and make the Application Web Service to be down, and sometimes it kill a backup process and affect production system backups..
The reason behind the utilization of OMM-killer is using an 32 bit Linux OS and installing a big sized memory(like 64 GBs)..


 The problem arises in Oracle/Redhat Linux 5 and 6 32 bit, as those operating systems can not handle the memory stably when the installed memory is more than 16GB..

16 GB memory however , can be used with Oracle/Redhat Linux 32 bit Operating Systems without any problems..
The reason behind this incapability is not being able to use HugePages.. As you may know, starting with Oracle Linux 5 for example; hugemem is not option..

So even if you can use PAE to address a big memory, and even if you can address a big memory like 64 bit, you will end up with OMM-killer killing your processes...
I mean even our UEK kernel supports PAE and can address those big sized RAMs, we 'll still have problems in runtime..
Note: Checking PAE support:
Go to /boot ;
cat config-2.6.32-300.10.1.el5uek. (your kernel) | grep CONFIG_X84_PAE;
If a line like CONFIG_X86_PAE=y is returned, yhen you kernel supports PAE and it is active right now.

What I try to say is;
OMM-killer kill your processes because it see a shortage in LowMem area. Hugepage option in earlier Linux releases address this problem by making us able to use a 4-4 split rather tan 1-3 memory split.
You will have a better idea , when you read my related post about Linux 32 bit memory addressing ,->
http://ermanarslan.blogspot.com.tr/2013/11/linux-ebs-11i-uek-kernel-pae-support32.html

Okay, lets list the recommended solution for this particular environment;

1) Why not go for 64 bit? EBS Release 12 supports 64 bit in Apps Tier too!
2) Deinstall those RAMs, and use 16GB Ram in your PAE enabled Linux 32 bit envrionment
       - Also, increase the lowmem zone protection ..
       For Oracle Linux 5/Rhel 5 and higher:
add the following line into /etc/sysctl.conf
vm.lowmem_reserve_ratio = 256 256 250
Also execute following for current setting.
echo "256 256 250" > /proc/sys/vm/lowmem_reserve_ratio
     
       For earlier releases :
echo "250" > /proc/sys/vm/lower_zone_protection
3)You may try with OMM disabled: (not recommended) --just in caseAdd a line to /etc/sysctl.conf and reboot 
vm.oom-kill = 0

No comments :

Post a Comment