Tuesday, January 6, 2015

Linux -- setting limits in .bash_profile vs 90-nproc.conf file

I already wrote a post about setting limits in Redhat/Oracle Linux 6..
http://ermanarslan.blogspot.com.tr/2014/11/ebslinux-fork-retry-resource.html
There is bug introduced by using nproc.conf file to prevent the fork bombs, and Application admins should add their limits settings into .bash_profile of the relevant Os user to workaround this problem..
But yesterday, I have seen a problem while making a switch user operation(su - osuser)..
The problem was that , altough we have the required settings in the user's .bash_profile, "su - osuser" can not be completed.. The error was something like resource unavailable.
Then I have started to investigate the problem , and end up with the following;

When executing su - osuser operation;
90-nproc.conf (which is causing the problems as explained above) is read after all other limits related files(for example : limits.conf)
After reading the file , the limits setting are done according to the values which are set in 90-nproc.conf file.

open("/etc/security/limits.d/90-nproc.conf", O_RDONLY) = 3  -> we read the file
fstat(3, {st_mode=S_IFREG|0644, st_size=316, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5726262000
read(3, "# Default limit for number of us"..., 4096) = 316
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f5726262000, 4096)            = 0
setrlimit(RLIMIT_NPROC, {rlim_cur=1024, rlim_max=1024}) = 0  -> we set the limits to the values which we read from the 90-nproc.conf file
setrlimit(RLIMIT_NOFILE, {rlim_cur=40384, rlim_max=40384}) = 0   -> we set the limits to the values which we read from the 90-nproc.conf file
setpriority(PRIO_PROCESS, 0, 0)         = 0

The important thing is the number of process setting in that file is 1024, and su - osuser sets it to 1024..
Thus , we get resource unavailable errors in our crowded systems while switching to our OS user.. 
--eventhough we have the needed limits settings in our .bash_profile..

Normally, what happens is ;

In Time 0  -> 
we issue a su - osuer command and during the su operation we have low limits  ..

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515291
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 40384
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

In Time 1 ->

But after the su - osuser operation completes, our .bash_profile  is read, and our high limit setting commands are executed , and our limits are set as we want them to be set. 
Because we have someting like following written in our .bash_profile..

if [ $USER = "your_os_user" ]; then

        if [ $SHELL = "/bin/ksh" ]; then

              ulimit -p 16384

              ulimit -n 65536

        else

              ulimit -u 16384 -n 65536

        fi

fi

In Time 2 ->
We start our application processes, and everyting works well..

But sometimes, we cant even complete our su - osuser operation.. 
If that is the case; the solution is to set the limits in 90-nproc.conf file , too..

Note that:  Being not able to make switch user does not affect the application in this case..
I mean, your application may run without problems, if you have defined the limits in the application OS user's .bash_profile.. 90-nproc.conf plays part in the su - osuser phase..

So we can say that; setting the limits in .bash_profile affects application and setting the limits in 90-nproc.conf affects Linux admins :)

Okay.. That's enough for this topic :). I hope you 'll find it useful.

No comments :

Post a Comment