This blog post will be about Huge pages on Linux.
I actually wrote a comprehensive article about Linux Memory Optimization (including Hugepages) earlier, but this blog post will be a little different.
Today, I want to make a demo to show you the Hugepages in real life and the memory locking mechanism that we need to get used to, when we enable Hugepages.
The thing that made me write this article was a question that one my collegues asked last week.
My collegue realized that after rebooting his database server, the memory directly becomes "used". Even before starting the database, he could see the memory is in use when he executed the "free" command.
This question asked me on the phone and I directly answered that "it is because huge pages".
However, I wanted to make a demo and see this statement in real life.
Well let's revisit my earlier blog post and recall the general information about the Huge pages:
(I strongly recommend you to read this blog post,as well ->
http://ermanarslan.blogspot.com.tr/2013/12/oracle-linux-memory-optimization.html)
When we use hugepages, we have smaller page tables in terms of size, because there will be less pages to handle, as Hugepages are 2MB(or more , it depends on the system) sized. In addition the hugepages are never swapped out, they are locked in memory.. Kernel does less work for bookkeeping of virtual memory, because of the larger page sizes.. Note that: Hugepages is not compatible with automatic memory management that Oracle does if configured to do..
Let's start our demo. (Note that, my demo env is an Oracle Linux 6.5 x86_64 and the kernel is an UEK 3.8.13-16.2.1.el6uek.x86_64)
HUGEPAGES OCCUPY MEMORY ONCE THEY ARE CONFIGURED (altough they are not used by any applications)
Firstly, I will show you the affect of hugepages. You will see the hugepages are never swapped out and when they are configured; they occupy memory, eventhough they are not used at all.
Initially, our hugepages are not configured as seen below;
[root@jiratemp ~]# cat /proc/meminfo |grep Huge
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Next, we sync and drop filesystem caches to have a clean environment in terms of memory. (we do this as we will use free command to see the affect of our actions)
[root@jiratemp ~]# sync;echo 3 > /proc/sys/vm/drop_caches ; free -m
[root@jiratemp ~]# free -m
total used free shared buffers cached
Mem: 7985 609 7375 0 4 47
-/+ buffers/cache: 557 7427
Swap: 8015 0 8015
Afterwards; we configure 2048 hugepages directly using proc fs and directly check the memory usage using free command;
[root@jiratemp ~]# echo 2048 > /proc/sys/vm/nr_hugepages (Hugepages are 2MB)
[root@jiratemp ~]# free -m
total used free shared buffers cached
Mem: 7985 4709 3275 0 4 49
-/+ buffers/cache: 4655 3330
Swap: 8015 0 8015
A quick explanation for the free command output:
Mem: total= Total physical memory
Mem: used = MemTotal - MemFree
Mem: free = Free memory
Mem: shared = meaningless nowadays, can be ignored
Mem: buffers Buffers
Mem: cached Cached memory
-/+ buffers/cache: used MemTotal - (MemFree + Buffers + Cached)
-/+ buffers/cache: free MemFree + Buffers + Cached
Swap: total Total swap
Swap: used SwapTotal - SwapFree
Swap: free Free Swap
You see 4709M are used. One page is 4K, one hugepage is 2M, so 2048 Hugepage makes 4096M
free command reports mb values when used with "-m" argument. See the used value is 4709 (609M was already used before we configure hugepage) . 4709-4096= 613 (almost equal to 609). So these used megabytes are caused by hugepages.
I remind you, we didn't use those hugepages, but once configured, they occupy memory as you see.
Well, it is certain that huge pages are reserved inside the kernel .
HUGEPAGES ARE NOT SWAPPED OUT EVEN UNDER PRESSURE (even when they are not used by any applications)
Hugepages can not be swapped out.. It is real. In order to test it, I wrote a python program. This program takes only one input , the memory size that we want it to allocate.
So , we use this program to create a memory pressure and to see if we can allocate hugepages when there is a memory pressure.
First, we configure 2048 hugepages;
[root@jiratemp ~]# echo 2048 > /proc/sys/vm/nr_hugepages (Hugepages are 2MB)
[root@jiratemp ~]# free -m
total used free shared buffers cached
Mem: 7985 4709 3275 0 4 49
-/+ buffers/cache: 4655 3330
Swap: 8015 0 8015
As you see above, there is only 3275 mb free, so almost all used memory is occupied by Hugepages.
Now, we execute our python program and try to allocate 4500MB of memory.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3740 root 20 0 4615m 3.4g 424 S 0.0 44.2 1:09.38 python ./erm 4500
While our program is running, we take the free command output in every second to see the system wide memory usage ;
[root@jiratemp ~]# free -m -s 1
total used free shared buffers cached
Mem: 7985 4701 3284 0 0 43
-/+ buffers/cache: 4656 3328
Swap: 8015 0 8015
total used free shared buffers cached
Mem: 7985 4701 3284 0 0 43
-/+ buffers/cache: 4656 3328
Swap: 8015 0 8015
total used free shared buffers cached
Mem: 7985 4703 3281 0 0 45
-/+ buffers/cache: 4657 3327
Swap: 8015 0 8015
total used free shared buffers cached
Mem: 7985 5750 2234 0 0 46
-/+ buffers/cache: 5704 2281
Swap: 8015 0 8015
total used free shared buffers cached
Mem: 7985 7928 56 0 0 46
-/+ buffers/cache: 7881 103
Swap: 8015 0 8015
....
.............
........................
You see , as our program allocates more memory in every second, free memory is getting closer to 0 (zero).
Moreover; because of this pressure, our server starts to hang and when we check the situation using top command (using our limited cpu cycles), we see that kswapd is aggresively running..
*COMMAND HANGS..
*SWAP DAEMON IS RUNNING AND SWAP USED IS INCREASED IN EVERY SECOND!!
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
60 root 20 0 0 0 0 R 11.2 0.0 0:16.84 kswapd0
[root@jiratemp 3740]# cat status|grep Swap
VmSwap: 1018512 kB
Morever, when we check our process; we see its Resident memory is 3.4G, as seen below;
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3740 root 20 0 4615m 3.4g 424 S 0.0 44.2 1:09.38 python ./erm 4500
However, the virtual memory of our process is 4615m, as you see above.
(VIRTUAL MEMORY= 4615 MB but RES=3.4G)
So , this is a little interesting right? Because we requested 4500 m memory, but our resident memory is 3.4G..
The situation is the same when we run a C program and try to allocate 5500 megabytes..
Program just slows down when it reaches the 3 GB of memory and swap activities are triggered.
Our application stuck at this point, but if we wait the swap daemon to swap out the memory , we can see that our program can actually allocate 5500 MB.. Look program says, I m allocating 5550 th MB ;
Currently allocating 5535 MB
Currently allocating 5536 MB
Currently allocating 5537 MB
Currently allocating 5538 MB
Currently allocating 5539 MB
Currently allocating 5540 MB
Currently allocating 5541 MB
Currently allocating 5542 MB
Currently allocating 5543 MB
Currently allocating 5544 MB
Currently allocating 5545 MB
Currently allocating 5546 MB
Currently allocating 5547 MB
Currently allocating 5548 MB
Currently allocating 5549 MB
But; when we look at the top output, we see the RES is only 3.6 G, however again VIRT is increased. So swap is there.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4191 root 20 0 5418m 3.6g 152 R 64.0 45.7 1:10.96 ./a.out
You see; when we look at the top output above, we see the RES is only 3.6 G, however VIRT is 5500. So swap is there...
So our page are swapped out! (Remember VIRT = The total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages that have been swapped out)
MAN TOP ->
VIRT -- Virtual Image (kb)
The total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages that have been swapped out. (Note: you can
define the STATSIZE=1 environment variable and the VIRT will be calculated from the /proc/#/state VmSize field.)
Well, if we disable the hugepages; we can allocate "resident" memory using the same program... Here is an example output of top for the same program; (hugepages disabled),.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4333 root 20 0 6623m 6.5g 376 S 67.2 82.9 0:03.33 a.out
You see RES=6.5g.. So you see it allocates from the RESIDENT memory. (a.out is a C program which continously and endlessly allocates and uses memory)
So, this proves that hugepages are not swapped out . They are not swapped out, even in the case of a memory shortage. Also, when we have a memory shortage and if almost all the memory is allocated by the Hugepages, then we can see the pages that we recently allocated by our program, are swapped out to make room for our program to allocate more memory :).
Another interesting thing is; if there is not enough free memory, Hugepages can not be configured properly.
I mean, we can allocate regular pages from a self written program and we can test this.
When we do such a test, we see the hugepages will not be allocated altough we issue the commands;
Well we allocate all the memory by using a self written application and then configure 2048 hugepages..
Interesting thing is that, our command doesn't encounter any errors but hugepages are not allocated at all..
[root@jiratemp ~]# echo 2048 > /proc/sys/vm/nr_hugepages
[root@jiratemp ~]# echo $?
0
[root@jiratemp ~]# hugeadm --pool-list
Size Minimum Current Maximum Default
2097152 1 1 1 *
[root@jiratemp ~]# grep Huge /proc/meminfo
HugePages_Total: 1
HugePages_Free: 1
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
As you see, setting hugepages is a task that should be done carefully. As they are not swapped out, system may hang in case of a memory shortage and the risk of memory shortage is actually increased when we you hugepages or let's say when we configure(not even used) hugepages.
Well, there is alternative way for configuring hugepages actually. User the overcommit configuration, we can at least decrease the memory allocation of our hugepages when they are not used by any process.
OVERCOMMIT SETTING FOR HUGEPAGES:
Let's introduce the overcommint setting for the hugepages first;
/proc/sys/vm/nr_overcommit_hugepages specifies how large the pool of
huge pages can grow, if more huge pages than /proc/sys/vm/nr_hugepages are
requested by applications. Writing any non-zero value into this file
indicates that the hugetlb subsystem is allowed to try to obtain that
number of "surplus" huge pages from the kernel's normal page pool, when the
persistent huge page pool is exhausted. As these surplus huge pages become
unused, they are freed back to the kernel's normal page pool.
So, if we set the hugepages to a lower value and set the overcommit hugepages to a large value (large enough to meet our peak hugepage requests); then we can have a dynamic hugepage allocation in our environments.
Let's make a demo and see how it is done and how it behaves;
We set 100 hugepages and we set 1000 overcommit hugepages
[root@jiratemp ~]# echo 100 > /proc/sys/vm/nr_hugepages
[root@jiratemp ~]# echo 1000 > /proc/sys/vm/nr_overcommit_hugepages
We check the /proc/meminfo and hugepage pool list and see only 100 hugepages are allocated (as no processes use any hugepages at the moment)
[root@jiratemp ~]# grep Huge /proc/meminfo
HugePages_Total: 100
HugePages_Free: 100
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@jiratemp ~]# hugeadm --pool-list
Size Minimum Current Maximum Default
2097152 100 100 1100 *
We sync and clear the caches to have a fresh start in terms of memory and allocate shared memory from the huge pages. (just like an Oracle Database does :)
[root@jiratemp ~]# sync;echo 3 > /proc/sys/vm/drop_caches ; free -m
total used free shared buffers cached
Mem: 7985 736 7249 0 0 21
-/+ buffers/cache: 713 7271
Swap: 8015 125 7890
Note: for allocating shared memory from the hugepages, I use the following C program:
#include<stdio.h>
#include<sys/shm.h>
#include<sys/stat.h>
#include<unistd.h>
int main(){
int segment_id_1;
char *shared_memory_1;
struct shmid_ds shmbuffer;
int segment_size;
const int shared_segment_size=0x40000000;
/*Allocate a shared memory segment*/
segment_id_1=shmget (IPC_PRIVATE,shared_segment_size,SHM_HUGETLB | IPC_CREAT | SHM_R | SHM_W);
shared_memory_1=(char*)shmat(segment_id_1,0,0);
sprintf(shared_memory_1,"ERMAN");
sleep(100);
return 0;
}
0x40000000 means 1GBytes, which means 512 Hugepages in Linux.
So , we tell our program to allocated 1GB ( 512 Hugepages) shared memory from the hugepages.
Remember, our hugepage count was 100, so there were 100 hugepages in our hugepage pool as shown earlier. On the other; we set 1000 overcommit hugepages.
Well, when we execute this program, we see 512 pages are allocated. So our pool has enlarged :)
[root@jiratemp ~]# grep Huge /proc/meminfo
HugePages_Total: 512
HugePages_Free: 511
HugePages_Rsvd: 511
HugePages_Surp: 412
Hugepagesize: 2048 kB
[root@jiratemp ~]# hugeadm --pool-list
Size Minimum Current Maximum Default
2097152 100 512 1100 *
[root@jiratemp ~]# free -m
total used free shared buffers cached
Mem: 7985 1567 6417 0 3 25
-/+ buffers/cache: 1538 6446
Swap: 8015 125 7890
Now, our free memory decreased by 1024 Mbytes.
So overcommit works perfectly. We had 100 hugepages at first. So our hugepages were occupying only 200 Mbytes initially. However, when we need more, we could allocate it (thanks to overcommit)
We got ourselves an environment which can do a dynamic hugepage allocation..
WHAT ABOUT ORACLE DATABASE? CAN IT USE OVERCOMMIT HUGEPAGES?
Let's try with the Oracle database;
First of all our limits.conf should be configured properly to use hugepages.. In other words; oracle Os user must be able to lock memory when it is instructed to use hugepages (especially Hugepages only!)
This can be done in 2 ways.
1) By setting cap for oracle binary with root account
cd $ORACLE_HOME/bin
setcap cap_ipc_lock=+ep oracle
2) By adding the following (change the values according to your needs) to the limits.conf
oracle soft memlock unlimited
oracle hard memlock unlimited
If we don't do one of these configurations; we end up with the following ORA-27137 ;
[oracle@jiratemp ~]$ sqlplus "/as sysdba"
SQL*Plus: Release 12.1.0.1.0 Production on Thu Nov 10 11:08:01 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup nomount;
ORA-27137: unable to allocate large pages to create a shared memory segment
Linux-x86_64 Error: 1: Operation not permitted
Additional information: 14680064
Additional information: 1
Well, suppose we configured our memory lock parameters or set cap for oracle binary,
and configured our memory related database parameters as follows;
--these parameters are used to configure the initial memory allocation of Oracle Database ,when it is started. (the parameter sga_target is for this actually)
sga_max_size = 1000M
sga_target = 500M
use_large_pages_only=ONLY --> this instruct oracle to use only the hugepages.
We set hugepage overcommit to 1000 and hugepage count to 100 ;
[root@jiratemp ~]# sync;echo 3 > /proc/sys/vm/drop_caches ; free -m
total used free shared buffers cached
Mem: 7985 205 7779 0 0 16
-/+ buffers/cache: 189 7796
Swap: 8015 30 7985
[root@jiratemp ~]# grep Huge /proc/meminfo
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@jiratemp ~]# hugeadm --pool-list
Size Minimum Current Maximum Default
2097152 0 0 0 *
[root@jiratemp ~]# echo 100 > /proc/sys/vm/nr_hugepages
[root@jiratemp ~]# echo 1000 > /proc/sys/vm/nr_overcommit_hugepages
[root@jiratemp ~]# hugeadm --pool-list
Size Minimum Current Maximum Default
2097152 100 100 1100 *
So, we startup our Oracle database (note that , starting the database in nomount mode is enough for this test) as follows;
oracle@jiratemp ~]$ sqlplus "/as sysdba"
SQL*Plus: Release 12.1.0.1.0 Production on Thu Nov 10 11:23:41 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved
Connected to an idle instance.
SQL> startup nomount;
ORACLE instance started.
Total System Global Area 1043886080 bytes
Fixed Size 2296280 bytes
Variable Size 876611112 bytes
Database Buffers 159383552 bytes
Redo Buffers 5595136 bytes
We check our parameters are set.
SQL> show parameter sga_target
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
sga_target big integer 500M
SQL> show parameter sga_max_size
sga_max_size big integer 1000M
Once, our database is started, we directly check the huge page usage;
Remember, our overcommit hugepage number was set to 1000 (2048 Mb) and our hugepage number was set to 512 (1024 Mbytes)
[root@jiratemp ~]# grep Huge /proc/meminfo
HugePages_Total: 501
HugePages_Free: 252
HugePages_Rsvd: 252
HugePages_Surp: 401
Hugepagesize: 2048 kB
You see? The hugepages the total hugepages count is now 501. (it was 100 earlier), and the Hugepages surplus is 401. So Linux let oracle to allocate more hugepages than configured by respecting the overcommit configuration. In other words; Linux let Oracle to allocate(reserver+allocate) 512 Hugepages by enlarging the Hugepage pool automatically and dynamically.
As we set sga_target parameter to 500M, Oracle allocated almost 250 Hugepages.(see Hugepages_Free is 252), also Oracle reserved 252 more hugepages as sga_max_size was set to 1024MB.
Now, we set sga_target parameter to 1000Mbytes and see if Linux will let Oracle to use all the 512 Hugepages.
SQL> alter system set sga_target=1000M scope=memory;
System altered.
[oracle@jiratemp ~]$ grep Huge /proc/meminfo
HugePages_Total: 501
HugePages_Free: 2
HugePages_Rsvd: 2
HugePages_Surp: 401
Hugepagesize: 2048 kB
Yes. Oracle allocated almost 500 Hugepages to build its 1024M sized SGA on top of these hugepages.
When we shutdown our Oracle database, we see the hugepage pool dynamically deallocated and the space occupied by Oracle 's Hugepages are freed as expected.
SQL> shu immediate;
ORA-01507: database not mounted
ORACLE instance shut down.
[root@jiratemp ~]# grep Huge /proc/meminfo
HugePages_Total: 100
HugePages_Free: 100
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
HOW ORACLE DOES IT?
Well, we see that, Oracle can go with the the hugepage overcommit setting(nr_overcommit_hugepages).
What actually Oracle does is, that it honours the sga_max_size in the instance startup and make commitment to allocate the necessary hugepages for satisfying the sga_max_size. However, actually it only allocates the necessary hugepages for satisfying the sga_target.
Although the configured number of hugepages is set to a lower value, Oracle can allocate the necessary amount of hugepages to satisfy its sga_max_size, because an hugepage overcommit configuration is in place.Furthermore, when Oracle database is shut down, the surplus of hugepages (overcommit count - number of hugepage count) is given back to OS.
CONCLUSION:
So, following are the conclusions;
- Hugepages are not swapped out in any circumstances.
- Hugepages are not given to any process that want to allocate regular pages.
- Hugepages occupy memory once they are configured , eventhough they are not used.
- nr_overcommit_hugepages is good thing in the relevant cases.
- nr_overcommit lets a process to allocate more hugepages than configured.
- When a process shuts down gracefully or release its memory, the hugepages used by that process are given to back to system and if that process uses overcommit hugepages, the nr_overcommit_hugapages value will be set to its default.
- Oracle will go with the nr_overcommit_hugepages parameter.
- Oracle will allocate(not use) shared memory using Huge pages based on the sga_max_size.(We can think like Oracle allocates shared memory using shmget in the instance startup, in other words; the segment size argument given in the shmget equals the value defined in sga_max_size) . So, when Oracle allocates shared memory in startup, it actually reserves the hugepages and the Reserved hugepage count increases. On the other hand, Oracle will use the number of hugepages based on the sga_target and that's why Hugepagess free(seen in /proc/meminfo) decreases accordingly.
- If we use hugepages (only), we should set sga_max_size equal to the sga_target. (in order to not to waste our memory)
Well, after knowing these; I want to give an example, where overcommit memory can be used to address a memory wastage problem.
QUESTION:
Suppose;
- We have 3 databases in a single server environment. Let's say these are TEST , DEV and UAT.
- We sometimes work only in TEST, sometimes in DEV and sometimes, in 2 of these databases and sometimes in all of these 3.
- We also have processes/session which do lots of PGA work.
- We have 100 Gigabytes of Ram and we want to reserver 10GB of it to OS.
- We want to have 20GB sized SGA for each of these databases.
- We want to use sga_max_size/sga_target (ASMM -- not AMM) and use Hugepages as well.
Now, consider which one of the following setting is a better idea?
1)
echo 10240 > /proc/sys/vm/nr_hugepages
echo 30720> /proc/sys/vm/nr_overcommit_hugepages
2)
echo 30720> /proc/sys/vm/nr_hugepages
echo 30720> /proc/sys/vm/nr_overcommit_hugepages
MY ANSWER:
*My answer is 1) .
The reason is explained earlier. :)
What is yours? please feel free to comment...
