Linux -- understanding the "free" command output, reclaimable vs free

Nowadays, we have processes running on Linux .. They allocate high amount of memory regions and sometimes we suffer from this high memory consumptions.
On the other hand, by just looking to output that are generated by tools like Top , is not enough to make a decision about the memory consumptions..
Nowadays, I have seen that even the developers use free command to analyze the memory consumption of the OS that their applications are running on.. I also have seen that sometimes the output of free command is interpreted wrongly, and most of the time it has been thought that the system is suffering from the memory.
That's why I m writing this post , and sharing output of free -m command in an understandable format that summarizes the information that can be used to analyze the free command output.
Dont forget that the memory is a resource that is meant to be used %100 and Linux memory concept is designed for efficiency.

 Example of free -m command output , in Linux;

[root@nagios ~]# free -m

                      Total                       Used                  Free             Shared               Buffers          Cached

Mem:              512                         392                    119                 0                           28             249
                       Total Phys             Used Phys             Free Phys      Shared mem                 Buffers    Fast access                                   Mem                 Mem               Mem         across procs            

-/+ buffers/cache:                             113                    398
                                                Actual Used           Actual Free 
                                                  Phys Mem              Phys Mem

Swap:               2975                         0                           2975
                  Total Swap               Used Swap              Free Swap

So, the memory for buffers and cached memory can reclaimed if needed. So they can be thought that they are free..

If we need to calculate the used memory and free memory we should use the following formula;

Used Memory = Used Phys - Buffers - Cached -> in this example  392 - 28 - 249 = 115

Free Memory  = Free Phys + Buffers + Cached -> in this example 119 + 28 + 249 = 396 

Look for the -/+ buffer cache lines now , seems familiar :) Used 113 (almost 115) , Free 398 (almost 396)

So if you use free -o command you wont have a line starting with -/+ buffers/cache 

free -o -m

                    total       used       free     shared    buffers     cached

Mem:           512        401        110          0         34        252

Swap:         2975          0       2975

So use "-o" option  if you want to calculate the actual used and free memory yourself :)

The -o switch disables the display of a "buffer adjusted" line.  If the -o option is not specified, free subtracts buffer memory from the used memory and adds it to

       the free memory reported.

Note that, Linux is designed to use swap, so if you some used Swap in free command output, dont panic. Analyze swap in/out operations using "sar" or a similar tool..

Also note that: If you want to drop those caches , you can use /proc/sys/vm/drop_cache.
First issue sync command (for dirty things) , execute echo 3 > /proc/sys/vm/drop_caches

Sometimes we use 3 sync commands repeatedly.. Do we need 3 syncs? That is another story :)

Another place to look for memory consumption is  /proc/meminfo

Here is an example meminfo:

MemTotal: 524288 kB
MemFree: 109992 kB
Buffers: 37736 kB
Cached: 258604 kB
SwapCached: 0 kB
Active: 141500 kB
Inactive: 203412 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 524288 kB
LowFree: 109992 kB
SwapTotal: 3047416 kB
SwapFree: 3047416 kB
Dirty: 340 kB
Writeback: 0 kB
AnonPages: 48564 kB
Mapped: 13780 kB
Slab: 20736 kB
PageTables: 9344 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 3309560 kB
Committed_AS: 192008 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 3600 kB
VmallocChunk: 34359734759 kB

As you see , it has similar values in terms of memtotal, memfree , buffers and cache

Okay we said that , the actual free memory is free + cache+ buffers, but we also know that cache and buffers are not free actually, on the other hand; they are reclaimable.

So is the free memory and reclaimable memory the same thing ?

In my opinion , they are not the same thing in theory, actually.

We can put anything to our free pages, but our file data and metadata are put on the reclaimable memory.

In other words, in a healthy system, reclaimable memory should be non-zero .. There should be room to grow in memory without effecting those caches.

That is , maybe we can meet the user process requirements by reclaiming the pages from the caches, but this may also effect negatively on performance in terms of IO , because we are freeing the cache ..(It is been said on some articles, that in the last 1MB of the freeing caches, there will be a noticable performance degregation in a crowded system.)