VMSTAT Explained

vmstat :

vmstat is a tool in Unix/Linux which is used to Report virtual memory statistics. It shows how much virtual memory there is, how much is free and paging activity. Most important, you can observe page-ins and page-outs as they happen.

vmstat  reports  information about processes, memory, paging, block IO, traps, and cpu activity.

> vmstat

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----

 r  b   swpd   free   buff  cache        si   so    bi    bo   in   cs us sy id wa st

 0  0  21688 12797396 850716 15372668    0    0     3    35    0    0  1  0 99  0  0

 0  0  21688 12789292 850716 15372668    0    0     0  1355  883  744  1  0 99  0  0

 1  0  21688 12784648 850716 15372676    0    0     0    15 1934 1292  6  0 93  0  0

 2  0  21688 12781356 850716 15372736    0    0     0    16 2222 2094  8  1 92  0  0

 0  0  21688 12809992 850716 15376880    0    0     0    16 1959 1621  6  1 93  0  0

 1  0  21688 12809992 850716 15376888    0    0     0    29 1090  860  1  0 99  0  0

 0  0  21688 12808936 850716 15376888    0    0     0   132  931  798  1  0 99  0  0

 0  0  21688 12809496 850716 15376888    0    0     0     5  868  768  1  0 99  0  0

 1  0  21688 12793492 850716 15376888    0    0     0     5 1701 1177  5  0 95  0  0

 1  0  21688 12767728 850716 15376928    0    0     0  1355 8553 2589 48  1 51  0  0

13  0  21688 12765680 850716 15376968    0    0     0    24 7619 3122 44  1 56  0  0

Fields explained as follows:

Under Procs we have

       r: The number of processes waiting for run time or placed in run queue or are already executing (running)

       b: The number of processes in uninterruptible sleep. (b=blocked queue, waiting for resource (e.g. filesystem I/O blocked, inode lock))

If runnable threads (r) divided by the number of CPU is greater than one -> possible CPU bottleneck

(The (r) coulmn should be compared with number of CPUs (logical CPUs as in uptime) if we have enough CPUs or we have more threads.)

High numbers in the blocked processes column (b) indicates slow disks.

(r) should always be higher than (b); if it is not, it usually means you have a CPU bottleneck

Note: “cat /proc/cpuinfo” dispalys the cpu info on the machine

cat /proc/cpuinfo|grep processor|wc -l

output: 16

Remember that we need to know the number of CPUs on our server because the vmstat r value must never exceed the number of CPUs. r value of 13 is perfectly acceptable for a 16-CPU server, while a value of 16 would be a serious problem for a 12-CPU server.

Whenever the value of the r column exceeds the number of CPUs on the server, tasks are forced to wait for execution. There are several solutions to managing CPU overload, and these alternatives are:

1.      Add more processors (CPUs) to the server.

2.      Load balance the system tasks by rescheduling large batch tasks to execute during off-peak hours.

Under Memory we have:

swpd: shows how many blocks are swapped out to disk (paged). The amount of Virtual memory used.
            Note: you can see the swap area configured in server using "cat proc/swaps"
>cat /proc/swaps

Filename                                Type            Size    Used    Priority

/dev/dm-7                               partition       16777208        21688   -1

free: The amount of Idle Memory
buff: Memory used as buffers, like before/after I/O operations
cache: Memory used as cache by the Operating System

Under Swap we have:

si: Amount of memory swapped in from disk (/s). This shows page-ins

so: Amount of memory swapped to disk (/s). This shows page-outs. The so column is zero consistently, indicating there are no page-outs.

In Ideal condition, si and so should be at 0 most of the time, and we definitely don’t like to see more than 10 blocks per second.

Under IO we have:

bi: Blocks received from block device - Read (like a hard disk)
bo: Blocks sent to a block device – Write

Under System we have:

in: The number of interrupts per second, including the clock.
cs: The number of context switches per second.

(A context switch occurs when the currently running thread is different from the previously running thread, so it is taken off of the CPU.)

It is not uncommon to see the context switch rate be approximately the same as device interrupt rate (in column)

If cs is high, it may indicate too much process switching is occurring, thus using memory inefficiently.

If cs is higher then sy, system is doing more context switching than actual work.

High r with high cs -> possible lock contention
Lock contention occurs whenever one process or thread attempts to acquire a lock held by another process or thread. The more granular the available locks, the less likely one process/thread will request a lock held by the other. (For example, locking a row rather than the entire table, or locking a cell rather than the entire row.)

When you are seeing blocked processes or high values on waiting on I/O (wa), it usually signifies either real I/O issues where you are waiting for file accesses or an I/O condition associated with paging due to a lack of memory on your system.

Under CPU we have:

These are percentages of total CPU time.

       us: % of CPU time spent in user mode (not using kernel code, not able to acces to kernel resources). Time spent running non-kernel code. (user time, including nice time)

       sy: % of CPU time spent running kernel code. (system time)

       id: % of CPU  idle time

       wa: % of CPU time spent waiting for IO.

Note: the memory, swap, and I/O statistics are in blocks, not in bytes. In Linux, blocks are usually 1,024 bytes (1 KB).

To measure true idle time measure id+wa together:
- if id=0%, it does not mean all CPU is consumed, because "wait" (wa) can be 100% and waiting for an I/O to complete

- if wait=0%, it does not mean I have no I/O waiting issues, because as long I have threads which keep the CPU busy I could have additional threads waiting for I/O, but this will be masked by the running threads

If process A is running and process B is waiting on I/O, the wait% still would have a 0 number.

A 0 number doesn't mean I/O is not occurring, it means that the system is not waiting on I/O.

If process A and process B are both waiting on I/O, and there is nothing that can use the CPU, then you would see that column increase.

- if wait% is high, it does not mean I have io performance problem, it can be an indication that I am doing some IO but the cpu is not kept busy at all

- if id% is high then likely there is no CPU or I/O problem


To measure cpu utilization measure us+sy together (and compare it to physc):
- if us+sy is always greater than 80%, then CPU is approaching its limits

 
- if us+sy = 100% -> possible CPU bottleneck

- if sy is high, your appl. is issuing many system calls to the kernel and asking the kernel to work. It measures how heavily the appl. is using kernel services.

- if sy  is higher than us, this means your system is spending less time on real work (not good)