[Linux] Understanding Load Average

I am writing this quick article to help understand the importance of Load Average in Linux for a Performance Engineer when debugging an issue.

Load average on a system can be obtained by different ways:

1.Using "top" command. This gives the current live load average.

2. Using "cat /proc/loadavg" command. This gives a snapshot of the load average as the command is executed.

3.Using command "uptime". This also gives a snapshot of the load average as the command is executed.

4.Using command "sar -q". This is very useful method, if you are trying to diagnose a problem that has already occurred and passed by. Since sar collects historic data, you can extract historic load average details using "sar -q" command.

Now that we know how to get the load average values, lets try to make sense of what these load average values mean. As you can see, in all the above ways load average is a set of three numbers. These three numbers are the load averages on the system in [last 1min] [last 5mins] [last 15mins]. These three sets of values help in understanding if the load on the system is increasing/decreasing or is steady over time.

In simple words, load average means the number of processes in active state waiting to be executed. So the values of load average tell us if the CPU's have a large number of waiting processes to be served or they are good to serve the processes without making them wait long. Note : In the above sentence by "CPU's" I mean logical CPU's. Details explanation of what logical/virtual CPU is can be found in my other article here.

Now that we know what load average values mean, how to decide if a load average value is good or bad? The answer to this depends on the capacity of CPU that you have on your system.

Lets assume that we have a 8 core CPU installed on our machine. Each of these cores can at any given time can serve one process. In this case a load average value of 8 would mean each core are serving a process and are running to their capacity. It's handling a consistent number of tasks that are equal to the number of tasks it can handle. If your load average is consistently more than the number of cores you have available, that's when you'd probably want to look into the situation. It's fine for your server to be at capacity every now and then, but if it always is, that's a cause for alarm.

It is important to note that CPU percentage on a system and load average are different. CPU percentage refers to the resources used by the process running on the CPU and load average refers to the number of processes in active state waiting to be served by the CPU.

Load average is a very important parameter as it gives a picture of run queue on a server. When the load averages are pretty high, you might want to take a bunch of thread dumps to see what is making this run queue grow and what the current served processes are doing on the CPU.

Hope this article helps.