[Performance] : What does CPU% usage tell us ?

[Edit] -- All my latest articles are not published at - https://performanceengineeringin.wordpress.com/

When you come across a system which is misbehaving, majority of the times the first metrics that we look at is CPU usage. But do we really understand what CPU usage of a system tells us ? In this article let us try and understand what X % usage of a system really means.

One of the easy ways to check on CPU is "top" command.

The "%Cpu(s)" metrics seen above is a combination of different components.

• us - Time spent in user space
• sy - Time spent in kernel space
• ni - Time spent running niced user processes (User defined priority)
• id - Time spent in idle operations
• wa - Time spent on waiting on IO peripherals (eg. disk)
• hi - Time spent handling hardware interrupt routines. (Whenever a peripheral unit want attention form the CPU, it literally pulls a line, to signal the CPU to service it)
• si - Time spent handling software interrupt routines. (a piece of code, calls an interrupt routine...)
• st - Time spent on involuntary waits by virtual cpu while hypervisor is servicing another processor (stolen from a virtual machine)

 Out of all the breakdowns above, we usually concentrate mainly on User Time (us) , System time(sy) and IO wait time (wa). User time is the percentage of time the CPU is executing the application code and System time is the percentage of time the CPU is executing the kernel code. It is important to note that System time is related to application time; if application performs IO for example, the kernel will execute the code to read file from disk. Also, any wait seen in IO will reflect in IO wait time. So us%, sy% and wa % are related.

 Now let's see if we understand this correctly on a whole.

My goal as a Performance Engineer would be to drive the CPU usage as high as possible for as short a time as possible. Does that sound far away from the "best-practice" ? Wait, hold your thought there.

The first thing to know is, the CPU usage reported by any command is always an average over an interval of time. If the CPU consumed by an application is 30% for 10minutes, the code can be tuned to make it consume 60% for 5minutes. Do you see what I mean by "driving the CPU as high as possible for as short time as possible"? This is doubling the performance. Did the CPU usage increase ? Sure, Yes. But is that a bad thing ? No. CPU is sitting there waiting to be used. Use it, improve the performance. High CPU usage is not a bad thing all the time. It may just mean that your system is used at its full potential. A good ROI. However, if you have your run-queue length increasing, where requests are waiting for cpu, then it definitely needs your attention.

In linux systems, the number of threads that are able to run (i.e, not blocked on IO or sleeping etc) are referred to as run-queue. You can check this by running "vmstat 1" command. The first number in each line refers to run-queue.

If the count of the threads in the above output is more than the available CPU's (count in hyper-threading if enabled), that means the threads are waiting for CPU and the performance will be less the optimal. Although a higher number is ok for a brief amount of time, but if the run-queue length is high for a significant amount of time, it is an indication that system is overloaded.

Conclusion :

• High CPU usage of a system is not a bad sign all the time. CPU is available to be used. Use it and improve the performance of the running application.
• If run-queue length is high for a significant amount of time, that mean the system is overloaded, and needs optimizations.