"The Ultimate Guide to CPU Analysis: Boosting Efficiency and Troubleshooting Performance"

In the world of system performance, understanding how the CPU behaves is crucial for identifying bottlenecks, optimising resource usage, and ensuring smooth operation of applications and services. Linux provides a range of powerful tools for analysing CPU performance, each with unique capabilities. This guide will explore key tools and techniques for CPU analysis on Linux

Generating CPU load is a crucial step in performance testing and analysis. It allows you to simulate high load conditions and observe how your system behaves under stress. Here’s a step-by-step guide on how to generate CPU load using a buggy application, a CPU simulator, and the sysbench tool.

Generating CPU Load

1. Using a Buggy Application (Ycrash)

Ycrash is an example of a buggy application that can be used to generate CPU load. While it is not a real-world tool for load generation, we'll simulate its use as a placeholder. If you have a specific application in mind that is known to cause high CPU usage, you can use that instead.

Install or Compile Ycrash: Assuming Ycrash is a dummy application for the example, ensure it’s available for your system. If you don't have a real application, you can use other methods to simulate load.

2. Using a CPU Simulator

A CPU simulator or stress test tool is designed to generate a controlled amount of CPU load. One popular tool for this purpose is stress or stress-ng.

Install stress:

sudo apt-get install stress

Run stress to Generate CPU Load:

stress --cpu 4 --timeout 60s

• --cpu 4: Specifies the number of CPU workers (adjust based on the number of cores).
• --timeout 60s: Specifies the duration of the load (60 seconds in this case).

Alternatively, use stress-ng:

sudo apt-get install stress-ng stress-ng --cpu 4 --timeout 60s

• --cpu 4: Number of CPU stressors.
• --timeout 60s: Duration of the test.

3. Using sysbench Tool

sysbench is a versatile benchmarking tool that can generate CPU load and perform various performance tests.

Install sysbench:

sudo apt-get install sysbench

Generate CPU Load Using sysbench:

sysbench cpu --cpu-max-prime=20000 run

• --cpu-max-prime=20000: The maximum prime number used in the CPU test. Larger values will generate more load.

Below are list of tool which helps to analyse CPU usage & find out bottleneck . We are delivering CPU analysis in 3 part

Part 1:

1. uptime
2. top
3. htop
4. vmstat
5. mpstat
6. pidstat

Part 2 :

1. perf linux profilling

Part 3:

BCC tools

1. execsnoop
2. exitsnoop
3. runqlat
4. runqlen
5. runqslower
6. cpudist
7. cpufreq
8. profile
9. offcputime
10. syscount
11. argdist
12. trace
13. funcount

uptime

top

From the above top output we can see that java process is consuming 300% CPU . This is combination of all 4 CPU available on the system

top -H

With top -H we can find out which specific thread are consuming how much CPU . From above snapshot, java process has almost 20 threads and out of that those mentioned 4 threads are consuming very high CPU

htop

htop is advanced utility which is give combine information of uptime + top + top - H + individual CPU utilisation

htop with filter & tree:

vmstat

from top & htop , we are sure that which process is consuming CPU , now we want to confirm that that CPU utilisation is from user level or system level.

from vmstat observation , we can confirm that here CPU utilisation is from user level and not from system level so later we have to troubleshoot application code

mpstat

from mpstat , we can confirm that out of 4 CPU which CPU is used java process is utilising. here all 4 CPU are being utilised by java

pidstat

At least from above observation , we are sure that which process , thread , how many CPU are responsible for high CPU utilisation

CPU utilisation is happening at user level not at system level

Next challenge is to find out where exactly CPU consumption is happening with Perf tool

Reference:

https://github.com/iovisor/bcc/blob/master/images/bcc_tracing_tools_2019.png

https://github.com/iovisor/bcc/blob/master/docs/tutorial.md

https://github.com/akopytov/sysbench?tab=readme-ov-file