How do you monitor and analyze the system behavior and resource utilization under stress testing?

Common objective of a software stress test is to identify the breaking point / weakness by pushing incremental user load to extreme levels. Typical Way to write a test case is to identify the peak user load test cases and incrementally amplify it

If I had a nickel for every time the requirements changed during the course of stress testing, I would be "Richie Rich" rich! Before starting the activity, answer the question - why. Define objectives. Capture relevant data. Stress testing is finding the breaking point of the system by subjecting it to extreme loads beyond normal operating capacity to proactively identify bottlenecks, failures, and stability issues. Often times we want to do unit sizing; we want to find out the load that your application can handle and instead of running tests on your "isolated" system, we make things more difficult for ourselves by stressing the entire system and then try to make sense of the results. Remember, Performance testing != stress testing

The most important step in defining a good perfornance test is to dive into the production scenario’s that are most likely to happen to the system under test. Is it peak load due to an increase of users? Or heavy traffic spikes after an emergency failover? Or do we just want to know how the application self-recovers after overload? There are many performance test scenario’s that can be applied to test specific system behaviour. Picking the right test scenario/pattern is key to reach a valuable outcome.

Stress testing is a vital process for identifying and addressing potential problems in a system before it goes live. By simulating heavy user loads, developers and testers can assess the system’s ability to handle extreme conditions, ensuring that it remains stable and reliable. Early detection and resolution of issues through stress testing help organizations avoid costly downtime and urgent fixes after deployment. This testing is particularly important for critical systems, such as servers, databases, and networks, as it directly impacts customer satisfaction. Monitoring and analyzing key metrics during stress testing allows teams to pinpoint weaknesses, prevent failures, and guarantee that the system can support high loads in production.

At my current workplace, we use LoadRunner to create test scripts and controller to run smoke tests. Along with that we have a licensed tool for Load Runner Enterprise 2023R1 on SaaS platform where we execute multiple tests across the State Of Maine. We gather the client-side metrics from LRE and correlate the graphs with server-side metrics where we Dynatrace or Datadog to monitor the systems. Dynatrace is insightful in identifying bottlenecks in our tests and provide good metrics with clean dashboard and it can dig up to the code level and pinpoint the bottlenecks.

Choose your tool based on the technology you are working on. Only a few Performance tools can test legacy applications or pure protocol-based transactions. Web-based applications There are a plethora of tools out there; some are free, and some are more than my home. No code, low code. My grandmother can test tools. Does the tool provide you with all the information to report based on your findings, and does it exercise the application in the way it was intended?

Choose tools wisely as you can't change them often.. I usually follow below methods to choose my tools for building a stress test: - Evaluate features - almost all tools can support stress tests, but some can support distributed load testing in open source versions - Scalability - Scripting language support and your team's skillset - - Reporting and integration. Popular open source options include JMeter, Gatling, k6 for testing and Grafana, ELK, Prometheus for monitoring. Always consider compatibility, conduct trials, and gather feedback before finalising on tools.

Ensure load testing framework doesn't break down before the system under test. Identify a tool that can generate more concurrency for the given resources. Though lot of APM tools available to measure the KPI's , I typically start with raw scripts (sar/vmstat/iostat etc) just to ensure I give enough available resources to the system under test and avoid additional overhead caused by the APM's

It's important to choose the right set of tools, be it for performance testing, tool like Jmeter, Loadrunner, Neoload, Blazemeter Cloud based ; or APM tool like JVisualVM, Grafana, Glowroot, Dynatrace, native log monitoring tools like sumologic. The selection of tools can depend on the expertise of the engineer with that tool and acceptance of the tool in that domain.

In a stress test, once you have chosen the tools, the next important step is to execute your tests and gather relevant metrics to make further decisions. I usually gather following metrics for each test run dividing collected metrics in 2 important bucket viz. before reaching the peak and post reaching each peak until the system breaks: - Response time - Throughput - Resource utilisation such as CPU, memory, disk I/O, and network bandwidth - Error rates - System stability: The ability of the system to maintain its performance and behavior under stress without crashing, freezing, or exhibiting other abnormal behaviours. The most important point from which every peak metrics becomes important are once your P95 SLA starts breaching.

Validate the test data before actual run, one thing missed by myself and causes a lot of unnecessary analysis later. Keep and eye on data that requires Real time capture like thread dump while high cpu, heap dump, error logs and prioritize saving that info. Other things like long DB queries, deadlocks can be captured later on. Also, good eyes that can do detailed analysis can relate the error with exact log trace as per timestamp or relevant id, provided they are not lost or overwritten.

For a stress test, these are the 2 primary information I would typically collect. 1. Load at which the system broke / Throughput stagnated . This is typically the current capacity of the system. 2. What is the first breaking point / hotspot ? This limits system scalability. It can typically be any of the system resources , network or in most cases inefficient application architecture. All other data are supplement to the above primary information.

Analyze test results to identify performance metrics, trends, patterns, and anomalies. Use LoadRunner graphs or external analysis tools to generate reports, charts, graphs, and dashboards for visualizing test data. Use APM tools to analyze server side metrics usch as CPU utilization, memory, memory leaks, Database tables, memory leaks, Garbage collection and others. Compare performance metrics against predefined thresholds or benchmarks to assess test success and identify areas for improvement.

Rerun tests multiple times to maintain consistency of results, Analyse peaks and their behaviour through your performance testing tool's report and APM that you might have setup for server side monitoring. Identify the bottlenecks by deep diving into application logs and traces of individual microservices, DBs, network etc. Also see if memory dumps and thread dumps needs to be analysed for any memory leakages.

Once the test reaches the inflection point (No increase in throughput under increasing user load), Identify the cause. Look for application errors, OOM's , Insufficient memory / CPU / Paging etc.

Personally, for testing, I find Jmeter to be able to serve most of the requirements, also being open source and having a lot of library integrations. One can also write a custom java code, package and bundle it as a jar, place in the lib/ext directory and use in jsr223 sampler. Use it to generate basic HTML reports. Monitoring wise, Dynatrace gives a holistic view of all services and apps in multi tier tech stack. Can provide pure path related to trace for long running APIs and DB queries. Grafana can provide monitoring dashboards with cool visualization and important metrics, also integrations with slack and other tools provides real time updates and health checks. I used Prometheus for VM info and InfluxDB for Jmeter results.

Once you have results analysed and in some cases root cause identified, you should report them to your dev squads and respective stakeholders with recommendation for short term and long term basis. Let's say if you found a memory leak issue which takes 2 sprints of time for devs to fix, do recommend a short term fix e.g. increase the infra temporarily in production to reduce impact on end users.

1. Monitor in Real-time: Keep a close eye on the monitoring dashboards during the stress test. Watch for any abnormal behaviour or performance degradation compared to baseline measurements. 2. Analyze Results: After the stress test, analyze the collected data to identify any performance bottlenecks or areas of concern. Look for correlations between different metrics and try to understand the root causes of any issues observed. 3. Tune and Optimize: Based on your analysis, make adjustments to your system configuration, code, or infrastructure to address any identified issues. This could involve optimizing algorithms, adding more resources, scaling out components, or making architectural changes.

For the stress tests results should be simple. It should just answer 1. What component in the system is weak under load / hotspot. 2. What is the maximum load it can handle. Recommendation is based on the type of weakness, But they typically include additional resources if the weakeness is hardware resources. If the weak ness is within application architecture , high level recommendation is to rearchitecture the component . But typically work with respective teams to address the weakness

Reporting the results, as is, is one part but then recommending is another game. As performance engineers we have to run iterations with the suggested optimization or fix before sharing with the stakeholders. So let's say we find that creating an index for a DB query works well for the related API, bringing down its response time significantly, we do not just share the insight for one API but we run the whole test again and monitor the overall effect. Other than that suggesting tweaks around db connection pool, xmx, xms, gc approach, infra level wise, cloud wise (aws alb v/s nlb) need to be thoroughly checked and then put forward not just for sake of recommending or as per previous experience or "industry good practices".