How can you identify performance bottlenecks in system administration?

There are some commands/tools that is always good to have handy. While some are shipped with the OS depending on the distribution, and others will need to be installed the ones below are part of my tool set. - ifconfig (network) - ethtool (network) - nethogs (network) - glances (network) - top (overall performance) - htop (overall performance) - vmstat (memory) - free (memory) - iostat (hdd performance) - nfsiostat (nfs performance) - df (disk space) - ps (processes) - netstat (services and ports) - ss (services and ports) Other dedicated monitoring solutions offer more capabilities, UI and historical data, to name some: - Nagios - Zabbix - Grafana - Thanos - Monit - Icinga - Cacti

Monitoring almost always ends up incremental as you learn more about your systems, so a flexible and composable monitoring system needs to be at your disposal. More importantly you need to understand how your application interacts with the system(s) it uses. Do you understand how your JVM works, thread pools work, GC, NUMA, network throughput and calls? These deeper level things are what is always looked at when you are looking for performance bottlenecks in system administration. You don't have telemetry telling you thread pools are near exhaustion - add it. Monitoring and the passion you put in is where you will get the most out. Instead of - I don't have that data - It's, I have that data but I just need to add this alert.

Monitoring is always challenge but before monitoring we need to understand what workload running on the system. First 60 second use tools like top, uptime, dmesg | tail, vmstat 1, mpstat P ALL 1, pidstat 1, iostat -xz 1, free -m, sar -n DEV 1 sar -n TCP,ETCP 1, try to understand where is the error. Then you may look into error rate if there is any IOT applications are running could be connectivity timeout etc.

Performance testing: This is the process of simulating different scenarios and loads on the system and measuring its response time, throughput, resource utilization, & (KPIs). Performance testing can help identify the system’s capacity, limitations, and potential bottlenecks under various conditions. Performance testing can be done using automated tools, such as LoadRunner, WebLOAD, or Apache JMeter1. Profiling: This is the process of analyzing the behavior and characteristics of a specific component or process within the system, such as an application, a database, a network device, or a CPU. Profiling can be done using tools that monitor and collect data on the component or process, such as code analyzers, debuggers, profilers, or tracers.

Utilizing an external monitoring system is key for identifying bottlenecks within a system and to gather the systems normal operating metrics.

It is imperative to distinguish the difference between software vs hardware limitations to limit downtime and cost effective solutions. Oftentimes the issue isn't hardware related, it may be software; i.e. timeout of interfaces / application crash, errors in reporting.

Identifying the root cause could be broken down as: 1. Diagnostic Perspective, to get a granular view of the problem using tools that inspect specific areas of a system. For example, system calls and network routes. This pinpoint exactly where the issue lies. 2. Historical Perspective from log & configuration analysis with tools like Splunk or ELK Stack to get the historical context of the issue. 3. Comparative Perspective with baseline vs current state from tools that have historical data comparison like Zabbix & Prometheus, comparing the problem state against a known good state. 4. Collaborative Perspective, from internal documentation and different team members who may have encountered similar issues.

Every performance bottleneck has a finite number of possible causes. Test the possible causes against the facts about the issue to eliminate false causes until you identify the most probable cause. Consider the following scenario: multiple users complaining about timeouts. Since it isn't just one user, don't waste time looking at their individual machines.

An ecommerce website experienced unexplained slowdown. Short-term monitoring via htop indicated high CPU & memory usage. Long-term monitoring historical data via Splunk ruled out internal server issues and showed instead, increased traffic to specific pages, unrelated to online purchases. Collaborative analysis with IT forums & team insights suggested bad traffic as the culprit. To mitigate this, we opted for a Web Application Firewall (WAF). Before enabling WAF filter rules, we verified the presence of bot traffic in WAF stats. Upon rule activation, the site's performance normalized. This approach aligns with industry best practices for identifying & mitigating non-legitimate traffic, ensuring both availability & security.

One proactive approach that allows you to address scalability issues before they impact users is to test how your system performs under different levels of load. By simulating heavy usage or traffic, you can identify how well your system handles stress and whether it's scalable. Load testing tools like Apache JMeter, Siege and many other tools can help you mimic real-world conditions and discover potential bottlenecks that might only show up when the system is under heavy demand.

Maintaining a connection between business users and ICT is a pivotal strategy in the battle against performance bottlenecks. While the we often talk about essential technical steps to prevent bottlenecks, the human element is often underestimated. Business users possess invaluable insights into the system’s real-world usage and can help bridge the gap between technical optimisation and user expectations, of which both are ever evolving. By fostering collaboration, business users can communicate their needs and pain points, leading to more effective alert setting and metric selection. They can provide context that metrics alone cannot convey, ensuring that thresholds and alerts align with the actual impact on business operations.

Upgrades & updates can introduce issues, risking destabilizing environments. To mitigate, incorporating a development platform for testing is crucial. This allows vetting updates & upgrades for compatibility & stability before deploying to production. For instance, the recent end-of-life announcement for the Linux distro CentOS poses potential future issues. Organizations relying on CentOS may face compatibility challenges. Using a development platform, new Linux distros can be tested for compatibility with existing systems. Once confirmed stable, updates can then be safely rolled out to the production environment. This approach minimizes risks & ensures a smooth transition, thereby maintaining system reliability & performance.

Regular load testing, emergency management drills, etc. allow for systems teams to surface issues before they occur in production. It is also crucial to routinely audit monitors, and evaluate if they measure what they are believed to, and if they will occur in a timely and useful way (for example, if by time an issue occurs, have other prerequisite and detectable failures likely to have also occurred that you are not yet monitoring for?)

"Bottleneck" is a wonderfully descriptive term. It describes an artificial constraint on some form of communication, interaction, or transfer of information. And it leads one to believe that some magical combination of luck, money, and ingenuity can smash that bottleneck and let all good things flow. The trouble with performance bottlenecks is that they can be tough to identify. Is it the CPU? The network? A clumsy bit of code? Often, the most obvious culprit is actually downstream of something larger and more mystifying. And when performance riddles remain unsolved, IT management may find itself faced with a Hobson's choice between admitting ignorance and making up excuses.

The IT department supports other departments. Understanding their workflows & requirements is crucial for an effective system management. Once the IT team gained insights into the marketing department's needs for A/B testing & PPC campaigns, which require fast page loading times, they tailored their system optimizations accordingly and scheduled maintenance & updates in a manner that minimized negative impact on crucial marketing activities. Essentially, understanding the operational needs of other departments helped us to avoid creating performance bottlenecks for A/B tests or PPC campaigns. Cross-departmental understanding is key to ensuring IT operations align with organizational objectives & department-specific requirements.