What are some of the key indicators of capacity issues in IT systems and how do you monitor them?

Key indicators of capacity issues in IT systems include high CPU and memory utilization, increased response times, and frequent system outages. I monitor these indicators using performance monitoring tools like Nagios and Grafana, which provide real-time insights into resource usage. Alerts are set for thresholds to proactively identify potential issues. Additionally, I track user load and application performance metrics, enabling a comprehensive view of capacity trends and facilitating timely interventions to prevent system overloads.

While monitoring the resources on essential parameters e.g. CPU, memory, buffer pool, swap, disk, network latency, get rate, put rate among others, is absolutely essential to understand utilization, but ensuring that these measures are captured when there are concurrent running processes during capacity test is paramount to provide a realistic view, else siloed resource utilization monitoring could result in misleading interpretations and incorrect decisions. While this sounds obvious but often remains uncommon with risk based testing approaches where team's complacency in dealing with testing constraints can result in oversimplified risk assessment, costly mistakes, service disruption, and unfavorable outcomes.

It is important to understand how the business units measure their level of success and align IT high level reporting with those. For example if your business has a high customer assistance component, average call handle time is important and by measuring the IT component of that helps the business understand IT's contribution. Then if you can reduce the IT component measure, the business understands the impact of your work. In a busy call center, a 5 second reduction in the average call handle time can translate into substantial savings.

One of the practical issue I see is in those business units / organizations where teams conceptualize an innovative product/service and rush the MVP (Minimum Viable Product) for product-market fit testing while failing to check all the boxes for operational readiness, and tend to compromise by leaning on existing production infrastructure capacity of any other live services. In absence of clarity on data growth projections of new product pending market testing, the approach is a non-starter. After product market fit testing and product taking off, teams unconsciously forget actions on scalability and are reminded (or disappear) when there are incidents on other service whose capacity was used.

There’s nothing worse than having to go to finance and ask them to approve a large purchase because you just ran out of disk space, and now you can’t meet your SLAs until they approve it. If you can forecast consumption, you can accrue funds. That way, there aren’t going to be any surprises when you need $1m for additional space. Regardless of the size of your business, you still need to manage cash flow. By forecasting, and accruing funds, you’ll be doing the business a favor.

When it comes time to add capacity, it is always a difficult conversation with the CFO. One way to turn that request in your favor is to measure the IT cost of business transactions. When adding capacity, you almost always benefit from economies of scale. By reporting by transaction, you can show the cost reduction and change the conversation from one of expense to one of investment in the future to reduce costs.

Peak time, and peak load spikes - systems do not see uniform traffic distribution throughout a day. For online systems, the business hrs typically ~07:00-15:00 hrs are critical, while for those systems that require heavyweight lifting in batch mode (e.g. end of business processes, general ledger update, core banking and payments), an overnight window is suited. So, carrying out system capacity testing for peak time and peak load is a key testing exit criterion (NFR outcome). The usual system design norm is to keep consideration for 2.5x-3x of expected peak load to assess system resilience in worst-case scenarios e.g. disruptive data growth beyond forecast, poisoned data in a self-healing system, disaster recovery failover (active/passive).