How can you make your system monitoring strategy more flexible?

In my experience one of the best way to start defining monitoring goals is to consider the Service Level Agreement (SLA) - Service Level Objective (SLO) - Service Level Indicator (SLI) relationship. In other words. Start from what we are promising customers: like performance. Then establish internal objectives to ensure we meet that SLA: like response time < 1 second, throughout >= 1000 queries/second. Then, your SLIs are the metrics you need to collect: throughout and response time. The SLIs are your most important metrics, which usually are your main alerting source. To ease troubleshooting, we can go deeper and collect metrics directly Or indirectly influencing the SLIs, such as CPU usage on a server if work is CPU-bound.

How about by starting to considering monitoring as business functional requirements instead of it being as a non functional requirements, giving equal importance and assigning business value to monitoring in todays world will have better outcomes than additional features without monitoring insights

One way of monitoring which can lead to elevated costs and inefficiency is “collecting all metrics”, and deciding later on which are important. If you have a distribution application using multiple container instances, you might collect by default application metrics such as response time, container level metrics such as memory/CPU, and OS metrics such as processes count or disk reads/writes per second. There is a very high chance you will never look at metrics on the number of open files on a container or number of disk writes/reads per second. But they will most certainly load your monitoring system, adding to its costs and required processing power. Maybe even amount to higher costs than the metrics you actually care about.

We are in a time in which it is easy to: - deploy applications inside containers - scale on-demand horizontally processing layers by adding/removing containers - use distributed databases that support adding/removing storage nodes at run-time (e.g. Cassandra, FoundationDB, etc) In such a volatile world in which containers or virtual machines/servers appear and disappear on-demand, the configuration of the monitoring system can't be static. We need to automate the discovery of system components, such that when a new machine, a new container, a new API gets deployed, the monitoring system picks them up and collects information without requiring any human input.

This is so very important in cybersecurity where the term “alert fatigue” is so prevalent that it’s now being used by sales people. A world class IT organization first use machine logic to filter out alerts that don’t matter. Once done, workflows and automation can be created to better handle the important alerts from monitoring.

Monitoring can be used to detect patterns in system behavior. By collecting throughput or response time information for a period of weeks/months, one can extract patterns. Understand which days or hour intervals are more loaded. And adjust the system scaling policies to match those patterns. Another possible use case is extracting correlations from monitoring data. While causality can be quite difficult to prove, extracting correlations is easier and can be very usefull. For example, noticing that database load increases when a particular type of API is called, or that response time increases seems correlated to CPU/DISK usage increases, can point you in which direction to look for optimizing your application.