Who is responsible for Monitoring?

There is an age long question on "who is responsible for monitoring" that is recurrent in a lot of tech, business, product and incident meetings that it has now become a cliché to the extent that we cannot answer the questions correctly without an argument due to the immense transformation of how todays Microservices have evolved from the Monolithic architecture we were running and the past and still running in some cases.

"Who is responsible for monitoring has now become a cliché"        

In a lot of cases I have observed that these questions hardly come up during product designs, sprints or prior to deployment where we can easily set traps, events, etc while identifying teams who will be responsible for monitoring as the product goes live. I have also observed that in some cases, we hardly transfer the discoveries of the application testers and quality assurance analysts through their work on APIs prior deployment to production to the monitoring teams to help in monitoring or observing performance there by impacting the teams learning curve, causing extended downtime and in some cases introducing software engineers into incident calls.

However, if you consider the Monolithic architecture, all you needed to do was monitor the virtual or physical machine (CPU, Memory, drives) hosting the application, the application webserver and the network then you are on-spot with monitoring without understanding the intricacies of the Monolithic application. During some incidents, all that is required is a of restart the server or webserver when you suspect a load which typically always triggers the resources, then the monitoring starts getting complicated with load balancing the; since it results into several nodes for same application then more complications when the same Monolithic application is converted into Microservices where same application is broken down into several components with potentials of running several instances on same node and across multiple nodes with multiple dependencies using #Kubernetes, #docker, #OpenShift.

When asked at some point during meeting "who is responsible for monitoring" I said everyone and further explained that although there is a central monitoring team, it was everyones responsibility to observe. I further explained with the. below;

1. The product manager/owner should have a view of how his/her product is performing.
2. The relationship managers should know how their customers are interacting with the product.
3. The Architects/Software engineers should have routine checkins for their software to take feedback.
4. The monitoring team should detect and resolve earlier.
5. The list goes on.

To imprope effectiveness in response to incidents or improve performance on Microservices, there has to be a shift from just monitoring to observability.        

It is worthy of note that observability is a journey, however the below are key to improving observability;

Steps to Improve Observability

1. Service Level Agreement (SLAs), Service Level Objectives (SLOs), Service Level Indicators (SLIs) must be setup.
2. Central Log Aggregation
3. Baselines must be set based on SLAs, SLOs, SLIs.
4. Targets must be set to continously improve SLAs and tune down errors.
5. The monitoring teams must set intelligent alerts to avoid alert burn out. Are the alerts timely? are the alerts relevant?

Alert burnout is the experience of a lot of teams as they are unable to act on critical issues due alert burn out.
Recall "the fokker 50 crash crew ignored multiple alerts" possible due to alert burnout.        

6. Intelligent and dynamic health endpoints for services.

7. Continuous product testing even while in production.

“If you don’t like testing your product, most likely your customers won’t like to test it either.” 
                             Anonymous                           

8. Continuous dashboard creation to make detection easy even when there is no incident.

9. Transfer of knowledge of the Microservice prior to deployment to production from testers/QAs to the monitoring team and enshrining through dashboard.

Imagine a scenario         

Summary

In my experience of reviewing incidents, some things like the below stick out easily and you can predict them to be knowledge gap, late discovery and other similar cliché and you may wonder why same recurrence in a lot of incidents.

However, it is important to note that while and incident is ongoing with calls blaring and emails piling with customer complaints, what matters is have a team with a mind shift from the regular monitoring to observing the the entire value chain, follow the complaint, the logs, events timestamps and hypothesis while we subsequently create dynamic dashboards to potentially solve the problems before the customer notices a downtime.