Decoupled Application Monitoring

What are we doing now

There is are a lot of new monitoring tools out there. The tools are becoming more sophisticated and there are more of them. How do you keep up with the new Application Monitoring tools when every time you want to try a new monitoring tool you have to update your applications? This slows down your speed of change. So why not decouple your applications from your monitoring tools.

It used to be that a lot of tooling is API based. This means you from your application send data to a API of your monitoring application. To do this your application has to talk the protocol that the Monitoring tool understands. And if you want to switch monitoring tools you will have to upgrade your application.

There are also more and more monitoring tools that use a pull system like Prometheus these pull data from your application. Your application needs to expose an endpoint to the Monitoring tool. This endpoint has to provide the data in a way that the monitoring tool understands. Again this means that if you want to switch your monitoring tool you have to upgrade your application.

All these scenarios have one thing in common the Application and Monitoring tool are tightly coupled. If we want to be able to move between Monitoring tools or do quick POC’s (Proof of concept) with new and upcoming tools we will need to change our approach to monitoring.

Decoupling your Monitoring

The best way I can see is to decouple your Monitoring tools from your application something like this

Application -> Bus Software -> Data Base -> Monitoring Tool

If you want an example of one of these chains:

Application -> KAFKA -> Elastic Search -> Kibana

I am not saying that this is THE chain I am just saying this is a way to setup a Monitoring system.

Some more advantages

If you do this on of the advantages is that what was a series of hard-wired (1:1) data sets, evolved into reusable (1:N) data sets, and is now converging on a common solution (M:N) for the overall IT data collection sector. And this 1:N data set can be used in other Monitoring tools. This would make running POC’s a lot easier and faster. The reason being you do not have to rewrite your application to connect to any new Monitoring Tool. You can even run multiple Monitoring tools from the same streaming data. All do I would not advise you to store the data multiple times but you could use it in more than one Monitoring Tool in real time. If you get most of your monitoring data in one BUS system then it also becomes a lot easier to start correlate your data. Lets say your application has a failure when talking to a Database this database uses a NAS and in that NAS there is a disk Failure. Before you have correlated your failure to the disk you could have been looking into multiple systems or talked t different teams. If all this data was on the BUS and storage in one place you could have found it a lot sooner. This links in with my earlier article on “THE INFLUENCE OF BIG DATA on OPERATIONS.” https://www.dhirubhai.net/pulse/influence-big-data-operations-marcel-koert/

The other major advantages are that your monitoring tools can be switch out without changing your application. Switching Elastic Search to HADOOP is easier now than when you would have to rewrite part of your application.

There are drawbacks

There are some drawbacks. The big one is that you have a extra application in between your monitoring tools and the Application the BUS. This will add latency and risk that the BUS application fails then you would not be able to monitor your application.

But when the setup of your BUS system is done correctly you can mitigate most of these issues.

Conclusion

In the current climate of DEV/OPS Agile way of working , we also need to be agile in our monitoring setup. We can not do this if we have tightly coupled monitoring arrangements. We need to be able to switch quickly and we need to be able to test new monitoring idea's quickly and good enough to see if they add value to our environment or we should not proceed with them. So a Decoupled Monitoring structure is the way forward.