The beauty of Cloud Native

A few days ago, our cloud provider decided to terminate 2 of our k8s worker nodes and while that should be (and was) uneventful for Kubernetes, it did make some waves for our Grafana LGTM stack (see above). Fortunately, the stack was built for the cloud and was able to recover on its own without intervention from an operator.

Orchestration is a joy to witness when an entire solution, from the server infrastructure to the application\service, is built with failure and scale in-mind. Not only did the Kubernetes and Grafana LGTM platforms recover, application-level rate-limiting kicked in and prevented the application from falling over while it was processing the backlog.

While not every app or service is instrumented to this level or built with this strategy, it's still important to understand them and how they respond to changes in the environment. Having a toolset (like the Grafana Labs LGTM stack) that provides reliable data to make operational and strategic decisions is invaluable to all stakeholders, no matter their function within the organization.