Why Enterprises that are running Kubernetes clusters at scale should adopt GitOps

In the era of digital disruption and with modern architectures we are constantly moving to Kubernetes at a rapid pace. In?technology ecosystem we are running multiple clusters across multiple clouds supporting the businesses day and night.

Without using GitOps we are introducing lot of configuration drift with time and these clusters become unmanageable soon. As the clusters mature more and more undocumented configuration changes are introduced and there is no straight forward way to tell the state of clusters in real time.

Lets discuss some of the challenges that we face:

Management Challenges and Cluster Sprawl

With introduction of cloud computing we have entered into an era of distributed computing and managing world full of VMs.??

Now, enterprises often deal with a chaotic environment of cluster and workload sprawl as large and distributed teams provision multiple Kubernetes clusters across their local workstations, data centers, public clouds, edge sites, and at times, on premises at end-customer sites.

Enterprise IT though wish for global policies been implemented across all clusters and not divert from desired configuration but achieving it real time is still not easy.

Configuration Sprawl

Kubernetes is well known for its declarative API: Every component in a cluster, from configuration settings to applications, no matter how small, is configured via a configuration snippet called resource, typically expressed as a YAML file. This means that fully describing and configuring a cluster may involve creating and maintaining dozens or hundreds of different resources, resulting in a management nightmare. The problem can be aggravated by the plethora of tools that introduce additional abstractions and features (e.g., templating) over those resources, such as Jsonnet, Helm, and Kustomize.

Pet Clusters

As enterprises start deploying multiple clusters and create many of them as they scale to multiple environments. These clusters though can be split into node groups but management of them gets tougher as they grow in scale.

Desired State Management in Kubernetes?

We know the biggest advantage of kubernetes being its declarative in nature. All deployments are specified in yaml format and when pod is removed it is added back based on configuration.?

By default, Kubernetes doesn’t have a mechanism to monitor changes to the cluster itself and its attributes and automatically reconcile the state. This is where GitOps comes into the picture.

E.g. if a namespace is deleted then kubernetes will not recreate it by itself.

?What GitOps will give you?

GitOps is an operational framework where one employs standard development best practices around version control and management of source code using central repositories for source code management and CI/CD, and extends them to management of your infrastructure.

In a multicloud and multicluster environment, GitOps can be a very valuable and effective process to automate configuration management, deployment, updates and policy management of your Kubernetes clusters and the surrounding infrastructure.

Enforcement with Flexibility

Templates can only be successful if there is an engine that always keeps a check on state of clusters and bring them back to what template mentions stored as desired state in Git.

Consider case of RBAC policies - The templates gets stored in your Git repository declaratively, and in an immutable manner, without forcing the user to access a YAML file. Once the enforcement engine is instructed to associate the cluster with the repository (and, typically, a path within the repository), the contents of the repository path define the “source of truth” for the RBAC settings are applied to the cluster, then periodically synchronized to ensure that they continue to be enforced over time.

Don’t get the DRIFT

The ability to easily and seamlessly audit changes made to clusters that might differ from the configuration on the Git repository is critical — known as “drift.” A properly implemented template system, described above, should provide “drift analytics,” where with a single command the system can determine and report whether there is a difference with the deployed cluster and the desired state captured in the template. Alerts should be sent when changes are made to relevant Ops owners depending on severity and importance of changes.

With?Flux?or?Argo CD, drift might happen when a particular object in Kubernetes has changed — whether inadvertently by an admin, for example, or during the course of a network attack.?

All changes should thus first be made in Git before they are rolled out into production, so that both the configuration on Git and in the clusters remain reconciled. The state of Git representing the immutable “single source of truth” is also a major component of cloud native security and compliance by meeting such mandates defined by the Sarbanes-Oxley act, for example.

The Threshold Starting Point

It is possible to manually handcraft and manage the configuration for a single or a small number of clusters. However, the pain becomes palpable when attempting to manually manage 10 or more Kubernetes clusters consistently, especially if the size of the individual cluster is large.

Ultimately, GitOps serves as the framework to take advantage of what Kubernetes was originally created to do: to offer significant computing advantages, resource-savings, and compatibility for apps deployed across multiple and highly distributed containerized environments.

Note: All above content is my personal opinion and has no relation to my firm and professional network.