Kubernetes - before you start

I've been looking for an accessible summary of what Kubernetes is, but so far I've only been able to find explanations of what Kubernetes does, or many even jump directly into how to use it to profit. In this article, I'll take a step back and focus on the basics, explain the core design, and finish with some higher level implications and further reading.

Kubernetes was born from Google's experience of running a humongous fleet of containerized applications, and having to keep them running on a humungous estate of generic hardware. It implements the patterns that they have discovered over many years, using a clean and extensible architecture, and is a good match to the problem of running sprawling microservices on a humongous estate of generic cloud resources. This is a common problem today, but not necessarily a good one to have.

The Cluster is a database

An installation of Kubernetes is called "a cluster". At the core of the cluster sits a document database (etcd). For those of you thinking about MySQL and Oracle, it is nothing like that - the Kubernetes database is good for very few things:

• storing json documents in a flat namespace (no tables, no primary keys)
• querying documents with simple WHERE clause (no joins)
• issuing events when a new document is added, updated, or deleted

Each document stored in the Kubernetes database needs to have a "kind" specified, and depending on its kind we expect certain structure of the actual data in the document (some of us would call it a "schema").

The API server controls the updates to the database

Doing anything in Kubernetes means creating, updating and deleting documents in the Kubernetes database. This is done through a web service called the API Server - normally we use a tool such as kubectl or Helm to access the API server, but in a pinch it is not too difficult to do with curl either. In the end all we are doing is creating, updating and deleting text documents.

So far so good - we have a database and we can keep data in it, but how does that help us orchestrate software?

Controllers do things

The meat of Kubernetes lies in its controllers - small processes that monitor the database, and when a document of a specific "kind" is updated, they do something fancy.

A good example of a controller would be the Pods controller - it guarantees for every document in the database that has specified 'kind: Pod', it will run a container in a way specified in that document. It will also watch over the containers it started, and update the database with the actual state it observes. For example:

• If a new Pod document is created, it will start a new container.
• If a Pod document is deleted, it will destroy the container.
• If a Pod document is updated, it will update, or re-create the container.
• If a container dies, but the corresponding document still exists, it will restart the container. If it crash-loops, it will update the database and issue an error event.

There are a few other controllers that control networks, storage and other physical resources. These form the core of Kubernetes.

Controllers controlling controllers

Being able to provision applications and resources by updating database state makes it easier to implement higher-level functionalities. For example, if you need load-balanced set of services, you can create a document with kind: ReplicaSet, describing how you want the load-balancing and scaling to work.

The ReplicaSet controller will not have to know how to run containers or monitor them - it will instead reach out to the database and create documents of 'kind: Pod' and monitor and respond to changes in the pod state.

In essence, what used to be a heavy system programming task has been reduced to arranging JSON documents in a database.

We can have one controller respond to multiple "kinds", or one "kind" handled by multiple controllers - this is uncommon, as it makes it difficult to reason about the system.

Operators

Often we would find ourselves automating the same sequence of state manipulations repeatedly (i.e. rolling update of container versions would require the update of many ReplicaSets or Pods).

Once we have discovered a good recipe that works well, we can make it part of Kubernetes, by adding a new "kind" describing the recipe parameters, and then creating a process that monitors the Kubernetes database, and performs the lower level database updates for us. For the rolling update scenario, this has actually happened - the corresponding kind is called Deployment.

The advantage of having the compound operation described as a new kind, and managed by controller process is that it becomes easily observable and reusable (i.e. more abstractions can be built on it). It also improves the performance and reduces the chance of scripts failing half-way through because we lost network connectivity.

The combination of new "kind" definition and the process reacting to that kind is called "operator" pattern. Most of Kubernetes is written in operators, because they are easy to reason about.

So, what did we learn?

By starting with a description of our desired state in a database (some may also call it "whiteboard"), and having detached controllers react on it and further update the database or the external world, we achieved a number of nice characteristics:

• The data model of the state is purely declarative and can be inspected, updated and analyzed independent of ordering.
• The system is transparent, as everything is in the database. This is helpful when dealing with utility compute farms, where you cannot log in to individual machines.
• Each controller is kept relatively simple.
• The data model is a common contract, and depending on the cluster configuration, it can abstract different resource providers. For example, depending on config, dynamically provisioned storage can come from NetApp or Amazon EBS, without any modification of application's deployment manifests.

Of course it is not all roses, for all the good things, we also get:

• Significant complexity for cluster administrators. Kubernetes has occasionally been described as "a framework for building a PaaS", and doing a good job is as complex as it sounds. Not doing a good job means it is more expensive and more vulnerable.
• Action at a distance. While the pattern of higher-order operators is elegant and scalable, for mere humans it may be overwhelming to trace the chain of events that lead to observable change.
• Reasoning about where to make a change to fix undesirable state is also complex. A very common issue is to make a change in a suboptimal place (implications: expensive, fragile, vulnerable).
• Abstractions leak - in the end we are dealing with real OS underneath and real hardware. As much as Kubernetes does a good job at making them look consistent and fungible, when there is a platform issue, the debugging is often exceedingly difficult and unpleasant
• Kubernetes is a very attractive attack vector for hackers - it is easier to find an exploit path when so many things are happening behind the scenes.
• Do the fundamental premises hold for you? Do you really need to split your software into microservices (see modular monolith)? Can you justify the cost you pay for renting your cloud? Are the benefits of a Google-scale framework outweighing the cost of hiring and training people who can operate it (and the cost of outages and lost productivity if you underhire)?

With all that said

Kubernetes is impressive! While it is still very young compared to established cluster schedulers, with Google's resources and wide indistry backing, it has achieved high degree of usability and functionality. Being widely adopted also means that it is getting easier to hire people with experience, and that its design choices are becoming canonical (similar to Linux kernel).

The biggest risk for Kubernets is the complexity and immaturity of tooling. Many vendors are trying to provide simpler experience by preconfiguring, and complementing with custom services, but eventually someone will find a way to provide 90% of the functionality that everyone needs, with 10% of the complexity and this is going to be a game changer. Not everything is destined to scale dramatically and unexpectedly.

Now you know - this is what Kubernetes is. If you are still interested in doing things with it, go ahead and read some of the other tutorials. Kubernetes The Hard Way is my favorite.