A Simple FAQ for Containers, Kubernetes and Related Technologies

Author: Brent Laster

Author's note: I had written this up for a website that decided they didn't need it after all, so thought I would share it here as a quick intro to some of these topics for anyone who was curious about them or wanted to understand the high-level concepts. Hope some find it useful.

What is a container?

A container is a way of packaging software so that applications and their dependencies have a self-contained environment to run in, are insulated from the host OS and other applications and are easily ported to other environments. 

A container is not a VM. A hypervisor program (such as VirtualBox) is not required to run a container. Instead the container is run by leveraging several core pieces of Linux technology to “carve out” a portion of the operating system’s resources to provide the container a self-contained space to run in.

A container is a running instance of an “image”.

What is an image?

In the context of containers, an image is a software stack where software has been pre-installed, along with any needed configuration and environment setup, as a basis for using in running containers. 

Images are built up in layers – where each layer roughly corresponds to an installed piece of software, data being copied in, or configuration being done.  Through leveraging Linux’s Union Filesystem functionality, the multiple layers can be treated as a single filesystem view – looking through from the topmost layer to the other layers.

Layers in an image are read-only.

Images can share layers between them.

To create a container, a thin read-write layer is added on top of an image’s layers.

What is Docker?

Docker is a technology for creating and managing containers. It leverages core pieces of Linux that actually provide the container functionality – cgroups, union filesystem, and namespaces. Docker provides a formal approach for leveraging these easily. It includes:

-         a restful interface for service

-         a description format for containers

-         an API for orchestration

-         a format for creating images by specifying content and operations (a Dockerfile)

-         a command line interface for working with containers and images

What is a Dockerfile?

A Dockerfile is a text file that defines how to construct a Docker image.

It generally consists of instructions like FROM, USER, RUN, COPY, etc. that start with a base image and then layer other applications, configuration and environment settings on top.  Most instructions result in a new layer being added to the image. 

The Docker command “build” is used to create an image from a Dockerfile.  The Docker command “run” is used to create a container from an image.

What is Kubernetes?

Kubernetes is a framework for managing sets of containers, aka a container orchestrator. It is responsible for creating and maintaining running sets of containers to conform to a desired deployment state. This includes starting up running instances, ensuring a set number of running instances always exist (spinning up a replacement if one goes down), scaling the number of instances to meet workload (if desired) and generally removing the need for any manual intervention to maintain the desired state.

Kubernetes has its own set of native object types to manage containers. These include pods (which wrap around one or more containers), deployments (which are responsible for ensuring a desired number of pods are available), and services (which provide a virtual way to transparently access one or more pods).

Managing Kubernetes is done through a declarative process. The types and configuration of native Kubernetes objects are defined via specifications in YAML or JSON files and supplied to Kubernetes through the command line interface or an API call. Kubernetes then works diligently to ensure that the set and state of objects running in its cluster of machines matches the desired set and state of objects declared in the specifications. The function of Kubernetes working to make sure the state of the cluster matches the declared specifications is called a “reconciliation loop”.

What kind of infrastructure does Kubernetes run on?

Kubernetes runs on a system known as a “cluster”. A cluster is made up of one or more “nodes”. A node is simply a virtual or real machine typically in supported with an HA environment. A cluster will always have a “master node” and can have zero to many “worker nodes”.

What is on a Kubernetes master node?

In Kubernetes, master nodes run a set of components that are collectively known as the “control plane”. This set consists of the following components:

?      etcd cluster – this is simply a distributed key-value store that stores critical information about the cluster

?      apiserver – an instance (at least one) of the scalable Kubernetes API

?      kube-controller-manager – responsible for checking the status of objects in the cluster

?      scheduler – responsible for scheduling pods to run on particular nodes

What is on a Kubernetes worker node?

In Kubernetes, the worker nodes host the actual running objects managing the underlying containers. When you tell Kubernetes to deploy an application, it schedules it to run on one of these nodes. The particular pieces a worker node has includes:

?      kubelet - a kubelet is the “agent” that manages the node and allows it to communicate back to the master (via the Kubernetes API) as well as watching for and running pods

?      container runtime – Docker or a compatible application to create, manage, and run the containers

?      kube-proxy – this component serves as a networking proxy and load balancer

What are the major Kubernetes objects?

Pods: Pods are the smallest deployable units in Kubernetes. Pods wrap around one or more containers and any volumes they have.  

Deployments: Deployments instantiate pods and also manage pod replicas. The deployment ensures that if the desired number of pods is less than the desired number of replicas, additional pods are spun up to get to the desired number.

Services: Services provide a virtual IP (and optionally load balancing) for a set of pods. Within Kubernetes, each pod gets it’s own IP address.  However, since pods may be terminated at some point or sharing workloads, we do not want to be tied to any particular pod’s IP address. The service allows us to have one IP address and to route traffic to one or more pods on the backend transparently.

Ingresses: Ingresses allow a way for external network traffic to come into the cluster.

Namespaces: Namespaces are simply a way to group different objects together into to a common “area” within the Kubernetes system. 

What other types of objects are in Kubernetes?

In addition to the utilitarian types of objects mentioned above, Kubernetes supports several others for specialized usage. These include:

Jobs: Jobs are similar to deployments but they are intended to make sure the pod runs to completion and then terminates rather than ensuring there is always a pod running.  These are useful for batch type of processes.

StatefulSets: StatefulSets differ from a deployment in that the pods are not interchangeable. Each pod has its own persistent identifier that it manages across rescheduling. And because it maintains this consistent identified, the pods so managed can also maintain a stable network id and stable storage connections.

DaemonSets:  DaemonSets ensure that a pod runs on all nodes of a cluster (thus the “daemon” part of the name). If a node is added or deleted to/from a cluster, the DaemonSet will automatically add or delete the pod.   A typical use for this can be to handle operations that need to run across all nodes of a cluster, such as log collection.

What are Kubernetes ConfigMaps and Secrets?

ConfigMaps are a type of Kubernetes object for storing data (such as name/value pairs) that might normally be stored in multiple specifications by means such as environment variables. By placing the data in ConfigMaps as separate objects, we can avoid having to duplicate data across other objects and just point those objects to the ConfigMaps.

Secrets are like ConfigMaps but are meant to be used for data that needs to be not be in plain text, such as user information. The basic form of a Secret simply obfuscates the data by encoding it in a base-64 format.

How does Kubernetes manage resources?

Kubernetes manages both pod and cluster health. If situations are identified where a pod is using too much memory or resources or a node in the cluster is running critically low on resources, one or more pods may be killed and removed to attempt to free up resources. Ideally, specifications for pods should include both the minimum amount of cpu and memory they require to be scheduled, and also the maximum amount of cpu and memory that the pod expects to consume. These are referred to as resource requests and limits, respectively.

If a system that is part of a Kubernetes cluster becomes critically low on memory, Kubernetes may invoke the OOM Killer (OOM = Out Of Memory) to more rapidly kill off objects and free up memory.

How does networking work in Kubernetes?

In the Kubernetes cluster, each pod has its own IP address that traffic can be routed to. Kubernetes requires a networking plugin to be installed to handle routing requests internally between hosts to the intended pod.  These plugins typically have “fabric” names such as Calico, Flannel, Weave, etc.

An ingress controller, load balancer, or service can provide external access to network traffic, which is then routed to the appropriate pod.

The actual routing within the cluster between nodes and between pods leverages iptables (tables of IP filter packet rules and chains specified in the Linux kernel) to handle networking/port forwarding rules. 

What are Kubernetes Probes?

As a best practice, pods in Kubernetes should have health probes defined as a way for Kubernetes to determine is a pod is still healthy and ready to receive traffic. There are two types of probes – liveness and readiness. Liveness is used to determine if a pod is still alive and functioning. Readiness is used to determine if a pod is ready to receive traffic. There are 3 approaches that can be used to implement either of these probes – a rest point that responds with a successful HTTP return code, an exec of some process on the underlying container, or a TCP port ping.

What does Stateless versus Stateful mean for applications?

Native objects in Kubernetes are intended to be deletable and recreatable at any point in time.  To do this cleanly, those objects must not need to maintain their state from the deletion of one object to the creation of another. 

Many modern apps rely on being able to maintain state over a long period of time and from one instantiation to another. To work well in Kubernetes, these apps must use additional pieces to ensure the state is managed appropriately and data persisted as required across Kubernetes events. Common approaches for managing data persistence include persistent volumes, persistent volume claims, and storage classes in Kubernetes. Common approaches for adding additional functionality to manage state across events include creating custom object types in Kubernetes with custom resource definitions and adding additional processing around these types via Kubernetes operators.

What is Data Persistence in Kubernetes?

As previously described, containers running inside of pods in Kubernetes are assumed to be ephemeral – that is they may be killed off or go away at some point. To ensure that data used by those containers continues to persist across different instantiations, Kubernetes supports defining persistent volumes. This, as the name implies, is an instantiation of a type of storage and location with set characteristics like provisioner, reclaim policy, etc.  Kubernetes administrators can create StorageClass type of objects that specify how these characteristics are defined and the values for them. Typical examples of storage classes might be NFS, local, data volumes on the cloud, etc.  

When a pod in Kubernetes requires a persistent volume to store data on, they can request one via a Persistent Volume Claim (PVC). Kubernetes will then work to fulfill the PVC by looking for available persistent volumes that match the requested characteristics.

What are Kubernetes Operators?

Kubernetes Operators are structured applications designed to respond to, and handle, standard Kubernetes events for custom objects in a Kubernetes cluster. The most common use case for an operator is managing a stateful application running in the cluster that needs additional functionality to maintain state across Kubernetes lifecycle events (create, delete, etc.)  The operator is implemented via a custom controller that runs a reconcile loop against the customer object’s spec and the existing state of the custom object in the cluster – attempting to make them match. This is similar to how the built-in controller in the Kubernetes master works for native Kubernetes objects.

Red Hat now provides an Operator Framework with an Operator SDK that makes implementing operators significantly easier. Operators may be written in Go for the most flexibility and full lifecycle management. They can also be written in Helm and Ansible though those are typically targeted for initial actions and not the entire lifecycle. Red Hat also manages a public hub/repository for operators, known as operatorhub.io. Operators can be shared here after going through review and getting approval.

What is Helm?

Helm provides a way to create templates for Kubernetes specifications, instantiate them with sets of values and apply the resulting manifests to a Kubernetes cluster.  Helm also provides a framework for  installing and managing related sets of Kubernetes specifications. These specifications can be for any valid Kubernetes objects such as services, deployments, etc. and can be for any number of them. So this is especially useful for managing specifications for multiple projects or products.

What is a Helm release?

Each installed set of specifications is called a “release”. Helm maintains information about each release, including it’s history. This allows releases to be rolled back to previous versions as well as being easily upgraded via the Helm command line.

 What is a Helm template?

Helm provides additional utility for Kubernetes specifications by provided a templating capability. This means that the traditional specification formats can have placeholders in them in the form of Go templates instead of hard-coded values. The values used to fill in the templates can come either from separate values files or be passed on the command line. The benefits of templating include:

-         Not having to hard-code values in the specifications

-         Not having to repeat values throughout similar specifications

-         The ability to override default values

-         Template reuse for multiple instances given the above

-         Ability to compute and modify rendered values in templates through the use of template functions

What kinds of functions can be used in Helm templates?

Template functions are provided by Go templates and the Sprig library of template functions.  Examples of functions include simple transformation ones such as changing text to upper case or quoting it in the rendered template. More complex functions can also be created to do things such as read in data in one form and fill in the template in a different format or do calculations such as adding an offset to a starting port passed in on the command line.

What is a Helm chart?

Helm organizes the set of Kubernetes specifications for an application in a structure known as a “chart”. Typically a chart may include the various specification files as templates in a templates directory and also simple name/version files as well as helper functions.

Charts may also specify other required charts through a requirements file.

Helm also provides a repository implementation and format for storing charts as packages. Alternatively, charts can simply be managed and stored as text files in a source repository such as Git.

What are the concerns with the Helm piece Tiller?

Helm currently includes a server piece named “Tiller” that resides inside the Kubernetes cluster to execute changes requested by the command line. Tiller is considered a security risk as it is difficult to secure and has significant access to do operations inside the cluster.  As such, use of Tiller is generally discouraged or banned. 

An alternative approach to using Tiller leverages the Helm “template” functionality to render the template files filled in with values but not execute the changes on the cluster. These rendered template files are then passed to the Kubernetes command line and the “apply” command to make the changes in the cluster.  While more secure, this approach does remove the ability to leverage other Helm features such as upgrade, rollback, etc.

What is Kustomize?

Kustomize ( with a “K” ) is an applications for generating copies of Kubernetes YAML specifications with customizations.  These customizations can be in different forms.  One example is adding in common information to different files (for example a common label to both a service and a deployment ). Another use case is patching in newer information using an “overlay” where only the changed information is provided and is applied to the designated YAML specifications to update them. In all cases, the original YAML is left unchanged – Kustomize generates new copies with the changes. And, unlike Helm, no templating is required.

 Kustomize functionality has recently been added to Kubernetes (in 1.14) via the -k option to kubectl.

What is Istio?

Istio is an implementation of a Kubernetes service mesh. A service mesh can be thought of as a way to connect, monitor, and secure applications running in Kubernetes without having to modify the applications in the cluster to implement those.

Istio consists of a control plane and a data plane. For the data plane, Istio injects an additional container into each pod. This pod is a special proxy called Envoy. The Envoy proxy in each pod gets the network traffic intended for the pod and can process it in lieu of the other applications in the pod having to deal with it.

The control plane of Istio consists of a number of pieces that implement policies around network traffic, security, etc. By being connected to all of the pods via the Envoy containers, the control plane establishes a service mesh across the deployments. (Note that a service here refers to a microservice, and not a standard Kubernetes service.)

Istio applications include adding encryption, traffic-shaping, canary testing, fault injection, timing delays, etc. into the network traffic flow within the cluster in Kubernetes. Istio can also provide metrics gathering and observability into the pods.  Since the functionality is handled by the sidecar proxy container, no changes to other containers within the pod are usually required.

Even if not used in production, Istio can play a valuable role in testing container workloads.

How are permissions and security managed in Kubernetes?

Kubernetes provides an object type called “Secrets” for data that needs to be secure. This does not actually secure the data, but rather obfuscates it by doing a base-64 encoding on it and not displaying details about it in standard Kubernetes operations. 

For true security around objects, such as credentials, certificates, etc. a better approach would be to use a tool like “Vault” should be implemented and used.

For securing access within the cluster, Kubernetes relies on Role-Based Artifact Controls, better known as “RBAC”.  RBAC consists of defining “roles” that map actions (Kubernetes commands) to objects in the cluster (such as pods, deployments, etc.). Once the roles are defined, role bindings (RoleBinding objects) map a user or process to that role.  

In short, roles define what kind of operations can be done to what kind of objects (deleting pods for example) and rolebindings bind the roles to who can do them (user or process).

Within a cluster, a service account (ServiceAccount) may also be needed to be setup to allow applications to be bound to roles.

What is Prometheus?

Prometheus is an open-source application that is used for event monitoring and alerts.  It uses a time-series database ( a database optimized for time stamped or time series data ) to store realtime metrics. The pieces that make up the Prometheus “platform” include

?      multiple “exporters” that run on the systems being monitored and export metrics

?      Prometheus itself to centralize and store the metrics

?      an alert manager to trigger alerts based on the metrics

?      a powerful query language called PromQL

?      a tool like Grafana to produce dashboards

What is Grafana?

Grafana is a tool for metric analytics and visualizing data. It provides a way to visualize time series and allows users to build dashboards with graphs, panels, and other components. Grafana can also accept data from many different data sources including Graphite, Elasticsearch, InfluxDB, Prometheus, and others.