Deploy Kubernetes Storage Service With Rook-Ceph

Introduction

There has been discussion around having suitable storage solution for Kubernetes and this discussion has always centred on the need that?storage should be run as its own service, to be consumed by deployments, rather than built as part of them using native components. Although we have a few different storage services available, but may really not be suitable for what we want to achieve:

• Cloud Storage?- perfect for object storage, but not particularly performant and will usually require application refactoring.
• Cloud Filestore?- expensive but reliable and performant managed NFS solution.
• Roll-your-own in Compute?- the last resort of a desperate system admin. One can get lost for days trying to string a GlusterFS cluster together and then comes the headache when there's need to?change?it.
• Vendor lock-in - some organisations that prefer to run their own self-manage Kubernetes cluster on-premises may not want to be locked into any cloud vendors and still wants a high-performing, highly scalable distributed storage system with no single point of failure.

What Storage Solution is then available to fill in this gap?

Rook

This is an operator and orchestrator for Kubernetes that automates the provisioning, configuration, scaling, migration and disaster recovery of storage. Rook supports several backend providers (such ceph, cassandra, etc.) and uses a consistent common framework across all of them. The Ceph provider for Rook is stable and production ready.

Rook consists of multiple components:

• Rook Operator?is the core of Rook.?The Rook operator is a simple container that automatically bootstraps the storage clusters and monitors the storage daemons to ensure the storage clusters are healthy.
• Rook Agents?run on each storage node and configure a FlexVolume plugin that integrates with Kubernetes’ volume controller framework. Agents handle all storage operations such as attaching network storage devices, mounting volumes on the host, and formatting the filesystem.
• Rook Discovers?detect storage devices attached to the storage node.

Rook also deploys MON, OSD and MGR daemons for the Ceph clusters as Kubernetes pods.

The Rook Operator enables you to create and manage your storage clusters through CRDs. Each type of resource has its own CRD defined.

• A Rook?Cluster?provides the settings of the storage cluster to serve block, object stores, and shared file systems. Each cluster has multiple pools.
• A?Pool?manages the backing store for a block store. Pools are also used internally by object and file stores.
• An?Object Store?exposes storage with an S3-compatible interface.
• A?File System?provides shared storage for multiple Kubernetes pods.

Ceph

Ceph is an open-source project that provides massively scalable, software-defined storage systems on commodity hardware. It can provide object, block or file system storage, and automatically distributes and replicates data across multiple storage nodes to guarantee no single point of failure.

Ceph consists of multiple components:

• Ceph Monitors (MON)?are responsible for forming cluster quorums. All the cluster nodes report to monitor nodes and share information about every change in their state.
• Ceph Object Store Devices (OSD)?are responsible for storing objects on local file systems and providing access to them over the network. Usually, one OSD daemon is tied to one physical disk in your cluster.?Ceph clients interact with OSDs directly.
• Ceph Manager (MGR)?provides additional monitoring and interfaces to external monitoring and management systems.
• Reliable Autonomic Distributed Object Stores (RADOS)?are at the core of Ceph storage clusters. This layer makes sure that stored data always remains consistent and performs data replication, failure detection, and recovery among others.

To read/write data from/to a Ceph cluster, a client will first contact Ceph MONs to obtain the most recent copy of their?cluster map. The cluster map contains the cluster topology as well as the data storage locations. Ceph clients use the cluster map to figure out which OSD to interact with and initiate a connection with the associated OSD.

Deployments

The Rook and Ceph deployment was done using the Rook Helm Chart located in this GitHub repo. The repo has ArgoCD configured to deploy Rook-Ceph to the Kubernetes cluster. The Rook-Ceph deployment can also be done using the operator and cluster model as illustrated in this quick guide.

You can visit the official Rook Helm Chart page if you will like to explore more on the Rook deployment

It should be noted that there are some requirements that must be met for Rook and Ceph to work properly.

Prerequisites

This documentation assumes you have a Kubernetes cluster that is ready for Rook and Ceph deployment. To make sure you have a Kubernetes cluster that is ready for Rook and Ceph, you can?follow these instructions.

In order to configure the Ceph storage cluster, at least one of these local storage options are required:

• Raw devices (no partitions or formatted filesystems)
• This requires lvm2 to be installed on the host. To avoid this dependency, you can create a single full-disk partition on the disk (see below)
• Raw partitions (no formatted filesystem)
• Persistent Volumes available from a storage class in block mode

You can confirm whether your partitions or devices are formatted filesystems with the following command.

lsblk -f
NAME                  FSTYPE      LABEL UUID                                   MOUNTPOINT
vda
└─vda1                LVM2_member       >eSO50t-GkUV-YKTH-WsGq-hNJY-eKNf-3i07IB
 ├─ubuntu--vg-root   ext4              c2366f76-6e21-4f10-a8f3-6776212e2fe4   /
 └─ubuntu--vg-swap_1 swap              9492a3dc-ad75-47cd-9596-678e8cf17ff9   [SWAP]
vdb
        

If the FSTYPE field is not empty, there is a filesystem on top of the corresponding device. In this case, you can use vdb for Ceph and can't use vda and its partitions.

Deploy The Rook Operator

The first step is to deploy the Rook operator. All that is needed is to have your?ArgoCD app point to the directory where the Rook operator helm and ensure that all needed lines referencing the Rook operator is uncommented out in the kustomization.yaml file. This way ArgoCD picks the relevant manifest files and sync it to the Kubernetes cluster for onward deployment.

You may want to change the Ceph FlexVolume Configuration in the values.yaml file before deploying the Rook operator. This is because some Kubernetes deployments require you to configure kubelet with a FlexVolume plugin directory that is accessible and read/write (rw). These steps need to be carried out on?all nodes?in your cluster. Rook needs to be told where this directory is in order for the volume plugin to work. However, there are platform specific configurations for this and you need to understand how to configure the FlexVolume for your specific platform. See this link for list of platforms and configuration to be done

In my case I am not using any of platform listed in the link above and the path?/usr/libexec/kubernetes/kubelet-plugins/volume/exec/?is read/write, so I don't need to configure anything. This is because?/usr/libexec/kubernetes/kubelet-plugins/volume/exec/?is the kubelet default FlexVolume path and Rook assumes the default FlexVolume path if not set differently.

It will take a while for all the pods involved in Rook’s operator to be ready. You can keep an eye on things with:

watch kubectl -n rook-ceph get pods        

When you have a rook-ceph-operator pod and 3 rook-discover pods, you’re ready to move on and you can CTRL-C out of watching the output.?

Deploy Ceph Cluster

Now that the Rook operator is running we can create the Ceph cluster. This is basically our cluster of Ceph agents that provide the storage service. This cluster manifest defines things like the version of Ceph we’re using, and where Ceph can store its data. For the cluster to survive reboots, we will set the?dataDirHostPath?property that is valid for our hosts. We specify the?dataDirHostPath?to tell it to use the local disk of the Kubernetes node it runs on, and on each node it will create a 5Gb block (this can be increased based on your requirement and available disk capacity). Like the Rook operator deployment, the Ceph cluster deployment is also done via ArgoCD and all you need to do is to have the ArgoCD app for Ceph cluster points to the directory where cluster manifest is and the deployment to Kubernetes cluster will be executed. Do not forget to also uncomment the necessary lines in the kustomization.yaml file. In my case, I have uncommented the line for baremetal-cluster.yaml since I am deploying to on-premesis environment.

You should be able to see the following pods once they are all running. The number of osd pods will depend on the number of nodes in the cluster and the number of devices configured. If you did not modify the baremetal-cluster.yaml above, it is expected that one OSD will be created per node. The CSI, rook-ceph-agent (flex driver), and rook-discover pods are also optional depending on your settings.

If the rook-ceph-mon, rook-ceph-mgr, or rook-ceph-osd pods are not created, please refer to the?Ceph common issues?for more details and potential solutions.

kubectl -n rook-ceph get pod
NAME                                                 READY   STATUS      RESTARTS   AGE
csi-cephfsplugin-provisioner-d77bb49c6-n5tgs         5/5     Running     0          140s
csi-cephfsplugin-provisioner-d77bb49c6-v9rvn         5/5     Running     0          140s
csi-cephfsplugin-rthrp                               3/3     Running     0          140s
csi-rbdplugin-hbsm7                                  3/3     Running     0          140s
csi-rbdplugin-provisioner-5b5cd64fd-nvk6c            6/6     Running     0          140s
csi-rbdplugin-provisioner-5b5cd64fd-q7bxl            6/6     Running     0          140s
rook-ceph-crashcollector-minikube-5b57b7c5d4-hfldl   1/1     Running     0          105s
rook-ceph-mgr-a-64cd7cdf54-j8b5p                     1/1     Running     0          77s
rook-ceph-mon-a-694bb7987d-fp9w7                     1/1     Running     0          105s
rook-ceph-mon-b-856fdd5cb9-5h2qk                     1/1     Running     0          94s
rook-ceph-mon-c-57545897fc-j576h                     1/1     Running     0          85s
rook-ceph-operator-85f5b946bd-s8grz                  1/1     Running     0          92m
rook-ceph-osd-0-6bb747b6c5-lnvb6                     1/1     Running     0          23s
rook-ceph-osd-1-7f67f9646d-44p7v                     1/1     Running     0          24s
rook-ceph-osd-2-6cd4b776ff-v4d68                     1/1     Running     0          25s
rook-ceph-osd-prepare-node1-vx2rz                    0/2     Completed   0          60s
rook-ceph-osd-prepare-node2-ab3fd                    0/2     Completed   0          60s
rook-ceph-osd-prepare-node3-w4xyz                    0/2     Completed   0          60s
        

To verify that the cluster is in a healthy state, connect to the?Rook toolbox?and run the Ceph status command.

• All mons should be in quorum
• A mgr should be active
• At least one OSD should be active
• If the health is not HEALTH_OK, the warnings or errors should be investigated

ceph status
 cluster:
   id:     a0452c76-30d9-4c1a-a948-5d8405f19a7c
   health: HEALTH_OK

 services:
   mon: 3 daemons, quorum a,b,c (age 3m)
   mgr: a(active, since 2m)
   osd: 3 osds: 3 up (since 1m), 3 in (since 1m)
...
        

If the cluster is not healthy, please refer to the?Ceph common issues?for more details and potential solutions.

Provision Storage Type

Now that we have deployed Rook operator and Ceph cluster, it's time to provision our desired storage service. Please refer to the link below to configure your storage type.

• Provision Object Storage via Ceph