Unlocking the Secrets of etcd: A Comprehensive Guide to Backup and Restore in Kubernetes

In this article, we will explore the ins and outs of etcd data management, focusing on the vital aspects of backup and restoration.

The etcd datastore contains crucial cluster configuration data, encompassing information about the number of pods or deployments running within our cluster, along with details about which ports are exposed on specific servers.

To back up or restore the etcd datastore, we first need to install the etcd client, known as the etcdctl tool.

Step 1:?To install this tool, execute the following commands in your terminal:

sudo apt update
sudo apt install -y etcd-client        

After successfully installing the etcdctl tool, we can verify the version using the following command:

etcdctl v        

The output of the etcdctl version command will provide version information, depending on the version you’ve installed.

Step 2:?Before performing any etcd related operations, it’s essential to set an environment variable. This can be done by running the following command:

export ETCDCTL_API=3        

With the environment variable correctly set, we are now prepared to create a backup of the etcd datastore.

Step 3:?The command for taking a backup of the etcd datastore in the current directory is as follows:

etcdctl snapshot save sssnapshotdbbb - cacert /etc/kubernetes/pki/etcd/ca.crt - cert /etc/kubernetes/pki/etcd/server.crt - key /etc/kubernetes/pki/etcd/server.key
#This is to save in current directory        

or

etcdctl snapshot save sssnapshotdbbb - cacert /etc/kubernetes/pki/etcd/ca.crt - cert /etc/kubernetes/pki/etcd/server.crt - key /etc/kubernetes/pki/etcd/server.key <backup-file-location>
#This is to save in a particular location        

Note:?We need to run this command from master node

Let’s break down the command for clear understanding!

1.?etcdctl: This is the name of the command-line tool we are running. It is used to interact with an etcd cluster, which is a distributed key-value store often used for configuration management and service discovery in Kubernetes clusters.

2.?snapshot save: This is the action we want etcdctl to perform, which is to save a snapshot of the etcd database. Snapshots are a way to back up the data in etcd for recovery and disaster scenarios.

3.?sssnapshotdbbb: This is the name we’ve chosen for the snapshot we’re creating. You can replace this with any name you prefer. This will be the file that stores the snapshot.

4.?--cacert /etc/kubernetes/pki/etcd/ca.crt: The -- cacert flag is used to specify the Certificate Authority (CA) certificate file. In an etcd cluster, this is the certificate authority that issues certificates for secure communication within the cluster. This flag is used to authenticate and encrypt communication with etcd. /etc/kubernetes/pki/etcd/ca.crt is the file path to the CA certificate.

5.?-- cert /etc/kubernetes/pki/etcd/server.crt: The --cert flag is used to specify the client certificate file. This certificate is used to authenticate the client, in this case, etcdctl, to the etcd server. /etc/kubernetes/pki/etcd/server.crt is the file path to the client certificate.

6.?--key /etc/kubernetes/pki/etcd/server.key: The --key flag is used to specify the private key file associated with the client certificate. The private key is used for cryptographic operations to establish secure communication. /etc/kubernetes/pki/etcd/server.key is the file path to the private key.

The command is used to create a snapshot of the etcd database in a secure manner. It specifies the location and names of the files required for secure communication and authentication with the etcd server. The resulting snapshot file will be named “sssnapshotdbbb” or whatever name you specify, and it will contain a backup of the etcd database’s data, which can be used for recovery or backup purposes in a Kubernetes cluster.

Step 4:?List the snapshots

ls | grep <snapshot-name>        

Step 5:?See if there’s any data in the snapshot file

etcdctl snapshot status <snapshot-name> - write-out=table        

We have effectively captured snapshots (backup) of the etcd datastore. Next, we will delve into the process of restoring etcd.

Restore etcd

To restore etcd run below command

etcdctl snapshot restore <snapshot-name> --dat-dir /var/lib/etcd-restore
#This will restore the snapshot to /var/lib/etcd-restore        

Now we need to change the location where Kubernetes looks for the etcd data. This will be yaml file of API server on the control plane node in /etc/kubernetes/manifests directory

Edit /etc/kubernetes/manifests/etcd.yaml file

Change the path for the volume from /var/lib/etcd to /var/lib/etcd-restore

Save the file & check if the deleted daemonset is restored or not (if you have deleted ds)

I hope we’ve gained valuable insights into the art of creating backups and restoring etcd using the etcdctl command-line tool. If you found this information valuable, please consider sharing this article with your peers. Don’t forget to follow me for more informative articles in the future??. Thank you for reading!???

#Kubernetes #DevOps #ContainerOrchestration #LearningNuggets #K8sTips #GameOfKubes #K8s #platformengineering #etcdctl #etcd