Decoding Velero and Restic: A Comprehensive Guide to Backup Sets and Kubernetes Architecture in Action
Swadhin Pattnaik
Cloud Infrastructure specialist || Technical writer || Devops Enthusiastic || Open-source contributor
Table of contents
Introduction
As we know, Velero is an open-source tool used for backing up and restoring resources in a Kubernetes cluster. It facilitates disaster recovery and enables resource and persistent volume migration between Kubernetes clusters.
However, understanding how Velero functions and the underlying backup process is essential. In this explanation, I will delve into the backup backend process and how it gets initiated with a single Velero command.
Velero and Restic Integration
Velero supports backing up and restoring Kubernetes volumes using Restic, a free open-source backup tool. However, Restic is not installed by default with Velero. You need to enable it during installation by using the --use-restic flag in the Velero install command.
When using Restic, Velero creates a Restic repository per namespace upon the first backup request for a given namespace.
Kubernetes Perspective
From a Kubernetes perspective, every pod containing a volume to be backed up using Restic must be annotated with:
backup.velero.io/backup-volumes: <volume-name>
This annotation tells Velero to include the specified volumes in the backup process.
Architecture flow
NOTE: THE ABOVE ARCHITECTURE IS GENERIC FOR GENERAL UNDERSTANDING PURPOSES ONLY, AND THE GIVEN DATA IS RANDOM, BASED ON MY PERSONAL LAB SETUP. IT IS NOT BASED ON ANY PRODUCTION OR UAT ENVIRONMENT.
Based on the architecture, Now, I'm going to classify the backend respository directory and their purpose.
this config file allows for easy verification of files for accidental modifications, like disk read errors, by simply running the program sha256sum and comparing its output to the file name. (meanwhile, the name for the file is the lower case hexadecimal representation of the storage ID, which is the SHA-256 hash of the file's contents)
Apart from the files stored within the keys directory, all files are encrypted with AES-256 IDs. (the integrity of the encrypted data is being secured by AES-256signature)
the file config is encrypted with this signature and contains a following parameters mentioned below.
{ "version": 1, "id": "5956a3f67a6230d4a92cefb29529f10196c7d92582ec305fd71ff6d331d6271b", "chunker_polynomial": "25b468838dcb75" }
After decryption, restic first checks that the version field contains a version number that it understands, otherwise it aborts. At the moment, the version is expected to be 1. The field id holds a unique ID which consists of 32 random bytes, encoded in hexadecimal.
The field chunker_polynomial contains a parameter that is used for splitting large files into smaller chunks . (the chunks file will be present in data directory , we will discuss after this)
Index files contain information about Data and Tree Blobs(No of data and it's length) and the Packs(combine of blobs) they are contained in and store this information in the repository.
When the local cached index is not accessible any more, the index files can be downloaded and used to reconstruct the index. The files are encrypted and authenticated like Data and Tree Blobs, so the outer structure is AES-256 again.
snapshots represent a directory with all files and sub-directories at a given point in timestamp.
All the chunk files, created by chunker_polynomial will be stored in data directory.
this is all about the basic structure of restic backup. we will continue exploring it to provide further insights and clarity.
My Lab Setup
Prequisite:
Lab setup details:
Steps For Configuration:
1. To download Velero, run the following command in the kubectl command-line interface:
[[email protected] ~]$ sudo curl -fsSL -o velero-v1.5.2-linux-amd64.tar.gz https://github.com/vmware-tanzu/velero/releases/download/v1.5.2/velero-v1.5.2-linux-amd64.tar.gz
2. Extract the .tar file by issuing the following command:
[[email protected] ~]$ sudo tar -xvf velero-version-linux-amd64.tar.gz
3. Before setting up the namespace backup with Velero, we will need to prepare S3 credentials.
Now, we will navigate to ceph rados gateway and create an Object Gateway user for Velero S3 credentials.
[[email protected] ~]$ sudo radosgw-admin user create --uid="velero" --display-name="Admin User for velero backup"
{
"user_id": "velero",
"display_name": "Admin User for velero backup",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "velero",
"access_key": "UD5CUZFI***Y17QO2O14",
"secret_key": "sneZFawpy0ihV*****JycnkWn2GmSbQG6R1YT1q"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}
4. Create bucket for velero backup.
Log in to the Dashboard. --->On the navigation bar, click Object Gateway --->Click Buckets and then click Create.--->In the Create Bucket window, enter a value for Name and select a user that is not suspended.---> Select a placement target.
5. Check the bucket stats
[[email protected] ~]$ sudo radosgw-admin bucket stats --bucket=velerobackupstorage
{
"bucket": "velerobackupstorage",
"num_shards": 11,
"tenant": "",
"zonegroup": "30c7dba0-dba4-4a3a-bc49-95fa2423ebd7",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "4ba2e434-37ac-4c82-95da-1318f71bbe99.189817.2",
"marker": "4ba2e434-37ac-4c82-95da-1318f71bbe99.189817.2",
"index_type": "Normal",
"owner": "velero",
"ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
"mtime": "0.000000",
"creation_time": "2023-11-19T05:17:06.450142Z",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#",
"usage": {},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
6. Now we can save the access and secret keys for use as a Kubernetes Secret. First, open up the credential-file file using your favorite editor. and save it.
[[email protected] ~]$ cat credential-file
[default]
aws_access_key_id = UD5CUZFIO****QO2O14
aws_secret_access_key = sneZFawpy0ihVm9**********2GmSbQG6R1YT1q
7. Now, connect to your k8 cluster, Change to the directory where the extracted .tar file is located and run the following command:
[[email protected] ~]$ sudo ./velero install --provider aws --plugins "projects.registry.vmware.com/tkg/velero/velero-plugin-for-aws:v1.5.3_vmware.1" --bucket mybucketdefault --secret-file ./credential-file --use-volume-snapshots=false --use-restic --default-volumes-to-restic --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=https://192.168.1.10:80
8. When the installation finishes, a message similar to the following message is displayed:
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: createdWaiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: created
Velero is installed! ? Use 'kubectl logs deployment/velero -n velero' to view the status.
8. Now, we can watch the deployment logs using the kubectl command from the output.
[[email protected] ~]$ sudo kubectl get deployment -n velero
NAME READY UP-TO-DATE AVAILABLE AGE
velero 1/1 1 1 20s
[[email protected] ~]$ sudo kubectl get daemonset -n velero
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
restic 23 23 23 23 <none>
9. Check that the secret has been created for default bsl:
[[email protected] ~]$ sudo kubectl get secret/velero-1658856389 -n velero
NAME TYPE DATA AGE
velero-1658856389 Opaque 1 50s
10. Now, we will create both Kubernete's secret and a new backup storage location(bsl) for Velero backup, respectively.
[[email protected] ~]$ sudo kubectl create secret generic -n velero bucket-credential --from-file=bsl=credential-file
secret/credential-file created
[[email protected] ~]$ sudo velero backup-location create cephlocation --provider aws --bucket velerobackupstorage --config region=minio --credential=credential-file=bsl --config region=minio,s3ForcePathStyle="true",s3Url=https://192.168.1.10:80 --prefix default
Backup storage location "cephlocation" configured successfully.
11. Check the status of both the secret and bsl using the following command below.
[[email protected] ~]$ sudo kubectl get secret -n velero
NAME TYPE DATA AGE
bucket-credential Opaque 1 21s
default-token-cr6bv kubernetes.io/service-account-token 3 196d
velero-restic-credentials Opaque 1 196d
velero-token-tr9ml kubernetes.io/service-account-token 3 196d
[[email protected] ~]$ velero backup-location get
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT
cephlocation aws velerobackupstorage/default Available 2023-11-20 11:33:39 +0000 IST ReadWrite true
12. Testing backup and Restore procedure:
12.1 create a nginx pod for backup test.
[[email protected] ~/nginx/deployment]$ cat nginx-deployment.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: nginx-test-velero
labels:
app: nginx
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nginx-logs
namespace: nginx-test
labels:
app: nginx
spec:
storageClassName: block-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-test-backup-velero
namespace: nginx-test
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
volumes:
- name: nginx-logs
persistentVolumeClaim:
claimName: nginx-logs
containers:
- image: nginx:stable
name: nginx
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/var/log/nginx"
name: nginx-logs
readOnly: false
---
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx
name: nginx-svc-net
namespace: nginx-example
spec:
ports:
- port: 80
targetPort: 80
selector:
app: nginx
type: LoadBalancer
12. Create the objects using kubectl apply:
[[email protected] ~/nginx/deployment]$ kubectl apply -f nginx.yaml
namespace/nginx-test created
persistentvolumeclaim/nginx-logs created
deployment.apps/nginx-test-backup-velero created
service/nginx-svc-net created
13. Check that the Deployment succeeded:
[[email protected] ~]$ kubectl get deploy -n nginx-test
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-test-deploy 1/1 1 1 23s
[[email protected] ~]$ kubectl get svc -n nginx-test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-svc LoadBalancer 10.245.148.62 159.103.58.172 80:30220/TCP 1m
14. then we will proceed with either velero backup or velero schedule .
14.1. create a schedule.
Here, we have set the total backup duration to 1 hour with a time-to-live (TTL) of 168 hours, allowing the system to retain backups for up to 7 days.
[[email protected] ~]$ sudo velero create schedule nginx-backup-1hr-168ttl --schedule="0 " --include-namespaces nginx-test-velero --storage-location objects --default-volumes-to-restic --ttl 168h
Schedule "nginx-backup-1hr-168ttl" created successfully.
#After one hour, the backup status for 'nginx-backup-1hr-168ttl' has been marked as completed.
[[email protected] ~]$ sudo velero backup get | grep nginx-backup-1hr-168ttl | grep 2023-11-24
nginx-backup-1hr-168ttl-20231124100156 Completed 0 0 2023-11-24 10:00:56 +0000 UTC 6d objects <none>
14.2. create a backup.
[[email protected] ~]$ sudo velero backup create nginx-test-backups --selector app=nginx-test
Backup request "nginx-test-backups" submitted successfully.
Run velero backup describe nginx-test-backups or velero backup logs nginx-test-backups for more details.
15. Running output. backup describe nginx-test-backups --details should provide the following output .
[[email protected] ~]$ sudo velero backup describe nginx-test-backups-20230112335032 --details
Output
Name: nginx-test-backups-20230112335032
Namespace: velero
Labels: velero.io/backup=nginx-backup
velero.io/pv=pvc-6b7f63d7-752b-4537-9bb0-003bed9129ca
velero.io/storage-location=cephlocation
Annotations: <none>
Phase: Completed
Namespaces:
Included: *
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: app=nginx-test
Storage Location: cephlocation
Snapshot PVs: auto
TTL: 168h0m0s
Hooks: <none>
Backup Format Version: 1
Started: 2023-11-20 23:45:30 -0500 EST
Completed: 2023-11-20 23:45:34 -0500 EST
Expiration: 2023-11-26 23:45:30 -0500 EST
Resource List:
apps/v1/Deployment:
- nginx-test-velero/nginx-test-deploy
apps/v1/ReplicaSet:
- nginx-test-velero/nginx-test-deploy-694c85cdc8
v1/Endpoints:
- nginx-test-velero/nginx-svc
v1/Namespace:
- nginx-test-velero
v1/PersistentVolume:
- pvc-6b7f63d7-752b-4537-9bb0-003bed9129ca
v1/PersistentVolumeClaim:
- nginx-test-velero/nginx-logs
v1/Pod:
- nginx-test-velero/nginx-test-deploy-694c85cdc8-vknsk
v1/Service:
- nginx-test-velero/nginx-svc
Persistent Volumes:
pvc-5c6g93d9-854c-8529-8dd0-003ves8259xq:
Snapshot ID: cgf823as-3de4-11ea-9ec0-0b97de14f134
Type: ext4
Availability Zone:
IOPS: <N/A>
this output indicates that the nginx-test-backups completed successfully.
16. now, we can test the restore procedure.
16.1. let's delete the nginx-test-velero Namespace.
[[email protected] ~]$ sudo kubectl delete namespace nginx-test-velero
16.2. verify that the nginx can no longer available at the svc,lb and deployment.
[[email protected] ~]$ sudo kubectl get deploy -n nginx-test-velero
No resources found in nginx-test-velero namespace.
16.3. we can now perform the restoration process, using below command
[[email protected] ~]$ sudo velero restore create --from-backup nginx-test-backups
Restore request "nginx-test-backups-20230112335032" submitted successfully.
Run velero restore describe nginx-backup-20200102235032 or velero restore logs nginx-test-backups-20230112335032 for more details.
16.4. Check the status of the restored Deployment ,service and pod.
[[email protected] ~]$ kubectl get deploy -n nginx-test-velero
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-test-deploy 1/1 1 1 3s
[[email protected] ~]$ kubectl get svc -n=nginx-test-velero
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-svc LoadBalancer 10.245.148.62 159.103.58.172 80:30220/TCP 1m
[[email protected] ~]$ kubectl get pod -n nginx-test-velero
NAME READY STATUS RESTARTS AGE
nginx-test-deploy-694c85cdc8-vknsk 1/1 Running 0 2m20s
Conclusion:
In this guide, we discussed the architecture details and configured the Velero Kubernetes backup tool on a Kubernetes cluster.
I hope this article proves helpful.
For additional reference, you can visit: https://velero.io/docs/v1.12/.
Ceph architect, consultant, and contributor, published author, technical leader, people manager, mentor, expert consultant, storage and server observability fanatic, infrastructure harmonizer and uplifter
1 周Ceph is where all the cool kids keep their data.