Kube Secret Data at Rest
Requirements
Introduction
Kubernetes is a open source container orchestration platform. For more info refer: https://kubernetes.io/docs/concepts/overview/
Why Kubernetes in short - Gordon Haff and William Henry said. “you can cluster together groups of hosts running Linux containers, and Kubernetes helps you easily and efficiently manage those clusters.”. Now it has evolved from ONLY Linux containers to multi OS containers.
Problem statement
The Kubernetes secret that is stored in the cluster is not that secure by default. The content is stored in plain text. With sufficient access, one can sniff the details.
There are multiple methods to read the content. Will share some of them here
Method1: Read content by logging into Master node
Step1: SSH into the master node
Step2: Install the ETCD client tools if not present
ETCD_RELEASE=$(curl -s https://api.github.com/repos/etcd-io/etcd/releases/latest|grep tag_name | cut -d '"' -f 4)
echo $ETCD_RELEASE
wget https://github.com/etcd-io/etcd/releases/download/${ETCD_RELEASE}/etcd-${ETCD_RELEASE}-linux-amd64.tar.gz
tar xvf etcd-${ETCD_RELEASE}-linux-amd64.tar.gz
cd etcd-${ETCD_RELEASE}-linux-amd64
sudo mv etcdctl /usr/bin
cd ..
rm -rf etcd-v3.4.23-linux-amd64*
etcdctl version
# etcdctl version: 3.4.23
# API version: 3.4
Step3: Create a secret to test.
kubectl create secret generic mysecret --from-literal=securekey=securedata
Step4: Call the etcd endpoint to fetch the required secret from namespace. In my case I have created it in the "default" namespace and the secret is know as "mysecret" . So the command looks as below. We will be printing it in Hex format.
ETCDCTL_API=3
sudo etcdctl \
--cert=/etc/kubernetes/pki/etcd/peer.crt \
--key=/etc/kubernetes/pki/etcd/peer.key \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--endpoints=127.0.0.1:2379 get /registry/secrets/default/mysecret | hexdump -C
00000000 2f 72 65 67 69 73 74 72 79 2f 73 65 63 72 65 74 |/registry/secret|
00000010 73 2f 64 65 66 61 75 6c 74 2f 6d 79 73 65 63 72 |s/default/mysecr|
00000020 65 74 0a 6b 38 73 00 0a 0c 0a 02 76 31 12 06 53 |et.k8s.....v1..S|
00000030 65 63 72 65 74 12 d8 01 0a b4 01 0a 08 6d 79 73 |ecret........mys|
00000040 65 63 72 65 74 12 00 1a 07 64 65 66 61 75 6c 74 |ecret....default|
00000050 22 00 2a 24 35 36 36 39 38 30 31 35 2d 61 62 39 |".*$56698015-ab9|
00000060 64 2d 34 37 33 61 2d 62 61 31 38 2d 36 34 39 38 |d-473a-ba18-6498|
00000070 65 30 33 61 36 36 34 33 32 00 38 00 42 08 08 e0 |e03a66432.8.B...|
00000080 f0 ed 9d 06 10 00 8a 01 66 0a 0e 6b 75 62 65 63 |........f..kubec|
00000090 74 6c 2d 63 72 65 61 74 65 12 06 55 70 64 61 74 |tl-create..Updat|
000000a0 65 1a 02 76 31 22 08 08 e0 f0 ed 9d 06 10 00 32 |e..v1".........2|
000000b0 08 46 69 65 6c 64 73 56 31 3a 32 0a 30 7b 22 66 |.FieldsV1:2.0{"f|
000000c0 3a 64 61 74 61 22 3a 7b 22 2e 22 3a 7b 7d 2c 22 |:data":{".":{},"|
000000d0 66 3a 73 65 63 75 72 65 6b 65 79 22 3a 7b 7d 7d |f:securekey":{}}|
000000e0 2c 22 66 3a 74 79 70 65 22 3a 7b 7d 7d 42 00 12 |,"f:type":{}}B..|
000000f0 17 0a 09 73 65 63 75 72 65 6b 65 79 12 0a 73 65 |...securekey..se|
00000100 63 75 72 65 64 61 74 61 1a 06 4f 70 61 71 75 65 |curedata..Opaque|
00000110 1a 00 22 00 0a |.."..|
00000115
Method2: Read content by logging into etcd pod
Step1: Loginto the k8s[Kubernetes] cluster
Step2: Lets use the secret created in Method1 ["mysecret" in default namespace]
Step3: List the resources and see the secret from etcd
kubectl -n kube-system exec -it etcd-master.k8s-lab.dev -- sh -c \
"etcdctl --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --cacert=/etc/kubernetes/pki/etcd/ca.crt --endpoints=https://127.0.0.1:2379 get /registry/secrets/default/mysecret"
Output is as shown below
k8s
v1Secret?
?
mysecret?default"*$56698015-ab9d-473a-ba18-6498e03a66432????f
kubectl-createUpdate?v???FieldsV1:2
0{"f:data":{".":{},"f:securekey":{}},"f:type":{}}B
securekey
securedata?Opaque?"
Method3: Read content by accessing the physical path in etcd - DB file
Step1: Loginto Master node
Step2: Navigate to the etcd db path and search for the data
ssh masternode
sudo -i
cd /var/lib/etcd/member/snap
cat db | less
# search for the "mysecret" and the data value of the secret
output will have unencrypted data. snipped of the content is as shown below
领英推荐
"/registry/secrets/default/mysecret^P?^Y^X?^Y ^A*<F1>^Ak8s^@
^L
^Bv1^R^FSecret^R<D8>^A
<B4>^A
^Hmysecret^R^@^Z^Gdefault"^@*$56698015-ab9d-473a-ba18-6498e03a66432^@8^@^H<E0><F0><ED><9D>^F^P^@<8A>^Af
^Nkubectl-create^R^FUpdate^Z^Bv<E0><F0><ED><9D>^F^P^@FieldsV1:2
0{"f:data":{".":{},"f:securekey":{}},"f:type":{}}B^@^R^W
securekey^R
securedata^Z^FOpaque^Z^@"^@^@^@^@^@^@^FK<D3>_^@^@^@^@^@^@^@^@
$/registry/minions/master.k8s-lab.dev^P<C9>^A^X?^Y <82>^G*<BE>%k8s^@
Solution
The solution is to use the EncryptionConfiguration. As part of this post we will be using the "aescbc" technique.
Step1: Generate the key and encode it with base64
head -c 32 /dev/urandom | base64
XPUWUXyoc94i7YesPR/rNFRfwq/uJCZ8zRnFOhoVnQE=
Step2: Create a folder to store the yaml configs
sudo mkdir /etc/kubernetes/enc
Step3: Create the "EncryptionConfiguration" yaml file. Here the secret value is the one generated from above step.
cat /etc/kubernetes/enc/enc-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: XPUWUXyoc94i7YesPR/rNFRfwq/uJCZ8zRnFOhoVnQE=
- identity: {}
Step4: Edit the Kube-apiserver manifest file. This is by default located at "/etc/kubernetes/manifests/kube-apiserver.yaml". Here we need to add the enc-config.yaml and then mount the "/etc/kubernetes/enc" folder. The snippet is as shown below
apiVersion: v1
kind: Pod
metadata:
annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.10.30.4:6443
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
...
- --encryption-provider-config=/etc/kubernetes/enc/enc-config.yaml # <-- add this line
volumeMounts:
...
- name: enc # <-- add this line
mountPath: /etc/kubernetes/enc # <-- add this line
readonly: true # <-- add this line
...
volumes:
...
- name: enc # <-- add this line
hostPath: # <-- add this line
path: /etc/kubernetes/enc # <-- add this line
type: DirectoryOrCreate # <-- add this line
Optional step -->Restart your API server if the changes are not picked-up
kubectl -n kube-system delete pod -l 'component=kube-apiserver'
Step5: Create a new secret and check if the data is encrypted.
kubectl create secret generic mysecret2 -n default --from-literal=securekey123=securedata123
kubectl -n kube-system exec -it etcd-master.k8s-lab.dev -- sh -c\
"etcdctl --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --cacert=/etc/kubernetes/pki/etcd/ca.crt --endpoints=https://127.0.0.1:2379 get /registry/secrets/default/mysecret2"
The output is as shown below
But you can see that the old secret "mysecret" is still unencrypted
kubectl -n kube-system exec -it etcd-master.k8s-lab.dev -- sh -c
"etcdctl --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --cacert=/etc/kubernetes/pki/etcd/ca.crt --endpoints=https://127.0.0.1:2379 get /registry/secrets/default/mysecret1"\
Step6: Lets encrypt the pre-existing secrets.
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
This will secure the kubernetes secret object. Thanks for reading.
Video version of the content
References