Keycloak Cluster High Availability in Kubernetes
Keycloak is an open source software product to allow single sign-on with Identity and Access Management aimed at modern applications and services. As of March 2018 this WildFly community project is under the stewardship of Red Hat who use it as the upstream project for their RH-SSO product.
Keycloak having features such as Single-Sign-On (SSO), Identity Brokering and Social Login, User Federation, Client Adapters, an Admin Console, and an Account Management Console.
Refer more Keycloak below links
Prerequisites For This Setup
3. PostgreSQL (Backend DB for Keycloak)
4. Helm (Kubernetes Package Manager)
5. Dynamic NFS Provisioning (Storage PVC For PostgreSQL DB)
6. MetalLB (LoadBalancer for Baremetal Kubernetes Cluster)
7. Nginx Ingress Controller (Route traffic by Ingress Controller)
8. Kubeseal (Encrypt the encoded Secrets)
* Ensure K8s cluster is up and ready status
All dependencies setup are automated by Ansible playbooks
* Clone the git repo ( https://github.com/avi1512/Keycloak-HA-K8s.git) and run the playbook to deploying MetalLB, Dynamic NFS Provisioning, Kubeseal, Nginx Ingress Controller using Helm charts.
Before run the playbook ensure the connectivity between Ansible to K8s control plane node.
* Deploying Dynamic NFS Provisioning
Dynamic NFS provisioning playbook will install if helm is not installed on K8s master node and we are using K8s master node as a NFS server and it will exports NFS path like /mnt/kubedata. And also NFS packages will installed on workers nodes as well.
Once the playbook completed without any issues we can see default StorageClass in Kubernetes cluster.
K9S Terminal Dashboard View
* Kubeseal Deployment
Secrets in Kubernetes are used to store sensitive data, like password, keys, certificates and token. Secrets are encoded in base64 and automatically decoded when they are attached and read by a Pod.
We can't store sensitive data's in public repository. Because of secrets objects are encoded format only not encrypted. so it will decoded easily and everyone see the data. Using Kubeseal we can prevent the issues. Like we can sealed the sensitive data.
Kubeseal will encrypt the actual encoded values and save it secret objects. So when ever deploying that secrets it will decrypt by kubeseal secret controller in server side. In client side we will use Kubeseal binary for encrypt the secret yaml file.
Kubeseal playbook will setup client and server side like it will download the Kubeseal binary and it will deploying the controller in Kubernetes cluster.
Once the playbook completed without any issues we can see Kubeseal secret controller container is up and running in Kubernetes cluster. Like below image
* MetalLB Deployment
MetalLB is a load-balancer implementation for bare metal Kubernetes?clusters, using standard routing protocols.
Here Playbook will ask the network range for our lab setup to configure the MetalLB and this IP range will be auto assign when service will expose as a LoadBalancer. For my network range will be 192.168.0.180-192.168.0.200
Once the playbook is completed without issues we can see MetalLB controller and Speaker DaemonSet container is up and running.
And also we can the Metallb IPAddressPool and L2Advertisement in metallb-system namespace.
* Nginx Ingress Controller Deployment
An?Ingress controller?is a specialized load balancer for Kubernetes (and other containerized) environments. Kubernetes?is the de facto standard for managing containerized applications. For many enterprises, moving production workloads into Kubernetes brings additional challenges and complexities around application traffic management. An Ingress controller abstracts away the complexity of Kubernetes application traffic routing and provides a bridge between Kubernetes services and external ones.
Once the playbook completed without any issues. We can see Nginx Ingress Controller container and service like below image.
Here Nginx Ingress Controller service is exposed as LoadBalancer and we can see the service is assigned to that external IP is 192.168.0.180 This is the IP was we mentioned in our MetalLB IPAddressPool network range.
* Encrypt the PostgreSQL and Keycloak DB connection String secret objects using Kubeseal
Example: kubeseal <secret-file-name.yaml> secret-file-name-encrypted.yaml
After sealed the secret we can see the sensitive data like passwords are encrypted. Like below image.
* KeyCloak HA Clustering
* Ok Let's divine in our main topic, We need to create dedicated namespace for Keycloak HA setup and also serviceAccount, Role and Rolebinding. This SA and Roles will get and list the pods in that namespace.
apiVersion: v1
kind: Namespace
metadata:
? name: sso
---
apiVersion: v1
kind: ServiceAccount
metadata:
? name: keycloak-kubeping-service-account
? namespace: sso
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
? name: keycloak-kubeping-pod-reader
? namespace: sso
rules:
- apiGroups: [""]
? resources: ["pods"]
? verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
? name: keycloak-kubeping-api-access
? namespace: sso
roleRef:
? apiGroup: rbac.authorization.k8s.io
? kind: Role
? name: keycloak-kubeping-pod-reader
subjects:
- kind: ServiceAccount
? name: keycloak-kubeping-service-account
? namespace: sso
Keycloak clustering are based on the discovery protocols of?JGroups?(Keycloak use?Infinispan?cache and Infinispan use JGroups to discover nodes).
Infinispan is an open-source?in-memory data grid?that offers flexible deployment options and robust capabilities for storing, managing, and processing data. Infinispan provides a key/value data store that can hold all types of data, from Java objects to plain text. Infinispan distributes your data across elastically scalable clusters to guarantee high availability and fault tolerance, whether you use Infinispan as a volatile cache or a persistent data store.
On Kubernetes cluster we are going to use JGroups discovery protocol like KUBE_PING These will discovery how many containers are running in that namespace. And it will get pod details using JGroups properties and it will sync up with each pods and also cache between one container to another using Infinispan concepts.
* Let's deploy first backend DB PostgreSQL
PostgreSQL Secrets after sealed by Kubeseal
{
? "kind": "SealedSecret",
? "apiVersion": "bitnami.com/v1alpha1",
? "metadata": {
? ? "name": "postgres-secret",
? ? "namespace": "sso",
? ? "creationTimestamp": null
? },
? "spec": {
? ? "template": {
? ? ? "metadata": {
? ? ? ? "name": "postgres-secret",
? ? ? ? "namespace": "sso",
? ? ? ? "creationTimestamp": null,
? ? ? ? "labels": {
? ? ? ? ? "app": "sso",
? ? ? ? ? "role": "backend"
? ? ? ? }
? ? ? },
? ? ? "data": null
? ? },
? ? "encryptedData": {
? ? ? "db-name": "AgCePcBeZ0+UcwWHT3axQwoGmtW3UPWyz8upVt5lFRDyEnd9nnQ9aeBcmhG2wt20kPs6MwIN9Kzw4aW1qE2C40LwSlTqZIBqArb9e1lSEnYGwSgLnCRL62nFQL/3SyaUrU+r/o9CRC1AOzqv/aJv3AYoDvPoX4J+Pjtk5cQyNo9GXkBVyutIeS7amxP182dSJVAEnOMS2qKpwR7BctQSJtGtcbASxqd18ItVnQ/V2dOk2hMMCZR6nRszMphk/PSpy2i6gEfOkuI7mnAawaUOXi/IXI5NvgYGLlDFW2FzSRLmIRG/Etopjnbb6pm0GwMkUosUi9BMPkg1n4pxk6n4rRZvEG22ZifTHP3ZXQ3VuQxK/u3AEsV3Nq0IscbRRa20Qd+XZO5XME+ej/clF4U20QHGnVGS9z+asLPhq/nNPbymCWRSXAw/WOQKe4fjg/onhvA5Z1f+QNC+ZWLLLAEGqZWfUN+QulWKo5/zXOBRQKam5Bro1G2FD5u2DUhJzOB5dpN6fa70d2mj4jrX+nSjm8TK1Ddm4PFJoBg5lCkThBsu0wGm/9VHYU6FMnig62n6LgGDyObEjKmrIx0X8TzeuADBdFEXxDTDpBikgeFSjLESROWqWCESP7I/2wMRhpHxw4UiJ0CC0pTpWzKx+z1dAznJdpRgqdlM7gh7Dt8nbSSxzDeyDNy9twNi/vYZmY6LQjGdcigR0qcqIg==",
? ? ? "db-password": "AgBrX6aiS5Hqz/BSvUXYNbe5hB7N2JtCD1iAGxEfJsw3ABUfY2+u/xN7EWXSm8oGEgGMBNSoDY+glgtasc4+5dUAU2YWoFNOLZ885yJEQHcCeIJh3QuVvQ8Q6MZpZzJGh9J/w+80XDZEV4USE80/L3edYbVguDc70jHkY2VXbXrVST41KBgLhknt/Dm3x0wUzwysi3zyeIEmaUvT9+60TRF3YlJ1lPTWefCxkTj6r0VxfEP7aADD4Zua3mt178+rQCVPZAPdW8PIIZSiZLji4F41eFlMg6zU1IJwIh97ktTqQsvIckK/eaBckmxs5ZroBywGSxss9LbMdY4IyudIba39ptpbtsPRjyZbF4xshkTT5dj6TsYfQl6qDqQlXoLBiZhxfxOl/YF+P8GhcUOP3H/YhGngiWQXf1lcNtJlAh15s7jipbnrArs1JjetFnLhJCFAzGPPgKIwER1jgLG2/48L6GZZVF4QrET9xEDkOvq4gSFVX6nMYdOPwrgDsrw5scnIk8tv1W1xsPuZz86j+lKksRKnIJZMuY//n6/F0ki/Hwjk0eSU1pUzdttuRBPP5UcrWyHBloxRpu6ntAG/o9LYKtLrjSXShnOq1oe7e9mUfQRYPKg3KsjlKPbf9E5+fbtXhsFgGb0mvuBL6iP6YjDL2cEHU0Xe41Vf2p71/m9gHqlLSZcD8HPjCMxHiW6oZsWuBFReGL1Aog==",
? ? ? "db-user": "AgDBfqP+QCrosewaspOUcsnhoJRaMpt5C3GIO9a+nbxGaQSxCYc3Tj/3f44+UOyZmyv0Q2Zc4SFzMqHCr805w7XY96ZjM8LrCdVJUfr75+GE3A7nho4IN1hR7A1Nnxfpg6fUV5/gaFHaqKXDpYVfkrbiBcE5JzmPHYv0kFe+sOtgFaTvSOe1stwLQf8RhCa4/JR/kMXrunNEnWS32ydHhgwr7izEXpxf4vgv0W1q9OZ9oq/ao1zyv7+yUK4GsK6ETMZl9K2ds7OFmTtmkqaSx+oOb53LMnA2qpid7sWEwh5IkXedo3lHWppP7dXoCd5oDNyPiJ6hYmTAKzKh9kehuf61YhZk2ocevtvz/KgsHDnp537SsFORzm6rlhyKtKNHN5TOEmAiEotEEMNF/qOPciknmCl+yWEpJMQtEutxG/45/y2rvQrTBBWSs6p5mCaolxuv4cU+AXIQ+trhBVQyyE8lDu0TSyb0wZfttb45HZ/WrQOah0dYIHfCRAcMXb60N+aCSy5svY/jdZG1oSh+uPpKiEWv4eOsHaLfaR3yzBsrTbu72b8wAgvAvqGQvC5idr1DAq3ezCrVlCOHyxq4CK8dxWbNtp8WBO9kO3u7pD9JZCjfYCFOgiT6QEcVbqE9uTXiDTkc588OCJpu1m810skGLXPZfTjj3123xQhgsRdErQkijtUgMFpTNSGxqF2KhlHf6IqsyxLwjg=="
? ? }
? }
}
PostgreSQL StatefulSet
--
apiVersion: v1
kind: Service
metadata:
? name: postgres-svc
? namespace: sso
spec:
? type: ClusterIP
? selector:
? ? app: sso
? ? role: backend
? ports:
? - port: 5432
? ? targetPort: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
? name: postgres
? labels:
? ? app: sso
? ? role: backend
? namespace: sso
? annotations:
? ? kubernetes.io/change-cause: "Deploying Postgres Latest Version"
spec:
? serviceName: postgres-svc
? replicas: 1
? selector:
? ? matchLabels:
? ? ? app: sso
? ? ? role: backend
? template:
? ? metadata:
? ? ? labels:
? ? ? ? app: sso
? ? ? ? role: backend
? ? spec:
? ? ? restartPolicy: Always
? ? ? containers:
? ? ? - name: postgres
? ? ? ? image: postgres:latest
? ? ? ? imagePullPolicy: IfNotPresent
? ? ? ? env:
? ? ? ? - name: POSTGRES_USER
? ? ? ? ? valueFrom:
? ? ? ? ? ? secretKeyRef:
? ? ? ? ? ? ? name: postgres-secret
? ? ? ? ? ? ? key: db-user
? ? ? ? - name: POSTGRES_PASSWORD
? ? ? ? ? valueFrom:
? ? ? ? ? ? secretKeyRef:
? ? ? ? ? ? ? name: postgres-secret
? ? ? ? ? ? ? key: db-password
? ? ? ? - name: POSTGRES_DB
? ? ? ? ? valueFrom:
? ? ? ? ? ? secretKeyRef:
? ? ? ? ? ? ? name: postgres-secret
? ? ? ? ? ? ? key: db-name
? ? ? ? ports:
? ? ? ? - containerPort: 5432
? ? ? ? volumeMounts:
? ? ? ? - name: db-data
? ? ? ? ? mountPath: /var/lib/postgresql/data
? ? ? ? ? subPath: data
? volumeClaimTemplates:
? - metadata:
? ? ? name: db-data
? ? spec:
? ? ? storageClassName: nfs-client
? ? ? resources:
? ? ? ? requests:
? ? ? ? ? storage: 1Gi
? ? ? accessModes:
? ? ? - ReadWriteOnce
Ensure all the workloads are fine in sso namespace.
* Keycloak deployment
Keycloak Secret after sealed by Kubeseal
{
? "kind": "SealedSecret",
? "apiVersion": "bitnami.com/v1alpha1",
? "metadata": {
? ? "name": "keycloak-secret",
? ? "namespace": "sso",
? ? "creationTimestamp": null
? },
? "spec": {
? ? "template": {
? ? ? "metadata": {
? ? ? ? "name": "keycloak-secret",
? ? ? ? "namespace": "sso",
? ? ? ? "creationTimestamp": null,
? ? ? ? "labels": {
? ? ? ? ? "app": "sso",
? ? ? ? ? "role": "frontend"
? ? ? ? }
? ? ? },
? ? ? "data": null
? ? },
? ? "encryptedData": {
? ? ? "keycloak-db-password": "AgBsjKXF7ouCX3vdCePdvKegHDaZZdL25aO8GVlTyxdNVxOgRLGf2gsWmrn/uP3iP2/M0nS9H1p1qlYeICQEdWK6skFP1u2Z8bZp8Ntt57TuIAivNCb7NlV9ienm95m0Z09aG2dHEYUiZ4sIUAW1aAAQN96oExnxjfSXw1tJEjk7x75za5Zw4Jr7tfT6X9cskV0NQ9zDM+LAXB1KqemxnrMO9TU3TeBYsrSPFNYw+Yehf9bmnLmijsKpE6WBgHMf7cdajXdcUZA2/QnTfZYBfyUixScqOx+zx4U9TZaL+waYLHLpGaXmiVoErlNpQb339YpSAIzKOHtjrtFCvOBZKRR7z5citO9kDDNCLNTPnh02KmIKgTJaAe7sporVpp35hY+kj/oF7cICR0z9/+ZdqHfs797Z2DII4tTlWUqk5+xPNZP1rf9SKtAM5tkEEno1hIznlv/oum8pFWlo06Vj7QPt716or75SPzLR0cGSLaALK8Go0iFw9QumrK2/2VN7gsbAOebobnXxD9sZHoCq+oPQXVuxExb+xlT8MMsT81wZAOsnFiwYWGTxeShfqTJamAMFLl8x5ZMUU6JFhk8Zseqj6z5hZCT5G/jeat0aXojzeHEdrMOzkTi7ItD9QdB/aQV7pWqB19hy9Za/8QvHjdkJiHcD5NRrstV2MRpVsUU0WAyiAisXG2at4OrhpU43lPV8ajjwvu8h8w==",
? ? ? "keycloak-db-user": "AgCk7/ey1AXDKDTxJ/AucbKw7PJ7fu5vEkZNbJdCGZ6myFN9O+hDvqTmW9uNY0+YuCOJ7hpo/cXBlyWugxypKICHs3vXHwhAFggwKRtZn+2kkfbtixfSTcGVjqEIeobqbJv+zrOL9nuWUjJ8zr57DZ2Yd0J3ONGf+quLItGF6kwI8TNIQ878Y0Ab7h1+7XJRC8Wc7vsa9Qg/M5DEB6E/a5De7cDT1zc6SDnWhn/nGU/h2qGU3kzdZ8tOc/3UxD2yROxquXZIAEn2T08zy6o5S8mBx0k++B/6s2921zqjt2xO1zxdXTcIqBp5knW87+akNIesz1FDRnG0XCVl/HDrv8VijtR4Iiw6YVvJqlRnDOE5BQP+uBtLuKpWnhabOGor7VoTeLZmJ/OumL1kbnmDCmDxaE9iJh3Cu53LfV2PcJIyLnTx/ztmJxtqvpStc+bbrA99J6f1A80dXk1Fj1wR92V3ahZrO0kWILzh0RtBf17d/Fd98wlyIqezO1wpe1+b+tKWZjbuRgvh34bHT/YAc8HSMyYSyagzbHO4HBgClRp+7QiOUBsU35C9QlVAtS+85limyX3Fxka2Xlxxt53N7SXZvwKbex7AdCksmhtWJ4HffrKO3Zx754T8PFEv08bwVe1f0mszbHhkD9iqRSLXDPX0uD654m6LHPdOOAdB80/8diBFCAO3eQu6Lb6LvTLmRXcLdwIkgoqLQA==",
? ? ? "keycloak-webui-password": "AgAUYVsVSDuotJefKiPDjWe8V8+nXEwtEqwWYOXYLZ4eAdK3LyOx75VQhXIOapQh0j+nZ5miBcji4XH/wy2DMJnddP1STwFZx9UN0ltnMv70PVbYnl6pBuZuhnn6mDWvPovnXlgT4yAxHGY6qA/ToyV7Cm8+dB5XW8x3FFf88fdsBf1Qnvp81tS36EIdp8EHL500Suf13F754Ey39RKmgKpopNkB9Pl+BarYzEYgy3eh5jxU+Huh1V3DBagxI5jJyoiAWVxUOgpUcaBv+eaUQUIBSCOWN1mds5VSbaRbUkL2NisVLzsh74uM8xGYSDl7eFV02lTMX23+pRgg4/sAGYpv7rkfEY17PJZsAzdS13SFVkDKSgJQB3B26X4diPInzmLe1NoEGjdyHnG289jmDMBgYJPHK75+Wrnoxfz9F+OwrXQk+37h0/msEtBQqh9JO2lbN2qOwFsO2R3Xt27yPii2+5hStpVZ0DFt1oWe7Hfpr+uBcQtiTwBDMZqYTO5RZ1fmSJOKAmc+FRVZbNKphyBy2OQI9Esm+ujfamD8b2cv+ZEUSNCAT4xGX3U6TBUyr0nekQn/GyM9p95LB0Joq+UwqJIIJAuB2oHIJ/b7FvI+8rnScb6sCqU/XS7/r5NbtNdsAXl4N6pFDM987mRwhEAMBWjkyMndiob3LoIbLmcwiXB3S3r2ofe18fd5b1AnGYrYFgrCNQ==",
? ? ? "keycloak-webui-user": "AgAm4gcZyX3E+aqY4qf90SwYrFFO5lDl5/sUwkZA5c1zzmeN7xD2c0VIvBWmg6DodRnkW9RmvIT8vgwX3441lG7DYAN47M1Gqz2hjltO0k45ktPNBfr1vsaie2VMuY9jQ2Ik1cmI9LL2AOpLOQyu/9ghxEBziWC96MR2YPqUGN0FBNZWJsqXPsvKvsgnd3JutOfU0vge/PyQXuIR05FmfhjtgarCsIyGQJxbBMXqpZdZCR1zz2FgcH7Bw6wsbfhvEs93GP2m/iRWZnQkL6Y90eF24EOkbGjBUWTYwLJVq+kphnaApL5UDDjHsvpdWFs7Kf1WpzLiLCGuz2dXV4pgRDkcoVaNR6BXdTcRqrnc2NylTh1MvMQ34yNSum/mSV/oTfOzBnhF+MXsvrlk0UYSMFfxy0kGV/QOJ0wIWbO6UzjnA1lahFzWupmo2ak+HEnrcTmywmc1XgEJBUUu+9jHkuhMFY7hh9f4eVRV0qpCpe+snVYOUwfOEjZtaF9vbUKyXHWGar9a+cmeDSDYzzn1cKAvL+1StM/r0Hz8qJuHaDykG+qOB1Vg/xrq33EgphRNfsccA+yRERMPuvtaxRTptGR58qvBfDNFH6hR+GtuzAfqK8THglQTdyIKFBxmwDlZ80h+r4HwQ1/hFLeBOVL6ECClwAuMG9UlrTnZiCBoMENR3wwEjAfT1tbWoCNr3R730507iojg4Q=="
? ? }
? }
}
Deployment
--
apiVersion: apps/v1
kind: Deployment
metadata:
? name: keycloak
? namespace: sso
? labels:
? ? app: sso
? ? role: frontend
? annotations:
? ? kubernetes.io/change-cause: "Deploying Keycloak Latest Version"
? ? kubernetes.io/Revision: "10"
spec:
? replicas: 2
? strategy:
? ? type: RollingUpdate
? ? rollingUpdate:
? ? ? maxUnavailable: 1
? selector:
? ? matchLabels:
? ? ? app: sso
? ? ? role: frontend
? template:
? ? metadata:
? ? ? labels:
? ? ? ? app: sso
? ? ? ? role: frontend
? ? spec:
? ? ? serviceAccount: keycloak-kubeping-service-account
? ? ? serviceAccountName: keycloak-kubeping-service-account
### Pod replicas will schedule differnet nodes using Pod-Anti-Affinity for Keycloak HA ###
? ? ? affinity:
? ? ? ? podAntiAffinity:
? ? ? ? ? preferredDuringSchedulingIgnoredDuringExecution:
? ? ? ? ? - weight: 100
? ? ? ? ? ? podAffinityTerm:
? ? ? ? ? ? ? labelSelector:
? ? ? ? ? ? ? ? matchExpressions:
? ? ? ? ? ? ? ? - key: app
? ? ? ? ? ? ? ? ? operator: In
? ? ? ? ? ? ? ? ? values:
? ? ? ? ? ? ? ? ? - sso
? ? ? ? ? ? ? topologyKey: kubernetes.io/hostname
? ? ? restartPolicy: Always
? ? ? containers:
? ? ? - name: keycloak
? ? ? ? image: jboss/keycloak:latest
? ? ? ? imagePullPolicy: IfNotPresent
#? ? ? ? resources:
#? ? ? ? ? limits:
#? ? ? ? ? ? cpu: 1000m
#? ? ? ? ? ? memory: 4096Mi
#? ? ? ? ? requests:
#? ? ? ? ? ? cpu: 500m
#? ? ? ? ? ? memory: 2048M
? ? ? ? env:
######Keycloak WEB UI Login And DB Login ENV Values From Secrets#########
? ? ? ? - name: KEYCLOAK_USER
? ? ? ? ? valueFrom:
? ? ? ? ? ? secretKeyRef:
? ? ? ? ? ? ? name: keycloak-secret
? ? ? ? ? ? ? key: keycloak-webui-user
? ? ? ? - name: KEYCLOAK_PASSWORD
? ? ? ? ? valueFrom:
? ? ? ? ? ? secretKeyRef:
? ? ? ? ? ? ? name: keycloak-secret
? ? ? ? ? ? ? key: keycloak-webui-password
? ? ? ? - name: DB_USER
? ? ? ? ? valueFrom:
? ? ? ? ? ? secretKeyRef:
? ? ? ? ? ? ? name: keycloak-secret
? ? ? ? ? ? ? key: keycloak-db-user
? ? ? ? - name: DB_PASSWORD
? ? ? ? ? valueFrom:
? ? ? ? ? ? secretKeyRef:
? ? ? ? ? ? ? name: keycloak-secret
? ? ? ? ? ? ? key: keycloak-db-password
######Common Keycloak Envrionment Values #########
? ? ? ? - name: PROXY_ADDRESS_FORWARDING
? ? ? ? ? value: "true"
? ? ? ? - name: DB_VENDOR
? ? ? ? ? value: "postgres"
? ? ? ? - name: DB_ADDR
? ? ? ? ? value: "postgres-svc"
? ? ? ? - name: DB_PORT
? ? ? ? ? value: "5432"
? ? ? ? - name: DB_DATABASE
? ? ? ? ? value: "keycloak"
? ? ? ? - name: KEYCLOAK_HTTP_PORT
? ? ? ? ? value: "80"
? ? ? ? - name: KEYCLOAK_HTTPS_PORT
? ? ? ? ? value: "443"
? ? ? ? - name: JGROUPS_DISCOVERY_PROTOCOL
? ? ? ? ? value: "kubernetes.KUBE_PING"
? ? ? ? - name: JGROUPS_DISCOVERY_PROPERTIES
? ? ? ? ? value: "dump_requests=false,namespace=sso"
? ? ? ? - name: CACHE_OWNERS_COUNT
? ? ? ? ? value: "2"
? ? ? ? - name: CACHE_OWNERS_AUTH_SESSIONS_COUNT
? ? ? ? ? value: "2"
? ? ? ? ports:
? ? ? ? - name: http
? ? ? ? ? containerPort: 8080
? ? ? ? - name: https
? ? ? ? ? containerPort: 8443
? ? ? ? readinessProbe:
? ? ? ? ? httpGet:
? ? ? ? ? ? path: /auth/realms/master
? ? ? ? ? ? port: 8080
---
apiVersion: v1
kind: Service
metadata:
? name: keycloak-svc
? namespace: sso
spec:
? selector:
? ? app: sso
? ? role: frontend
? type: ClusterIP
? ports:
? - port: 8080
? ? targetPort: 8080
* Deploying Keycloak secret, deployment and service
Here above image we can see keycloak container as replicated 2 as per deployment and the important thing is each container is running different nodes using POD-ANTI-AFFINITY. If one node having any issue or maintenance. Other one will be available to serve the connection. So high availability is implemented on node level.
领英推荐
But another important thing is we need to verify the both container is setup as clustering like each pod have sync up with between the data's and cache using JGroups discovery protocol like KUBE_PING and infinispan conecpts.
Let's verify the logs one by one.
JGroups Discovery setup and enable cache logs
Infinispan Full Logs
17:02:49,787 INFO? [org.jgroups.protocols.kubernetes.KUBE_PING] (ServerService Thread Pool -- 56) namespace sso set; clustering enable
17:02:54,201 INFO? [org.infinispan.CONTAINER] (ServerService Thread Pool -- 58) ISPN000128: Infinispan version: Infinispan 'Taedonggang' 12.1.7.Final
17:02:54,205 INFO? [org.infinispan.CONTAINER] (ServerService Thread Pool -- 59) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.spi.marshalling.InfinispanProtoStreamMarshaller'
17:02:54,206 INFO? [org.infinispan.CONTAINER] (ServerService Thread Pool -- 56) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.marshalling.jboss.JBossMarshaller'
17:02:54,203 INFO? [org.infinispan.CONTAINER] (ServerService Thread Pool -- 60) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.spi.marshalling.InfinispanProtoStreamMarshaller'
17:02:54,209 INFO? [org.infinispan.CONTAINER] (ServerService Thread Pool -- 58) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.spi.marshalling.InfinispanProtoStreamMarshaller'
17:02:54,299 INFO? [org.infinispan.CONTAINER] (ServerService Thread Pool -- 57) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.marshalling.jboss.JBossMarshaller'
17:02:54,511 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 59) ISPN000078: Starting JGroups channel ejb
17:02:54,532 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 60) ISPN000078: Starting JGroups channel ejb
17:02:54,533 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 58) ISPN000078: Starting JGroups channel ejb
17:02:54,534 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 56) ISPN000078: Starting JGroups channel ejb
17:02:54,512 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 57) ISPN000078: Starting JGroups channel ejb
17:02:54,541 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 56) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-dp9k9|0] (1) [keycloak-69c658c867-dp9k9]
17:02:54,552 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 57) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-dp9k9|0] (1) [keycloak-69c658c867-dp9k9]
17:02:54,553 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 59) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-dp9k9|0] (1) [keycloak-69c658c867-dp9k9]
17:02:54,561 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 56) ISPN000079: Channel ejb local address is keycloak-69c658c867-dp9k9, physical addresses are [10.244.8.7:7600]
17:02:54,563 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 58) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-dp9k9|0] (1) [keycloak-69c658c867-dp9k9]
17:02:54,565 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 60) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-dp9k9|0] (1) [keycloak-69c658c867-dp9k9]
17:02:54,572 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 59) ISPN000079: Channel ejb local address is keycloak-69c658c867-dp9k9, physical addresses are [10.244.8.7:7600]
17:02:54,577 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 57) ISPN000079: Channel ejb local address is keycloak-69c658c867-dp9k9, physical addresses are [10.244.8.7:7600]
17:02:54,581 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 60) ISPN000079: Channel ejb local address is keycloak-69c658c867-dp9k9, physical addresses are [10.244.8.7:7600]
17:02:54,624 INFO? [org.infinispan.CLUSTER] (ServerService Thread Pool -- 58) ISPN000079: Channel ejb local address is keycloak-69c658c867-dp9k9, physical addresses are [10.244.8.7:7600]
17:02:55,206 INFO? [org.infinispan.CLUSTER] (thread-5,null,keycloak-69c658c867-dp9k9) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-dp9k9|1] (2) [keycloak-69c658c867-dp9k9, keycloak-69c658c867-78kc7]
17:02:55,210 INFO? [org.infinispan.CLUSTER] (thread-5,null,keycloak-69c658c867-dp9k9) ISPN100000: Node keycloak-69c658c867-78kc7 joined the cluster
17:02:55,211 INFO? [org.infinispan.CLUSTER] (thread-5,null,keycloak-69c658c867-dp9k9) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-dp9k9|1] (2) [keycloak-69c658c867-dp9k9, keycloak-69c658c867-78kc7]
17:02:55,211 INFO? [org.infinispan.CLUSTER] (thread-5,null,keycloak-69c658c867-dp9k9) ISPN100000: Node keycloak-69c658c867-78kc7 joined the cluster
17:02:55,211 INFO? [org.infinispan.CLUSTER] (thread-5,null,keycloak-69c658c867-dp9k9) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-dp9k9|1] (2) [keycloak-69c658c867-dp9k9, keycloak-69c658c867-78kc7]
17:02:55,212 INFO? [org.infinispan.CLUSTER] (thread-5,null,keycloak-69c658c867-dp9k9) ISPN100000: Node keycloak-69c658c867-78kc7 joined the cluster
17:02:55,212 INFO? [org.infinispan.CLUSTER] (thread-5,null,keycloak-69c658c867-dp9k9) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-dp9k9|1] (2) [keycloak-69c658c867-dp9k9, keycloak-69c658c867-78kc7]
17:02:55,212 INFO? [org.infinispan.CLUSTER] (thread-5,null,keycloak-69c658c867-dp9k9) ISPN100000: Node keycloak-69c658c867-78kc7 joined the cluster
17:02:55,212 INFO? [org.infinispan.CLUSTER] (thread-5,null,keycloak-69c658c867-dp9k9) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-dp9k9|1] (2) [keycloak-69c658c867-dp9k9, keycloak-69c658c867-78kc7]
17:02:55,212 INFO? [org.infinispan.CLUSTER] (thread-5,null,keycloak-69c658c867-dp9k9) ISPN100000: Node keycloak-69c658c867-78kc7 joined the cluster
17:02:55,361 INFO? [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 58) WFLYCLINF0002: Started work cache from keycloak container
17:02:55,453 INFO? [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 56) WFLYCLINF0002: Started offlineSessions cache from keycloak container
17:02:55,457 INFO? [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 61) WFLYCLINF0002: Started authenticationSessions cache from keycloak container
17:02:55,461 INFO? [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 62) WFLYCLINF0002: Started offlineClientSessions cache from keycloak container
17:02:55,477 INFO? [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 59) WFLYCLINF0002: Started clientSessions cache from keycloak container
17:02:55,480 INFO? [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 57) WFLYCLINF0002: Started keys cache from keycloak container
17:02:55,482 INFO? [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0002: Started loginFailures cache from keycloak container
17:02:55,484 INFO? [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 66) WFLYCLINF0002: Started realms cache from keycloak container
17:02:55,485 INFO? [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 67) WFLYCLINF0002: Started authorization cache from keycloak container
17:02:55,487 INFO? [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 60) WFLYCLINF0002: Started users cache from keycloak container
17:02:55,495 INFO? [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 63) WFLYCLINF0002: Started sessions cache from keycloak container
17:02:55,499 INFO? [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) WFLYCLINF0002: Started actionTokens cache from keycloak container
Infinispan logs joining cluster one container to another
Here above image we can see starting of logs it's enabled the infinispan clustering using JGroups protocols ( kubernetes.KUBE_PING ) and namespace for ( sso ) and also container ( keycloak-69c658c867-dp9k9 ) joined to another container ( keycloak-69c658c867-78kc7 ) as cluster.
* Rebalance logs read and write between two containers. like context keys cache, users cache, sessions cache, realms, authorization, authenticationSessions, clientSessions, loginFailures, offlineSessions, actionTokens, etc..
* Nginx Ingress Rules
* Let's deploy the Ingress rule for Keycloak and will verify the high availability with practically.
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
? name: keycloak-ingress
? namespace: sso
? annotations:
? ? kubernetes.io/ingress.class: nginx
? ? nginx.ingress.kubernetes.io/rewrite-target: /
spec:
#? tls:
#? - hosts:
#? ? - sso.local.com
#? ? secretName: keycloak-tls
? rules:
? - host: sso.local.com
? ? http:
? ? ? paths:
? ? ? - path: /
? ? ? ? pathType: Prefix
? ? ? ? backend:
? ? ? ? ? service:
? ? ? ? ? ? name: keycloak-svc
? ? ? ? ? ? port:
? ? ? ? ? ? ? number: 8080
* Verify the Keycloak Ingress as deployed
* Let's verify the web UI
Need to add the local host entry in /etc/hosts file like ( 192.168.0.180 sso.local.com ) and check the url in browser.
After login Keycloak UI
And also we can see Nginx Ingress controller logs both container having serve connections now.
* Verify High Availability
* Finally now we are going to practically test the high availability of the keycloak application. We can do one thing like drain the K8s worker node1 and we can see still able to access the application and main thing is users caches, session caches.
* Attempt 1:
After draining the worker node1 (k8sworker1). We can see keycloak container ( keycloak-69c658c867-78kc7 ) is evicted and also other containers ( MetalLB controller & Sealed Secret Controller ) is evicted.
Logs of worker node2 (k8sworker2) container ( keycloak-69c658c867-dp9k9 ). We can see container ( keycloak-69c658c867-78kc7 ) left from keycloak cluster due to we drained the worker node1 (k8sworker1).
When I check keycloak web UI is not working ??. and it's showing unable to connect. Because of when ever node1 drained it's evicted also MetaLB controller container due to that Nginx Ingress Controller also not ready status.
We can see below image ( keycloak-69c658c867-dp9k9 ) and postgreSQL container is running fine but MetalLB Controller and Nginx Ingress Controller is not ready status.
Based on this first attempt we learned before drain the nodes ensure all other dependencies containers are not running on that node. And also we have setup for this demo 1Master and 2worker nodes.
* Attempt 2:
For this time I was cordon on worker node1 on start of this setup because of when we deploying depending resources like ( Dynamic NFS Provisioning, Kubeseal, MetalLB, Nginx Ingress Controller, PostgreSQL ). So this will prevent any containers will not scheduled on worker node1.
Below image we can see all depending resources containers are running on worker node2.
This time we have keycloak container ( keycloak-69c658c867-hcfkt ) is running on worker node1 (k8sworker1) and ( keycloak-69c658c867-sjcww ) is running on worker node2 (k8sworker2).
* Infinispan Clustering Logs
* Let's drain worker node1
For this time we can see keycloak container is only evicted from worker node1(k8sworker1).
And also we can see container ( keycloak-69c658c867-sjcww ) logs. container ( keycloak-69c658c867-hcfkt ) is left from the cluster.
Below logs we can see there is new keycloak container scheduled on worker node2 (k8sworker2) and it's joined to the cluster.
But unfortunately this time also keycloak UI is not working ????. Due to MetalLB controller container is having issue, So Nginx Ingress controller container also not route the traffic to backend keycloak service.
* Attempt 3:
I hope this time will works. In Kubernetes cluster setup added one more worker nodes. worker node3 (k8sworker3)
So we followed the same steps to setup all other prerequisites like ( Dynamic NFS Provisioning, Kubeseal, MetalLB, Nginx Ingress Controller ).
Let's try this time,
Below image we can see keycloak container ( keycloak-69c658c867-2w6d6 ) is running on worker node1(k8sworker1) and container ( keycloak-69c658c867-wj6fd ) is running on worker node2 (k8sworker2).
Infinispan Cluster Logs before testing both container in cluster setup
Before drain the nodes we can verify what are the pods are running on each nodes.
Let drain the worker node1 (k8sworker1) because our keycloak container ( keycloak-69c658c867-2w6d6 ) is running on node1 only.
Below image we can see keycloak container ( keycloak-69c658c867-2w6d6 ) is evicted. As well Sealed secret controller and Nginx Ingress controller containers.
Below image we can see clearly after worker node1 (k8sworker1) is in drained state and the new keycloak container is running and ready on worker node3 (k8sworker3).
* Finally Keycloak HA Clustering works on third attempt !!! ??????
We can see the old container ( keycloak-69c658c867-2w6d6 ) is left from the cluster and new keycloak container ( keycloak-69c658c867-cbl2f ) is scheduled on worker node3 (k8sworker3) and it's joined to the keycloak cluster.
* Infinispan clustering full logs
08:13:33,461 INFO? [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on https://127.0.0.1:9990/managemen
08:13:33,461 INFO? [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on https://127.0.0.1:9990
08:47:06,083 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) [Context=sessions] ISPN100008: Updating cache members list [keycloak-69c658c867-wj6fd], topology id 6
08:47:06,150 INFO? [org.infinispan.CLUSTER] (thread-62,ejb,keycloak-69c658c867-wj6fd) [Context=authenticationSessions] ISPN100008: Updating cache members list [keycloak-69c658c867-wj6fd], topology id 6
08:47:06,151 INFO? [org.infinispan.CLUSTER] (thread-64,ejb,keycloak-69c658c867-wj6fd) [Context=offlineSessions] ISPN100008: Updating cache members list [keycloak-69c658c867-wj6fd], topology id 6
08:47:06,154 INFO? [org.infinispan.CLUSTER] (thread-62,ejb,keycloak-69c658c867-wj6fd) [Context=actionTokens] ISPN100008: Updating cache members list [keycloak-69c658c867-wj6fd], topology id 6
08:47:06,155 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) [Context=work] ISPN100008: Updating cache members list [keycloak-69c658c867-wj6fd], topology id 6
08:47:06,172 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) [Context=loginFailures] ISPN100008: Updating cache members list [keycloak-69c658c867-wj6fd], topology id 6
08:47:06,182 INFO? [org.infinispan.CLUSTER] (thread-62,ejb,keycloak-69c658c867-wj6fd) [Context=offlineClientSessions] ISPN100008: Updating cache members list [keycloak-69c658c867-wj6fd], topology id 6
08:47:06,184 INFO? [org.infinispan.CLUSTER] (thread-62,ejb,keycloak-69c658c867-wj6fd) [Context=clientSessions] ISPN100008: Updating cache members list [keycloak-69c658c867-wj6fd], topology id 6
08:47:06,288 INFO? [org.infinispan.CLUSTER] (thread-62,ejb,keycloak-69c658c867-wj6fd) [Context=org.infinispan.CONFIG] ISPN100008: Updating cache members list [keycloak-69c658c867-wj6fd], topology id 6
08:47:06,346 INFO? [org.infinispan.CLUSTER] (thread-62,ejb,keycloak-69c658c867-wj6fd) [Context=http-remoting-connector] ISPN100008: Updating cache members list [keycloak-69c658c867-wj6fd], topology id 6
08:47:06,384 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-wj6fd|2] (1) [keycloak-69c658c867-wj6fd]
08:47:06,385 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN100001: Node keycloak-69c658c867-2w6d6 left the cluster
08:47:06,385 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-wj6fd|2] (1) [keycloak-69c658c867-wj6fd]
08:47:06,385 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN100001: Node keycloak-69c658c867-2w6d6 left the cluster
08:47:06,392 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-wj6fd|2] (1) [keycloak-69c658c867-wj6fd]
08:47:06,392 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN100001: Node keycloak-69c658c867-2w6d6 left the cluster
08:47:06,392 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-wj6fd|2] (1) [keycloak-69c658c867-wj6fd]
08:47:06,393 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN100001: Node keycloak-69c658c867-2w6d6 left the cluster
08:47:06,393 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-wj6fd|2] (1) [keycloak-69c658c867-wj6fd]
08:47:06,393 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN100001: Node keycloak-69c658c867-2w6d6 left the cluster
08:47:58,619 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-wj6fd|3] (2) [keycloak-69c658c867-wj6fd, keycloak-69c658c867-cbl2f]
08:47:58,620 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN100000: Node keycloak-69c658c867-cbl2f joined the cluster
08:47:58,620 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-wj6fd|3] (2) [keycloak-69c658c867-wj6fd, keycloak-69c658c867-cbl2f]
08:47:58,621 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN100000: Node keycloak-69c658c867-cbl2f joined the cluster
08:47:58,621 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-wj6fd|3] (2) [keycloak-69c658c867-wj6fd, keycloak-69c658c867-cbl2f]
08:47:58,621 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN100000: Node keycloak-69c658c867-cbl2f joined the cluster
08:47:58,621 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-wj6fd|3] (2) [keycloak-69c658c867-wj6fd, keycloak-69c658c867-cbl2f]
08:47:58,622 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN100000: Node keycloak-69c658c867-cbl2f joined the cluster
08:47:58,622 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN000094: Received new cluster view for channel ejb: [keycloak-69c658c867-wj6fd|3] (2) [keycloak-69c658c867-wj6fd, keycloak-69c658c867-cbl2f]
08:47:58,622 INFO? [org.infinispan.CLUSTER] (thread-57,ejb,keycloak-69c658c867-wj6fd) ISPN100000: Node keycloak-69c658c867-cbl2f joined the cluster
As Verified the keycloak web UI. I can access without any issues. And main important thing is my user session is still available and I can use the application with same user cache sessions.
* On first and second attempt for this Keycloak HA cluster testing having issues after drained the worker node1. Because of we don't have resources for evicted pods from that drained node1 to schedule on any other nodes. And also other pods are having issue for scheduling to available worker node2. And we used prerequisites for Kubernetes cluster is 1 Master & 2 worker nodes .
And also after drained the worker node1 in Kubernetes cluster level. we can see some of pods are not comping up as well as container continuously restarting in first and second attempt. But in third attempt like after added one more worker nodes in Kubernetes cluster level. All pods are up and running fine.
* Conclusion
Based on the third attempt we understand the prerequisites for Kubernetes cluster part. We need minimum three worker nodes for Keycloak HA clustering setup. Then only if having any issues on any of the worker nodes. Pods will schedule on another healthy nodes.
############################ Happy Learning ##############################