Reload secrets in Kubernetes

Nowadays, we see how widely Kubernetes is used as an efficient and automated way to manage containerized applications in production environments. In these dynamic, cloud-native environments, managing secrets (like API keys, tokens, or credentials) securely is a cornerstone of application security. Saving these as Kubernetes secrets is indeed a very secured way. But, recently I encountered a use case where we needed to reload these secrets in run time. Due to some admin or superuser requests, such secrets needed to be updated in the secret store, and we needed to ensure that our application pods now use these updated values. Much to my surprise, this became a challenge in itself. Hence, I decided to pen down my experience and possibly discuss a viable solution.

1.?????? Pod Restart:

To refresh secrets inside kubernetes, one way is to restart the pod. Once, the new pod comes up it will refer the updated secrets. But, this would mean downtime for my application.

2.?????? Rolling Update:

Another way would be a rolling update. But again, it is not an efficient approach.

3.?????? Kubernetes watcher:

?Kubernetes does have a native approach to this. There is a utility called kubernetes watcher which can be configured to watch any change in secrets and can then update the context inside the app so that it refers to the updated version of secrets.

a.?????? This is also called sidecar container approach where an additional container keeps running in a background and checks for any update to secrets and then sends an event to the app to refer to updated secrets.

b.????? However, in this case, the application needs to implement some additional logic. Say for eg. If it is a Spring Boot app, it will need “@RefreshScope” annotation to listen to “/refresh” endpoint to refresh the application context on secret update.

c.?????? This particular approach may not be suitable to every application. Like in our case, refreshing the application context was not a convenient approach as it was coinciding with one of the responsibilities of the app.

Solution:

Since, none of the above approaches were suitable, I implemented something similar to the 3rd solution above but without a sidecar. When I read the working of Kubernetes watcher, I found that for my case, I was going for the overkill.

1.?????? First, I mounted the secrets as a volume.

2.?????? Then, wrote a simple utility method to check the secret file mounted on volume for any change every few seconds which was configurable. I used “@scheduled” annotation for this.

3.?????? And, then the utility would call the normal application’s logic of referring the updated secrets.

Suddenly, the whole problem statement became very trivial in terms of solution. Needless to say I did waste a lot of time to come to this conclusion lol.