Friday fun - Impersonation (in a good way)
All of us know that impersonation - the assumption of another person's identity, be it for good or bad - is not a good idea. In most cases, identity impersonation is with bad intent. Hence, impersonation is a crime.
But, in few scenarios, impersonation is required.
In December 2023, our customer data platform went into production using Databricks. At that point in time, things we a bit liberal on the QA environment. Liberal in who was allowed to execute code on the QA environment. As it was our first release, all developers had required permissions on the QA environment. Due to the frenzy of development, testing and going live, developers executed their notebooks on the QA environment using their permissions - at least for the first few notebooks. Then we decided to streamline the process and execute everything with the permissions of one person - yours truly. We created jobs using my credentials.
After sufficient testing on QA, we promoted the code to Prod. This time, we ensured that everything was executed using Service Principal credentials.
Six months after the move to production and a couple of releases later, we were in a situation where we faced team churn. A few project members decided to leave. While their access was revoked from the system, it still retained objects in their name. If we had to make a change to that object, we had to inform the administrator to change permissions in the best case or drop and re-create in the worst case.
Over period of time, we changed permissions of all objects over to Service Principal - tables and job definitions. A side effect of this positive change is that we are not able to know the user who performed the action - irrespective the action is good or bad. This is because the job executes as Service Principal.
In summary, impersonation in the Databricks environment was beneficial as we could get away from maintenance headaches.
#databricks #data_engineering #impersonation #service_principal