Cluster Scoped Init Scripts

Managing cluster configurations and custom installations in Databricks can be streamlined and more reliable by favoring Cluster Scoped Init Scripts over global or named scripts. Here's why:

Init Script Modes:

• Global: Placed in the /databricks/init folder, these scripts run on every cluster creation or restart, affecting all users in the workspace.
• Cluster Named (Deprecated): Limited to specific clusters by placing them in /databricks/init/<cluster-name>/.
• Cluster Scoped: Specified in the cluster’s configuration, either written directly in the UI or stored in DBFS (excluding /databricks/init), providing greater flexibility and control.

Why Prefer Cluster Scoped Init Scripts?

1. Targeted Execution: Unlike global scripts, cluster scoped scripts are specified per cluster, minimizing unintended impacts on other clusters.
2. Enhanced Debugging: Specifying scripts in the cluster configuration makes it easier to identify them during troubleshooting, reducing the chances of overlooked errors.
3. Reduced Errors: Scripts run on each cluster node, and success depends on timely, error-free execution. Cluster scoped scripts help manage this complexity better than global scripts.
4. Custom Control: Offers greater control over where and how scripts are executed, making it easier to manage different configurations for different workloads.

Best Practices:

• Use Cluster Scoped Init Scripts: Whenever possible, specify init scripts in the cluster configuration to avoid the pitfalls of automatic script execution.
• Manage Cluster Logs Effectively:

- Default DBFS Location: Logs are sent to the default DBFS with a 30-day retention policy.

- Custom Blob Store: For extended retention and better control, use the Cluster Log Delivery feature to send logs to a blob store within your subscription.

Why Send Logs to a Custom Blob Store?

1. Extended Retention: Avoid the default 30-day purging policy, ensuring compliance with longer retention requirements.
2. Integration Flexibility: Logs in your storage account can be easily integrated with other log analytics tools, unlike those in the default DBFS protected by Azure’s read lock.

By adopting these practices, you can enhance the reliability, manageability, and compliance of your Databricks environments. Embrace the flexibility and control offered by Cluster Scoped Init Scripts and ensure your cluster logs are securely managed.