FinOps & Databricks (Episode 1)

With the rise of SAAS Data plarforms users and/or admins are authorized to create on demand ressources (compute / RAM, GPU) which is great but may lead to situations where costs run out of control???

This is why it is important to proactively monitor costs and this is called "FinOps"

FinOps

FinOps (short for Financial Operations) is a management practice for managing cloud finances, but it also emphasizes a cultural change within the organization that promotes collaboration between finance, engineering, and business teams.

• Shared Responsibility: This means everyone involved (finance, IT, business) has a stake in optimizing cloud costs.
• Data-Driven Decisions: By proactively monitoring costs, organizations can identify areas for optimization and make informed choices.
• Business Value: This is achieved by balancing cost optimization with business needs.?

FinOps is a way to manage your cloud finances effectively :

• Catch waste early
• Optimize resource utilization
• Stay within budget
• Make informed decisions

FinOps in Databricks

When you run a Databricks workspace you mainly pay for 3 things : DBU (Databricks Unit), Compute (Virtual Machine from the cloud provider) and Storage (also from the cloud provider).

Each of these metrics should be kept in control and proactively monitored

Zoom on Compute : DBU pricing

DBU price depends on type of task you are running, and it impacts directly the cost. Sometimes you don't have choice, but sometimes you can for example to run a task on Job Cluster ($0.07/DBU) or All Purpose cluster ($0.40/DBU) and it makes a big difference.

Hint : You can monitor SKU type of cluster from the system table called "Clusters"

Conclusion

There is a lot to say on FinOps on Databricks, I hope this article is a good starting point for your journey.

This is the first article of a series on "FinOps & Databricks", stay tuned

For more information & help (my costs are out of control ??) : [email protected]

Website : https://accure.fr

Previous Articles

Why transition to a Lakehouse Architecture in 2024?

Put LLMs under control in your company : AI-Gateway

Microservices in Databricks ? (Episode 1)

IA / ML / Deeplearning (FRENCH)