Databricks Serverless Performance Notes
Josue A. Bogran
VP of Data + AI @ zeb | Advisor to Sigma, Kythera Labs, and Lumel | Databricks Product Advisory Board Member & Databricks MVP
Since around July, I've been running the same workflow from time to time to test Databricks ' Serverless performance, using mostly analytical type queries on roughly 10B rows of dummy data. Here is a break down of some of my observations as it relates to performance and costs.
Please note, I've included costs calculated at both the interactive and jobs standard rates, not the promotion rates. Standard rates for my region at the time of writing are $0.75 per DBU for interactive & $0.35 for jobs. Additionally, I've included a comparison against a SQL Serverless Warehouse (XL), which runs at $56 per hour.
Observation 1
Execution time for Serverless is marginally improving & but costs are decreasing noticeably.
Observation 2
SQL Serverless performs significantly better compared to interactive Serverless in terms of $, but Serverless is likely to save you $ on compute when it comes to orchestrated jobs.
Observation 3
The XL SQL Serverless Warehouse was considerably faster than Serverless compute in completing the full workflow. Details in Observation 4.
Observation 4
Generally, the SQL Serverless XL Warehouse outperformed Serverless in terms of execution time in 10 out of 11 queries. I found that the most noticeable performance gaps were in the simplest of queries. Serverless usually performed on par with my SQL Serverless Warehouse when it came to more complex queries.
Thank You For Reading!
CEO @ Skypoint | AI-Powered Action for Every Decision | HITRUST r2 Certified | AI Agents for Healthcare
4 个月Very informative Sayandip Sarkar
FinOps Engineer at 84.51°
4 个月Great article! I've noticed the same in terms of performance, but when it comes to cost, nothing seems to beat a well-optimized jobs cluster. It would be great to see that included in your comparison.
Product Architecture & Engineering Mgmt., Deloitte Consulting LLP
4 个月Rishabh Srivastava!
Co-Founder, CEO at Sync Computing
4 个月Very cool! I am curious how an optimized classic cluster would perform here… :)
Senior BI Data Developper (Analytics Engineer) at Coveo
4 个月Thanks for sharing! If the trend continues serverless should get cheaper and be even less than sql for jobs. That being said sql is priced nicely and is cheaper than an equivalent all purpose cluster. If you look at classic configurations they don’t charge dbus for the driver as the cost only scales with the workers. The enterprise rate also is the same as premium which is a bit surprising (could it be an error on the page?). https://www.databricks.com/product/pricing/databricks-sql