Snowflake vs Databricks: TPCS-DS Benchmark Wars – Who Cares?

This article discusses the relevance of the recent TPC-DS “results” to potential customers.

Let’s start with disclaimers:

1.??????Unlike my past articles, this article contains opinions and speculations in addition to technical commentary. These opinions and speculations are strictly my own and do not represent the views of any of my clients or any database vendors.

2.??????I make my living as a Consulting Data Architect specializing in Snowflake. I’m also a huge fan going back to its stealth days. This article was written without any communication or input from Snowflake.

3.??????Nothing in this article should be construed as a commentary on Databricks technology. It is primarily about the use of TPC-DS as a marketing vehicle.

A Little History

RDBMs vendors using TPC benchmarks goes back decades. It started as benchmarks for transaction processing around 1990. At its peak, 33 vendors were publishing TPC benchmarks; that was a looong time ago. (See TPC?History).

This was frankly marketing. In those days, the similarities between RDBMS offerings were much stronger; Snowflake and Databricks are completely different technologies.

Those of us in the profession at the time could only LOL; we knew how easy it was to optimize the benchmarks. The Dewitt Clause was included in licensing agreement to suppress vendors publishing benchmarks on competitors’ products. We shall see why this was desirable. Despite the DeWitt clause the so called TPC wars were filled with accusations about misleading pronouncements.

Bold Statements

A customer purchasing an RDBMS based solely or even most heavily weighted on performance benchmarks has far more serious issues facing them. This isn’t the 1990s, when RDBMs ran in fairly similar hardware and largely offered the same features.

Today’s products offer vastly different features sets and typically run on very different data and compute sources. Performance is only one of many factors to consider.

Even more important, the TPC-DS Power test, as noted in TPC-DS 3.2.0 Specification, paragraph 7.1.10, A Power Test consists of exactly one query session running a single query stream . Basing performance decisions on this core benchmark ignores the critical issues of concurrency and scalability, not to mention functional aspects. The Power Test is the benchmark in dispute.

There are as many vendors striving to achieve a new TPC-DS World 100,000 GB Record as competitors for most Guinness Book World Record. According to the TPC website, the number of competitors for this award is two (2): Alibaba and Databricks.

Conflicting Statements

Databricks opened with Databricks Sets Official Data Warehousing Performance Record claiming superior performance and cost compared to Snowflake.

Snowflake responded with their own benchmark, Industry Benchmarks and Competing with Integrity, with their own numbers on price and performance.

Databricks responded with Snowflake Claims Similar Price/Performance to Databricks, but Not So Fast! ?In which they ran the TPC-DS Power Test provided in Snowflake’s Tutorial 4: TPC-DS 100 TB Complete Query Test.

Technical Commentary

In their response, Databricks “noticed that the Snowflake pre-baked TPC-DS dataset had been recreated two days after our benchmark results were announced”.

I had a twofold reaction:

1.??????If you’re going to publish a benchmark, you want to show your best technology and results.

2.??????Technically, it was about time!

Snowflake Sample data has been available since day one. I noticed when writing my article, Snowflake Micro-partition vs Legacy Macro-partition Pruning, that the TPC-DS data was poorly and not realistically clustered. In the real world, data would typically be loaded on a regular basis, e.g., daily, and tend to be naturally clustered. The intent of that article was to demonstrate micro-partitioning, and while the performance of the unclustered data was fine, it didn’t clearly demonstrate the point of the article.

A second critical observation is that earlier this year, Snowflake completely changed the underlying data storage structure of its product. Data stored prior to this change remained in the old format! Recreating the TPC-DS data not only improved performance, it also resulted in more realistic data!

Databricks claims to have reloaded the data themselves and come up with different results than Snowflake. There are several potential issues with how this was done. In particular:

1.??????“After you define a clustering key for a table, the rows are not necessarily updated immediately”.

2.??????“Clustering Key… is useful for very ?large tables where the ordering was not ideal (at the time the data was inserted/loaded) or extensive DML has caused the table’s natural clustering to degrade"

Databricks’ results are only relevant if the clustering depth at the time its queries were executed match those of Snowflake’s. Simply adding the cluster key columns is not sufficient.

The clustering efficiency can be determined by SYSTEM$CLUSTERING_DEPTH function.

Snowflake’s table is clustered ?on columns ss_sold_date_sk and ss_item_sk. ss_sold_date_sk is the primary clustering column. The clustering depth on this column is effectively perfect. Cluster key on both columns is, as would be expected, not as good.

All large tables I examined were nearly perfectly clustered on the relevant data column.

Unless these numbers match, Databricks’ results are not comparable!

Simply running Powers queries with no further elaboration on the setup isn’t helpful.

“Final” Battle?

Shortly after the above exchange, Snowflake announced significant performance improvements, which probably make the above exchanges obsolete. No surprise, Snowflake uses TPC-DS for benchmarking their own product and provides very detailed setup information.

I sincerely hope this is the end of the TPC-DS “bench-marketing” wars. Nobody should make purchasing decisions based on these types of exchanges.

Copyright ?2021, Jeffrey Jacobs & Associates, LLC