What Are SDF Labs and Quary? A Dive Into Last Week’s Big Data Acquisitions

It’s big news when one company gets acquired, but in the data industry, we just saw it happen twice in the same week. Last Monday, dbt Labs announced it would be acquiring SDF Labs . The very next day, Tobiko announced its acquisition of Quary (YC W24)

These acquisitions send a clear signal about the future of transformation frameworks. Both tools are written in Rust, signaling that high-speed execution is a top priority across the board. Both tools feature database-agnostic runtimes allowing users to run code in a single language and not worry about the database syntax they're targeting. Lastly, both products feature an enhanced user interface not offered by either of the acquiring companies. If both parent companies merge their products with those of the companies they're acquiring, then this will raise the bar for expected features in a data framework. Even more so if they choose to release the features as open-source.

What is SDF Labs?

SDF Labs is a transformation framework that offers local compilation as a stand-out feature. It has support for running queries on a variety of different compute engines such as Snowflake, Bigquery, and S3. It seeks to abstract away the differences that come with running queries on different engines, so a developer using the tool can write everything in one common dialect of SQL.

Although the primary engine code is closed source, SDF has an open-source piece. There are a handful of libraries that can be installed with SDF to extend its capabilities. Using these packages allows you to create custom materializations, tests, and other utils that are necessary for your scenario. Unfortunately, due to the acquisition, the code can no longer be downloaded. We will need to wait and see what dbt Labs decides to include as open-source code vs keeping as proprietary.

What is Quary?

Quary is a data transformation engine like dbt, sqlMesh, and SDF Labs, but with a few key differences. It can be accessed in two convenient ways: the VS Code custom add-on or the CLI tool. The improvement in speed and improved user interface means that working in Quary is a very pleasant experience. There’s no need to switch back and forth to a web-based UI when everything you want to see is in VS Code.

Another key feature is the ability to generate BI dashboards right in your IDE and store the definition as code. With Quary, you can build the basics of an entire data pipeline without even deploying your code. The flexibility and attention to developer experience that Quary brings to their product is a great sign for future integration with Tobiko Labs. These two products complement each other in ways that make them a natural fit to be merged.

The Role of Rust in High-Performance Data Tools

We still don’t know how dbt and Tobiko will integrate these products into their tools, but here are a few inferences we can make:

• Data pipelines are about to get a lot faster. Rust is becoming the new standard language for high-performance data applications. Both tools being acquired are built in Rust, and both have the
• Improving user experience is top of mind for both companies. Tobiko would like to bring the interface closer to the developer environment in VS code, while dbt Labs is focused on bringing the dynamic framework of expansion packages that the SDF framework offers.

Why These Acquisitions Matter

This M&A activity signals a possible future for the data software industry:

• The arms race for features is on. Leading companies cannot simply build the features they believe in - they should be searching the market for similar companies and merging platforms to improve quickly
• Highly-performant code will be the standard from here on out. Rust will dominate the data software landscape as the de-facto programming language, and any projects written in Python will begin to show their age.
• Open-source companies like Quary are now targets for acquisition. Any founders who can show a high-quality, innovative product in data have now reached the threshold to have their companies appeal to buyers.

We should expect to see both dbt and SQLMesh add new features in the coming months as a direct result of this announcement. However they will each face a test - do they keep the code open source, or hide it as proprietary? Said another way, will you be able to get these features as a user of the open-source version of the product, or will you be required to pay for their SaaS tool?

Open Source vs Proprietary: What’s Next?

In one scenario, the two companies would absorb all the code into their open-source products and make the features publicly available. While this would lead to greatly increased adoption and satisfaction with these platforms, it would be a hefty investment to acquire a company just to release its code for free.

In the other timeline, each company makes the new features part of their paid offering. The cloud users for both will be happy to have more reasons to justify paying for the subscription fee. However, the divide between the cloud and core user bases will grow wider. An interesting scenario opens up with Quary - since it's already open and publicly available, the community will always have this current copy. If Tobiko were to close-source the future updates, we could see a fork of the current Quary maintained elsewhere and remain open.

Looking Ahead: The Future of Data Transformation

Regardless of what’s decided, if you are a dbt or sqlMesh user, you should be thrilled about this news. Your transformations are about to get an upgrade. At Database Tycoon, we hope to see both teams continue their commitment to open-source by releasing these features to the community. The teams that have been acquired this month are both innovators in the data space and have made their contributions to the open-source world. We're excited to see how these innovators will enhance both the core and cloud products offered by dbt Labs and Tobiko.

Partner with Database Tycoon

Here at Database Tycoon, we are committed to staying current on the new releases from tools like this. Follow our page for more updates like this as new announcements are released.

To work with a trusted partner who can help you start or upgrade a dbt or sqlMesh project, contact us! We are a full-stack data consultancy offering help with projects ranging from staff augmentation to support a needy pipeline, or a business planning session to scope out an expansion of your data team. Reach out here for a free 30-minute consultation chat!