Why We’re Excited About dbt Labs’ Acquisition of SDF Labs

dbt Labs’ acquisition of SDF Labs is a significant announcement in the data landscape, with promises of a bright future for all dbt users — both core and Cloud.

Supercharging the Developer Experience

SDF’s capabilities, built on top of Apache DataFusion, an open-source query planner and execution engine, enable a better dbt user experience, resulting in faster engineering velocity. It does this through a handful of key features:  

• Speed: First off, SDF is built in Rust (just like DataFusion). Rust is known for being fast and SDF is no exception. It can parse and compile large dbt projects orders of magnitude faster than dbt-core. Faster parsing and compile times mean a shorter feedback loop, which makes engineers happier and enables them to find errors and bugs quicker. 
• Static Analysis & SQL Comprehension: SDF brings robust SQL comprehension to dbt, understanding the objects and semantics that your SQL code represents. At the same time, SDF keeps a full definition of your data warehouse and can statically analyze it as code! This allows real-time validation of SQL code as it is written — providing IntelliSense style auto-completion, detecting syntax and other errors without ever connecting to a remote database, and more. As SDF validates your code in real-time the developer feedback loop is shortened even more, further increasing velocity. 
• Local Execution: Apache DataFusion enables SDF to build full logical plans based on the SQL you write and its understanding of your data warehouse. That logical plan can be executed locally, no connection to your data warehouse necessary. You’ve probably guessed already, but this means even shorter feedback loops and increased velocity. As a bonus, it means lower warehouse compute costs as you’re using less compute than before!  

This is a huge advancement in what dbt will be capable of, driving them closer to realizing their vision of being a comprehensive data control plane. We’re excited to see SDF integrated into dbt over the coming months and what other amazing capabilities dbt will gain based on this acquisition.   

In addition to higher data quality other benefits to the dbt community include optimizing data platform costs, improving data transformation, and enhancing data classification. One way to describe SDF's technology is that it emulates the SQL compilers native to data platforms in the way that virtual machines (VMs) emulate physical hardware. This approach can address many of dbt user's biggest pain points as it improves the dbt user experience, expedites data velocity, and enhances analytics engineering capabilities.

Realizing the Power of dbt

At Brooklyn Data, we’ve been dbt power users and experts right from the very beginning. Our expertise in designing impactful data models, combined with our extensive experience implementing them via dbt, positions us perfectly to help you leverage SDF’s full potential as it gets integrated into dbt. We understand the intricacies of data modeling and transformation, the importance of clear and accessible metrics, and the importance of providing a stellar analytics engineering development experience. When combined these elements produce one thing: a high-executing data team that's a force multiplier for your organization.

dbt Labs' acquisition of SDF Labs will have a big impact on all dbt users, but especially those with large, complex data models and transformation pipelines that require faster data velocity. Brooklyn Data is excited to partner with dbt Labs and our clients to power enterprise data practices and unlock as much value from this acquisition as possible.  

Want to learn more about the benefits dbt Labs’ acquisition of SDF will offer? Reach out. Our team of analytics engineering and data engineering experts would be happy to chat with you.