Three Key Themes from Coalesce 2024

Last month, dbt Labs hosted the fifth annual Coalesce conference, gathering data professionals from around the world. I loved going as a first-timer this year!

Each day was structured like this: a keynote from dbt Labs, then a variety of seminars and training sessions from some of the brightest minds in data. From my learnings at the conference, I’ve come up with three key themes that I think will significantly impact data professionals in the next few years. In this post, I’ll provide my thoughts on these themes and share links to some Coalesce talks that touched on each one.

Iceberg Breaks the Surface

dbt Labs is now the most recent enterprise data vendor to embrace a future heavily involving Iceberg, an open table format. Open table formats such as Iceberg are becoming increasingly popular for data management, allowing organizations to continue managing and querying their SQL-based tables and datasets, but with the freedom to store these anywhere — not just in their data warehouse’s managed storage service. Iceberg is also great because it’s extremely performant when handling large-scale data.

At Coalesce, dbt Labs announced cross-platform interoperability in dbt Cloud through their newly added support for Iceberg. Organizations can now use dbt Cloud to build dbt projects and models that reference data across multiple data warehouses!

For example, in dbt Cloud an organization can now create dbt models built in Snowflake that reference data from upstream dbt models built in Redshift and stored using Iceberg. This feature will be paramount for enterprises who need to break down data silos arising from having different business areas operate on separate data platforms.

Amy Chen of dbt labs shared more about dbt Cloud’s new Iceberg capabilities in this kickoff keynote starting at the 36-minute mark.

LLMs are Here to Stay

Large Language Models (LLMs) such as OpenAI’s ChatGPT or Anthropic’s Claude will continue to disrupt many industries, and data is not immune to such changes. While the exact impact is to be determined, it’s naive to think the role of a data professional in five years will look the same as it does today. As LLMs improve, the coding part of a data professional’s job will get increasingly automated, shifting a data professional’s responsibilities from spending most of their time coding, to spending most of their time framing problems and planning solutions.

However, the “AI analyst” isn’t yet ready for a full-time role, as LLMs are still prone to hallucinations and have trouble understanding the intricacies of a data warehouse and the business’s data modeling layer. “Business Context” will be one of the last things to be automated by AI. Data professionals, in addition to honing SQL skills, should continue to focus on being knowledgeable about the business, how it makes money or provides services, and how building and maintaining data platforms addresses the business’s challenges.

I enjoyed two talks from the Hex team on the future of LLMs in data: Bryan Bischoff’s Data Person 2029 and Erika Pullum and Amanda Fioritto’s Surfing the LLM Wave.

BI Tools are Evolving

Business intelligence (BI) platforms are a crowded space, with various up-and-comers such as Lightdash, Preset, and Zenlytic looking to compete with more established names such as Tableau and Looker. To stand out from their competitors, vendors are adding new capabilities at a rapid pace.

For example, some (including Hex, Zenlytic, getdot.ai) are betting on AI by adding LLM-enabled agents inside their tool so users can have a back-and-forth conversation with their data. Others (including Preset, Evidence.dev) are providing greater support for embedding their visualizations into customers’ internal and external apps. And many BI vendors at Coalesce are integrating with a Semantic Layer to cleanly define key business metrics; some such as Steep.app or Cube.dev are rolling out their own semantic layer features to avoid vendor lock-in.

Katie Hindson from Lightdash did a deep dive on the Semantic layer in her Coalesce Talk.

Looking Ahead

The congregation of data people from around the world at Coalesce helps surface what the common themes are throughout our industry. But it also sparks so many exciting ideas for the future of it as well. I’m looking forward to seeing what the next 12 months brings to data professionals. It’ll be fun to review this list come Coalesce 2025!

In the meantime, if you want to know more about how your organization can capitalize on these exciting developments in the world of data, get in touch. Our team would be happy to help you understand how you can leverage these technologies to unify your organization’s data and harness the power of it to accelerate growth.