Coalesce 2024: What to Consider as AI Agents Develop

At Coalesce this year, AI assistants were everywhere, showcased in almost every talk and at most booths. At times, their prevalence felt like a meme, but the excitement was real as data practitioners mused over how these tools could help their day-to-day work and provide value to stakeholders more quickly. We’ll dig into the evolution of analytics tooling, and why these AI tools might be the next step forward if they can overcome a few critical hurdles.

How We Got Here and Where We’re Going

Many platforms are positioning their AI agents as challengers to traditional business intelligence tooling redefining how individuals and companies interact with data. Compared to AI as a feature, these are platforms that are AI-first. Let’s review how reporting tooling has evolved:

• Traditionally, analysts relied on labor-intensive methods, copying data from Excel into PowerPoint to meet reporting requests.
• Then came modern BI tools which delivered automatically refreshed and interactive dashboards that have proven to be a significant improvement over flat-file reporting. These innovations allowed analysts to spend less time on manual and repeated requests, freeing their time to focus on more complex challenges.
• Now, new AI-driven agents make data insights and analytics accessible through natural language interfaces. Companies pushing these agents claim they’ll offer the same value as traditional BI tools — allowing users to slice and dice data — but without the burden of development and maintenance.

Three Challenges to Greater AI Adoption

Will AI agents reduce your reliance on traditional BI tooling? Yes, but three challenges hinder your adoption of them: agent, accuracy, documentation, and governance.

Accuracy

We don’t need to explain why AI agents must generate accurate results; nobody wants to work with the wrong numbers! Although some AI models achieve impressive accuracy, even a small error rate can undermine their credibility and slow their adoption. Companies need ways to validate and correct AI outputs to instill confidence in these tools. One innovative solution comes from Dot, a virtual data assistant with a built-in capability to critique its model’s results. This allows you to test the agent with questions you may anticipate from stakeholders, allowing you to redirect requests to the relevant assets.

Documentation

AI agents are only as good as the metadata you feed them. They need high-quality, consistent documentation, and as data practitioners, we all know maintaining documentation over time is challenging. However, these AI agents could produce unreliable results without clear data definitions and standardized naming conventions. We recommend that your company maintains organized documentation to ensure accurate and actionable information. Beyond being imperative to well-functioning AI agents, this is just best practice.

Governance

Seamless integration with data warehouse security standards is essential for AI agents to achieve successful adoption and scaling. Ideally, these tools should inherit permissions directly from the underlying data source, ensuring secure access for users at all levels. For example, if you have data masking policies, will the AI agent understand which users have permission to see which data? This consideration is vital if your organization prioritizes robust governance or deals heavily with PII data.

Looking Ahead to the Future

Currently, we believe that AI agents are unlikely to replace traditional BI tools. Although they provide substantial value as complementary tools. If you’re a business leader who frequently handles ad hoc data requests, these tools can streamline workflows, reducing your reliance on an army of analysts. They are also well-suited for exploratory analysis, enabling quick insights without writing SQL.

Ultimately, as the market develops, we recommend identifying the specific problems your business needs to solve first, and then assessing how these agents can effectively address those challenges before you adopt them.

Want help determining whether you should adopt an AI agent or wait until the technology matures more? Contact us. We can help you evaluate AI agents and determine if they can provide you with accurate data.