# AI SQL Data Analyst

Paul van der Boor & Donne Stevenson, Prosus Group | Agents in Production Series | 53:57
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=hwgoNmyCGhQ
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/ai-sql-data-analyst
Published: 2025-02-28
Tags: agents, evals, guardrails, tool-use

## TL;DR
- The Toqan Data Analyst gives people a fast first point of contact for recurring questions about company data, while human analysts handle harder requests and supervise its answers.
- A separate clarity pre-check asks whether a question contains enough information before the SQL agent runs, because the model often answers ambiguous questions with unjustified confidence.
- Prosus chose plain-text filtering for relevant schema information instead of embeddings because the context changed frequently and the simpler process was easier to maintain.

## Summary
Paul van der Boor and Donne Stevenson describe how Prosus Group productionized a Toqan Data Analyst agent that answers questions about company data in shared Slack channels. The agent is attached to a specific company and data use case, with access to selected schemas and a SQL connector. It handles frequent questions, while human analysts review its answers and take over complex work. Paul says response time fell by about 74% for the measured query-to-response interval, and connected data sets saw two or three times as many generated insights. Donne explains that SQL generation was relatively easy. The harder problems were interpreting business context, refusing ambiguous questions, preventing unsafe or expensive queries, and recovering from small SQL errors without restarting the whole process. The team added a separate clarity check before the agent, maintained development and production versions, and filtered schema metadata with plain-text LLM calls. They avoided embeddings because changing documentation would make that system harder to maintain.

## Key ideas
### The agent joins existing data analyst teams instead of replacing them
[01:29](https://www.youtube.com/watch?v=hwgoNmyCGhQ&t=89s)
Paul van der Boor describes the Toqan Data Analyst as an addition to existing data analyst teams. It sits in shared Slack channels where human analysts already work. The agent becomes the first point of contact because it is always available and responds quickly. Humans can review its answers, flag incorrect results, and handle questions it cannot answer. Paul gives iFood as an example. Employees ask about restaurant orders, ZIP codes, order IDs, and performance over recent months. These recurring questions are suitable for the agent, while complex queries remain with human analysts.

### Each deployment is scoped to a company, data set, and SQL connector
[07:54](https://www.youtube.com/watch?v=hwgoNmyCGhQ&t=474s)
Prosus does not turn the agent on for every use case with one switch. Onboarding takes a few days and requires integrations with systems such as Snowflake, Databricks, and BigQuery. An instance is logically separated by company, and teams also separate use cases such as marketing, logistics, and grocery data. Donne Stevenson explains that each agent has a fixed database connection and SQL dialect. Slack channels provide the operational separation, so the agent knows whether it is working against marketing or logistics data without choosing among many databases dynamically.

### The biggest model failure was answering questions it should have rejected
[18:50](https://www.youtube.com/watch?v=hwgoNmyCGhQ&t=1130s)
Donne says SQL generation was not the main problem. General OpenAI models were already good at producing valid SQL. The difficult failure was overconfidence. When a question lacked enough context, the model usually assumed it could answer. It could produce a plausible response, including numbers, even when no query had run. Prosus addressed this with a separate pre-check. The system first asks an isolated LLM call whether the question and available context are sufficient. Only a positive decision starts the agent, while an unclear question goes back to the user. The team also built a clarity data set to test how well models can doubt their own answers.

### Business metadata must state assumptions and formulas explicitly
[24:44](https://www.youtube.com/watch?v=hwgoNmyCGhQ&t=1484s)
Donne identifies context creation as the largest onboarding bottleneck. Schema information includes table descriptions, columns, types, and descriptions, but metadata written for human analysts often relies on shared knowledge. Terms such as MQL, contribution margin, and GM1 need explicit definitions and formulas. The agent cannot reliably infer how a metric is calculated or how tables connect. Prosus learned to write descriptions without ambiguity or assumed history. Donne compares the process to onboarding a new analyst on their first day, except the goal is to give the system the context of someone with much more experience.

### SQL generation moved into the main agent, with repair tools for local errors
[30:16](https://www.youtube.com/watch?v=hwgoNmyCGhQ&t=1816s)
The first design generated SQL in a separate tool and then passed it to another tool for execution. As use cases became harder, the separate generator did not receive enough context to produce valid queries. The current design lets the main agent generate SQL and pass it to an execution function. That design has a drawback: a small error can force the agent back to the beginning. Prosus made the execution tool more capable by allowing it to update the SQL after an error. A date-function mistake can then be fixed from the error message without repeating the full reasoning process.

### Plain-text schema filtering was easier to maintain than embeddings
[33:10](https://www.youtube.com/watch?v=hwgoNmyCGhQ&t=1990s)
For larger documentation sets, the system cannot send every table and column to the agent at once. Donne says Prosus filters metadata in a sequence of plain-text LLM calls. The question and structured metadata are used to select relevant tables, columns, and other information before the smaller context is given to the SQL agent. The team considered embeddings and retrieval, but documentation changed often while analysts were setting up use cases. Embeddings would need continual updates and introduced another system to evaluate. Donne also found that short user questions did not always match the way table descriptions were represented, while plain text was easier to inspect when retrieval went wrong.

### Analysts release narrow use cases through separate development and production versions
[36:37](https://www.youtube.com/watch?v=hwgoNmyCGhQ&t=2197s)
Prosus found it difficult to define SQL accuracy with one exact expected query because several SQL queries can produce the same result. The team began with fixed question, SQL, and output test sets, then narrowed releases to specific use cases. Analysts classify use cases by difficulty and decide whether each is ready for users. Every analyst has a development version where they add tables, change context, and test questions, plus a production version for trusted use cases and users. Donne and Paul say close work with the analysts made adoption possible because those analysts supplied the domain knowledge and wrote the business context that the agent could not infer.

### The agent has a ceiling on complex joins and still acts as an assistant
[42:50](https://www.youtube.com/watch?v=hwgoNmyCGhQ&t=2570s)
Donne says the agent tends to choose the simplest query, which makes complicated joins, CTEs, and multi-step SQL difficult. That limits the range of questions it can answer. The product is therefore aimed at people who understand enough about the data to know what they want and need help retrieving it. Human analysts remain responsible for difficult queries and higher-value analytical work. Paul also lists production issues beyond SQL quality, including loops, unsafe or expensive queries, database overload, chart generation, file exports, and behavior changes after model upgrades. The team found that prompts and context could be overfit to a particular model.

## Notable quotes
- Demetrios Brinkmann: "It's an internal project that made it through the great filter of your projects that have had success." (00:41)
- Donne Stevenson: "The biggest failure case was that it would continue to answer when it shouldn't." (19:30)
- Donne Stevenson: "When it's written for humans, it doesn't mean it's written for agents." (26:26)
- Donne Stevenson: "You can overfit the prompt." (48:57)
- Donne Stevenson: "It's really that middle ground where it's people who need some assistance with the data and they have a little bit of understanding of what they're looking for." (44:26)

## Tools & references mentioned
- MLOps Community
- Toqan Data Analyst
- Prosus Group
- iFood
- OLX
- Glovo
- Slack
- Snowflake
- Databricks
- BigQuery
- OpenAI
- Redshift
- PostgreSQL
- DeepSeek R1

## Who should watch
- You are building a natural-language SQL agent and need to decide how to scope its database access, schema context, and user channels.
- Your model produces plausible answers for ambiguous questions, and you need a separate validation step that can reject them before execution.
- You are deciding whether to add retrieval and embeddings or keep a simpler text-based context pipeline while schema documentation is still changing.

## Related talks

- [Hundreds of Users Love Our Data Analyst AI Agent](https://mlopstalks.com/talks/hundreds-of-users-love-our-data-analyst-ai-agent) (Ioannis Zempekakis & Donné Stevenson, 29:00)
- [Dynamic Contextual Retrieval in Enterprise Analytics](https://mlopstalks.com/talks/dynamic-contextual-retrieval-in-enterprise-analytics) (Dirk Petzoldt, explai.com, 20:06)
- [How AI Agents Are Transforming Data Analytics](https://mlopstalks.com/talks/how-ai-agents-are-transforming-data-analytics) (Ines Chami, Numbers Station, 24:26)
- [AI Data Engineers: Data Engineering After AI](https://mlopstalks.com/talks/ai-data-engineers-data-engineering-after-ai) (Vikram Chennai, Ardent AI, 48:07)
- [Before Building AI Agents Watch These Hard Earned Lessons](https://mlopstalks.com/talks/before-building-ai-agents-watch-these-hard-earned-lessons) (Nishikant Dhanuka, Prosus Group, 52:37)