The data analyst agent became more consistent when predictable checks moved out of the language model and into deterministic workflows.
2
Data experts need to maintain the documentation and test cases that guide a specialised agent.
3
The system uses hard limits, restricted database access, and tool-level controls to prevent infinite loops and unsafe SQL activity.
Summary
Ioannis Zempekakis and Donné Stevenson describe how their English-language data analyst agent moved from an unreliable prototype to a product used by hundreds of people. The agent turns questions into SQL, executes the queries, and can visualise the results. Its early failures included answering vague questions incorrectly, generating different answers for the same question, repeating responses, and producing SQL with nonexistent columns. The team addressed these problems with deterministic preprocessing, schema checks, expert-maintained documentation, hard limits, soft guidance, and more focused tools. Data experts can update the agent's documentation through their own pipelines and interfaces. The team also limits database permissions, adds query timeouts, and stops excessive tool calls. They built their own agent framework and use different language models for different steps, choosing larger models where errors carry a high cost. The talk is practical about where agent autonomy helps and where ordinary software controls are needed.
Deterministic checks should handle predictable agent behaviour
Donné Stevenson says the team initially expected the agent to decide whether it had enough information to answer a question. In practice, users asked vague questions and the language model produced answers that were wrong or varied between runs. The team added a preprocessing step that evaluates the question and its available context before the agent tries to solve it. They used the same approach for SQL validation. Before executing a query, the system checks whether each referenced column exists. This catches a predictable error without sending unnecessary queries to a user's database or spending extra agent cycles correcting it.
Experts must keep the agent's specialised knowledge up to date
The team first collected documentation and schemas from data experts, then returned test results for review. The experts often found details that were wrong, but the engineering team could not infer general fixes because each problem depended on a particular use case. They created pipelines and user interfaces that let data experts update the documentation themselves. This reduces dependence on engineers and lets experts shape documentation for agent consumption rather than human reading. Stevenson treats the agent as a data product that needs rigorous testing, even though its answers are written in understandable language.
Production systems need hard limits alongside guided autonomy
Once people outside the development process started using the agent, they asked unexpected questions and pushed beyond its intended role. Stevenson says this is normal for a product exposed to users, so the system must handle inputs that exceed its design assumptions. The team combines hard limits, which tell the agent when it must stop, with soft limits that steer it toward a useful path. This preserves some ability to solve problems and self-correct while preventing infinite loops and unproductive rabbit holes.
Tool design can reduce the context and work required from the main agent
The schema tool originally returned broad information because early deployments had only a few small tables. As more use cases appeared, the tool had to select what was relevant. Concentrated context reduces distraction and prevents the main agent from carrying unrelated information through its reasoning. The team is applying a similar idea to SQL execution. The execution tool can fix simple, context-independent errors, such as SQL dialect issues, before returning to the analyst. This saves calls between the agent and its tools and keeps bad intermediate information out of the main context.
Database safety depends on permissions and execution controls
In the questions, Stevenson explains that SQL tools run against user databases, so queries that run for too long could lock systems. The team uses timeouts and limits the credentials supplied by customers. Excessive tool calls also trigger a hard stop. Access is handled case by case because customers manage databases differently. The team works with each customer's access owners and also manages secrets on its own side. These controls limit the damage caused by an incorrect query or an agent that keeps calling tools.
The team built its own framework and assigns models by task
Ioannis says the team developed its own framework from the beginning rather than adopting an existing one. They had tried other frameworks early on, but the project was exploratory and they had already invested in a framework that had been tested at scale. The system can call different models for different tools. Smaller calls can handle narrower tasks such as reflection, while the execution component needs more capable models. The team weighs model complexity against cost and says it prioritises precision because users may make decisions from the returned data.
AI-first still includes ordinary software and human oversight
The talk ends with the principle that AI-first does not mean AI-only. Deterministic preprocessing, schema validation, timeouts, access restrictions, expert review, and tool-specific logic all sit around the language model. The agent remains responsible for creative problem solving and recovery, but predictable behaviour is handled by code. Experts continue to define the documentation and review results. This division gives the system room to work through ambiguous problems without allowing it to decide or execute everything without constraints.
"We had to find a way to solve this, and because it was a required behavior, we knew the agent needed to go, can I answer this question given the information I have right now, and because it was required, we could hard code it."Donné Stevenson07:43
Who should watch
You are building an agent that writes or executes SQL and need practical controls for vague questions, invalid queries, and repeated tool calls.
Your team owns a specialised AI product whose domain experts need to maintain schemas, documentation, and test cases without waiting for engineers.
You need to decide where a language model should act autonomously and where deterministic checks, permissions, and timeouts should take over.
