Wobby became useful to business teams after moving its AI analysts into Slack, where onboarding, peer examples, and visible support helped people trust the system.
2
Text-to-SQL accuracy depends on business context, so Wobby replaced a growing collection of query templates with a semantic layer and separate steward agent.
3
Early products should turn known failure modes into ordinary tests, then use evals to compare agent performance across different tasks and trajectories.
Summary
Quinten Rosseel describes how Wobby deployed AI analysts for business teams and learned what made the system useful in daily work. The team moved from a web interface to Slack and used shared conversations, onboarding flows, and visible debugging to build trust. It also learned that text-to-SQL benchmarks did not solve the real problem. SQL syntax was rarely the issue. Business definitions, changing processes, and missing context caused failures. Wobby responded by building a semantic layer, with an analytics agent using logical business representations and a steward agent helping maintain them. Quinten argues that early teams should write ordinary tests for known bugs before investing heavily in evals. Their eval system compares different agent paths with composite scores. He also covers prompt-based logic, framework scepticism, schema compression, message pruning, caching, and observability wrappers. The talk is candid about the work still required from data teams and business users.
Slack adoption depends on shared use and visible recovery from mistakes
Quinten says agent adoption is more than placing a chat interface in front of users. Wobby moved its logistics and finance analysts into Slack, where business users could see other people asking questions and getting useful answers. Slack also allowed onboarding workflows that explained how to query the agent and use different modes. Customer support staff watched the channel, investigated bad answers, and turned failures into eval items. This made the agent's problems visible and gave users a way to see that the team was fixing them. Quinten says that peer examples and this feedback loop helped skeptical business users trust the system.
Text-to-SQL fails when the agent lacks business context
Wobby initially focused too much on text-to-SQL benchmarks. Quinten says the main difficulty was not SQL syntax. An agent with access to a data catalog still needs to understand how the company defines metrics and how its data is used. The team tried retrieval with parameterized query templates and ended up with more than 100 similar SQL queries for analysts to review. That approach did not fit the day-to-day BI process. Quinten's lesson is to choose abstractions that match the business domain and the way data teams actually work.
A semantic layer gives agents a logical view of business data
The second design gave the analytics agent a logical representation of metrics and dimensions instead of raw SQL. Wobby also added a steward agent that could inspect the physical data layer and help business users create or refine concepts in the logical layer. The product eventually needed its own semantic layer because it had to integrate with several external semantic systems. Quinten describes an integration layer with APIs and a SQL compiler that maps logical entities to different SQL dialects. He says the need for this layer emerged gradually because reliable agent output depended on business context and guardrails embedded in the business process.
Direct control was more useful than adopting every agent framework
Quinten was sceptical of frameworks after finding LangChain's abstractions and changing APIs difficult to work with. Wobby needed control over the boundary between its application layer and agent layer, so the team used clean-code interfaces and kept its own abstractions. Useful patterns included an event loop, a semaphore for concurrency control, message history in the OpenAI Harmony format, state stored in message metadata, a factory for creating agents, and wrappers around changing APIs. He says MCP was not useful for their setup because a separate tool server added little value. ReAct prompting and multi-agent networks also created more context and made debugging harder.
Known agent failures belong in tests before they become evals
Quinten argues that early projects can over-invest in evals before they have enough ordinary test cases. When a tool returns the wrong data, a semantic search fails, a token has the wrong access, or a service is unavailable, the team should make that failure a fixed test. These tests can run in CI and prevent the agent from shipping with known bugs. Evals answer a different question: how well does the agent perform across a sample of tasks and configurations? Wobby uses a layered Swiss cheese model, with simple tests catching routine failures and later eval layers handling broader agent behaviour.
Composite scoring makes different agent trajectories comparable
Wobby's text-to-SQL evals use input configurations such as clear and ambiguous tasks, then record the agent's trajectory, including which tools it calls and when. The team uses several judges, asks for reasoning before structured judging output, and attaches comments to make scores easier to inspect. Because an agent can end in different states, Wobby combines relevant signals into one weighted score. Quinten compares this to overall equipment effectiveness in manufacturing. The single score does not replace inspecting traces, but it gives the team a common signal for comparing runs and testing whether a change improved the system.
Prompt-based logic can be easier to change and evaluate than fixed workflows
Wobby moved some logic out of fixed workflows and into prompts. Quinten says workflows are more deterministic, but every branch and sub-path needs testing, which makes the source of a failure harder to find. A prompt can be changed, compared, and evaluated as one configuration. This approach depends on language models following instructions well, so it is not a universal rule. His claim is limited to their agentic text-to-SQL setting. The team found that prompt-based logic made experiments and deployment more portable than embedding every decision in a large workflow.
Latency work focused on reducing context and unnecessary calls
Quinten lists several ways Wobby reduced response time. The team removed fixed intent-detection workflows and used progressive disclosure instead. It compressed large schemas, pruned message history, and compressed JSON output. It also cached large system prompts and tools. In some cases, the agent could make a tool call while placing its explanation in the content field, which avoided an extra call for reasoning or explanation. Quinten warns that simple questions can still require longer answers, so users sometimes need an explanation when a response takes more time.
Reliable agent deployment still requires data modelling and organisational work
In the discussion, Quinten says an agent cannot be dropped onto poorly understood data and expected to produce trustworthy answers. Teams need to define business domains, model dimensions, and capture the ontology of the business in a reusable form. YAML may be practical for developers, while more formal enterprise ontology approaches can be harder to query. The industry still lacks a shared standard for semantic-layer interoperability across systems such as dbt, Databricks, and Snowflake. Wobby works pragmatically with a bounded data warehouse today. Cross-source joins remain a limitation, and customers still need data, governance, and business teams to agree on definitions.
