Enterprise agents are commonly used for document understanding, knowledge assistance, custom language models, and workflows that chain these components together.
2
Agent development requires manual choices around prompts, architectures, tools, models, and evaluation, while cost and quality usually trade off against each other.
3
Agent Bricks automates evaluation and optimization across prompts, models, tools, fine-tuning, and reinforcement learning methods to improve an agent's cost and quality.
Summary
Krista Opsahl-Ong describes how enterprises are using agents for document understanding, knowledge assistance, custom language model tasks, and chained workflows. She focuses on the work needed to move these systems into production. Teams must try different prompts, architectures, tools, and models, while also defining evaluations for tasks that may lack labelled data. Cost adds another constraint, especially for high-volume information extraction. Agent Bricks addresses this by taking a task description and data sources, generating evaluations, and sweeping different system configurations. Its optimization methods include automatic prompt optimization, fine-tuning, reinforcement learning, and choices of tools and models. In an information extraction benchmark, Krista says optimization reached the same performance at about a tenth or a sixteenth of the cost, or achieved a 20% quality increase at a similar cost. She also explains that Databricks uses DSPy and a label-free reinforcement learning method called Tao behind the platform.
Enterprise agents often start with document understanding
Krista describes document understanding as a common enterprise use case. A company may receive tens of thousands of invoices each month and need to extract information for systems such as SAP. Another example is an e-commerce platform with millions of product specifications buried in PDFs. An agent can extract the relevant fields so the company can publish product information on its website. These tasks can involve large volumes, which makes inference cost an important part of the design.
Knowledge assistants must fit a specific source and use case
Knowledge assistance includes internal and external applications. An internal team might use a deep research tool over company documents to help analysts. A public chatbot might answer questions from a website while following specific brand guidelines. Krista also mentions custom language models for tasks such as generating article titles. Different components may then be chained together into a larger workflow.
Agent development contains many manual design decisions
Teams building a performant agent must choose prompts, architectures, tools, and models. Krista describes the usual process as manual setup and trial and error. Engineers plug different tools and models in and out, test them, and compare the results. This process is time-consuming, especially when a company does not have an in-house research team working on every problem.
Cost and quality create a practical production trade-off
Krista shows a graph with cost on a logarithmic x-axis and quality on the y-axis. In her information extraction benchmark, out-of-the-box models with a reasonably optimized prompt become more expensive as quality improves. That trade-off is difficult for high-volume workloads. She argues that moving the curve toward higher quality and lower cost requires research that many enterprises cannot do internally.
Agent Bricks starts from a task description and data source
Agent Bricks takes a high-level task choice, a natural-language description of the desired agent, and basic inputs such as internal documents. The platform includes bricks for information extraction, knowledge assistants, a multi-agent supervisor, Genie for text-to-SQL, and custom language models. The multi-agent supervisor can connect tools such as MCP servers and other bricks.
The platform generates evaluations before optimization
For a knowledge assistant, Agent Bricks creates language-model judges for properties that matter in question answering. Krista names completeness, groundedness, relevance, and safety. The platform can generate responses and score them so users can check whether the judges match their expectations. These evaluations provide a way to measure quality for tasks that may not have standard labels.
Optimization sweeps several parts of the agent system
The optimization step appears as a button in the product, while the platform tests different system configurations behind the scenes. It can optimize prompts and weights, compare models, choose tools, apply fine-tuning, and use reinforcement learning methods. Krista says the goal is to improve the cost-quality curve for the particular task rather than rely on one fixed recipe.
Databricks uses automatic prompt optimization and label-free reinforcement learning
Krista explains that automatic prompt optimization systematically proposes and evaluates prompts instead of relying on manual prompt engineering. She also describes Tao, a label-free reinforcement learning method from the research team. When users do not provide labels, Tao can adapt the program over time using new queries. She confirms that DSPy is used in Agent Bricks and that the research behind the platform continues to change.
"The first step on the Agent Bricks side that we'll automatically do for you is create a set of evals."07:04
Who should watch
You are building an enterprise agent for document extraction, internal research, or a branded chatbot and need a production workflow.
Your team is spending time testing prompts, models, tools, and architectures by hand, especially for a high-volume workload where inference cost matters.
You need evaluations for an agent task without reliable labels and want to compare quality against cost.
