# Advanced Context Engineering

 | Agents in Production 2025 | 28:42

Source: https://www.youtube.com/watch?v=2yi4mAN3CtE
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/advanced-context-engineering
Published: 2025-08-13
Tags: context-engineering, evals, structured-outputs, tool-use

## TL;DR
- Frameworks can get an agent to roughly 80% quality quickly, but production work often requires owning the software, prompts, state, and control flow underneath.
- Context engineering means controlling the tokens that enter an LLM call, including prompts, memory, retrieval, history, structured outputs, and error context.
- Small, focused agents work better when deterministic code owns most of the workflow and the LLM handles tasks involving natural language, judgment, or human interaction.

## Summary
Dex argues that production agents are mostly software with LLM calls placed where they add value. Frameworks help teams get an early system running, but they can hide prompt construction, tool injection, and control flow. After speaking with around a hundred people building successful production agents, he presents the 12-factor agents approach as a set of engineering practices. The central idea is context engineering: manage every token sent to the model and treat the model as a function that turns context into structured output. Dex recommends writing the loop, switch statements, state handling, prompts, and context builders yourself when the abstraction gets in the way. He also recommends small agents, deterministic workflows, explicit human interaction, and external state. The talk includes examples of asynchronous human handoffs, XML context, error recovery, and deterministic evaluations for an open-source Linear issue agent. His advice is practical and candid: use agents where they help, and write ordinary software everywhere else.

## Key ideas
### An 80% agent is easy to build and often unsuitable for customers
[01:10](https://www.youtube.com/watch?v=2yi4mAN3CtE&t=70s)
Dex describes a common path: choose a product problem, pick an agent framework, and assemble enough off-the-shelf components to excite a CEO. That approach can produce something that is roughly 80% good very quickly. The remaining quality gap exposes the framework's internals. Engineers end up deep in a Python call stack asking how prompts are built and where tool calls are injected. Dex also gives a make-file agent as an example of poor problem selection. He spent about two hours making GPT-4 call tasks in the correct order, then realized a bash script would have solved the known workflow in about 90 seconds.

### Agents should be built from ordinary software components
[03:44](https://www.youtube.com/watch?v=2yi4mAN3CtE&t=224s)
Dex proposes a set of conventions for agents, similar to the 12-factor app conventions that became normal for cloud software. He advises against throwing away an existing system to build seven new agents. Teams should take small, modular ideas from agent building and apply them to existing codebases. The focus is not model training or fine-tuning. It is getting the most from the best model available today. His framing is that an agent can be understood from first principles as software that transforms input into structured output, chooses a next step, and repeats when needed.

### Structured output is the basic bridge from an LLM to useful software
[09:10](https://www.youtube.com/watch?v=2yi4mAN3CtE&t=550s)
Dex calls the transformation of a string into JSON the most important factor in making an LLM-powered system useful. The JSON can tell deterministic code to fetch data, change data, or send data elsewhere. He treats tool use in the same way. A model outputs JSON, ordinary code does something with it, and the result may be returned to the model. The tool is not a special alien capability. It is a software boundary. This perspective lets engineers use switch statements and other familiar constructs instead of hiding the whole system inside an agent abstraction.

### Large agent loops become weaker when their context grows
[11:17](https://www.youtube.com/watch?v=2yi4mAN3CtE&t=677s)
Dex describes an agent as an event and goal entering an LLM loop, with the model selecting tools and receiving an updated context window at each iteration. That loop can dynamically traverse a graph instead of following a hand-written DAG. However, he says larger loops do not work as well. Even as models improve, a small, focused prompt and context usually gives better results than a long context window. He recommends managing state outside the LLM and separating the pieces of the loop so teams retain control over how data is processed.

### Owning prompts and context lets teams improve the model's decisions
[13:16](https://www.youtube.com/watch?v=2yi4mAN3CtE&t=796s)
Dex says that, apart from changing the model or its settings, the tokens placed in the prompt are what influence agent performance. Framework templates may be good, but owning the prompt lets a team test alternatives and find better results. The same applies to context construction. A system can represent history as tool calls or place the information in a single user message if the model can understand what happened and choose the next step. Dex says his team often serializes traces as XML because it packs meaning more densely than JSON. Prompt, memory, retrieval, history, and structured output all belong to context engineering.

### Small focused agents should handle narrow parts of deterministic workflows
[16:56](https://www.youtube.com/watch?v=2yi4mAN3CtE&t=1016s)
Dex recommends using LLMs for the parts of a process that involve natural language, human decisions, or turning plain text into a more useful structured result. He gives HumanLayer's deployment agent as an example. A pull request is merged, the system pulls it in and tests it, and the LLM handles a narrower part of the process. He advises against asking an LLM to run an entire workflow. His rule of thumb is fewer than 100 tools and fewer than 20 steps for good results.

### Human interaction belongs inside the agent's control flow
[16:16](https://www.youtube.com/watch?v=2yi4mAN3CtE&t=976s)
Dex describes tool calls that let an agent declare its intent to request clarification, provide a final answer, or ask for another type of human response. HumanLayer puts those interactions in Slack or email, where users already work. This is harder than a synchronous browser flow because email and Slack are asynchronous. The agent therefore needs stateful control flow that can pause, persist context, wait for a response, and resume. His implementation stores context and a state ID, launches a long-running job, receives a webhook, appends the result, and continues the loop.

### Explicit traces make agent evaluation more deterministic
[18:24](https://www.youtube.com/watch?v=2yi4mAN3CtE&t=1104s)
Dex argues that treating LLMs as functions with tokens in and tokens out produces a clean input-output pair for evaluation. He shows an open-source agent that manages Linear issues. The agent has around ten structured outputs, including several ways to contact a human. Its tests assert decisions such as choosing to list teams after the model hallucinates a team ID, and choosing its final answer once the thread is complete. This lets the team change a prompt while checking that the agent's decision-making has not changed unexpectedly. He mentions BoundaryML's BAML as the tool used for these tests.

## Notable quotes
- "Not every problem needs an agent." (03:05)
- "The things that makes agents feel magical is the ability to take a string like this and turn it into JSON that looks like this." (09:10)
- "Agents are software. You can write a for loop. You can write a switch statement. You can serialize a string. So write the dang software." (20:36)
- "Everything in building great agents is context engineering." (15:09)
- "If you can have the model write code in the way that it wants to write code, which is the way code is written in the training set, that's better." (25:34)

## Tools & references mentioned
- 12-factor agents
- HumanLayer
- MCP
- GPT-4
- Airflow
- Prefect
- LangChain
- LangGraph
- Claude Code
- Shopify
- Cognition
- Andre Karpathy
- NotebookLM
- BoundaryML
- BAML
- Linear
- Ruby on Rails

## Who should watch
- You are building an LLM-powered product and the framework abstraction is making it hard to inspect prompts, tool calls, or state.
- Your agent needs to pause for people in Slack or email, resume later, and keep a testable record of its decisions.
- You want a practical design for using small LLM components inside a mostly deterministic workflow instead of making the whole workflow agentic.

## Related talks

- [Beyond Prompting: The Emerging Discipline of Context Engineering Reading Group](https://mlopstalks.com/talks/beyond-prompting-the-emerging-discipline-of-context-engineering-reading-group) (Adam Becker, HeadOn & Matt Squire, Fuzzy Labs & Rohan Prasad, EvolutionIQ, 1:00:00)
- [Expanding context engineering to the tooling layer](https://mlopstalks.com/talks/expanding-context-engineering-to-the-tooling-layer) (Frank Wittkampf, Databook, 26:18)
- [Context Engineering pitfalls for our e-commerce agent](https://mlopstalks.com/talks/context-engineering-pitfalls-for-our-e-commerce-agent) (Nishikant Dhanuka & Chiara Carateli, Prosus, 28:06)
- [Context Engineering 2.0](https://mlopstalks.com/talks/context-engineering-2-0) (Simba Khadder, Redis, 45:34)
- [Context Engineering 2.0: MCP, Agentic RAG & Memory](https://mlopstalks.com/talks/context-engineering-2-0-mcp-agentic-rag-memory) (Simba Khadder, Redis, 24:35)