Large language models will become reasoning engines inside software, coordinating tools, models, and outside data sources.
2
LLM systems fail in more places than traditional software, including prompts, tool calls, workflows, outputs, training, and model alignment.
3
Better foundation models will help, but production systems also need middleware, watcher models, grounding, synthetic data, model patching, and continual learning.
Summary
Dan Jeffries describes a future in which intelligent agents are embedded throughout software. He imagines small teams producing complex games because large thinking models coordinate art, writing, animation, and other workflows. The path to that future is difficult because LLMs hallucinate, make logic errors, choose incorrect actions, leak information, and respond badly to prompt injection or social engineering. Jeffries breaks failures into training, prompting, model-to-tool interactions, outputs, and workflow steps. He argues that better models alone will not solve the problem. Production systems will need middleware that checks inputs and outputs, watcher models, grounding in external sources, synthetic data, faster fine-tuning, adapters, and continual learning. His practical advice is aimed at engineers working in the middle of the stack, where models interact with tools and applications. He is optimistic about engineering progress, but clear that autonomous agents will require much lower error rates and stronger controls than chat interfaces do.
Large thinking models could coordinate entire software workflows
Jeffries opens with a 2033 example of a small game team using large thinking models, or LTMs, to produce a major game. The models are descendants of LLMs with reasoning, large memory, grounding, and links to outside programs and sources of truth. They coordinate many stages of work behind the scenes. In his example, an artist asks for iterations of a battle robot, the system runs a long asset pipeline, and a writer later turns the character into multiple story drafts. Human artists and writers still make decisions and corrections, while the model handles much of the repetitive orchestration.
Jeffries says people often treat LLMs as if they were databases, but their main strength is rudimentary reasoning. They can collect unstructured information from places such as Discord, Slack, websites, blogs, and podcasts, then extract, summarise, merge, and organise it. He gives the example of finding every discussion of artificial intelligence across thousands of podcast episodes and putting the results into a spreadsheet with author information. These tasks were previously too large for ordinary software or a human researcher to complete quickly.
Autonomous agents make small error rates expensive
LLMs can hallucinate, choose the wrong answer, make logic mistakes, and execute the wrong action. Jeffries compares this with traditional software, where a login system has a more limited set of failure modes. An autonomous agent can make a mistake at one step in a long chain, reveal internal information, offend a customer, or produce an unsafe recommendation. Even a 0.1 percent error rate can have an outsized cost when the agent sends a damaging email or affects a major deal. People tolerate a bad chat answer more readily than a bad autonomous action.
LLM failures need to be classified across the production stack
Jeffries divides failures into training and fine-tuning problems, prompt-time failures, errors between the model and the user, errors between the model and tools or other models, output failures, and breakdowns during a workflow or DAG. This classification gives engineering teams places to investigate instead of treating every bad result as a generic model problem. He expects the middle of the stack to contain many failures because that is where LLMs interact with software and become more autonomous.
Prompt rules help in the short term but do not generalise far enough
Jeffries discusses constrained prompts, prompt padding, rules, heuristics, and watcher models that inspect inputs or outputs. These methods can limit how users ask questions or block known prompt injections. He compares them with signature-based spam filters and says they are useful while better systems are being built. They can also create unwanted effects. A prompt intended to improve demographic diversity might cause a request for Mario to produce an unexpected result instead of a male Italian plumber. Jeffries argues that general methods based on learning and search will eventually outperform collections of hand-built rules.
Middleware is a major opportunity between models and tools
Jeffries expects many serious failures to occur when an LLM is chained with commands, APIs, tools, and other models. The system may choose the wrong order, take the wrong step, or enter a text-generating loop. He proposes middleware that checks inputs and outputs at every stage. The likely design is a hybrid of traditional code and smaller watcher models that inspect results and judge whether they make sense. These errors often look like correctly formatted responses, so ordinary HTTP-style error handling cannot detect them.
Synthetic data and model-assisted labelling can speed up fixes
Jeffries says fine-tuning is slow because data labelling and human scoring create bottlenecks. Foundation-model data is often poorly labelled, and it is impossible to label every item by hand. He expects models to generate synthetic data, predict human preferences, and send only a small subset of examples to people for review. He uses an image dataset example where a clever caption about diabetes does not describe the pictured objects. Better labels at scale could improve foundation models and make it faster to fix specific failures.
Grounding connects model output to external facts and systems
Jeffries describes grounding as attaching model knowledge to text, code, databases, and external information sources. He points to code comments, vector databases, Wolfram Alpha, and systems that retrieve information from Wikipedia as examples of ways to anchor generation. He expects natural-language labels and overlays to appear across many kinds of data. The model can then use those labels and external sources while producing an answer, rather than relying only on patterns stored during training.
Adapters and continual learning could turn models into lifelong systems
Jeffries presents adapters such as LoRA as an early form of model patching. They can add capabilities without retraining the whole model, although many adapters may increase memory use or eventually hurt performance. He expects models to use large numbers of patches and may need methods to compress them. He also argues for continual learning, where models add tasks and updates without scaling memory requirements or suffering catastrophic forgetting. That would let a model keep improving as it encounters new data and failures.
"If it's an autonomous agent that has the ability to do a lot of steps without a human in the loop, even if we got the error rate down to, say, 0.1 percent, you might think that's perfect."09:25
Who should watch
You are building an LLM application that calls tools or runs multi-step workflows, and you need a way to reason about failures beyond ordinary software errors.
Your team is working on prompt controls, evaluation, fine-tuning, or middleware and wants a broad map of where production systems can break.
You are deciding whether to wrap a foundation model or work on the infrastructure that checks, grounds, patches, and improves it.
