# Why Planning is the New Search | MLOps Community | 27:48 Source: https://www.youtube.com/watch?v=_U5ovDn50mw Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/why-planning-is-the-new-search Published: 2024-12-06 Tags: human-in-the-loop, planning, search, workflows ## TL;DR - Knowledge work is largely information retrieval followed by decisions, reasoning, and actions across open-ended company processes. - Large language models struggle with autonomous planning, especially when real work depends on private company context, precedent, and several acceptable outcomes. - Interloom grounds plans in observed workflows, knowledge graphs, clustering, and human review before assigning subtasks to people or agents. ## Summary Fabian argues that much knowledge work is repeated searching across inboxes, files, business systems, and conversations, followed by decisions and actions. This work is difficult to automate because company processes depend on private context, employee knowledge, and precedent that is rarely present in public training data. Large language models can generate plans, but Fabian says they perform poorly on autonomous planning and cannot reliably choose among several feasible outcomes. Interloom approaches the problem as a work navigation system. It builds plans from past cases using knowledge graphs, process-mining-like patterns, reranking, and hierarchical clustering. Humans can inspect and adjust the proposed plan, while agents can be assigned to individual subtasks. The Pathfinder view shows how similar cases moved through task clusters. Fabian expects smaller, well-contextualized tasks to produce more reliable agent behavior, while companies gradually collect enough precedent to automate selected work. He is clear that customers still want control for higher-risk processes. ## Key ideas ### Knowledge work is repeated search followed by action [01:59](https://www.youtube.com/watch?v=_U5ovDn50mw&t=119s) Fabian defines knowledge work as retrieving information, refining it through a decision or reasoning step, and moving it toward a productive outcome. People search inboxes, Google, files, Slack, Teams, case-management systems, and Salesforce. He says this work makes up a large share of activity across back offices, including tax processing, insurance claims, finance, and IT support. Much of it remains manual because the relevant information is spread across language-based systems and employee conversations. ### Planning has to account for company-specific context [06:21](https://www.youtube.com/watch?v=_U5ovDn50mw&t=381s) A work plan depends on how a particular company actually handles its cases. Fabian describes this as corporate memory and know-how, including the patterns used by a railroad company or a support desk. Those patterns are usually absent from public data because they can be a competitive advantage. The same request can have several acceptable outcomes, and the best choice may depend on whether the company values speed, customer value, or another business constraint. ### Language models struggle with autonomous planning [07:49](https://www.youtube.com/watch?v=_U5ovDn50mw&t=469s) Fabian says language models are not built for reliable autonomous planning. He points to Blocks World and Simple Bench as examples where models perform poorly on planning tasks. Real cases are harder because they are open-ended and depend on context. A request involving a delayed purchase order may require weeks or months of work, hundreds or thousands of subtasks, and coordination among several people rather than a single linear answer. ### Work can be treated as a navigation problem [12:07](https://www.youtube.com/watch?v=_U5ovDn50mw&t=727s) Fabian compares Interloom's approach with Google Maps. A destination can have many possible routes, but the system presents feasible options based on context such as the mode of travel or an added stop. He proposes a similar navigation system for work. A case becomes a destination, and the system proposes a feasible sequence of tasks based on the case context and what happened in similar cases. ### Interloom builds plans from precedent and workflow patterns [14:23](https://www.youtube.com/watch?v=_U5ovDn50mw&t=863s) In the product demonstration, a new order case includes system data, an email, and information from a colleague about earlier customer delays. The system combines a knowledge graph, process-mining-like analysis, clustering, and a language model to propose a plan. One plan routes triage to a colleague. After Fabian adds the context that the customer is Carpenter Solutions, the system replans and adds a 10% discount based on a similar precedent case. ### The Pathfinder view makes the source of a plan visible [17:47](https://www.youtube.com/watch?v=_U5ovDn50mw&t=1067s) Interloom's Pathfinder view visualizes how similar cases moved through task clusters. The size of each dot shows the size of a cluster, and language models label clusters such as invoice management, client follow-up, and case triage. Users can follow the current case through the graph and inspect which cases went to which people. Fabian says this visibility matters because large companies do not immediately want completely opaque agent automation. ### Small tasks with better context make agents more useful [20:20](https://www.youtube.com/watch?v=_U5ovDn50mw&t=1220s) Once a plan is split into subtasks, Interloom can assign each subtask to a human or an agent. Fabian demonstrates assigning an agent called Odin to draft an apology email that includes a 10% discount. The agent can search the internet, internal repositories, and company knowledge bases. Fabian's view is that agents become more accurate when the problem is smaller and the context is more faithful, much like assigning a task to a person with the right skills. ### Automation should increase gradually as companies collect precedent [25:22](https://www.youtube.com/watch?v=_U5ovDn50mw&t=1522s) Fabian says companies do not currently want a fully self-driving platform for every process. Interloom plans an accuracy-control system that would compare a proposed action with similar precedent cases and estimate whether the result is safe to release. The initial approach keeps humans involved and speeds up end-to-end work. Over time, as the system collects more company-specific precedent, selected simpler tasks could be automated with less supervision. ## Notable quotes - "Knowledge work is nothing else but literally working with information retrieval, which is nothing else but searches and taking that information somehow refining it by making a decision or reasoning over it and then enriching it and then pushing it along to towards a productive outcome." (02:20) - "The problem is that LLMs are just not really good at planning." (07:49) - "From our perspective it's now possible for the first time to actually build a navigation system of work." (13:03) - "We believe they're already good enough if you make the problems small enough, if you assign them to the right agents with the right capabilities, just as you assign them tasks to the right humans with the right capabilities and skills." (23:43) ## Tools & references mentioned - Interloom - Air Street Capital - Google - Google Maps - Hyperscience - Blocks World - Simple Bench - Odin - GPT - knowledge graphs - process mining ## Who should watch - You are building agents for enterprise workflows and need a way to ground their plans in private company data. - Your team is deciding how much human review to keep in case management, claims, logistics, or support processes. - You want to see how clustering, workflow history, and knowledge graphs can turn unstructured employee activity into reusable task plans. ## Related talks - [Everything We Got Wrong About Research-Plan-Implement](https://mlopstalks.com/talks/everything-we-got-wrong-about-research-plan-implement) (Dexter Horthy, HumanLayer, 26:46) - [Agents as Search Engineers](https://mlopstalks.com/talks/agents-as-search-engineers) (Santoshkalyan Rayadhurgam, Meta, 29:38) - [How Agentic Workflows Will Change Everything](https://mlopstalks.com/talks/how-agentic-workflows-will-change-everything) (Raj Rikhy, Microsoft, 49:13) - [Before Building AI Agents Watch These Hard Earned Lessons](https://mlopstalks.com/talks/before-building-ai-agents-watch-these-hard-earned-lessons) (Nishikant Dhanuka, Prosus Group, 52:37) - [A New Way of Building with AI](https://mlopstalks.com/talks/a-new-way-of-building-with-ai) (Jiquan Ngiam, Lutra AI, 1:04:46)