# Managing Small Knowledge Graphs for Multi-agent Systems

Tom Smoker, WhyHow.ai | MLOps Podcast | Episode 236 | 1:04:41
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=LhWtpV-ZEeI
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/managing-small-knowledge-graphs-for-multi-agent-systems
Published: 2024-05-28
Tags: agents, multi-agent, rag, reliability, testing

## TL;DR
- Tom Smoker uses knowledge graphs to constrain retrieval, expose errors, and make RAG pipelines more repeatable.
- Small graphs should capture the context needed for a specific problem, rather than trying to model an entire domain in one graph.
- Multi-agent systems need individually grounded, tested modules because uncertainty compounds across sequential model calls.

## Summary
Tom Smoker explains why WhyHow.ai uses knowledge graphs alongside vector and relational databases in RAG systems. His focus is reliability in high-value settings where an answer cannot simply be approximately correct. A graph can add context, structure, provenance, and a way to inspect and correct extracted information. Tom argues for small, well-connected graphs built around a particular use case, with schemas that can change as the problem becomes clearer. He describes graphs as an indexing layer over embedded chunks, allowing retrieval to return more focused context. The conversation then moves to multi-agent systems. Tom prefers decomposing workflows into small, typed, tested components, with each agent grounded in the information it needs. He is cautious about trusting agents with consequential actions such as booking travel. The practical goal is to use knowledge graphs as part of workflows that solve valuable business problems, rather than spending large amounts of engineering time perfecting chatbots that have little business value.

## Key ideas
### Knowledge graphs add control when RAG answers cannot be wrong
[06:12](https://www.youtube.com/watch?v=LhWtpV-ZEeI&t=372s)
Tom Smoker describes working on legal RAG questions where an incorrect answer could trigger the wrong contractual interpretation. In that setting, the team needed more than a likely answer from a model. They used representation and knowledge graphs to constrain the system, inspect what it had produced, and adjust the result. Tom says the aim is repeatability and some degree of determinism. He does not expect an LLM itself to be deterministic, so the surrounding workflow has to narrow the scope and provide structure that can be checked.

### Being exactly wrong gives the team something it can improve
[10:17](https://www.youtube.com/watch?v=LhWtpV-ZEeI&t=617s)
Tom explains his phrase "exactly wrong" as a practical first step toward reliability. A graph can reveal that its representation does not match what the team intended. If the same wrong answer is produced repeatedly, the team has a stable failure to diagnose. That is more useful than a system that is probably right but changes its answer unpredictably. Tom sees this as a framework for iterative improvement, where the schema, inputs, and retrieval process can be changed until the output is acceptable.

### Graphs, vectors, and relational databases fit different parts of one pipeline
[08:46](https://www.youtube.com/watch?v=LhWtpV-ZEeI&t=526s)
Tom does not describe a knowledge graph as a replacement for a vector database. WhyHow.ai uses vector search for semantic similarity, then uses the retrieved material to build or query a graph. He also uses relational databases when the data is already well structured and the query is specific. In his view, information retrieval is a process inside a larger system, and different stores can contribute at different stages. A graph is useful when the system needs structure, changing schemas, focused context, and repeatable retrieval.

### The right graph depends on whether its data and schema change
[12:42](https://www.youtube.com/watch?v=LhWtpV-ZEeI&t=762s)
Tom describes a four-part way to think about graph design: the underlying data can be static or dynamic, and the schema can also be static or dynamic. A veterinary example separates relatively stable disease information from changing patient information. He prefers separate graphs when the two kinds of data have different update patterns. This keeps the representation manageable and lets the team bring in only the diseases or patient details that matter to the current question.

### Context determines whether a small graph is sufficient
[25:40](https://www.youtube.com/watch?v=LhWtpV-ZEeI&t=1540s)
Tom says a graph is too small when it lacks enough context to answer the problem. Small does not mean that the domain itself is narrow. It means starting with a well-connected subgraph and expanding it as needed. He contrasts a graph's ability to return a complete set with vector search's tendency to return the most similar items. In a diagnosis setting, a professional may need every available diagnosis to compare against patient history, rather than only the most likely result.

### A graph can filter retrieval and preserve provenance
[31:06](https://www.youtube.com/watch?v=LhWtpV-ZEeI&t=1866s)
Tom describes the graph as an indexing layer over stored information. Each graph node connects back to the embedded chunk from which it came. After the graph narrows the relevant concepts and relationships, the system can return the selected chunks, node summaries, properties, document names, or provenance. This helps avoid context poisoning, where a top-K result contains a few useful chunks mixed with unrelated material. Tom's aim is to give the model only the information needed for the next step.

### Multi-agent workflows need small, testable modules
[36:07](https://www.youtube.com/watch?v=LhWtpV-ZEeI&t=2167s)
Tom prefers decomposing an agentic task into steps that the team understands and can test. He describes using structured grounding for each agent, along with typed inputs and outputs, rather than giving a system one broad natural-language instruction. His workflows use multiple model calls and ordinary software components, with unit, input-output, and integration tests. This also makes it possible to compare model changes in a pipeline and inspect latency and behavior across individual steps.

### Uncertainty compounds across agents, so each agent needs its own grounding
[40:09](https://www.youtube.com/watch?v=LhWtpV-ZEeI&t=2409s)
Tom gives a simple example of five sequential agents that each perform at 95 percent. Even without assuming perfectly independent failures, the combined process can be much less reliable than any individual call. He wants each agent to have a specific schema, graph, or source of grounded information, then iteratively improve its inputs and outputs. He sees this modular structure as a path toward larger workflows such as procurement, shipping, and supply-chain operations, while admitting that these systems are not yet trustworthy enough for many consequential actions.

### Production value matters more than a perfect chatbot
[56:10](https://www.youtube.com/watch?v=LhWtpV-ZEeI&t=3370s)
Tom is skeptical that asking questions in natural language and receiving an answer is the main destination for this technology. He wants to work on business processes where faster or more reliable information has a clear effect, such as radiology reporting, legal work, healthcare operations, or industrial workflows. He gives the example of an HR chatbot that could answer a leave-policy question but might have little value compared with the engineering effort required. WhyHow.ai treats knowledge graphs as one part of a workflow that helps useful systems reach production.

## Notable quotes
- Tom Smoker: "Being exactly wrong is the first step to being exactly right." (10:17)
- Tom Smoker: "Rice to a farmer is different to rice to a chef." (23:03)
- Tom Smoker: "I just want to give the LLM only the information it needs and no more." (34:30)
- Tom Smoker: "It's easy to build more agents than it is to build a better agent right now." (45:27)
- Tom Smoker: "I don't know if people writing natural language and getting an answer back is the future of this stuff." (56:30)

## Tools & references mentioned
- WhyHow.ai
- BHP
- Tim French
- Wayo Melingowitz
- Andre Karpathy
- KuzuDB
- DuckDB
- Neo4j
- ArangoDB
- LlamaIndex
- LangChain
- Cohere
- LangSmith
- Pydantic
- Instructor
- FastAPI
- AutoGPT
- OpenAI
- GPT-3
- GPT-4 Omni
- TensorFlow
- Jason Liu
- Joseph Halpern
- Air Canada
- Linux

## Who should watch
- You are building a RAG system where likely answers are not sufficient and you need to inspect, constrain, and repeat retrieval behavior.
- Your multi-agent workflow chains several model calls and you need a practical way to reduce propagated errors and test each step.
- You are deciding whether improving a chatbot will create business value, or whether the real opportunity lies in a workflow such as legal review, healthcare reporting, or industrial operations.

## Editor's note

Tom Smoker says the aim of these graph-backed workflows is repeatability and some degree of determinism. ZenML records each pipeline step, its inputs, outputs, and code version, so teams can trace a result to the run that produced it and compare changes across runs. Its dashboard lets them inspect those runs and artifacts without rebuilding that history by hand.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Graphs and Language](https://mlopstalks.com/talks/graphs-and-language) (Louis Guitton, 11:28)
- [Building a Decentralized Knowledge Graph for AI](https://mlopstalks.com/talks/building-a-decentralized-knowledge-graph-for-ai) (Tomaž Levak, OriginTrail, 52:07)
- [Integrating Knowledge Graphs & Vector RAG for Efficient Information Extraction](https://mlopstalks.com/talks/integrating-knowledge-graphs-and-vector-rag-for-efficient-information-extraction) (Nehil Jain, Stealth AI Startup & Sonam Gupta, AICamp & Matt Squire & Valdimar Eggertsson, Snjallgögn (Smart Data inc.) & Binoy Perera, MLOps Community, 48:47)
- [The Intersection of Graphs and Large Language Models](https://mlopstalks.com/talks/the-intersection-of-graphs-and-large-language-models) (Anthony Alcaraz, Fribl, 15:08)
- [Unleashing Unconstrained News Knowledge Graphs to Combat Misinformation](https://mlopstalks.com/talks/unleashing-unconstrained-news-knowledge-graphs-to-combat-misinformation) (Robert Caulk, Emergent Methods, 1:15:25)