# Knowledge is Eventually Consistent Devin Stein, Dosu | MLOps Podcast | Episode 335 | 55:15 Hosted by Demetrios Brinkmann Source: https://www.youtube.com/watch?v=HvtzIx1vgmc Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/knowledge-is-eventually-consistent Published: 2025-08-17 Tags: agents, coding-agents, human-in-the-loop, memory ## TL;DR - Dosu's fact-based reasoning agent stores claims with supporting evidence after a human or maintainer confirms the answer, then reuses those facts in later conversations. - Knowledge automation should move from human approval to draft suggestions and eventually to automatic updates when the system has enough confidence. - Code is a useful source of truth because changes can be compared with documentation, allowing an agent to find stale or conflicting knowledge. ## Summary Devin Stein describes how Dosu evolved from answering questions and triaging issues into a fact-based reasoning agent for product and engineering knowledge. The agent researches a question, records claims with evidence, and saves confirmed facts for later use. Stein explains why audience matters when deciding whether an agent should answer, how aliases and context help resolve company-specific language, and why concise responses reduce the cost of mistakes. He favors a gradual path to automation: explicit human approval first, proactive drafts next, and automatic saving only when confidence is high. Stein argues that written knowledge matters more as coding agents become common because agents are capable but forgetful. Code can provide a system of record against which canonical documentation is checked. Dosu also aims to provide trusted knowledge to other agents, such as coding and root-cause-analysis tools, while keeping runbooks and other operational information current. ## Key ideas ### Dosu learns reusable facts from confirmed answers [00:42](https://www.youtube.com/watch?v=HvtzIx1vgmc&t=42s) Devin Stein explains that Dosu began by answering questions and triaging issues across Slack and GitHub, using code, commits, conversations, and tickets. Its fact-based reasoning agent records claims supported by evidence during each investigation. When a user or maintainer confirms the response, those facts enter the knowledge base. A later question starts by checking what the agent already knows, then researches only what is missing. Stein describes this as a learning loop: repeated use gives the agent more facts and can make responses faster. ### The audience helps determine when an agent should respond [03:50](https://www.youtube.com/watch?v=HvtzIx1vgmc&t=230s) Stein says that deciding when to intervene is difficult in general, but the audience provides a useful signal in Dosu's domain. A maintainer may need only a pointer to a code location or recent work, or may not need a response. A new or less technical user usually benefits from information that moves them toward an answer. Dosu currently distinguishes experts, who curate knowledge, from normal users who are less familiar with the domain. Stein says future audience profiles could distinguish non-technical users from engineers who understand code but do not know a particular codebase. ### Company-specific language needs context and relationships [08:40](https://www.youtube.com/watch?v=HvtzIx1vgmc&t=520s) Dosu handles terms that have different meanings across companies, products, and codebases by storing aliases and synonyms. Stein gives examples where a front-end name differs from a historic back-end name, and where the word task can refer to a Celery task, an LLM task, or a background task. The agent also has to infer implied meaning in shared conversations. A question such as 'How do we do this?' requires understanding who 'we' refers to and what the channel is discussing. Stein calls this conversation implicature a difficult but useful part of deciding what someone wants. ### Concise answers and confirmed facts limit agent noise [12:30](https://www.youtube.com/watch?v=HvtzIx1vgmc&t=750s) Demetrios Brinkmann points out that an agent can add cognitive load by producing a long answer to a simple question. Stein agrees that language models tend to be more verbose than human counterparts and says Dosu has invested in shorter responses. Linking to source documents can reduce repetition. Knowing when the model does not know is harder. Stein mentions research from FAIR at Meta suggesting that reasoning agents can become worse at recognizing uncertainty because they talk themselves into an answer during extended reasoning. Dosu uses the questioner's audience and confirmed facts as signals for whether a response is appropriate. ### Knowledge should move through gradual stages of automation [16:17](https://www.youtube.com/watch?v=HvtzIx1vgmc&t=977s) Stein says Dosu first tried generating facts automatically from all ingested data, but that approach created an unsafe middle ground where an incorrect claim might spread. The current path begins with a human explicitly saving a fact, reviewing a preview, and editing it. The next stage has Dosu identify a new or conflicting claim and prepare a draft for approval. The eventual goal is automatic extraction when the agent is confident, with uncertain cases sent to review. Stein says experts are willing to approve facts because saving one answer can prevent them from answering the same question again. ### Coding agents make written knowledge more important [21:35](https://www.youtube.com/watch?v=HvtzIx1vgmc&t=1295s) Stein argues that coding agents currently have poor memory and do not learn from doing in the same way people do. They work better when documentation explains concepts such as billing and points directly to the directories where those concepts are implemented. Agents can consume dense text without needing the presentation style that helps people. At larger projects, asking a coding agent to change an unfamiliar system can be dangerous because the engineer may not understand the impact well enough to review the result. Documentation gives the engineer enough context to act as a reviewer and partner to the coding agent. ### Code can expose stale canonical documentation [28:30](https://www.youtube.com/watch?v=HvtzIx1vgmc&t=1710s) Stein separates temporal records, such as meeting notes, from knowledge intended to remain a source of truth. In product and engineering work, code provides a clear record of how the product works and how that state changes. Dosu can compare code changes with the current knowledge base and detect when documents or facts may no longer match. This distinction avoids treating every old document as something that must be rewritten. The goal is to keep canonical knowledge current while preserving records of what happened at a particular point in time. ### Dosu aims to provide maintained knowledge to other agents [32:24](https://www.youtube.com/watch?v=HvtzIx1vgmc&t=1944s) Stein agrees with Demetrios Brinkmann's idea that Dosu could work alongside agents for root-cause analysis and coding. Dosu can provide knowledge to tools such as Cursor or Cleric, while keeping runbooks from becoming stale. A wrong runbook could cause another agent to take an inappropriate action. Stein compares documentation with a knowledge cache: someone has already searched across conversations and sources, then compiled the result so later agents do not repeat that work. He also expects documentation formats and information organization to change when agents become major consumers. ## Notable quotes - Devin Stein: "Code is truth; it really tells you how your product actually works." (01:44) - Devin Stein: "The quality of your knowledge, I think, also dictates the quality of your responses." (14:55) - Devin Stein: "Agents are very, very smart, but they are forgetful." (22:06) - Devin Stein: "Documentation is as like a knowledge cache in some ways." (33:42) - Devin Stein: "Maintaining knowledge is just the effort to do that to be able to monitor all the different changes conversations happening in organization and reflect that in documentation is near impossible." (38:27) ## Tools & references mentioned - Dosu - MLOps Community - GitHub - Slack - ClickUp - Jira - Confluence - Notion - Celery - FAIR - Meta - Cursor - Cleric - MLflow - Databricks - Hyperbolic - Nvidia H100 ## Who should watch - You are building an internal agent that needs to decide when to answer, when to ask for approval, and when to stay quiet. - Your team has useful knowledge spread across code, tickets, chat, pull requests, and old documentation, and you need a way to keep canonical answers current. - You are adding coding or root-cause-analysis agents and want them to use runbooks and product knowledge without trusting stale documents. ## Related talks - [How Agent Memory Will Change Your Organization](https://mlopstalks.com/talks/how-agent-memory-will-change-your-organization) (Devin Stein, Dosu, 24:33) - [AI Agents: The Evolution of Our Digital Friends](https://mlopstalks.com/talks/ai-agents-the-evolution-of-our-digital-friends) (, 10:16) - [Beyond the Gold Standard: Evaluating and Trusting Agents in the Wild](https://mlopstalks.com/talks/beyond-the-gold-standard-evaluating-and-trusting-agents-in-the-wild) (Sanjana Sharma, Prosus, 24:45) - [Why Agents Are Stupid & What We Can Do About It](https://mlopstalks.com/talks/why-agents-are-stupid-what-we-can-do-about-it) (Dan Jeffries, Kentauros AI, 31:58) - [9 Commandments Building AI Agents](https://mlopstalks.com/talks/9-commandments-building-ai-agents) (Paul van der Boor & Dmitri Jarnikov, Prosus Group, 1:20:34)