# Overcoming Agentic Memory Management Challenges Biswaroop Bhattacharjee, Prem AI | MLOps Community | 50:35 Hosted by Demetrios Brinkmann Source: https://www.youtube.com/watch?v=XZ0LkF_rFs0 Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/overcoming-agentic-memory-management-challenges Published: 2025-10-17 Tags: context-engineering, memory, multimodal, tool-use ## TL;DR - Cortex builds memory relationships with named relationship types, scores, bidirectional links, merging, and retained history. - Cortex combines global semantic search with hierarchical auto-collections to reduce irrelevant retrieval results. - Biswaroop wants agents to use memory as a tool and choose retrieval settings such as date ranges, search depth, and recency. ## Summary Biswaroop Bhattacharjee explains how he built Cortex after studying agentic memory systems and human memory. The design stores summaries, keywords, tags, and relationships between memories instead of treating every memory as an isolated vector. Cortex names relationships, assigns relationship strength, supports merging, keeps history, and adds bidirectional links for retrieval. Its newer auto-collections organize memories into hierarchical topics such as work, projects, and programming languages. Retrieval combines a global search with a narrower collection search, using a composite score to control noise. Cortex also supports temporal queries, date ranges, short-term versus long-term memory, and configurable retrieval depth. Biswaroop argues that memory should eventually be an agent-controlled tool, potentially exposed through MCP, rather than a system whose many settings are manually tuned. He also describes planned multimodal memory that can retain more context from video, images, and audio than text summaries do. ## Key ideas ### Agentic memory connects memories instead of storing isolated chunks [00:21](https://www.youtube.com/watch?v=XZ0LkF_rFs0&t=21s) Biswaroop began with AMM, which analyzes each incoming memory before storing it. The system creates a context summary, keywords, and tags, then searches for similar existing memories. An LLM examines those representations and decides whether relationships exist. If it finds one, the system updates the existing metadata and adds richer metadata to the new memory. Retrieval also uses an LLM to analyze the query, then performs parallel searches with the query and its extracted keywords. This adds processing cost, but AMM prioritizes memory quality over latency. ### Human memory inspired a separation between short-term and long-term storage [06:52](https://www.youtube.com/watch?v=XZ0LkF_rFs0&t=412s) Biswaroop describes short-term memory as a constantly changing window of recent interaction. Long-term memory grows in the background, where information can be processed and connected over time. He relates this to human consolidation during rest, while noting that an AI system does not need to sleep before processing memories. The discussion also covers spaced repetition and forgetting. Some researchers argue that AI should remove outdated information, while others prefer archiving everything and retrieving only the most useful material. ### Cortex gives relationships names, strengths, and reversible merges [14:02](https://www.youtube.com/watch?v=XZ0LkF_rFs0&t=842s) Cortex extends the relationship structure by naming links such as "extends" and "definition of." The system can assign a relationship strength between zero and one, but only records a strong score when the LLM is confident that a meaningful relationship exists. It can also decide whether two memories should be merged. An update concatenates memories into a new memory, while the metadata retains the prior history so the pieces can potentially be separated later. Cortex also changes links from one-way to bidirectional, allowing retrieval to use both sides of a relationship. ### Hierarchical auto-collections narrow searches that are ambiguous in a flat memory graph [20:15](https://www.youtube.com/watch?v=XZ0LkF_rFs0&t=1215s) Biswaroop says current memory systems often use a flat structure with connected clusters, while human thinking also uses categories and subcategories. Cortex creates smart auto-collections such as work, a current company, a project, or Python. Memories receive categories in the background, and repeated categories eventually form collections with descriptions and query helpers. A query such as "fix this" can then be interpreted through collection context instead of matching every memory containing the word "fix." The purpose is higher-quality retrieval with less irrelevant material, rather than faster search. ### Cortex combines global search and constrained collection search [30:58](https://www.youtube.com/watch?v=XZ0LkF_rFs0&t=1858s) Cortex runs a global search and an auto-collections search in parallel. The collection path first finds relevant collections through their descriptions, then searches inside them with a modified query. Collection relevance receives 30 percent of the composite score, while the memory-level search receives 70 percent. The resulting memories are combined with global results. Operators can control how many memories are returned, disable global search with a code flag, or increase the retrieval limit when relevant information is missing. ### Temporal queries adjust retrieval toward recent or date-bounded memories [35:34](https://www.youtube.com/watch?v=XZ0LkF_rFs0&t=2134s) Cortex supports queries for relative periods such as today, last week, or last year, as well as explicit date ranges. A temporal weight between zero and one controls how much retrieval favors recency. At one, the system searches the short-term window for the most recent information. Intermediate values expand the long-term search window and combine temporal behavior with the normal composite scoring. Currently, keyword checks determine the weight, although Biswaroop says an LLM could make that decision if extra latency is acceptable. ### Biswaroop wants agents to control memory retrieval as a tool [43:12](https://www.youtube.com/watch?v=XZ0LkF_rFs0&t=2592s) The current Cortex design exposes many parameters that a person could tune, but Biswaroop wants an agentic system to call Cortex and choose those settings itself. The agent could decide the date range, retrieval depth, and other parameters based on the request. He connects this idea to MCP and to context engineering, where controlling tool definitions and injected information affects task completion. He also says giving an agent too many tools can reduce accuracy and increase input-token cost, so the memory interface should limit unnecessary freedom. ### Multimodal memory should preserve context that text summaries discard [47:16](https://www.youtube.com/watch?v=XZ0LkF_rFs0&t=2836s) Biswaroop is working on vision memory for Cortex. He expects agentic memory to include video, images, and audio rather than relying only on text. Converting a video or recording into a text summary makes it searchable, but loses context. He points to work from Memories AI on indexing video with a vision-oriented memory model. His preferred direction is a shared representation based on vectors or other numerical forms, with text acting as a higher-level view rather than the only stored representation. ## Notable quotes - Biswaroop Bhattacharjee: "So the key differentiating factor was it sort of forms relationships." (01:12) - Biswaroop Bhattacharjee: "So there are a few people who actually believes that yeah AI systems need to forget because information gets outdated." (10:44) - Biswaroop Bhattacharjee: "So imagine memory as a tool." (43:11) - Biswaroop Bhattacharjee: "But you're losing context there. So this is not a lossless compression." (48:48) ## Tools & references mentioned - Cortex - AMM - GraphRAG - Obsidian - Zettelkasten - Anki - MCP - Memories AI - Prem AI ## Who should watch - You are designing memory for an agent and need more than a vector store with similarity search. - Your retrieval results contain too much irrelevant context, especially when one phrase can refer to several parts of a user's life or work. - You are deciding how much control to give an agent over memory search, temporal filters, and tool use. ## Related talks - [Multi-Agent Personalization with Shared Memory: From Email to Website to Proposal](https://mlopstalks.com/talks/multi-agent-personalization-with-shared-memory-from-email-to-website-to-proposal) (Hamed Taheri, Personize.ai, 16:57) - [Context Engineering 2.0: MCP, Agentic RAG & Memory](https://mlopstalks.com/talks/context-engineering-2-0-mcp-agentic-rag-memory) (Simba Khadder, Redis, 24:35) - [AI Needs Memory: Here's How It Works](https://mlopstalks.com/talks/ai-needs-memory-heres-how-it-works) (Ben Labashian, Workhelix, 29:02) - [How Agent Memory Will Change Your Organization](https://mlopstalks.com/talks/how-agent-memory-will-change-your-organization) (Devin Stein, Dosu, 24:33) - [The Real Reason Your AI Models Forget](https://mlopstalks.com/talks/the-real-reason-your-ai-models-forget) (Jeff Huber, Chroma, 44:55)