# Building AI That Remembers You

Sarah Wooders, Letta | Agent Hour | 19:35
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=VgttOIWfoeo
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/building-ai-that-remembers-you
Published: 2025-02-17
Tags: context-engineering, deployment, memory, tool-use

## TL;DR
- Letta treats context management as the central job of an agent framework, deciding what information reaches the language model and how it is compacted.
- Letta agents persist message histories, use in-context and external memory, and can learn user preferences from offline data before making recommendations.
- Letta runs agents as services with APIs, custom tools, sandboxed tool execution, local deployment through Letta Desktop, and cloud deployment through Letta Cloud.

## Summary
Sarah Wooders presents Letta as an open source framework and platform for stateful agents. Its main responsibility is context management: collecting agent state, memories, tools, and runtime information, then packing the most useful material into the language model's context window. Letta automatically persists message histories, compacts information, summarizes or evicts older messages, and helps avoid context overflow. The agent can also write to in-context memory and an external vector store. Wooders describes applications such as personalized email and recommendations based on offline processing of transaction histories, user data, and company guidelines. Letta agents run as API services rather than remaining inside notebooks. The platform supports custom tools, authentication through Compos, sandboxing for insecure tools, local execution through Letta Desktop, and managed deployment through Letta Cloud. Wooders is direct that the language model itself chooses what to store, although developers can prompt or replace the memory tools when they need a different structure.

## Key ideas
### Letta puts context management at the centre of an agent framework
[01:03](https://www.youtube.com/watch?v=VgttOIWfoeo&t=63s)
Sarah Wooders says the main engineering problem is deciding what data to feed the language model and how to feed it. She calls this context management. The framework should compile the context and interface with the model so developers can get better performance from agents that call tools and retain memory. She also treats deployment as part of the framework. Agents that begin as Python scripts or notebooks need to become services that can run autonomously and expose an API. Letta therefore combines context handling with agent hosting rather than leaving both tasks to the application developer.

### Too much context can make an agent less useful
[05:04](https://www.youtube.com/watch?v=VgttOIWfoeo&t=304s)
Wooders describes the model context window as the main input developers can control. Adding more information does not always improve an agent. With too much data, a language model can become distracted, lose track of what matters, or summarize details too generally. With too little data, it lacks relevant information and is more likely to make something up. Applications such as recommendation systems may contain product offerings, company policies, transaction histories, and user data. Letta's job is to select and organize this material instead of placing everything in the context window.

### The Letta service stores state and compiles it into model context
[07:20](https://www.youtube.com/watch?v=VgttOIWfoeo&t=440s)
The Letta stack places a service between an application and the language model. Applications connect through REST APIs or SDKs, while the Letta service manages memory, agent state, tool execution, and additional data sources. It can process state with another language-model call to make the context more concise, then pass that compiled context to the model. When the model produces a tool call, Letta translates it into an update to the agent's state. The state is stored in a model-agnostic way, so an existing agent with its history and memories can change model providers.

### Letta supports personalization through persistent memory and offline processing
[08:16](https://www.youtube.com/watch?v=VgttOIWfoeo&t=496s)
Wooders gives an example of an agent that reviews interactions and other user data to build a personalized memory for writing emails. In another example, a payments platform has transaction histories, user data, and marketing guidelines that cannot fit into a single context window. Letta processes the data offline and repeatedly updates learned user preferences and historical trends. The resulting memories can then support real-time offer recommendations. This approach is presented as a way for agents to learn from large collections of data rather than relying only on a retrieval step at request time.

### Agents are exposed as services from the start
[11:24](https://www.youtube.com/watch?v=VgttOIWfoeo&t=684s)
Letta treats every agent as an agent service, so developers do not need a separate deployment step to turn a notebook experiment into an API. Its API can create agents and send messages, while also providing granular access to memory, conversation history, and step history. Developers can modify agent state through the API. Wooders says Letta provides autogenerated Python and TypeScript SDKs. The agent development environment gives developers a place to inspect what enters the context window and iterate on prompts, tools, and model settings.

### Memory combines message history, external storage, and in-context information
[14:18](https://www.youtube.com/watch?v=VgttOIWfoeo&t=858s)
In response to a question from Demetrios Brinkmann, Wooders explains that Letta automatically persists message histories and lets the agent read and write to an external vector store. It also allocates part of the context window for in-context memory. The agent uses tools to write to both forms of memory, while the language model orchestrates how memory changes over time. The design follows the approach described in the MemGPT paper, with the model helping decide what information should remain available.

### Developers can override how the model writes memories
[17:13](https://www.youtube.com/watch?v=VgttOIWfoeo&t=1033s)
Wooders says Letta memories are written by the language model through tools such as core memory replace and core memory append. Prompting can influence what the model stores. Developers can also replace Letta's memory tools with custom tools when memories need a specific structure or format. She argues that text is a practical memory representation because language models work with text directly. She also describes a newer multi-agent approach, where one agent specializes in managing another agent's memory and can revise the full memory over time instead of making only incremental changes.

### Tool security and deployment can be handled inside standard infrastructure
[15:54](https://www.youtube.com/watch?v=VgttOIWfoeo&t=954s)
Letta integrates with Compos for tool connectors and authentication scopes. Wooders says insecure tools run in a sandbox automatically, which helps prevent dangerous tools from running directly in the server. Teams with stricter requirements can deploy the Letta server themselves as a Docker image. She says it can also run on Kubernetes, allowing the service to fit into existing infrastructure. Letta's deployment options include an open source server, Letta Desktop for fully local agents, and Letta Cloud for building, testing, versioning, and deploying agents without managing infrastructure.

## Notable quotes
- Sarah Wooders: "We want to basically solve this key engineering challenge of what data do we feed to the LLM and how." (01:03)
- Sarah Wooders: "The intelligence of your agent is bounded by what you put into the LLM context window." (05:04)
- Sarah Wooders: "The memory management, the memories are actually all written by the LLM itself." (17:13)
- Sarah Wooders: "We basically telling it, remember to remember important things." (17:54)

## Tools & references mentioned
- Letta
- MemGPT
- UC Berkeley
- Rise Lab
- Postgres
- FastAPI
- Compos
- Letta Agent Development Environment
- Letta Desktop
- Letta Cloud
- Docker
- Kubernetes
- Python
- TypeScript
- R1
- Llama

## Who should watch
- You are building an agent that needs to retain information across sessions and want the agent service to manage state instead of your application.
- Your current agent is a notebook or Python script, and you need an API service with persistent memory, tool execution, and deployment options.
- You need to control how an agent remembers user information, inspect its context, or run tools inside a sandbox or private deployment.

## Related talks

- [AI Needs Memory: Here's How It Works](https://mlopstalks.com/talks/ai-needs-memory-heres-how-it-works) (Ben Labashian, Workhelix, 29:02)
- [Overcoming Agentic Memory Management Challenges](https://mlopstalks.com/talks/overcoming-agentic-memory-management-challenges) (Biswaroop Bhattacharjee, Prem AI, 50:35)
- [The Real Reason Your AI Models Forget](https://mlopstalks.com/talks/the-real-reason-your-ai-models-forget) (Jeff Huber, Chroma, 44:55)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlopstalks.com/talks/mem0-building-production-ready-ai-agents-with-scalable-long-term-memory) (Nehil Jain, Stealth AI Startup & David DeStefano, Adonis, 58:25)
- [AI Agents: The Evolution of Our Digital Friends](https://mlopstalks.com/talks/ai-agents-the-evolution-of-our-digital-friends) (, 10:16)