# When Agents Learn to Feel: Multi-Modal Affective Computing in Production

Chenyu Zhang, GlowingStar | Coding Agents Conference 2026 | 20:34

Source: https://www.youtube.com/watch?v=1H0HO1yOGNs
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/when-agents-learn-to-feel-multi-modal-affective-computing-in-production
Published: 2026-02-22
Tags: agents, memory, multimodal, privacy

## TL;DR
- Affective agents add emotional perception and modeling to the usual planning, memory, tool, and action loops.
- Production systems can combine voice tone, facial signals, response latency, silence, and interaction patterns, but conflicting signals can produce incorrect emotional conclusions.
- Emotion-aware systems need explicit consent, privacy controls, and validation across cultures and demographics because emotional inferences are sensitive and often unreliable.

## Summary
Chenyu Zhang argues that AI agents need to respond to a user's emotional state as well as the user's words. He proposes adding a multimodal perception layer and an explicit emotional modeling module to existing agent architectures. These components can estimate confusion, frustration, disengagement, or motivation and feed those estimates into planning, memory, and interaction. In tutoring, an agent might remember that a learner struggled with recursion while frustrated, then change its pacing later. Zhang also describes the production difficulties: signals can conflict, emotional data is highly sensitive, and recognition systems can fail across cultures and demographics while expressing high confidence. He argues for consent, transparency, and cautious deployment. In the question period, he says both large models and ensembles of smaller models could support affective sensing, and that future systems could process signals beyond text, audio, and images, including smell, taste, and touch.

## Key ideas
### Emotional attunement is an unfinished part of agent design
[02:04](https://www.youtube.com/watch?v=1H0HO1yOGNs&t=124s)
Zhang says current agents are moving from tools toward autonomous systems, but they still lack emotional attunement. Humans use affective cues to judge safety, trust, confusion, and engagement. Without those cues, an agent may give correct information in a way that fails when a user is frustrated, disengaged, or excited. He places emotion alongside thinking and doing as part of how people learn, collaborate, and make decisions. His claim is that future agents need to sense and adapt to how people feel, not only what they say.

### Affective agents add perception and emotional modeling to standard loops
[05:00](https://www.youtube.com/watch?v=1H0HO1yOGNs&t=300s)
The Google agent architecture Zhang discusses includes orchestration, short- and long-term memory, reasoning and planning, and tool use. It does not include a representation of affective context. Zhang's proposed architecture makes multimodal perception and emotional modeling first-class components. Perception can ingest voice, facial expressions, response latency, and interaction patterns. The emotional model estimates the user's state and passes it into reasoning and planning. In a multi-agent system, one agent could detect frustration, another could critique an explanation, and another could rewrite it in a calmer style.

### Production sensing depends on fusing imperfect modalities
[08:17](https://www.youtube.com/watch?v=1H0HO1yOGNs&t=497s)
Zhang describes affective perception as a multimodal problem involving vision, voice, interoception, and context. Production systems use simpler signals such as voice tone, visual action units, typing latency, and silence. The difficulty is combining those signals without overfitting or drawing the wrong conclusion about a person. He also assigns tools an affective role. With MCP and other tool access, an agent could retrieve user-specific data or store a long-term history of emotional patterns. Tool orchestration therefore affects how the system interprets recurring frustration or disengagement.

### Emotion-tagged memory can change how an agent teaches
[09:46](https://www.youtube.com/watch?v=1H0HO1yOGNs&t=586s)
Zhang says agents currently treat inputs too uniformly, with weak forgetting and weak abstraction. Emotional information can disappear because it is not tagged or prioritized. He proposes shallow episodic memory that records both what happened and how the user felt. In tutoring, if a learner struggled with recursion and showed frustration the previous week, the agent could remember that episode and adjust future pacing. Emotional context can also act as an internal motivation signal. A disengaged learner may need re-engagement, while an excited learner may be ready for a harder challenge.

### Emotion data requires stronger privacy and consent controls
[11:48](https://www.youtube.com/watch?v=1H0HO1yOGNs&t=708s)
Zhang says emotional data can reveal internal states that people may not consciously express, making it more sensitive than ordinary behavioral data, personally identifiable information, or personal health information. Users may not know when their cues are being analyzed or how the resulting inferences are stored and used. He warns that detecting fear, confusion, or enthusiasm could let systems influence people in ways they did not choose. He also mentions risks to minors using AI companionship apps and the possibility that outsourcing emotional labor could affect human relationships and emotional resilience.

### Emotion recognition can be confident and wrong
[13:18](https://www.youtube.com/watch?v=1H0HO1yOGNs&t=798s)
Affective systems do not generalize reliably across cultures, contexts, and demographics. Zhang says misclassification is common even when systems express high confidence. He connects this problem to current evaluation practices, where datasets may be labeled by a small number of annotators who cannot represent every individual's experience. In the question period, he says existing benchmarks can improve and that privacy should be the default unless a person gives explicit consent for their data to be used for training.

### Future affective agents may process more than the usual three modalities
[16:18](https://www.youtube.com/watch?v=1H0HO1yOGNs&t=978s)
Zhang says current systems mainly use text, audio, and sometimes images, but he does not view those as a fixed limit. He names smell and taste as possible future inputs and connects the idea to embodied agents and robots. He expects similar training processes could apply if these signals can be modeled and collected at scale. He returns to emotion as a separate concern from human senses, calling it an underdeveloped area of agent development rather than simply another input channel.

## Notable quotes
- "For agents to behave intelligently with humans, emotional signals must shape their decisions just as strongly as goals or instructions do." (07:50)
- "The opportunity is to introduce shallow episodic memory with emotional tagging. Not only what happened, but how the user felt." (10:22)
- "Unless you gave the consent explicitly to allow the agents to train on your data, by default, we should say no." (20:00)
- "Emotions are the missing piece in the AI agent development or is the underdeveloped area right now." (18:48)

## Tools & references mentioned
- MLOps Community
- MIT Media Lab
- Stanford HAI
- Harvard
- GlowingStar
- Google
- MCP
- Plato
- Aristotle

## Who should watch
- You are designing an agent that needs to react to confusion, frustration, motivation, or disengagement rather than treating every user interaction alike.
- You are building education or other sensitive-domain systems and need a concrete account of consent, emotional-data privacy, and evaluation limits.
- You are deciding whether affective sensing belongs in one large model or in an ensemble of smaller, specialized components.

## Related talks

- [Agentic Relationship Management](https://mlopstalks.com/talks/agentic-relationship-management) (Ashe Magalhaes, Hearth AI, 10:01)
- [The Coming Revolution of AI Agents](https://mlopstalks.com/talks/the-coming-revolution-of-ai-agents) (Jazmia Henry, Iso AI & Rogerio Bonatti, Microsoft & Julia Kroll, Deepgram & Joshua Alphonse, PremAI, 28:59)
- [Why Emotion Matters More Than Sound](https://mlopstalks.com/talks/why-emotion-matters-more-than-sound) (Anoop Dawar, Deepgram & Ajeet Grewal, Sierra, 26:44)
- [9 Commandments Building AI Agents](https://mlopstalks.com/talks/9-commandments-building-ai-agents) (Paul van der Boor & Dmitri Jarnikov, Prosus Group, 1:20:34)
- [What's Special About Meta's Multi-Agent Systems](https://mlopstalks.com/talks/whats-special-about-metas-multi-agent-systems) (Aditya Gautam, Meta, 28:46)