# Designing AI Agents for the Complex Realities of Healthcare

Dr. Sarah Gebauer, Validara Health | MLOps Community | 16:03

Source: https://www.youtube.com/watch?v=n8FT8WPOeTw
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/designing-ai-agents-for-the-complex-realities-of-healthcare
Published: 2025-09-19
Tags: agents, evals, healthcare, human-in-the-loop

## TL;DR
- Medical residents provide a useful model for deploying AI agents with supervision, escalation paths, and ongoing evaluation.
- Healthcare agents must recognize when to ask for help, respond within the time available, and communicate information in a format that fits clinical work.
- Safe deployment requires local testing, workflow measurement, post-deployment monitoring, and human review of overrides and near misses.

## Summary
Dr. Sarah Gebauer compares AI agents in healthcare with medical residents. Residents make decisions with partial training, supervision, and a duty to ask for help. She argues that agents need the same structure. Their evaluation should include whether they recognize critical moments, escalate appropriately, communicate in a clinically useful format, and act within the time available. Healthcare also creates problems with incomplete inputs, incomplete outputs, disagreements between agents and clinicians, bias, legal responsibility, and differences between local patient populations. Gebauer says products must be tested on local data and measured in real workflows through user satisfaction and time studies. After deployment, teams should track performance, review near misses, study cases where humans overrode an agent, and capture feedback. In the discussion, she says clinical use remains limited, while sepsis prediction and ambient scribes are more established. She advises engineers to involve clinicians and present tools as assistants rather than replacements for diagnosis.

## Key ideas
### Medical residents offer a working model for supervised agents
[00:29](https://www.youtube.com/watch?v=n8FT8WPOeTw&t=29s)
Gebauer uses medical residents as an analogy for AI agents because residents begin work with partial training and make decisions under supervision. They may score well on exams yet struggle at the bedside. The medical system has developed safeguards around this situation over a long period. Those safeguards include supervision, continued learning, and expectations that trainees will ask for help. Gebauer argues that healthcare AI can borrow this structure instead of treating agents as either fully autonomous systems or unusable risks.

### Escalation is part of the agent's job
[02:55](https://www.youtube.com/watch?v=n8FT8WPOeTw&t=175s)
A useful agent must recognize critical junctures and contact the right person at the right time. Gebauer compares this with residents, where a strong trainee knows when to ask for help. Failure to escalate can matter when a patient is beginning to deteriorate. The agent also needs to match its response to the time available. A stroke alert may call for a short answer because treatment is urgent, while a rare-disease discussion in a neurology clinic can support a longer explanation with references.

### Communication needs to fit clinical context
[04:19](https://www.youtube.com/watch?v=n8FT8WPOeTw&t=259s)
Gebauer says communication is a major source of errors and patient harm in healthcare, so AI integration does not remove that risk. Agents need to provide the right information to the right person at the right time. She gives the example of an emergency patient with heart failure symptoms after a salt-heavy meal. When the facts arrive in an expected format, a physician can quickly form a diagnosis and treatment plan. A long, poorly ordered story slows the clinician down and makes the tool frustrating to use.

### Healthcare agents face missing information and disagreement
[06:02](https://www.youtube.com/watch?v=n8FT8WPOeTw&t=362s)
Clinicians gather many signals from a patient, including observations made while simply walking past a room. AI agents may not have comparable access to context, so a small missing detail can change a diagnosis or treatment decision. Agents can also omit information in their output, much as residents sometimes do. The supervising clinician must recognize when important facts are missing. Gebauer also raises unresolved questions about disagreements between agents, disagreements with clinicians, and the liability attached to accepting or rejecting an agent's recommendation.

### Local testing and workflow fit determine readiness
[09:20](https://www.youtube.com/watch?v=n8FT8WPOeTw&t=560s)
An agent should be tested on the data, patients, and settings where it will operate. Small differences between institutions can change which illnesses are common and how patients present. Gebauer says workflow integration will determine whether doctors, nurses, and other healthcare workers accept a product. Teams should track user satisfaction and conduct time studies, while checking whether the agent reaches the right person with the right information. Testing should also cover edge cases, failure modes, training, user acceptance, reliability, and robustness.

### Post-deployment monitoring can borrow from medical training
[10:38](https://www.youtube.com/watch?v=n8FT8WPOeTw&t=638s)
Healthcare needs stronger monitoring after deployment. Gebauer compares agent performance tracking with resident evaluations and case-log reviews. Near misses can be examined in a way similar to morbidity and mortality conferences. Teams can review cases where a human overrode the agent and capture feedback about what the agent failed to consider. This creates a way to find problems after launch instead of relying only on pre-deployment testing.

### Clinical AI adoption is still limited
[12:02](https://www.youtube.com/watch?v=n8FT8WPOeTw&t=722s)
In the discussion, Gebauer says she does not currently have AI tools in her own day-to-day clinical work. Her experience includes health technology consulting and AI model evaluation at RAND, while healthcare adoption has moved slowly. Current use cases are concentrated in administrative work such as billing and revenue cycle management. In clinical settings, she mentions predictive models for identifying patients who may develop sepsis and ambient scribes that turn doctor-patient conversations into notes.

### Clinicians should be treated as partners, not replaced
[14:34](https://www.youtube.com/watch?v=n8FT8WPOeTw&t=874s)
Gebauer advises engineers to seek clinical input and work on problems that affect real people. She says doctors often dislike tools that suggest diagnoses because figuring out a diagnosis is a valued part of the job. A product may be easier to adopt when it is presented as an assistant that supports clinical reasoning. She contrasts that with claiming that the tool will diagnose everyone, which can make clinicians feel that the most satisfying part of their work is being taken away.

## Notable quotes
- "The difference between a good resident and a bad resident is knowing when to ask for help." (04:58)
- "We still have to have excellent communication to take good care of patients." (04:38)
- "This is going to be the key to making doctors, nurses, healthcare workers happy with your product or not is if it fits into the workflow." (09:58)
- "Doctors tend to hate tools that suggest diagnosis to them because that's the one fun part of their job and they don't want to lose it." (14:55)

## Tools & references mentioned
- medical residents
- US medical licensing exam
- RAND
- LinkedIn
- Substack
- ML for MDs

## Who should watch
- You are building a clinical AI product and need to decide what supervision, escalation, and monitoring should look like.
- Your model performs well in tests but has not been evaluated on the local patients, workflows, and time constraints where it will run.
- You want clinicians to adopt an assistant without making them feel that the system is taking over diagnosis.

## Related talks

- [Building Agents for Healthcare](https://mlopstalks.com/talks/building-agents-for-healthcare) (Lars Maaløe, Corti, 30:34)
- [AI in Healthcare](https://mlopstalks.com/talks/ai-in-healthcare) (Eric Landry, Zeteo Health, 51:06)
- [Governance for AI Agent Deployment](https://mlopstalks.com/talks/governance-for-ai-agent-deployment) (Spencer Reagan, Airia, 54:18)
- [Everything Hard About Building AI Agents Today](https://mlopstalks.com/talks/everything-hard-about-building-ai-agents-today) (Shreya Shankar & Willem Pienaar, Cleric, 47:03)
- [The Future of Healthcare: AI is Here](https://mlopstalks.com/talks/the-future-of-healthcare-ai-is-here) (, 24:56)