# Productionizing Health Insurance Appeal Generation

Holden Karau, Netflix | AI in Production 2024 | 27:11

Source: https://www.youtube.com/watch?v=lfoZPp6tLm4
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/productionizing-health-insurance-appeal-generation
Published: 2024-04-10
Tags: fine-tuning, gpus, model-serving, synthetic-data

## TL;DR
- Holden Karau fine-tuned a model to turn health insurance denials into appeal letters using public California medical review records and synthetic training data.
- She deployed the model on an on-premises Kubernetes cluster because buying hardware was cheaper than renting enough GPU capacity for long-term inference.
- The project exposed practical limits of small-scale ML operations, including data access, model compatibility, hardware fit, power use, version pinning, and failed demo infrastructure.

## Summary
Holden Karau describes a personal project to generate health insurance appeals. Her starting point is the difficulty of obtaining training data. Insurance companies have little reason to help, medical offices face privacy rules, and online appeals are too scarce for fine-tuning. She uses California independent medical review records to create synthetic denials and appeals, then fine-tunes a model with rented GPU capacity. For inference, she combines a desktop with an RTX 3090, an on-premises Kubernetes cluster, and a Django front end. The talk gives a detailed account of the costs and awkward parts of this setup, including GPU hardware that does not fit server chassis, ARM compatibility problems, power limits, ephemeral storage, and containers that break when the latest tag changes. The live demo fails because the machine running the model and Kubernetes head node are down. Karau is direct about the system's early state and says it should not yet be used for real appeals.

## Key ideas
### The project starts with a personal health insurance problem and a small budget
[03:04](https://www.youtube.com/watch?v=lfoZPp6tLm4&t=184s)
Karau frames the system as a personal attempt to make health insurance disputes easier to handle. She says Anthem Blue Cross, her insurer, denied around 20% of claims in 2019, and describes dealing with medical bills after being hit by a car and having surgeries. She is clear that this is not a Netflix project and that the budget is limited. The system is an imperfect technical response to a larger social problem. It takes an insurance denial as input and produces an appeal, then adds a front end so people do not need their own GPU, Python installation, or model files.

### Public medical review records provide training material when direct records are unavailable
[05:31](https://www.youtube.com/watch?v=lfoZPp6tLm4&t=331s)
The hardest part is finding usable training data. Insurance companies are unlikely to provide records for a system intended to challenge their decisions. Doctor's offices would need to redact patient information manually, which creates substantial work and legal risk. Appeals posted online are useful but too limited for fine-tuning. Karau uses California's publicly downloadable independent medical review records instead. She says other states have similar records, although access and commercial-use restrictions vary. The published records omit patient names and much of the identifying information, so they can be used to create synthetic examples.

### Synthetic data makes fine-tuning possible, but it needs filtering and license checks
[08:00](https://www.youtube.com/watch?v=lfoZPp6tLm4&t=480s)
Karau uses language models to turn the medical review inputs into possible denial letters and appeals. This produces a larger training set than the original public records alone. The approach has two clear costs. Generated data costs money, even if it is cheaper than asking people to create it, and the generated examples may be poor. She therefore filters the data before using it. She also checks the licenses of the models used to generate the synthetic records. The fine-tuning model is selected based on both its performance and whether it will fit the hardware available.

### Fine-tuning is affordable when the work is moved to rented GPU capacity
[08:50](https://www.youtube.com/watch?v=lfoZPp6tLm4&t=530s)
Karau first used Dolly and shell scripts, then moved to Oxal, which abstracts much of the setup into Python scripts. Lambda Labs supplies the rented GPU capacity because her RTX 4090 can run inference but cannot fine-tune the model. The process still includes shell scripts for moving data and configuring machines, along with configuration for the base model, data location, sequence lengths, and sliding windows. She puts the fine-tuning cost at about $112. Oxal distributes the work across multiple GPUs and selects batch sizes that keep them heavily occupied.

### On-premises inference turns hardware compatibility into part of the ML work
[10:35](https://www.youtube.com/watch?v=lfoZPp6tLm4&t=635s)
Serving the model requires more than a model server. Karau has ARM GPU devices from Nvidia, but they do not work well with many of the tools she uses. An RTX 3090 does not fit properly in either of her servers, so she puts a desktop computer at the bottom of the equipment rack. The setup uses more power than expected, and she has only 15 amps available. Her Kubernetes deployment targets amd64 hosts and uses the Nvidia runtime class so the container can access the GPU. She also needs ephemeral storage for downloading and holding the model.

### Unpinned containers can break a deployment without an obvious source-code change
[11:50](https://www.youtube.com/watch?v=lfoZPp6tLm4&t=710s)
Karau shows a deployment that pulls the latest container tag, then explains why this is a bad practice. The container maintainers sometimes change how prompts are generated before releasing the corresponding source-code changes. A deployment can therefore stop working after an image update, even though the user did not change their own code. She recommends pinning a specific version. This is one of the small operational details that matters when a personal cluster is running the only copy of a model.

### The failed demo exposes the cost of owning the infrastructure
[14:06](https://www.youtube.com/watch?v=lfoZPp6tLm4&t=846s)
The live demo fails because the Kubernetes cluster returns no pods. Karau traces the problem to the computer running the model, then finds that the machine named Jumba is down. The cluster's head node is also the node running model inference, so the front end cannot work while that machine is unavailable. She points out that on-premises systems take real time to reboot, unlike rented infrastructure where replacement capacity can often be requested through a service. The model and deployment code are available in the project's repositories, so someone with a working Kubernetes cluster can try the setup.

### GPU inference is preferred because CPU paths are slow and incompatible with her ARM nodes
[20:19](https://www.youtube.com/watch?v=lfoZPp6tLm4&t=1219s)
Karau has tried CPU inference, but says it has not produced particularly good results. The fine-tuned model already works only moderately well, and losing additional precision makes the output less useful. The bitsandbytes library, which is used by many quantization workflows, does not compile on her ARM nodes. CPU inference on an x86 node might work, but it would be slow and those machines already consume more power. She uses vLLM because its batching could improve GPU utilization when more than one user runs an appeal request, although the current batch size is one.

## Notable quotes
- "We're going to make an ML model that'll take health insurance denials and produce appeals for people." (05:08)
- "The generated data might not be very good, so then we have to do some filtering on top of it." (08:26)
- "You should actually pin to a specific version." (12:10)
- "Please don't use this right now in real life. It is not super ready for production usage." (14:06)
- "The downsides of on-prem is it takes real time to reboot computers instead of fake time." (16:53)

## Tools & references mentioned
- MLOps Community
- Netflix
- Apache Spark
- Anthem Blue Cross
- Hugging Face
- Oxal
- Dolly
- Lambda Labs
- Kubernetes
- Nvidia
- Django
- Docker
- vLLM
- bitsandbytes
- Hurricane Electric
- Lilo and Stitch
- Timbit

## Who should watch
- You are building a small ML system and need to understand the practical costs of data collection, fine-tuning, serving, and hardware.
- You are weighing rented GPU capacity against an on-premises setup and want a candid account of power, compatibility, and maintenance problems.
- You want to see how a real prototype behaves when the infrastructure fails, rather than only hearing about a successful deployment.

## Editor's note

Holden Karau had to split fine-tuning across rented GPUs and inference across on-premises Kubernetes, while dealing with hardware and compatibility limits. ZenML lets a pipeline use a configured stack, so the same Python workflow can run on a laptop, Kubernetes, Airflow or Kubeflow, or a cloud provider's services without rewriting its steps.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Tecton Round-table // Get your ML Application Into Production](https://mlopstalks.com/talks/tecton-round-table-get-your-ml-application-into-production) (Kevin Stumpf, Derek Salama, Eddie Esquivel & Isaac Cameron, Tecton, 55:42)
- [War Stories Productionising ML](https://mlopstalks.com/talks/war-stories-productionising-ml) (Nick Masca, Marks and Spencer, 50:48)
- [Productionizing AI: How to Think From the End](https://mlopstalks.com/talks/productionizing-ai-how-to-think-from-the-end) (Annie Condon, 11:11)
- [From Idea to Production ML](https://mlopstalks.com/talks/from-idea-to-production-ml) (Lex Beattie, Spotify, 53:18)
- [Don't Listen Unless You Are Going to Do ML in Production](https://mlopstalks.com/talks/dont-listen-unless-you-are-going-to-do-ml-in-production) (Kyle Morris, banana.dev, 51:30)