# Designing ML Infra for ML & LLM Use Cases

Amritha Arun Babu, Klaviyo & Abhik Choudhury, IBM | MLOps Podcast | Episode 221 | 1:00:18
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=KbSu6DqzzEk
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/designing-ml-infra-for-ml-llm-use-cases
Published: 2024-03-29
Tags: governance, model-serving, monitoring, platform-teams

## TL;DR
- MLOps maturity moves from manual experimentation and deployment toward standardized pipelines, model versioning, monitoring, and automated retraining.
- The right MLOps metrics connect platform work to business outcomes, especially the time it takes to move an idea from ideation to production.
- LLM systems add concerns around data access, GPU costs, evaluation, scalability, bias, serving, and compliance with third-party models.

## Summary
Abhik Choudhury describes MLOps maturity as a progression from manual work to standardized experimentation, packaged pipelines, model registries, continuous monitoring, and automated responses. He says many organizations remain between the earliest levels because business constraints make full automation difficult. Amritha Arun Babu explains how a product leader can connect platform work to business value by identifying data scientists, ML engineers, and analytics teams as customers, then mapping their pain points to outcomes such as shorter development and retraining cycles. The conversation also compares traditional ML with LLM use cases. LLM systems need large, continuously refreshed data sources, can create significant GPU costs, and are harder to evaluate with a single accuracy measure. Abhik also stresses scalability, model serving, bias, and compliance, especially when teams call third-party models. Both guests argue that platform teams should define metrics leadership understands, including developer time, deployment speed, model performance, and the business effect of poor or outdated predictions.

## Key ideas
### MLOps shortens a machine learning lifecycle that is otherwise too long for daily business use
[07:37](https://www.youtube.com/watch?v=KbSu6DqzzEk&t=457s)
Abhik Choudhury says organizations are starting to understand MLOps as a way to apply continuous integration and continuous delivery to machine learning. A machine learning pipeline can take a long time from data work through deployment, which makes it difficult to support changing business needs. MLOps helps teams shorten and streamline that lifecycle. He also places MLOps behind DevOps in maturity because the field has had fewer years to develop common practices. His experience is that many organizations, including technically mature companies, remain somewhere between the earliest maturity levels rather than reaching full automation.

### MLOps maturity begins with manual work and progresses through standardization and automation
[09:49](https://www.youtube.com/watch?v=KbSu6DqzzEk&t=589s)
Abhik describes maturity level zero as an organization that has started thinking about MLOps but still handles experimentation, versioning, and deployment largely by hand. At level one, data preparation, modeling, training, evaluation, and validation use standardized templates and the pipeline is automated. Teams package and deploy the whole pipeline, rather than treating it like ordinary application code. He describes the highest level as full continuous integration and delivery, although he says many organizations have not reached it. A customer he began working with in 2018 moved from level zero toward level two over several years, but constraints prevented complete automation.

### A platform's main business metric is how quickly it moves an idea from ideation to production
[15:35](https://www.youtube.com/watch?v=KbSu6DqzzEk&t=935s)
Amritha Arun Babu says the North Star for an ML platform is enabling scientists to take an idea from ideation to production in as little time as possible. That goal can be broken into measures for each stage. Standardized libraries, SDKs, templates, and data-quality information can reduce the time spent exploring data and training models. Abhik gives a customer example involving customer segmentation in wholesale distribution. The team initially accepted a lifecycle lasting several months, then discovered that changing behavior and new biases had made the deployed model outdated. That business problem created a reason to increase automation and shorten the cycle.

### Model registries and monitoring make automated systems easier to change when business needs shift
[19:39](https://www.youtube.com/watch?v=KbSu6DqzzEk&t=1179s)
Abhik says a more mature pipeline combines data preparation, training, evaluation, and validation into one package, then deploys it through orchestration. Registering model versions lets a team return to an earlier model when a new version performs poorly or when the business needs to change. At a more automated level, teams monitor data quality, model metrics, and other KPIs, then trigger updates when conditions change. One approach retrains on a schedule with new hyperparameters. Another responds to data skew or other discrepancies. Amritha adds that monitoring at early maturity levels may require sampling model outputs and checking them against ground truth by hand, because teams may lack continuous data pipelines and monitoring tooling.

### LLM operations need continuously refreshed data access and careful control of compute costs
[23:50](https://www.youtube.com/watch?v=KbSu6DqzzEk&t=1430s)
Amritha says LLM operations differ from early-stage traditional ML operations because LLM use cases often require access to very large internal and external datasets. At level zero, a scientist may receive one batch of data. At level one, the platform should provide catalog information, lineage, schemas, sampling, and continuously refreshed access. Abhik adds that GPU costs can grow quickly when clients use models such as GPT-4 for experimentation. That cost makes teams cautious about moving directly to complete automation. The guests also point out that traditional ML remains more established in many businesses, while LLM use cases are still being tested and evaluated.

### Designing an ML platform starts by treating its users as customers with distinct pain points
[29:15](https://www.youtube.com/watch?v=KbSu6DqzzEk&t=1755s)
Amritha applies product design thinking by identifying the primary and secondary users of an ML platform, then working backward from their use cases and problems. Her examples include data scientists building fraud detection or recommendation models, ML engineers supporting them, and analytics or BI engineers. The platform team can map a problem such as difficult retraining to a business consequence, such as less precise recommendations or a weaker ability to compete. This process connects work on data exploration, training, evaluation, and release to an outcome that leadership can understand. It also helps the team decide which platform improvements deserve attention.

### Teams should translate developer friction into time, cost, and risk
[34:03](https://www.youtube.com/watch?v=KbSu6DqzzEk&t=2043s)
Abhik explains that data scientists may be comfortable with Python but struggle when they need Spark for large-scale or distributed computation. The platform team can express this problem in terms of the skills people already have, the skills they need to acquire, and the hours required to learn them. Those hours can be connected to the cost of the scientists' time and multiplied across a larger organization. The calculation still has uncertainty because priorities change and people may misuse a technology after learning its basics. Even so, expressing friction through hours, money, and risk gives product and engineering teams a shared way to discuss platform investment.

### AI engineers must account for scale, model behavior, serving, and compliance before production
[48:45](https://www.youtube.com/watch?v=KbSu6DqzzEk&t=2925s)
For engineers moving from full-stack development into AI products, Abhik identifies several production concerns. Data volume can grow with seasonality and trends, so the data pipeline and cloud infrastructure must scale before teams optimize it. Models also need checks for bias and outliers. Serving has to meet the business requirement as data volume and model performance change. Compliance and privacy need attention when a product uses a third-party model through an API, since the team may not know whether its handling of data matches company policy. Amritha recommends security and legal reviews for releases, especially when data crosses regions with different rules.

## Notable quotes
- Amritha Arun Babu: "The North Star metric underneath is to make sure that you as a platform can enable scientists to get their idea from ideation phase to production as short a time as possible." (15:35)
- Amritha Arun Babu: "In level zero, you would just be able to get one batch of data. In level one, I would expect my scientist to be able to get all these data in a continuously refreshed manner." (24:55)
- Abhik Choudhury: "You need to be really wary about the scalability because data multiplies and particularly if there is a lot of seasonality and trend, it can easily overshoot what you are thinking and your whole pipelines are capable of handling." (49:07)
- Abhik Choudhury: "As soon as you put it into MLOps, you have to come up with your chief architect or someone who has the bigger picture." (53:55)

## Tools & references mentioned
- MLOps Community
- DevOps
- MLOps
- LLM Ops
- Docker
- Kubernetes
- Spark
- GPT-4
- OpenAI
- LatticeFlow AI

## Who should watch
- You are building an ML platform and need to explain its value beyond adding more automation.
- Your team is moving from manual model deployment toward standardized pipelines, registries, and monitoring.
- You are taking an LLM feature into production and need to think through data access, evaluation, GPU cost, scaling, and compliance.

## Editor's note

Abhik Choudhury describes teams whose customer-segmentation models became outdated because their machine learning lifecycle lasted several months. ZenML records each pipeline run's steps, inputs, outputs, and code version, so a team can trace a model to what produced it and compare runs when business conditions change. That record supports the versioning and monitoring process discussed here.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Engineering MLOps](https://mlopstalks.com/talks/engineering-mlops) (Emmanuel Raj, TietoEvry, 51:55)
- [Design and Development Principles for LLMOps](https://mlopstalks.com/talks/design-and-development-principles-for-llmops) (Andy McMahon, Barclays Bank, 1:10:18)
- [Enterprises Using MLOps, the Changing LLM Landscape, MLOps Pipelines](https://mlopstalks.com/talks/enterprises-using-mlops-the-changing-llm-landscape-mlops-pipelines) (Chris Van Pelt, Weights & Biases, 47:51)
- [MLOps - The Blind Men and the Elephant](https://mlopstalks.com/talks/mlops-the-blind-men-and-the-elephant) (Saurav Chakravorty, Brillo, 55:02)
- [MLOps vs LLMOps](https://mlopstalks.com/talks/mlops-vs-llmops) (Richa Sachdev, JPMorgan Chase & Willem Pienaar, Feast & Chris Van Pelt, Weights & Biases & Aparna Dhinakaran, Arize AI & Alex Ratner, Snorkel, 34:57)