# The cost of production ML

A pack of 7 sessions from the MLOps Community YouTube channel, in the order to watch them. 4h 17m of video.
Page: https://mlopstalks.com/packs/the-cost-of-production-ml

The bill is growing, but buying cheaper machines is only one possible response. First identify which project caused the spend and what useful work it purchased. Cleo supplies a small review process; a streaming account exposes the price of aggressive freshness and recovery settings. Lyft then asks whether a larger model earns its repeated training cost. Shared GPUs make idle reservations visible, while an inference benchmark separates throughput, responsiveness and hardware price. The LLM panel adds a simpler intervention: avoid calls that do not need an expensive model. Finish with Replit's interrupted servers, where cheaper capacity only became useful after recovery improved. These accounts span data pipelines, classical forecasting and language-model services. Their historical prices and speedups are examples, not estimates for your bill.

## This pack is for you if

- Your shared cloud bill rose and you cannot attribute the increase to a project or workload.
- You need cheaper production predictions while preserving the quality and response time the task requires.
- Cheaper capacity looks attractive, but downtime and recovery could erase the savings.

## The talks, in order

### 1. How To Cut Your Data Infrastructure Costs in Half

Jose Navaro, Cleo | 12:33 | MLOps Community
Video: https://www.youtube.com/watch?v=JNIWCGPCXC8
Summary: https://mlopstalks.com/talks/how-to-cut-your-data-infrastructure-costs-in-half.md

Why first: Cost work needs a specific owner and an observable change. Navaro shows how project and environment tags turn a shared-service increase into an investigation, then keeps the process small with a weekly review. Start there so later optimizations target a known bill rather than whichever expensive resource attracts attention.

### 2. Streaming Ecosystem Complexities and Cost Management

Rohit Agrawal, Tecton | 47:39 | MLOps Podcast
Video: https://www.youtube.com/watch?v=0axFbQwHEh8
Summary: https://mlopstalks.com/talks/streaming-ecosystem-complexities-and-cost-management.md

Why here: A high streaming bill may reflect settings rather than necessary workload. Agrawal connects checkpoint frequency and retained stream history to recovery and freshness needs. Those are explicit cost controls: decide how much recent state must be recoverable and which history can move to batch storage before treating every write as unavoidable.

### 3. Real-Time Forecasting Faceoff: Time Series vs. DNNs

Josh Xi, Lyft | 53:42 | MLOps Podcast
Video: https://www.youtube.com/watch?v=zeTnQAKb_2U
Summary: https://mlopstalks.com/talks/real-time-forecasting-faceoff-time-series-vs-dnns.md

Why here: The model itself can impose an unnecessary recurring bill. Xi's Lyft comparison puts CPU-based autoregressive training beside a spatial DNN and explains why a small, adjustable model fits the forecasting task. This adds quality and update frequency to the cost calculation; the reported training-cost difference belongs to that workload, not every classical-versus-neural comparison.

### 4. GPU For Machine Learning

Ronen Dar & Gijsbert Janssen van Doorn, Run:ai | 1:03:33 | MLOps Coffee Sessions
Video: https://www.youtube.com/watch?v=clU3Z18eyeY
Summary: https://mlopstalks.com/talks/gpu-for-machine-learning.md

Why here: Allocated capacity is not the same as used capacity. Dar's 2022 account describes exclusive GPU reservations and fractional allocation for workloads too small to fill an accelerator. After questioning how much computation the model needs, inspect how much hardware the allocation policy charges it for. The historical scheduler discussion illustrates the mismatch without ranking current products.

### 5. Exploring the Latency/Throughput & Cost Space for LLM Inference

Timothée Lacroix, Mistral | 30:25 | MLOps Community
Video: https://www.youtube.com/watch?v=mYRqvB1_gRk
Summary: https://mlopstalks.com/talks/exploring-the-latency-throughput-cost-space-for-llm-inference.md

Why here: A cheaper device is useful only if it meets the workload's throughput and response-time requirements. Lacroix separates those objectives and compares candidate hardware through short direct benchmarks. Use his 2023 examples to design a workload-specific comparison, rather than carrying their prices or preferred device into a new purchase.

### 6. Cost Optimization and Performance

Lina Weichbrodt & Luis Ceze, OctoML & Jared Zoneraich, Prompt Layer & Daniel Campos, Neeva & Mario Kostelac, Intercom | 36:06 | LLMs in Production 2023
Video: https://www.youtube.com/watch?v=wxq1ZeAM9fc
Summary: https://mlopstalks.com/talks/cost-optimization-and-performance.md

Why here: Sometimes the least expensive inference is a call you avoid. Kostelac describes putting a simple classifier before a costly model, while the panel asks whether request volume justifies a specialist team. This checks the total operating decision after hardware tuning: both unnecessary calls and the people maintaining an optimization belong in the comparison.

### 7. Preemption Chaos and Optimizing Server Startup

Bradley Heilbrun, Replit | 12:42 | LLMs in Production 2023
Video: https://www.youtube.com/watch?v=ofcJIrhadfc
Summary: https://mlopstalks.com/talks/preemption-chaos-and-optimizing-server-startup.md

Why last: A discount buys weaker capacity guarantees too. Heilbrun's 2023 Replit account makes preemption, zone shortages and slow replacement startup part of the economics, then describes the recovery improvements that made the cheaper nodes practical. Finish by measuring useful service after interruptions, so a lower rental price does not disguise a system that spends its time restarting.