Pack · 7 talks · 4h 17m to watch, 41 min to read

The cost of production ML

The bill is growing, but buying cheaper machines is only one possible response. First identify which project caused the spend and what useful work it purchased. Cleo supplies a small review process; a streaming account exposes the price of aggressive freshness and recovery settings. Lyft then asks whether a larger model earns its repeated training cost. Shared GPUs make idle reservations visible, while an inference benchmark separates throughput, responsiveness and hardware price. The LLM panel adds a simpler intervention: avoid calls that do not need an expensive model. Finish with Replit's interrupted servers, where cheaper capacity only became useful after recovery improved. These accounts span data pipelines, classical forecasting and language-model services. Their historical prices and speedups are examples, not estimates for your bill.

2
Rohit Agrawal, Tecton · 47:39 · MLOps Podcast
Streaming Ecosystem Complexities and Cost Management

Why here: A high streaming bill may reflect settings rather than necessary workload. Agrawal connects checkpoint frequency and retained stream history to recovery and freshness needs. Those are explicit cost controls: decide how much recent state must be recoverable and which history can move to batch storage before treating every write as unavoidable.

3
Josh Xi, Lyft · 53:42 · MLOps Podcast
Real-Time Forecasting Faceoff: Time Series vs. DNNs

Why here: The model itself can impose an unnecessary recurring bill. Xi's Lyft comparison puts CPU-based autoregressive training beside a spatial DNN and explains why a small, adjustable model fits the forecasting task. This adds quality and update frequency to the cost calculation; the reported training-cost difference belongs to that workload, not every classical-versus-neural comparison.

4
Ronen Dar & Gijsbert Janssen van Doorn, Run:ai · 1:03:33 · MLOps Coffee Sessions
GPU For Machine Learning

Why here: Allocated capacity is not the same as used capacity. Dar's 2022 account describes exclusive GPU reservations and fractional allocation for workloads too small to fill an accelerator. After questioning how much computation the model needs, inspect how much hardware the allocation policy charges it for. The historical scheduler discussion illustrates the mismatch without ranking current products.

5
Timothée Lacroix, Mistral · 30:25
Exploring the Latency/Throughput & Cost Space for LLM Inference

Why here: A cheaper device is useful only if it meets the workload's throughput and response-time requirements. Lacroix separates those objectives and compares candidate hardware through short direct benchmarks. Use his 2023 examples to design a workload-specific comparison, rather than carrying their prices or preferred device into a new purchase.

6
Lina Weichbrodt & Luis Ceze, OctoML & Jared Zoneraich, Prompt Layer & Daniel Campos, Neeva & Mario Kostelac, Intercom · 36:06 · LLMs in Production 2023
Cost Optimization and Performance

Why here: Sometimes the least expensive inference is a call you avoid. Kostelac describes putting a simple classifier before a costly model, while the panel asks whether request volume justifies a specialist team. This checks the total operating decision after hardware tuning: both unnecessary calls and the people maintaining an optimization belong in the comparison.

7
Bradley Heilbrun, Replit · 12:42 · LLMs in Production 2023
Preemption Chaos and Optimizing Server Startup

Why last: A discount buys weaker capacity guarantees too. Heilbrun's 2023 Replit account makes preemption, zone shortages and slow replacement startup part of the economics, then describes the recovery improvements that made the cheaper nodes practical. Finish by measuring useful service after interruptions, so a lower rental price does not disguise a system that spends its time restarting.