# Tecton Round-table // Get your ML Application Into Production

Kevin Stumpf, Derek Salama, Eddie Esquivel & Isaac Cameron, Tecton | MLOps Coffee Sessions | 55:42
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=sT3T03s8wJk
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/tecton-round-table-get-your-ml-application-into-production
Published: 2023-09-29
Tags: cost, feature-engineering, feature-stores, platform-teams

## TL;DR
- The biggest production bottleneck is often the handoff from data scientists to understaffed ML, data, or software engineers, especially when teams need both batch and real-time features.
- Teams should preserve historical data, separate operational databases from analytical systems, and make data access secure and auditable before building production models.
- A recommendation system can start with a simple heuristic and batch pipeline, then move toward custom models and real-time infrastructure once the business value justifies the added complexity.

## Summary
The roundtable examines why traditional ML applications remain difficult to put into production. Eddie Esquivel describes friction between data scientists and the teams that must productionize their feature definitions. The panel also discusses upstream data failures, organizational scale, cost attribution, training-serving skew, and the difficulty of reusing features when teams do not trust shared pipelines. Kevin Stumpf recommends preserving historical states in a warehouse or lake, separating analytical data from operational databases, and adding streaming only when the use case needs fresh predictions. Derek Salama explains that infrastructure cost must be connected to business value, using an Uber pricing example where an expensive model did not make economic sense in every market. For a new recommendation product, the panel favors starting with a simple heuristic and batch data, measuring its value, then adding model complexity and real-time systems as the use case earns them.

## Key ideas
### The handoff between data science and engineering creates a large production bottleneck
[03:36](https://www.youtube.com/watch?v=sT3T03s8wJk&t=216s)
Eddie Esquivel says data scientists often build models and features on their laptops, then throw a feature recipe over the wall to an understaffed ML, data, or software engineering team. That team must turn the recipe into live production features. Batch features are difficult enough, while streaming features add more complexity. Kevin Stumpf adds that teams can face an earlier obstacle: collecting the right data and giving people secure access to it. Derek Salama says the best realistic target may initially be making ML projects as manageable as ordinary software projects, although ML still brings extra data and skill requirements.

### A model's infrastructure cost has to be compared with the value of its use case
[10:27](https://www.youtube.com/watch?v=sT3T03s8wJk&t=627s)
Derek Salama gives an Uber pricing example from around 2015 or 2016. A sophisticated deep learning model generated additional rides, but cost about 50 cents in extra compute per ride. That could make sense in North America, where the profit margin per ride was a few dollars, but not in India, where the margin could be about 10 cents. The team chose not to roll the model out in places where its extra value did not cover its infrastructure cost. Derek recommends tagging infrastructure and bringing together costs from ETL, databases, model serving, and other services so teams can calculate the cost of a particular use case.

### Large and small ML teams face different kinds of production constraints
[17:28](https://www.youtube.com/watch?v=sT3T03s8wJk&t=1048s)
Eddie Esquivel says larger, more advanced teams have more requirements around security, service-level agreements, uptime, and related controls. Smaller teams can be more nimble, although they still face production challenges. Derek Salama adds that a larger organization usually has more varied use cases, such as recommendations, fraud, and pricing. A platform built for one use case may struggle to satisfy all of them. The panel warns that putting very different requirements into one system can create problems, even when the larger organization has more staff and resources.

### Historical data and clear separation from operational systems should come before production ML
[20:26](https://www.youtube.com/watch?v=sT3T03s8wJk&t=1226s)
Kevin Stumpf says every ML use case needs training data and prediction data, and training requires knowing what the world looked like at earlier points in time. Without that history, teams risk data leakage or label leakage. A simple approach is taking daily snapshots of an operational database into separate partitions in a warehouse or lake. A more advanced approach appends immutable events whenever a transaction or state changes. Kevin also calls for centralized historical data, secure and audit-logged access, and a standard environment where data scientists can train models. Isaac Cameron warns that driving features directly from an online transactional database is a red flag.

### Streaming should be added when fresher data has measurable value
[24:20](https://www.youtube.com/watch?v=sT3T03s8wJk&t=1460s)
The panel distinguishes batch use cases such as lead scoring and churn outreach from real-time cases such as fraud detection, recommendations, and insurance quotes. Real-time use cases often need a streaming layer, using systems such as Kafka or Kinesis, so downstream ML applications can process events as they arrive. Isaac Cameron explains that moving from batch processing to streaming introduces new data engineering problems, including changed-data processing and late-arriving data. Derek Salama recommends simulating the benefit before building the extra infrastructure. Teams can compare offline metrics such as area under the curve or log loss under daily updates versus an assumed streaming setup.

### Training-serving skew makes separate batch and online pipelines hard to maintain
[30:21](https://www.youtube.com/watch?v=sT3T03s8wJk&t=1821s)
Eddie Esquivel describes training-serving skew as a common problem. A real-time system may calculate features for an online model, while a separate pipeline creates training data. Those paths may use different languages or logic, which can produce inconsistent features. The panel describes a shared approach in which the same computation can support batch and streaming features and write to online and offline stores. Real-time features computed only when a prediction is requested also matter, especially when the transaction context is known only at inference time. Their value depends on the use case, but search terms and a user's immediate context can strongly affect recommendations.

### Repeatable design and requirements work matter more than a complex architecture
[35:39](https://www.youtube.com/watch?v=sT3T03s8wJk&t=2139s)
The panel says scaling from one model to many requires a repeatable process for gathering requirements, designing features, testing them, and deploying them. Eddie Esquivel treats a diagram with many boxes and arrows as a warning sign, especially when it contains multiple warehouses, streaming queues, or ways to generate training data. Derek Salama says teams often underinvest in the design phase. They should define data granularity, entities, table structure, feature requirements, and expected value before building. This can expose features that take weeks to develop but have little business value. Legacy migrations are often harder because teams must reverse-engineer poorly documented, interconnected code.

### Feature reuse depends on trust, ownership, and automated dependency handling
[41:12](https://www.youtube.com/watch?v=sT3T03s8wJk&t=2472s)
Derek Salama observes that leaders want feature reuse, while individual data scientists often prefer their own version of a feature even when an existing version is nearly identical. Kevin Stumpf says reuse requires agreement on definitions and governance, such as one shared meaning for an active customer or transactions in the last 90 days. It also requires trust that a shared feature will remain available and behave consistently. Dependency detection and notifications should be automated through CI pipelines or similar tooling. Without that protection, teams tend to create duplicate copies so they are not surprised by a downstream model failure.

## Notable quotes
- Derek Salama: "I think the best we can aspire to is, you know, it's as hard as like a software project." (05:58)
- Derek Salama: "You can't manage what you don't measure." (16:41)
- Eddie Esquivel: "You don't get extra points for more boxes." (37:05)
- Kevin Stumpf: "You have to have the right tooling in place to ensure that data pipeline reuse actually takes place." (45:08)
- Kevin Stumpf: "Start with the heuristic to get something out there." (52:43)

## Tools & references mentioned
- Tecton
- Uber
- AWS
- Spark
- Snowflake
- Kafka
- Kinesis
- Confluent
- S3
- Jupyter Notebook
- Postgres
- AWS Cost Manager

## Who should watch
- You are handing feature definitions from data scientists to an ML or data engineering team and need to reduce the production delay.
- Your team is deciding whether fresher streaming features justify more infrastructure and operational cost.
- You are building a recommendation system from limited data and want a staged path from a heuristic to a custom real-time model.

## Editor's note

Eddie Esquivel says data scientists often hand an understaffed engineering team a feature recipe that must be rebuilt for production, with separate batch and streaming paths creating training-serving skew. ZenML lets teams define workflows as Python pipelines and run the same code on different infrastructure through a configurable stack. Each run records its steps, inputs, outputs, and code version for tracing.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [From Idea to Production ML](https://mlopstalks.com/talks/from-idea-to-production-ml) (Lex Beattie, Spotify, 53:18)
- [Just Build It! Tips for Making ML Engineering and MLOps Real](https://mlopstalks.com/talks/just-build-it-tips-for-making-ml-engineering-and-mlops-real) (Andy McMahon, NatWest Group, 48:17)
- [Don't Listen Unless You Are Going to Do ML in Production](https://mlopstalks.com/talks/dont-listen-unless-you-are-going-to-do-ml-in-production) (Kyle Morris, banana.dev, 51:30)
- [War Stories Productionising ML](https://mlopstalks.com/talks/war-stories-productionising-ml) (Nick Masca, Marks and Spencer, 50:48)
- [Scaling AI in Production](https://mlopstalks.com/talks/scaling-ai-in-production) (Srivatsan Srinivasan, AIEngineering, 51:56)