Team structure and existing platform choices often determine which models can reach production.
2
Batch scoring is the simplest serving pattern, while online prediction needs careful handling of latency, features, caching, and cost.
3
A practical design starts with the simplest architecture that delivers value and can be maintained by the available team.
Summary
Rebecca Taylor explains why model-serving designs in industry often differ from the architectures taught in courses. The available team, approved technology stack, cloud platform, business requirements, latency targets, and maintenance budget all affect what can be deployed. She walks through several patterns, starting with batch scoring, where a job reads data and writes predictions to tables. A database-backed API can expose those results to other services. For online prediction, the model can run inside an API when the caller already has the required features. Historical or slower-moving features require additional data pipelines, an online feature store, or caching, which introduces cost and latency decisions. Taylor also describes a pattern that combines precomputed predictions for existing users with a lighter real-time model for users who lack historical data. Her advice is to keep the first design simple, confirm that it creates value, and account for who will operate it.
Team structure determines how much of a model survives into production
Taylor says companies hand models between teams in very different ways. Data scientists may build notebooks and proof-of-concepts before an ML Ops or DevOps team rebuilds them. In other cases, they provide a model artifact and configuration files, or the model is tested in one language and later rewritten in another. A mixed product team can own the system end to end, including parts of the frontend, backend, data engineering, feature-store contents, monitoring, and alerts. That structure improves collaboration, but the same people still have to maintain older models while building new ones.
A design must fit the tools and cloud platform that a company already permits
Taylor describes cases where engineers join a company and discover that their preferred tools are unavailable. Cloud services may only work on a particular provider, and inherited architecture decisions can leave teams with combinations that are difficult to integrate. Engineers may spend substantial time connecting systems instead of building model functionality. She recommends asking detailed questions about the technology stack during interviews, including whether the company maintains an internal ML platform or uses an existing product such as ZenML or Azure ML.
Business requirements decide whether a simple data lookup is enough
A model that reads features from cloud storage and returns a prediction may be adequate for one use case. Web and banking applications can need much lower latency, especially when predictions depend on recent user activity. Taylor says teams may need caches in the model API as well as caches in the frontend. If event data is not available quickly enough, the serving design is constrained before the model is even considered. Teams then have to find another way to provide business value within those platform limits.
In a batch scoring job, the inputs are data from one or more tables and the outputs are written to a table. Downstream systems can consume those results for forecasting, pricing, reporting, marketing, or fraud screening. Taylor presents this as a straightforward pattern with a clear interface between the scoring job and its consumers. A variation exposes the stored predictions through an API, allowing another service to retrieve a result for one entity or a group of entities.
Online prediction becomes harder when the caller lacks historical features
For real-time prediction, the easiest case is one where the frontend or calling service already has the information needed to calculate the features. The API can contain the model, run prediction, and return an answer. The harder case includes historical or slower-moving features that must be retrieved elsewhere. Data engineering jobs may prepare those values, while an online feature store or caching layer provides access at the required load and latency. Taylor says cost becomes an important part of tuning this setup.
Precomputed predictions and a lighter model can cover different user states
Taylor describes an e-commerce or banking pattern with existing and new users. Batch scoring can create predictions for users with enough historical data. When an existing user makes a request, the API retrieves the stored prediction instead of recalculating it. A separate model can handle users without historical features by using the real-time information available from the frontend. That model produces a result when the stored prediction cannot be used.
The first production design should stay as simple as the use case allows
Taylor closes by advising teams to think through the operational and business constraints before choosing a serving architecture. They should keep the design simple, build on it gradually, and check that it is delivering value. The people who will operate the system matter as much as the model itself. A design that cannot be maintained, monitored, retrained, or funded may prevent the intended model from reaching production.
"You have to really think about many things when you're doing your design, keep things as simple as you can, build on that, make sure you're getting value and speak to people that have done stuff before."16:59
Who should watch
You are choosing between batch scoring, an API, and a real-time feature setup for a model that needs to reach production.
Your team is inheriting a platform with approved tools, cloud restrictions, or integration problems and you need to understand how those limits affect serving design.
You need a practical way to serve predictions for users with rich history and users who only have current-session data.
