# Extending AI: From Industry to Innovation

Sophia Rowland & David Weik, SAS | MLOps Podcast | Episode 247 | 1:01:37
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=vQAiUiqQgDA
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/extending-ai-from-industry-to-innovation
Published: 2024-07-12
Tags: deployment, evals, governance, monitoring

## TL;DR
- Traditional MLOps is moving from batch scoring toward real-time application integration, which requires data scientists to work with application developers and understand APIs.
- Organizations often keep unused models in production because they lack ownership, usage metrics, or confidence that retiring them is safe.
- Generative AI projects need use-case-specific evaluation, privacy controls, model selection, and monitoring before organizations depend on them in production.

## Summary
Sophia Rowland and David Weik describe the practical problems they see when organizations move models into production. Traditional MLOps work now includes real-time fraud decisions, stream processing, application APIs, dependency management, and large fleets of models. They discuss models that are recoded incorrectly, environments that drift, and unused "zombie models" that consume resources and create security risk. Their examples include worker-safety computer vision, solar-farm optimization, and industrial systems with thousands of machine-specific models. The conversation then turns to generative AI. David describes document question answering, multilingual email routing, and process automation, while both guests discuss cost, latency, privacy, model changes, vector databases, and evaluation. They are skeptical of forcing chatbots into simple workflows that would work better with a button or slider. Successful teams communicate across functions, run small experiments, standardize repeatable processes, and build clear paths from development to retraining and production.

## Key ideas
### Real-time deployment changes who data scientists must work with
[08:08](https://www.youtube.com/watch?v=vQAiUiqQgDA&t=488s)
David says classic use cases such as fraud detection and churn prediction are moving from batch processing into real-time applications. A data scientist who once worked mainly with data engineers may now need to work with application developers integrating a model into Java, COBOL, or another application. The model may need an OpenAPI interface, and every input becomes a field that a front-end application must collect. David uses a small JavaScript application to make this visible to data scientists. If a model requires 100 inputs, the production interface becomes difficult to use, which can prompt better variable selection.

### Throwing a model over the fence leaves the organization with an unusable handoff
[09:53](https://www.youtube.com/watch?v=vQAiUiqQgDA&t=593s)
Sophia recalls a data science master's project with a healthcare provider. The team worked with 300 rows, 1,200 columns, and a messy dataset, then handed over training code instead of a trained model object or scoring code. She describes this as a lasting regret. In her later work, she saw that production requires data scientists, MLOps engineers, IT, data engineers, business users, end users, and model-risk teams to work toward the same decision. Her goal became getting a model into a form and location where it can be used and monitored.

### Model recoding and environment drift create avoidable production failures
[12:55](https://www.youtube.com/watch?v=vQAiUiqQgDA&t=775s)
Sophia describes organizations that develop a model in one language and then recode it for production in another. The recoding can introduce errors, so teams must check that the outputs still match the original model. Dependency management creates another source of failure when the development version of Python or its packages differs from production. David adds an example of a model operated through a manual chain of CSV exports, a specific laptop, a Jupyter notebook, and email. The process worked only because someone knew the exact environment and steps.

### Unused production models consume resources and hide ownership problems
[13:55](https://www.youtube.com/watch?v=vQAiUiqQgDA&t=835s)
The conversation covers "zombie models" that remain in production even though nobody knows whether they receive traffic or drive revenue. Demetrios Brinkmann describes a Yandex project where analyzing production models led to removing 10 percent of them. Sophia says unused models still cost compute and storage and can create security exposure. Teams should check whether production models are being used and retire those that are not. David adds that maintaining valueless models also occupies skilled people who could work on more useful systems, which can make the work less engaging.

### Industrial AI creates both scale and expensive errors
[18:01](https://www.youtube.com/watch?v=vQAiUiqQgDA&t=1081s)
Sophia gives examples from worker safety and industrial operations. Computer vision models watch assembly-line cameras for unsafe behavior and missing lockout tagout procedures, then alert someone on the floor because the physical lock is still needed to prevent the machine from starting. Clearblade and SAS also work on solar-farm optimization, adjusting operations based on the sun and clouds. In manufacturing, one model may be built for each machine or sensor. Sophia describes an organization running 10,000 models and planning to reach 20,000. Monitoring and alert thresholds are needed because no team can watch each model manually, while false maintenance predictions can also take valuable machinery offline.

### Generative AI helps business users test ideas, then raises production questions
[27:47](https://www.youtube.com/watch?v=vQAiUiqQgDA&t=1667s)
David says multimodal models let people test computer-vision ideas without waiting for a data science team. Business users may send a prompt and result to the data science team and ask whether it can go into production. He sees document question answering and process automation as common areas, including routing multilingual email with a human in the loop. The production discussion quickly turns to model cost, changing API versions, latency, retrieval augmentation, and whether a smaller self-hosted open-source model is better for a particular task. David says an interactive application may need a response in less than 800 milliseconds, while calls to large providers can already take around a second before extra retrieval or API steps.

### Generative AI governance and evaluation are still immature
[35:09](https://www.youtube.com/watch?v=vQAiUiqQgDA&t=2109s)
Sophia says enterprises must control where sensitive patient records, source code, and other private data go when employees use external models or copilots. She describes trustworthy AI governance as an emerging area and mentions the National Institute of Standards and Technology AI Risk Management Framework. David adds operational questions around vector-database service levels, backups, department isolation, and restricting retrieval to authorized data. Evaluation is also difficult. Teams may use thumbs-up and thumbs-down feedback, batch review, or another model to score text, but these signals do not fully capture whether the system answered correctly, retrieved the right information, or led to the right action.

### Simple interfaces and small agents often work better than ambitious chatbot systems
[47:04](https://www.youtube.com/watch?v=vQAiUiqQgDA&t=2824s)
The guests argue that AI should not be added to a workflow just because it is available. They describe a system that used a chatbot to collect a value from 1 to 10, even though a slider would have been clearer. David has seen a customer replace a complex multi-agent design with a simple agent that chooses between two or three actions because the larger system broke too often. Sophia says agents can make advanced software easier to use, but success depends on anticipating specific tasks and user needs. Their examples suggest that constrained interfaces and narrow workflows are easier to test than open-ended chat.

### High-performing teams share ownership and build repeatable paths to production
[57:34](https://www.youtube.com/watch?v=vQAiUiqQgDA&t=3454s)
Sophia sees strong communication, shared responsibility, small proofs of concept, and flexible standardization in effective teams. They start with work that has potential business value, grow through phases, and retire models when they do not deliver value. David says data scientists and data engineers should plan together for production scoring and retraining, with clear integration points instead of a new handoff for every project. Teams that succeed across more use cases develop a platform approach, so they can reuse the path from development to deployment rather than reinventing it each time.

## Notable quotes
- Sophia Rowland: "We need to make sure that we can put this into where we need it to be in just a few clicks." (23:47)
- Sophia Rowland: "You do not need that model out there, especially if it's not actually bringing any value to the company." (15:28)
- David Weik: "It is not like there's one clear use case winner to me. It's really about the question, okay, we have a lot of ideas, but how do we get them into production and keep them in production?" (34:26)
- Sophia Rowland: "Just because we can use AI doesn't mean we should." (48:39)
- David Weik: "When the data scientists work closely with the data engineers, then I always see that be successful." (59:51)

## Tools & references mentioned
- SAS
- MLOps Community
- Yandex
- Michelangelo
- Clearblade
- JavaScript
- Jupyter notebook
- ONNX
- TensorFlow
- PyTorch
- OpenAI
- Anthropic
- Gemini
- National Institute of Standards and Technology AI Risk Management Framework
- AI Incident Database

## Who should watch
- You are moving batch models into real-time applications and need to understand the engineering work around APIs, dependencies, retraining, and handoffs.
- Your organization has many production models but lacks clear usage metrics, ownership, or a process for retiring models that no longer matter.
- You are evaluating generative AI, agents, or retrieval systems and need practical questions about privacy, latency, evaluation, monitoring, and interface design.

## Editor's note

Sophia Rowland describes how a model can be developed in one language and recoded for production in another, introducing errors when teams do not check that outputs still match. ZenML records each pipeline run's steps, inputs, outputs, and code version, so teams can trace a model to the data and code that produced it. The same pipeline can then run on different infrastructure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Scaling AI in Production](https://mlopstalks.com/talks/scaling-ai-in-production) (Srivatsan Srinivasan, AIEngineering, 51:56)
- [MLOps at the Age of Generative AI](https://mlopstalks.com/talks/mlops-at-the-age-of-generative-ai) (Barak Turovsky, Scale Venture Partners, 56:56)
- [Managing Data for Effective GenAI Application](https://mlopstalks.com/talks/managing-data-for-effective-genai-application) (Anu Arora & Anass Bensrhir, QuantumBlack AI by McKinsey, 51:01)
- [GenAI in Production - Challenges and Trends](https://mlopstalks.com/talks/genai-in-production-challenges-and-trends) (Verena Weber, Verena Weber, 48:43)
- [Productionizing AI: How to Think From the End](https://mlopstalks.com/talks/productionizing-ai-how-to-think-from-the-end) (Annie Condon, 11:11)