# Moving Deep Learning from Research to Prod Using DeterminedAI & Kubeflow

David Hershey, Determined AI | MLOps Meetup | Episode 36 | 56:06
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=-UymDRk5ISY
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/moving-deep-learning-from-research-to-prod-using-determinedai-kubeflow
Published: 2020-10-02
Tags: experiment-tracking, model-registry, orchestration, training-pipelines

## TL;DR
- Determined AI is an open-source deep learning training platform that handles distributed training, hyperparameter search, and experiment tracking on compute ranging from a laptop to a large GPU cluster.
- A model registry gives research and ML engineering teams a consistent handoff by storing the model code, weights, metadata, and training history behind an API.
- Kubeflow Pipelines can call Determined APIs to trigger training, retrieve versioned models, and run reusable batch inference or deployment workflows.

## Summary
David Hershey explains how Determined AI and Kubeflow can connect research work to production workflows. Determined focuses on training rather than data management or model serving. It lets data scientists run distributed training and hyperparameter searches while tracking code, checkpoints, metrics, hyperparameters, machines, and data versions. Its model registry packages the code, weights, and metadata needed to load a particular model version, which reduces the manual handoff between researchers and ML engineers. Kubeflow Pipelines can call Determined APIs to start training, retrieve a named model version, run batch inference, compare a new model with the previous version, and deploy an approved model with Seldon. Hershey describes a gradual path from disposable proof-of-concept code to structured training, recurring retraining, and production deployment. He is also clear that feature transformation is often harder than loading a deep learning model, especially for real-time predictions. Feature stores can help keep training and serving transformations aligned.

## Key ideas
### Determined AI focuses on the training part of the machine learning workflow
[05:18](https://www.youtube.com/watch?v=-UymDRk5ISY&t=318s)
David Hershey describes Determined AI as an open-source deep learning training platform. It installs on a compute cluster, which can be a laptop, a GPU box, or a large cluster. Researchers continue to write models in familiar frameworks such as PyTorch and TensorFlow. Determined adds distributed training, hyperparameter search based on the ASHA algorithm, and tracking for the work involved in training models. Hershey places its boundary around training. It does not manage upstream data in the way Pachyderm does, and it does not mainly host or serve models, perform batch inference, or provide Spark services.

### Determined AI tracks training work as part of the training system
[07:47](https://www.youtube.com/watch?v=-UymDRk5ISY&t=467s)
Hershey contrasts Determined with MLflow and Weights & Biases, where users add API calls to track metrics and artifacts. Determined asks for more information about the model, optimizer, and data loading code. It then uses that structure to optimize work for the cluster and run sophisticated experiments without requiring researchers to rewire their code. Artifact tracking, metric tracking, hyperparameters, and checkpoints come from the training setup. Researchers can still use upstream data tools, including DVC or Pachyderm, and record dataset versions alongside the training results.

### Kubeflow is a collection of components with different levels of maturity
[11:31](https://www.youtube.com/watch?v=-UymDRk5ISY&t=691s)
Hershey rejects the idea that Kubeflow is dying, while acknowledging that its components have developed at different speeds. He speaks especially positively about Kubeflow Pipelines, which lets machine learning users define workflows in Python. He also calls TFJob and MPIJob useful building blocks for running machine learning on Kubernetes. His experience with Katib was less positive, particularly for hyperparameter tuning. His view is that users should assess the individual Kubeflow components rather than treat the project as one uniform product.

### A model registry turns a difficult team handoff into an API call
[19:18](https://www.youtube.com/watch?v=-UymDRk5ISY&t=1158s)
Hershey describes the handoff from a researcher to an ML engineering team as more than sending over a file. The receiving team needs the model code, the correct weights, metadata, and a way to instantiate the model. Without a registry, this can lead to weeks of meetings and emails, followed by the same process for every new model version. Determined's registry stores the code that creates the model, its weights, and its metadata. A downstream user can request a named model and version through an API, load it, and use it without reconstructing the environment by hand. The current approach is Python-based.

### Kubeflow Pipelines can orchestrate versioned Determined models
[23:16](https://www.youtube.com/watch?v=-UymDRk5ISY&t=1396s)
In the demo, Hershey starts with a question-answering model built with Hugging Face and shows the problems with manually managed checkpoints and metrics. The Determined version records model code, checkpoint locations, hyperparameters, and metric curves. A script can request a particular experiment or a version from the model registry and use it for inference. Hershey then shows a Kubeflow Pipeline that creates a shared volume, downloads data, and calls the inference script. Changing the model version lets the same pipeline use a newer research result without rewriting the workflow.

### The path to production starts with structure after the proof of concept
[42:14](https://www.youtube.com/watch?v=-UymDRk5ISY&t=2534s)
Hershey says teams often begin by testing a new paper or model against their own data. At that stage, the code can be disposable because the question is whether the approach is worth further work. Once the model is worth pursuing, the team needs more structure, distributed training, experiment tracking, and artifact management. Determined requires researchers to describe the model, optimizer, and data loading code in a consistent way. Hershey sees that organization as an early step toward production because it creates a form that can later be translated into a production system.

### Recurring retraining needs APIs that connect new data to deployment
[44:07](https://www.youtube.com/watch?v=-UymDRk5ISY&t=2647s)
After a model proves useful, some organizations need to retrain it regularly as new data arrives. Hershey gives the example of an autonomous vehicle company collecting data from vehicles and maintaining a labeling process. Kubeflow Pipelines can trigger Determined training through APIs, and a pipeline can compare the new model with the previous version before deploying it with Seldon. Jenkins or another continuous delivery tool can trigger the pipeline when model code, data, or deployment code changes. This creates a path from a repository change to training, evaluation, and deployment.

### Feature transformation is often harder than loading a deep learning model
[36:09](https://www.youtube.com/watch?v=-UymDRk5ISY&t=2169s)
For many deep learning models, inference requires the model code, trained weights, and sometimes extra artifacts such as a language model vocabulary. The harder case involves extensive preprocessing. Researchers may transform data from SQL into features and apply further Python transformations before training. Reproducing exactly those transformations for live, large-scale inference is difficult, especially when a prediction must be ready within a fraction of a second. Hershey points to feature stores such as Tecton, Hopsworks, and Feast as attempts to provide shared pipelines for the transformations used during training and serving.

## Notable quotes
- David Hershey: "Determined is a open source deep learning training platform." (05:18)
- David Hershey: "When you train a model or say something in the model registry, the whole history of that training is tied along with it." (18:26)
- David Hershey: "At its core what that means is you need to have tools to solve that in place." (47:26)
- David Hershey: "The thing that i don't think people talk about enough is how hard it is to recreate your data in production." (50:18)
- David Hershey: "Essentially all of Determined, everything that I briefly showed you today, is open source." (54:40)

## Tools & references mentioned
- Determined AI
- Kubeflow
- Kubeflow Pipelines
- MLflow
- Weights & Biases
- DVC
- Pachyderm
- Seldon
- Kubernetes
- PyTorch
- TensorFlow
- Hugging Face
- ASHA
- TFJob
- MPIJob
- Katib
- Jenkins
- Argo CD
- Feast
- Tecton
- Hopsworks
- Google
- Ford Motor Company
- Feudal Networks for Hierarchical Reinforcement Learning

## Who should watch
- You are trying to move deep learning experiments from notebooks or research repositories into repeatable training and inference workflows.
- Your research and ML engineering teams spend time exchanging weights, code, metrics, and environments by hand.
- You are comparing Kubeflow with a training platform and need a concrete example of how APIs can connect training, registries, batch inference, and deployment.

## Editor's note

David Hershey says moving a model from research to production requires more than passing along a file, since the receiving team needs the code, weights, metadata, and a way to instantiate it. ZenML records each run's inputs, outputs, and code version, with lineage that traces a model or artifact back to the data and code that produced it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Kubeflow vs MLflow](https://mlopstalks.com/talks/kubeflow-vs-mlflow) (Byron Allen, Servian, 54:57)
- [MLflow vs Kubeflow 2022](https://mlopstalks.com/talks/mlflow-vs-kubeflow-2022) (Byron Allen, Contino, 1:05:40)
- [Team Aurora: Accelerating ML with Kubeflow](https://mlopstalks.com/talks/team-aurora-accelerating-ml-with-kubeflow) (Maurizio Vitale, Vinay Anantharaman & Ankit Aggarwal, Aurora, 54:57)
- [Optimizing Your ML Workflow with Kubeflow 1.0](https://mlopstalks.com/talks/optimizing-your-ml-workflow-with-kubeflow-1-0) (Josh Bottum, Arrikto, 1:03:41)
- [Bring Your On-Prem ML Use Cases to Production on Google Cloud using Kubeflow](https://mlopstalks.com/talks/bring-your-on-prem-ml-use-cases-to-production-on-google-cloud-using-kubeflow) (Chanchal Chatterjee, Google, 20:28)