Podcast

How Hera is an Enabler of MLOps Integrations

Flaviu Vadan, Dyno TherapeuticsEpisode 115 · 41:33 · Aug 2022 · 520 viewsHosted by Vishnu Rachakonda
Thumbnail for How Hera is an Enabler of MLOps Integrations Watch on YouTube
TL;DR
  1. 1

    Dyno Therapeutics uses machine learning to design adeno-associated virus sequences for gene therapy delivery.

  2. 2

    Dyno's internal ML platform focuses on helping researchers train, store, access, and compare many models rather than serving external APIs.

  3. 3

    Hera gives Dyno's teams a Python interface for Argo Workflows, reducing YAML complexity and helping scientists work independently.

Summary

Flaviu Vadan explains how Dyno Therapeutics combines machine learning, lab experiments, and biological expertise to design better gene therapy vectors. The company works with adeno-associated viruses, whose sequence, physical structure, targeting behavior, and immune response create difficult design constraints. Machine learning narrows the search through the enormous space of possible sequences. Dyno's internal ML platform is built around experimentation. Researchers can submit models through Python, configure data and training settings, use different training stacks, access GPUs, track metrics, store models, and connect biological metadata to experiments. Argo Workflows and Kubernetes provide the underlying execution layer. Hera, an open-source Python SDK for Argo, was created after Dyno found Kubeflow and an earlier Argo DSL did not provide enough independence or ease of use for its teams. Flaviu also discusses the practical work of open sourcing an internal project, including legal review, maintenance, security, and finding internal supporters. He is candid that Dyno implemented only the ML practices its current needs justified.

Key ideas
03:37

Dyno designs the delivery vehicle for gene therapy

Flaviu describes Dyno Therapeutics as a company using machine learning and lab experimentation to design gene therapy vectors. These vectors are transport mechanisms that carry a therapy to a specific cell type. Dyno focuses on adeno-associated viruses, or AAVs, because naturally occurring AAVs do not target cells, organs, and tissues effectively enough. They can also trigger immune responses, which may require higher doses and increase the risk of side effects. Flaviu works across MLOps, DevOps, core infrastructure, compute, and machine-guided design of viruses in silico.

10:37

Machine learning narrows an enormous biological search space

The AAV sequence Flaviu discusses has about 730 positions, with 20 possible amino acids at each position. That creates 20 to the 730 possible configurations. Lab protocols and physical biology prevent researchers from testing every combination. Candidate viruses must also be manufactured, fold into the right shape, and have useful targeting properties. Dyno uses machine learning to model the relationship between a generated sequence and its function. This guides the search toward AAVs that are physically viable and better at targeting particular cell types.

14:09

Dyno's MLOps platform is built for experimentation

Flaviu says Dyno's MLOps work is currently more concerned with access to model training than with external model deployment. The platform lets data scientists train models implemented in PyTorch, scikit-learn, or custom NumPy code. It also provides access to trained models, artifacts, datasets, transformation metadata, training hyperparameters, and visualizations of training runs. Since Dyno has no external-facing APIs for these models, the platform is designed to give researchers independence while they try different approaches to biological problems.

21:05

A simple Python interface hides several training systems

A user of Dyno's training interface calls a function with a model specification and a configuration object. The configuration can control epochs, datasets, transformations, data locations, gradient aggregation, distributed GPU training, and optimizers. Defaults handle repeated setup so users can focus on a smaller number of parameters. The submitted model is routed to training infrastructure suited to its framework. PyTorch models can use distributed training, while NumPy models use custom training loops. The resulting workflows run through Argo Workflows on Kubernetes.

22:07

Integrations connect training to the rest of the platform

Flaviu describes integrations as the part of Dyno's MLOps system that connects model training with storage, monitoring, metadata, hyperparameter tuning, and high-scale experimentation. Dyno stores models automatically through a mechanism built on Google Cloud Firestore. An integration with Aim or Aimstack visualizes experimental and production training runs and stores metadata linking biological context to models. Horovod provides access to multiple GPUs for distributed training. Metrics can be selected from the TorchMetrics catalog, tracked in Aim, and aggregated during distributed training. Hera Workflows ties these pieces together.

27:36

Kubernetes and Argo fit Dyno's need to run many kinds of workloads

Flaviu chose Kubernetes because containerized applications can run on it with flexible resource access and changing workloads. He values its connection to the Cloud Native Computing Foundation, since related projects can integrate into a Kubernetes-based stack. Dyno considered workflow tools including Airflow and Prefect, but Argo Workflows fit the team's preference for Kubernetes-native constructs. Dyno also needed to train many models in parallel, sometimes across four GPUs. Flaviu accepts Kubernetes' steep learning curve because it gives the platform room to run different workloads and integrations.

30:59

Hera was created to make Argo usable by Dyno's researchers

Dyno had previously used Kubeflow and its Kubeflow Pipelines SDK, but the team encountered observability problems, GPU access issues, serialization behavior, and confusing decorators and parameters for parallel jobs. Argo Workflows offered a better fit for the infrastructure team, but its YAML configuration was error-prone, difficult to test, and hard to use without Argo knowledge. Dyno first used an Argo workflows DSL and then built Hera after finding gaps in that approach. Hera provides a Python SDK for constructing Argo workflows and was open sourced under the Argo project labs.

36:04

Open sourcing required a tailored internal case

Flaviu says engineers need to understand the audience when proposing an open-source project. At Dyno, one risk was exposing prior company information through the release. He recommends finding internal champions who can explain benefits such as recruiting value, then working with legal staff to define the risks. The team also had to consider who would maintain the project, how much time maintenance would take, and how external contributions could be reviewed and integrated safely. The decision depended on the project's net benefit to Dyno.

39:31

Dyno implements MLOps practices in small steps

Flaviu rejects the idea that Dyno deliberately designed a complete MLOps system in advance. The team started with current organizational needs and solved problems as they appeared. Its initial loop was mainly about getting access to data, training models, storing models, and using them for biological problems. Deployment and monitoring were less urgent because experiments and datasets change over time. He prefers building small increments, learning from them, and avoiding systems that look useful but are not yet needed.

"The TLDR is that ML allows us to encode this extreme level of complexity into some mathematical representation that's much more effective than simple search."Flaviu Vadan12:05
Who should watch
  • You are building an internal ML platform for scientists who need to run many experiments rather than serve a single external model.
  • Your team is deciding between Kubernetes-native workflow tools and higher-level ML platforms, and you want to hear how Dyno made that choice.
  • You have an internal developer tool that might be useful outside the company and need a practical approach to legal review, maintenance, and open-source support.