# How Hera is an Enabler of MLOps Integrations

Flaviu Vadan, Dyno Therapeutics | MLOps Coffee Sessions | Episode 115 | 41:33
Hosted by Vishnu Rachakonda

Source: https://www.youtube.com/watch?v=wkmxKZBEahA
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/how-hera-is-an-enabler-of-mlops-integrations
Published: 2022-08-14
Tags: open-source, orchestration, platform-teams, training-pipelines

## TL;DR
- Dyno Therapeutics uses machine learning to design adeno-associated virus sequences for gene therapy delivery.
- Dyno's internal ML platform focuses on helping researchers train, store, access, and compare many models rather than serving external APIs.
- Hera gives Dyno's teams a Python interface for Argo Workflows, reducing YAML complexity and helping scientists work independently.

## Summary
Flaviu Vadan explains how Dyno Therapeutics combines machine learning, lab experiments, and biological expertise to design better gene therapy vectors. The company works with adeno-associated viruses, whose sequence, physical structure, targeting behavior, and immune response create difficult design constraints. Machine learning narrows the search through the enormous space of possible sequences. Dyno's internal ML platform is built around experimentation. Researchers can submit models through Python, configure data and training settings, use different training stacks, access GPUs, track metrics, store models, and connect biological metadata to experiments. Argo Workflows and Kubernetes provide the underlying execution layer. Hera, an open-source Python SDK for Argo, was created after Dyno found Kubeflow and an earlier Argo DSL did not provide enough independence or ease of use for its teams. Flaviu also discusses the practical work of open sourcing an internal project, including legal review, maintenance, security, and finding internal supporters. He is candid that Dyno implemented only the ML practices its current needs justified.

## Key ideas
### Dyno designs the delivery vehicle for gene therapy
[03:37](https://www.youtube.com/watch?v=wkmxKZBEahA&t=217s)
Flaviu describes Dyno Therapeutics as a company using machine learning and lab experimentation to design gene therapy vectors. These vectors are transport mechanisms that carry a therapy to a specific cell type. Dyno focuses on adeno-associated viruses, or AAVs, because naturally occurring AAVs do not target cells, organs, and tissues effectively enough. They can also trigger immune responses, which may require higher doses and increase the risk of side effects. Flaviu works across MLOps, DevOps, core infrastructure, compute, and machine-guided design of viruses in silico.

### Machine learning narrows an enormous biological search space
[10:37](https://www.youtube.com/watch?v=wkmxKZBEahA&t=637s)
The AAV sequence Flaviu discusses has about 730 positions, with 20 possible amino acids at each position. That creates 20 to the 730 possible configurations. Lab protocols and physical biology prevent researchers from testing every combination. Candidate viruses must also be manufactured, fold into the right shape, and have useful targeting properties. Dyno uses machine learning to model the relationship between a generated sequence and its function. This guides the search toward AAVs that are physically viable and better at targeting particular cell types.

### Dyno's MLOps platform is built for experimentation
[14:09](https://www.youtube.com/watch?v=wkmxKZBEahA&t=849s)
Flaviu says Dyno's MLOps work is currently more concerned with access to model training than with external model deployment. The platform lets data scientists train models implemented in PyTorch, scikit-learn, or custom NumPy code. It also provides access to trained models, artifacts, datasets, transformation metadata, training hyperparameters, and visualizations of training runs. Since Dyno has no external-facing APIs for these models, the platform is designed to give researchers independence while they try different approaches to biological problems.

### A simple Python interface hides several training systems
[21:05](https://www.youtube.com/watch?v=wkmxKZBEahA&t=1265s)
A user of Dyno's training interface calls a function with a model specification and a configuration object. The configuration can control epochs, datasets, transformations, data locations, gradient aggregation, distributed GPU training, and optimizers. Defaults handle repeated setup so users can focus on a smaller number of parameters. The submitted model is routed to training infrastructure suited to its framework. PyTorch models can use distributed training, while NumPy models use custom training loops. The resulting workflows run through Argo Workflows on Kubernetes.

### Integrations connect training to the rest of the platform
[22:07](https://www.youtube.com/watch?v=wkmxKZBEahA&t=1327s)
Flaviu describes integrations as the part of Dyno's MLOps system that connects model training with storage, monitoring, metadata, hyperparameter tuning, and high-scale experimentation. Dyno stores models automatically through a mechanism built on Google Cloud Firestore. An integration with Aim or Aimstack visualizes experimental and production training runs and stores metadata linking biological context to models. Horovod provides access to multiple GPUs for distributed training. Metrics can be selected from the TorchMetrics catalog, tracked in Aim, and aggregated during distributed training. Hera Workflows ties these pieces together.

### Kubernetes and Argo fit Dyno's need to run many kinds of workloads
[27:36](https://www.youtube.com/watch?v=wkmxKZBEahA&t=1656s)
Flaviu chose Kubernetes because containerized applications can run on it with flexible resource access and changing workloads. He values its connection to the Cloud Native Computing Foundation, since related projects can integrate into a Kubernetes-based stack. Dyno considered workflow tools including Airflow and Prefect, but Argo Workflows fit the team's preference for Kubernetes-native constructs. Dyno also needed to train many models in parallel, sometimes across four GPUs. Flaviu accepts Kubernetes' steep learning curve because it gives the platform room to run different workloads and integrations.

### Hera was created to make Argo usable by Dyno's researchers
[30:59](https://www.youtube.com/watch?v=wkmxKZBEahA&t=1859s)
Dyno had previously used Kubeflow and its Kubeflow Pipelines SDK, but the team encountered observability problems, GPU access issues, serialization behavior, and confusing decorators and parameters for parallel jobs. Argo Workflows offered a better fit for the infrastructure team, but its YAML configuration was error-prone, difficult to test, and hard to use without Argo knowledge. Dyno first used an Argo workflows DSL and then built Hera after finding gaps in that approach. Hera provides a Python SDK for constructing Argo workflows and was open sourced under the Argo project labs.

### Open sourcing required a tailored internal case
[36:04](https://www.youtube.com/watch?v=wkmxKZBEahA&t=2164s)
Flaviu says engineers need to understand the audience when proposing an open-source project. At Dyno, one risk was exposing prior company information through the release. He recommends finding internal champions who can explain benefits such as recruiting value, then working with legal staff to define the risks. The team also had to consider who would maintain the project, how much time maintenance would take, and how external contributions could be reviewed and integrated safely. The decision depended on the project's net benefit to Dyno.

### Dyno implements MLOps practices in small steps
[39:31](https://www.youtube.com/watch?v=wkmxKZBEahA&t=2371s)
Flaviu rejects the idea that Dyno deliberately designed a complete MLOps system in advance. The team started with current organizational needs and solved problems as they appeared. Its initial loop was mainly about getting access to data, training models, storing models, and using them for biological problems. Deployment and monitoring were less urgent because experiments and datasets change over time. He prefers building small increments, learning from them, and avoiding systems that look useful but are not yet needed.

## Notable quotes
- Flaviu Vadan: "The TLDR is that ML allows us to encode this extreme level of complexity into some mathematical representation that's much more effective than simple search." (12:05)
- Flaviu Vadan: "MLOps overall is the glue of everything that may be needed to facilitate the practice of ML within Dyno's context." (19:29)
- Flaviu Vadan: "The primary goal from the start was and it will always be to empower the team at Dyno to be independent." (34:37)
- Flaviu Vadan: "Rather than over engineer, building small increments and then learn from that is probably the best thing we've done so far." (40:56)

## Tools & references mentioned
- Dyno Therapeutics
- Hera
- Argo Workflows
- Kubernetes
- Kubeflow
- Kubeflow Pipelines SDK
- Python
- PyTorch
- scikit-learn
- NumPy
- Google Cloud Firestore
- Aim
- Aimstack
- Horovod
- TorchMetrics
- Airflow
- Prefect
- Cloud Native Computing Foundation
- KubeCon
- Argo project labs
- CRISPR
- adeno-associated viruses
- Demetrios Brinkmann
- Jesse Johnson
- Dewpoint Therapeutics
- Ryan Russon

## Who should watch
- You are building an internal ML platform for scientists who need to run many experiments rather than serve a single external model.
- Your team is deciding between Kubernetes-native workflow tools and higher-level ML platforms, and you want to hear how Dyno made that choice.
- You have an internal developer tool that might be useful outside the company and need a practical approach to legal review, maintenance, and open-source support.

## Editor's note

Flaviu Vadan explains that Argo's YAML was error-prone, difficult to test, and hard for researchers to use without Argo knowledge. ZenML lets teams define workflows as Python pipelines, so researchers can work with steps and configuration without writing workflow YAML. Its stack configuration also keeps the workflow code separate from the execution infrastructure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [How to Leverage ML Tooling Ecosystem](https://mlopstalks.com/talks/how-to-leverage-ml-tooling-ecosystem) (Mariya Davydova, Neu.ro, 55:57)
- [I Am Once Again Asking "What is MLOps?"](https://mlopstalks.com/talks/i-am-once-again-asking-what-is-mlops) (Oleksandr Stasyk, Synthesia, 1:06:22)
- [Kubernetes, AI Gateways, and the Future of MLOps](https://mlopstalks.com/talks/kubernetes-ai-gateways-and-the-future-of-mlops) (Alexa Griffith, Bloomberg, 50:12)
- [Why is MLOps Hard in an Enterprise?](https://mlopstalks.com/talks/why-is-mlops-hard-in-an-enterprise) (Maria Vechtomova & Basak Eskili, Ahold Delhaize, 55:06)
- [Aggressively Helpful Platform Teams](https://mlopstalks.com/talks/aggressively-helpful-platform-teams) (Stefan Krawczyk, Stitch Fix, 51:52)