# MLflow Open Source Framework Hermione Demo

Neylson Crepalde, A3Data | MLOps Meetup | Episode 17 | 17:39
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=NfEigZ5ayJE
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/mlflow-open-source-framework-hermione-demo
Published: 2020-06-12
Tags: experiment-tracking, open-source, testing

## TL;DR
- Hermione creates a shared structure for machine learning projects so teammates can find data, preprocessing, training, tests, and model code in consistent places.
- Hermione uses Conda, includes project setup and helper functions, and provides example classes for data sources, preprocessing, trainers, metrics, and model wrappers.
- A Hermione project can train several models, run tests, log metrics and models to MLflow, and compare runs in the MLflow interface.

## Summary
Neylson Crepalde demonstrates Hermione, an open-source machine learning project framework built on top of MLflow. The demo starts by installing Hermione with Conda and creating a project. The generated structure includes folders for raw data, outputs, tests, source code, configuration, preprocessing, metrics, trainers, model wrappers, and notebooks. Using a Titanic example, he shows spreadsheet loading, data cleaning, category encoding, cross-validation, and training with scikit-learn algorithms. The training script logs algorithms, metrics, and models to an MLflow server running in AWS. The MLflow interface then allows the team to compare runs and inspect the registered models. Hermione also provides an example notebook and test structure. Neylson presents standardization as a practical response to team problems: when several people use the same layout, another person can locate code, debug it, and continue work more easily. The framework grew from repeated projects and mistakes rather than from a single design exercise.

## Key ideas
### Hermione starts with a shared project structure
[00:00](https://www.youtube.com/watch?v=NfEigZ5ayJE&t=0s)
Neylson installs Hermione with Conda and creates a new project. The generated project has folders for raw data, outputs, requirements, configuration, source code, tests, and notebooks. The source area includes places for an API, data sources, preprocessing, metrics, trainers, and model wrappers. The structure gives a team a common place for each part of an ML project instead of leaving every project organized differently.

### The framework is designed for teams to take over each other's work
[07:17](https://www.youtube.com/watch?v=NfEigZ5ayJE&t=437s)
Demetrios Brinkmann points out that a consistent structure helps when the person who knows a project best is unavailable. Neylson agrees and says his team has many people working on projects. If everyone follows the same structure, any teammate can reach any part of the project and help with debugging. Hermione therefore standardizes where code and project information live, rather than depending on one person's memory.

### Hermione supplies extension points for data, preprocessing, training, and models
[04:16](https://www.youtube.com/watch?v=NfEigZ5ayJE&t=256s)
The generated code includes a base data-source class, with a spreadsheet implementation shown as an example. It also includes preprocessing classes such as a normalizer and text vectorizer, a metrics class, and a trainer example for scikit-learn. Neylson says teams can add trainer classes for other algorithms or methods. The model wrapper keeps the model, preprocessing, metrics, columns, and run information together, and provides prediction and save methods.

### The Titanic example turns the structure into a working project
[08:28](https://www.youtube.com/watch?v=NfEigZ5ayJE&t=508s)
In the implemented Titanic project, the spreadsheet data source reads only the required columns. Preprocessing cleans the data, transforms it, drops missing values, and encodes categories. The trainer uses the existing scikit-learn implementation and performs cross-validation. Tests cover data reading, cleaning, category encoding, and whether the project was built correctly. Neylson presents these tests as a starting structure that teams can customize for their own data science projects.

### The training command connects project code to MLflow
[11:27](https://www.youtube.com/watch?v=NfEigZ5ayJE&t=687s)
The main training function reads the data, runs preprocessing, and tests random forest, gradient boosting, and logistic regression models. Neylson sets the MLflow experiment name from the project name, logs the algorithms and metrics, and logs the model. Running the Hermione training command from the source folder performs the training and saves the results to MLflow. The example uses an MLflow server hosted in AWS.

### MLflow lets the team compare the resulting runs
[12:50](https://www.youtube.com/watch?v=NfEigZ5ayJE&t=770s)
After training, Neylson opens MLflow and shows the three model runs. The interface allows the team to compare runs and inspect their metrics. He says accuracy is not his preferred classification metric and mentions F1 and AUC as alternatives. In the displayed example, gradient boosting appears to be the winning model. He also shows that a model can be registered and used through an API with a terminal command.

### Hermione grew through repeated project work
[14:58](https://www.youtube.com/watch?v=NfEigZ5ayJE&t=898s)
Neylson says Hermione was not planned as a one-day project. It developed over several years of projects, mistakes, and lessons. The repository includes an example notebook with extensions, imports, data checks, preprocessing, train-test splitting, model training, predictions, and tests. The notebook also shows how to use the framework's classes and how to move the resulting code into the training script.

## Notable quotes
- Neylson Crepalde: "If we all use the same structure, the same idea for building them all, anyone can get to any point and help." (07:56)
- Neylson Crepalde: "If we have to debug it, you know exactly where you are, and it's easy to debug." (07:56)
- Neylson Crepalde: "We work a lot in Jupyter notebooks, JupyterLab, so this is just an example for you to structure your Jupyter notebook." (16:40)
- Neylson Crepalde: "It was born like, for some years, project over project, making mistakes, learning from them, and gradually Hermione was made." (15:23)

## Tools & references mentioned
- MLflow
- Conda
- scikit-learn
- AWS
- Jupyter
- JupyterLab
- Plotly
- Titanic

## Who should watch
- You have several data scientists working on related ML projects and need a common layout for code, tests, data handling, and training.
- Your team already uses MLflow and wants a project scaffold that connects training code with experiment logging and model storage.
- You are building internal ML tooling and want to see a small framework demonstrated through a scikit-learn Titanic example.

## Related talks

- [What is the open source ML framework Hermoine](https://mlopstalks.com/talks/what-is-the-open-source-ml-framework-hermoine) (Neylson Crepalde, A3Data, 16:04)
- [The Challenges of ML Operations & How Hermione Helps Along the Way](https://mlopstalks.com/talks/the-challenges-of-ml-operations-how-hermione-helps-along-the-way) (Neylson Crepalde, A3Data, 1:01:30)
- [Making MLflow](https://mlopstalks.com/talks/making-mlflow) (Corey Zumar, Databricks, 59:11)
- [MLflow Leading Open Source](https://mlopstalks.com/talks/mlflow-leading-open-source) (Databricks' Corey Zumar, 58:24)
- [MLOps Engineering Labs Recap, Part 1](https://mlopstalks.com/talks/mlops-engineering-labs-recap-part-1) (John Savage, Overstock & Alexey Naiden & Varuna Jayasiri & Michel Vasconcelos, Bank of Nordeste, 59:26)