# So Fresh and So Data Clean

Tommy Dang, Mage | MLOps Meetup | Episode 107 | 49:58
Hosted by Ben Epstein

Source: https://www.youtube.com/watch?v=6Iyt9Wip3C4
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/so-fresh-and-so-data-clean
Published: 2022-07-25
Tags: data-pipelines, data-quality, feature-engineering

## TL;DR
- Mage combines interactive coding with reusable Python files for data transformation and machine learning pipelines.
- Each pipeline block can be reviewed, tested, committed, connected through YAML metadata, and run from another environment with one command.
- Mage is focused on the prototyping stage, while planned integrations include cloud resources, Spark, Airflow, Prefect, and custom data visualizations.

## Summary
Tommy Dang introduces Mage, an open source code editor for transforming data and building machine learning pipelines. He demonstrates a Titanic pipeline that loads data, profiles missing values, separates numeric, text, and binary columns, imputes missing values, encodes features, trains a logistic regression model, and scores it on a test set. Mage turns each reusable step into a Python file while keeping interactive exploration through scratchpads, reports, charts, and block-level state. The pipeline is represented by YAML metadata that records blocks and dependencies, which makes it easier to review and commit than a large notebook. Dang then runs the same project on a remote server with Mage's command-line runner. He describes Mage as a tool for the stage between notebook exploration and production, with planned work around cloud resources, large data sets, Spark, custom reports, unstructured data, and integrations with orchestration tools.

## Key ideas
### Mage combines notebook interaction with code that can be reused in pipelines
[03:03](https://www.youtube.com/watch?v=6Iyt9Wip3C4&t=183s)
Tommy Dang describes Mage as an open source code editor for transforming data and building machine learning pipelines. It keeps the interactive feel of notebooks for exploration, visuals, and scratch work, while adding the structure of a text editor. Data loaders, transformers, and exporters map to individual Python files. Scratchpads remain available for throwaway experiments. Once code is ready to keep, a user can move it into a reusable block that can be committed, unit tested, reviewed, and reused in production.

### Reports expose data quality problems before modeling starts
[08:02](https://www.youtube.com/watch?v=6Iyt9Wip3C4&t=482s)
After loading the Titanic data, Mage displays the returned data frame with charts and reports. Dang uses these reports to inspect distributions and identify missing values in columns such as embarked, age, and cabin. He then creates separate transformer blocks for numeric, text, and binary columns. The numeric block fills missing values with the median, while the text block fills missing values with a simple replacement. The reports update to show complete data after these transformations.

### Separate blocks make feature transformations reusable
[09:09](https://www.youtube.com/watch?v=6Iyt9Wip3C4&t=549s)
Dang extracts numeric, text, and binary columns into separate blocks because each group needs different transformations. Hardcoding all of those columns inside one transformer would make the code less reusable. The binary encoder handles values such as male and female, while the existing survived values are already represented as ones and zeros. Text columns receive their own encoding block. Since blocks are connected through dependencies, the same transformations can be reused by another pipeline in the project.

### The demo turns cleaned data into a trainable model pipeline
[16:14](https://www.youtube.com/watch?v=6Iyt9Wip3C4&t=974s)
The pipeline uses an exporter to combine the encoded data, create train and test sets, and save them to disk. A following block loads the training data, instantiates a logistic regression model, trains it, and saves the model. A final block loads the model and test set, makes predictions, and calculates a score. Dang first reports 70% accuracy during the local demonstration. Later, after running the project on his server, he reports 79% accuracy.

### Pipeline files and YAML metadata make review easier
[21:10](https://www.youtube.com/watch?v=6Iyt9Wip3C4&t=1270s)
Dang commits the pipeline and opens a pull request to show how Mage represents the work outside the editor. The individual loaders, transformers, and exporters are ordinary Python files. Each pipeline is a folder containing a YAML file that links those files to the pipeline and records their dependencies, IDs, status, and upstream relationships. The structure lets teammates inspect focused files instead of trying to review a large Python notebook.

### A committed Mage project can run in another environment
[20:50](https://www.youtube.com/watch?v=6Iyt9Wip3C4&t=1250s)
After merging the code, Dang connects to an online server, pulls the project, installs the dependencies from its requirements file, and runs the Titanic survivors pipeline with the Mage command-line runner. The project contains multiple pipelines and the command selects the one to execute. Mage runs the loaders, transformers, and exporters in the remote environment, creates the train and test data and model, and prints the resulting accuracy.

### Mage fits between exploration and broader production systems
[24:35](https://www.youtube.com/watch?v=6Iyt9Wip3C4&t=1475s)
Dang says Mage is intended for work that has moved beyond exploration and needs versioning, code review, maintenance, and regular execution. He places it in the prototyping part of a larger machine learning stack. Teams can load data from their existing data management systems and write transformed data back to a warehouse or other destination. They can also call services such as SageMaker for training, export models to storage, and use an orchestration system for broader scheduling and monitoring.

### Blocks remain flexible while the editor adds data-specific help
[39:14](https://www.youtube.com/watch?v=6Iyt9Wip3C4&t=2354s)
Mage has scratchpad, data loader, transformer, and data exporter blocks. Scratchpads allow arbitrary experiments. Data loaders bring data into a pipeline, transformers operate on inputs, and exporters write results without returning data. Dang says the blocks are intentionally permissive, so users can write other kinds of code when needed. The editor adds data-focused reports, charts, suggestions, templates, and dependency-aware parallel execution without restricting what a block can do.

### Planned features extend Mage beyond tabular data and local execution
[30:12](https://www.youtube.com/watch?v=6Iyt9Wip3C4&t=1812s)
Dang describes planned work for running Mage on cloud resources, handling large data sets and Spark, adding templates for sources such as Snowflake and Redshift, and reducing boilerplate for feature engineering. He also describes customizable reports and charts, including workspace-specific visualizations. For audio and image data, a chart source could convert an input into a form a chart presenter can display. Planned orchestration work includes generating Airflow tasks and deeper integrations with Prefect and other tools.

## Notable quotes
- Tommy Dang: "Mage, it's an open source tool that's a code editor, helps you transform your data and build ML pipelines." (03:06)
- Tommy Dang: "The moment you know what you want to do, and you actually want to take your work and put it in production, you want to take your work, you want to build something that impacts your users, or runs on a regular basis, that needs to be versioned, needs to be code reviewed, and maybe iterated on, maintained, and evolved, then you want to switch to using something like Mage." (24:55)
- Tommy Dang: "We fit right into that stack in the middle where you know what you're about to do, you got your data, you know where to load it, and you're ready to just explore and train." (38:27)
- Tommy Dang: "It does one thing or a few things very, very well. It's very focused on a singular data set or multiple data sets, right? In one. So, it's very data-centric." (31:35)

## Tools & references mentioned
- Mage
- Docker
- pip
- GitHub
- pandas
- Titanic
- matplotlib
- S3
- logistic regression
- Jupyter
- Airflow
- Prefect
- Spark
- PySpark
- Snowflake
- Redshift
- SageMaker

## Who should watch
- You have notebook code that needs to become reviewed, reusable pipeline code without rewriting every step in a separate text editor.
- Your team wants data profiling, transformation, model training, and export steps represented as connected Python files.
- You are evaluating how a data-centric pipeline tool could fit alongside an existing warehouse, model service, or orchestration system.

## Related talks

- [Building an ML Platform from scratch](https://mlopstalks.com/talks/building-an-ml-platform-from-scratch) (, 1:46:08)
- [The Post Modern Stack](https://mlopstalks.com/talks/the-post-modern-stack) (Jacopo Tagliabue, Coveo, 1:04:58)
- [Scaling your data and AI from 0-100 with open source](https://mlopstalks.com/talks/scaling-your-data-and-ai-from-0-100-with-open-source) (Maarten Breddels, Pycafe & Pranav Aurora, Mooncake & Simba Khadder, Featureform, 1:09:51)
- [Building a Modern Data Analytics Stack](https://mlopstalks.com/talks/building-a-modern-data-analytics-stack) (Jeff Katz, Jigsaw Labs, 55:21)
- [Learnings from Live Coding: An MLOps Project on Twitch](https://mlopstalks.com/talks/learnings-from-live-coding-an-mlops-project-on-twitch) (Felipe Campos Penha, Cargill, 50:12)