Mage combines interactive coding with reusable Python files for data transformation and machine learning pipelines.
2
Each pipeline block can be reviewed, tested, committed, connected through YAML metadata, and run from another environment with one command.
3
Mage is focused on the prototyping stage, while planned integrations include cloud resources, Spark, Airflow, Prefect, and custom data visualizations.
Summary
Tommy Dang introduces Mage, an open source code editor for transforming data and building machine learning pipelines. He demonstrates a Titanic pipeline that loads data, profiles missing values, separates numeric, text, and binary columns, imputes missing values, encodes features, trains a logistic regression model, and scores it on a test set. Mage turns each reusable step into a Python file while keeping interactive exploration through scratchpads, reports, charts, and block-level state. The pipeline is represented by YAML metadata that records blocks and dependencies, which makes it easier to review and commit than a large notebook. Dang then runs the same project on a remote server with Mage's command-line runner. He describes Mage as a tool for the stage between notebook exploration and production, with planned work around cloud resources, large data sets, Spark, custom reports, unstructured data, and integrations with orchestration tools.
Mage combines notebook interaction with code that can be reused in pipelines
Tommy Dang describes Mage as an open source code editor for transforming data and building machine learning pipelines. It keeps the interactive feel of notebooks for exploration, visuals, and scratch work, while adding the structure of a text editor. Data loaders, transformers, and exporters map to individual Python files. Scratchpads remain available for throwaway experiments. Once code is ready to keep, a user can move it into a reusable block that can be committed, unit tested, reviewed, and reused in production.
Reports expose data quality problems before modeling starts
After loading the Titanic data, Mage displays the returned data frame with charts and reports. Dang uses these reports to inspect distributions and identify missing values in columns such as embarked, age, and cabin. He then creates separate transformer blocks for numeric, text, and binary columns. The numeric block fills missing values with the median, while the text block fills missing values with a simple replacement. The reports update to show complete data after these transformations.
Separate blocks make feature transformations reusable
Dang extracts numeric, text, and binary columns into separate blocks because each group needs different transformations. Hardcoding all of those columns inside one transformer would make the code less reusable. The binary encoder handles values such as male and female, while the existing survived values are already represented as ones and zeros. Text columns receive their own encoding block. Since blocks are connected through dependencies, the same transformations can be reused by another pipeline in the project.
The demo turns cleaned data into a trainable model pipeline
The pipeline uses an exporter to combine the encoded data, create train and test sets, and save them to disk. A following block loads the training data, instantiates a logistic regression model, trains it, and saves the model. A final block loads the model and test set, makes predictions, and calculates a score. Dang first reports 70% accuracy during the local demonstration. Later, after running the project on his server, he reports 79% accuracy.
Pipeline files and YAML metadata make review easier
Dang commits the pipeline and opens a pull request to show how Mage represents the work outside the editor. The individual loaders, transformers, and exporters are ordinary Python files. Each pipeline is a folder containing a YAML file that links those files to the pipeline and records their dependencies, IDs, status, and upstream relationships. The structure lets teammates inspect focused files instead of trying to review a large Python notebook.
A committed Mage project can run in another environment
After merging the code, Dang connects to an online server, pulls the project, installs the dependencies from its requirements file, and runs the Titanic survivors pipeline with the Mage command-line runner. The project contains multiple pipelines and the command selects the one to execute. Mage runs the loaders, transformers, and exporters in the remote environment, creates the train and test data and model, and prints the resulting accuracy.
Mage fits between exploration and broader production systems
Dang says Mage is intended for work that has moved beyond exploration and needs versioning, code review, maintenance, and regular execution. He places it in the prototyping part of a larger machine learning stack. Teams can load data from their existing data management systems and write transformed data back to a warehouse or other destination. They can also call services such as SageMaker for training, export models to storage, and use an orchestration system for broader scheduling and monitoring.
Blocks remain flexible while the editor adds data-specific help
Mage has scratchpad, data loader, transformer, and data exporter blocks. Scratchpads allow arbitrary experiments. Data loaders bring data into a pipeline, transformers operate on inputs, and exporters write results without returning data. Dang says the blocks are intentionally permissive, so users can write other kinds of code when needed. The editor adds data-focused reports, charts, suggestions, templates, and dependency-aware parallel execution without restricting what a block can do.
Planned features extend Mage beyond tabular data and local execution
Dang describes planned work for running Mage on cloud resources, handling large data sets and Spark, adding templates for sources such as Snowflake and Redshift, and reducing boilerplate for feature engineering. He also describes customizable reports and charts, including workspace-specific visualizations. For audio and image data, a chart source could convert an input into a form a chart presenter can display. Planned orchestration work includes generating Airflow tasks and deeper integrations with Prefect and other tools.
"The moment you know what you want to do, and you actually want to take your work and put it in production, you want to take your work, you want to build something that impacts your users, or runs on a regular basis, that needs to be versioned, needs to be code reviewed, and maybe iterated on, maintained, and evolved, then you want to switch to using something like Mage."Tommy Dang24:55
Who should watch
You have notebook code that needs to become reviewed, reusable pipeline code without rewriting every step in a separate text editor.
Your team wants data profiling, transformation, model training, and export steps represented as connected Python files.
You are evaluating how a data-centric pipeline tool could fit alongside an existing warehouse, model service, or orchestration system.
