Predibase lets users define machine learning pipelines through configuration, a SQL-like language, a Python SDK, or a UI instead of rewriting code for each project.
2
Declarative systems can reuse model logic across data types and tasks while still allowing experienced engineers to add custom encoders, losses, and training strategies.
3
Machine learning engineers can shift toward platform development, platform extensions, data quality, and data-centric work as training and deployment become easier to use.
Summary
Piero Molino explains why he built Predibase around declarative machine learning. At Uber, he saw teams rewrite bespoke code for each project, creating technical debt that was difficult to maintain. Predibase uses configuration, a SQL-like language called PQL, a Python SDK, and a UI to let users define, train, evaluate, and query models. It connects to data in Snowflake and can fit into dbt workflows. Ludwig handles model assembly and training, while Horovod and Ray abstract distributed execution on Kubernetes. Molino argues that this approach can make machine learning available to more people without removing the ability to customize the system. He is clear about the boundary: business-specific joins and feature logic remain outside Predibase, while common transformations belong in the platform. He also describes how machine learning engineers can adapt by building platforms, extending them, or focusing more on data quality and labeling.
Molino traces Predibase's design to his time at Uber, where each new machine learning project often required an almost complete rewrite. Projects used different data, tasks, and metrics, then became separate bodies of technical debt that someone had to maintain as libraries and data changed. A flexible configuration system lets teams solve the general problem once and express new projects as changes to configuration. New architectures, losses, or metrics can be added to the underlying system and reused by the rest of the team. Molino says this also keeps institutional knowledge in one place instead of scattering it across individual projects and organizations.
Predibase brings model training into the modern data stack
Predibase connects to a Snowflake instance without copying or storing the data, according to Molino. Users can define models through the UI, a Python SDK, or PQL, which is a SQL-like language for predictive queries. The platform also provides dbt operators for training, evaluation, and querying, so machine learning steps can be used alongside dbt transformations. This gives data teams a way to apply models close to the data workflows they already use, rather than creating a separate process for every machine learning project.
Molino describes PQL as a way to select the data that should receive predictions. A user can target a slice of rows, such as records that arrived in the database the previous night, and run batch inference. The same interface can accept a new data point and return a prediction. Queries can request outputs such as confidence or explanations. Users can define model inputs and outputs while leaving the training details to the system, or specify the model and its parameters when they need more control.
Ludwig and Horovod hide much of the training infrastructure
The training layer in Predibase uses Ludwig to assemble and train a model from the declared inputs, outputs, and optional settings. Users can ask the system to choose a model or specify preprocessing, optimization, model parameters, and other training choices. Horovod, combined with Ray, runs distributed training on Kubernetes and can use many GPUs without requiring the user to manage the individual machines. Molino describes Predibase, Ludwig, and Horovod as layers that connect business data to model structure and training execution.
The platform keeps expert control through extension points
Molino expects experienced engineers to worry about being limited by a configuration system. He says the open-source foundations keep the implementation visible and editable. Predibase also provides hooks for adding custom image encoders, losses, and training strategies. Once an engineer adds such an extension to the underlying code, it becomes an option in the configuration system and the other interfaces. Less experienced users can stay at the higher level, while experts can extend the platform when a use case needs something outside the standard options.
Predibase draws a boundary around bespoke data preparation
Molino separates common machine learning preparation from business-specific feature logic. Predibase handles operations such as mapping text to tokens, filling missing values, creating categorical vocabularies, and producing vector representations. The platform does not take responsibility for every join or custom transformation needed to combine a company's tables. Those operations can be written with SQL and dbt in the underlying data stack. Molino says the line is based on what can be reused across use cases and what depends on knowledge of one company's data.
Ludwig grew from one reusable codebase into an open-source project
Molino originally created Ludwig as a personal codebase while working on several Uber projects. Rather than maintain four separate codebases, he made one system broad enough to cover the different tasks. Other people at Uber began using it for additional problems, which suggested that it could work outside the company. Ludwig was open sourced in 2019 and donated to the Linux Foundation in 2020. Molino also describes a move to PyTorch, which would let Ludwig use new models as they are implemented in the surrounding Python ecosystem.
Machine learning engineers can move closer to platforms and data
Molino compares the change to the move from COBOL to SQL and from hand-coded games to game engines. Engineers who once built models and services directly can help build the platforms that make those capabilities broadly available. They can also extend platforms with company-specific functionality. Another path is to work on data quality, data labeling, and the wider data process. As distributed training and deployment become platform capabilities, Molino expects the quality of the data entering the system to matter more to the resulting models.
A small startup should avoid rebuilding its whole machine learning platform
Answering a question about a startup using Flask, Kubernetes, Airflow, and a data warehouse, Molino says teams do not need to build every part themselves. A homegrown system requires specialists, ongoing maintenance, and work whenever dependencies change. He also points out that Flask may not be the most efficient way to serve a model simply because an engineer already knows it. His comparison is to databases and game engines: teams often use infrastructure built by specialists rather than recreating it for each project. Predibase is his example of a platform that can provide much of this machinery.
"I believe that today we are starting to get into the mindset of, I can use a machine learning platform that someone built for me if it does what is needed for me, but I don't have to build it from scratch every single time."Piero Molino49:50
Who should watch
You are deciding whether machine learning should live inside an existing SQL, Snowflake, and dbt workflow.
Your team keeps rewriting training and deployment code for each new model and wants a reusable platform with room for custom extensions.
You are a machine learning engineer thinking about moving toward platform engineering, data quality, or data-centric work.
