MLOps has shared practices, such as version control, but no single process works across every machine learning use case.
2
Ibis lets people write a Python data frame API that compiles to backend-specific code, so the same work can run on systems such as DuckDB, Snowflake, BigQuery, and PySpark.
3
Apache Arrow and Ibis reduce the cost of switching data systems, although open standards need broad adoption and outside contributors to remain useful.
Summary
Cody Peterson argues that MLOps depends heavily on the data layer, where teams still face problems with data versioning, cleaning, access, audits, scale, and handoffs between data scientists and engineers. He describes Ibis as an open-source data frame interface that separates the API from execution. Users can write Python-style code while Ibis compiles it to SQL or backend-native code for systems such as DuckDB, Snowflake, BigQuery, and PySpark. Apache Arrow provides a shared in-memory data format that lets systems exchange data without each tool inventing its own representation. Cody connects these projects to composable data systems, where teams can choose different engines for different workloads. He is also cautious about standards becoming fragmented and says adoption requires support from many organizations. Near the end, he gives a restrained view of AI hype and says practical deployment remains harder than building impressive demos.
MLOps has common practices, but its process still depends on the use case
Cody says MLOps has shared threads such as version control, while teams still debate issues such as whether notebooks belong in production. He compares MLOps with software engineering, where a broad discipline cannot be reduced to one process. Recommender systems have more established practices because companies have worked on them for longer. Other machine learning applications may have less history, so the right approach depends on the problem and the industry. He carries this lesson from Azure ML, where he worked with customers deploying end-to-end machine learning systems.
Machine learning systems need data processes that teams can audit and maintain
Cody describes data as a central part of an ML system because models receive new data and need updates over time. Teams have to manage and version that data, then explain decisions during audits in regulated settings. The work includes transforming, cleaning, and providing access to data. He says the rise of LLMs can make these problems less visible, but advanced use cases still require people who understand data engineering. The hard data problems do not disappear because a higher-level AI interface hides them.
Ibis separates the data frame API from the engine that runs the computation
Ibis began with a different approach to the pandas model. Pandas is closely tied to NumPy and in-memory, single-threaded execution. Ibis lets users write data frame code, then compiles that code into SQL or another backend-specific form. The backend performs the computation and manages the data. Cody says Ibis supports at least 20 backends, including DuckDB, Polars, Snowflake, BigQuery, ClickHouse, PySpark, Dask, and pandas. This lets users work with a familiar API while choosing where the work should run.
Ibis reduces the handoff between Python users and data engineers
Cody describes data scientists who sample a small part of a Snowflake dataset, write pandas code locally, and then hand it to an engineer to make efficient on the warehouse. With Ibis, they can connect to Snowflake and work against the data through the Ibis API. The same code can run locally on DuckDB or Polars for experimentation, then scale to the larger backend. This gives the data scientist and engineer a shared artifact instead of a code handoff that may work only on one machine.
Apache Arrow gives data systems a shared in-memory format
Cody explains Apache Arrow as an open specification for an in-memory data format. DuckDB and Polars use it, and pandas is increasingly taking a dependency on it. A new data system can adopt Arrow instead of creating a separate memory layout. When data moves between systems such as DuckDB and Polars, the systems can use the same representation rather than converting every type and layout independently. Cody describes this as a zero-copy conversion, which makes communication between tools more direct.
Open standards make data systems easier to swap, while adoption remains difficult
Cody says open standards help users avoid being tied to one vendor's memory format or data frame API. They can choose the right tool for a particular job and build a more composable system. A batch model can later be replaced with a model updated through streaming without forcing the whole data system to be redesigned. He also acknowledges the problem of creating yet another standard. Apache Arrow gained traction through support from multiple organizations that were facing similar problems. Ibis needs the same kind of ecosystem support to keep growing.
Voltron Data uses Ibis as an open interface around a proprietary engine
Cody describes Voltron Data's approach as open periphery. The company sells a distributed GPU database engine called Theseus for very large data workloads, while Ibis provides its interface. Most users do not need an engine that runs across many GPUs, so the same interface can cover smaller workloads on DuckDB or Polars, medium workloads on systems such as ClickHouse or Snowflake, and larger workloads on Theseus. Ibis is not owned by Voltron Data, even though many Ibis engineers work there and Theseus uses it.
Choosing a data system starts with latency, size, and organizational constraints
Cody recommends first asking whether the workload is batch or streaming and what latency it needs. Teams should then consider data size. For data that fits on a laptop or a modest virtual machine, he recommends a single-node analytical engine such as DuckDB or Polars and says to avoid distributed systems unless they are needed. Once data grows beyond roughly one terabyte, systems such as PySpark, Trino, and ClickHouse may make more sense. Governance needs and the tools already used by the organization also affect the decision.
Ibis ML aims to run machine learning preprocessing across different backends
Cody says the planned Ibis ML package applies the Ibis idea to machine learning preprocessing. Libraries such as pandas and scikit-learn often assume that data fits in memory and runs in a single-threaded environment. Ibis ML is intended to let a team experiment locally on DuckDB, then run the same pipeline on Snowflake, PySpark, ClickHouse, Trino, or Theseus. It includes machine learning details such as retaining values needed during retraining and providing wrappers for operations such as one-hot encoding. It is a preprocessing interface, not an orchestration tool like Airflow, Dagster, or Prefect.
"If we have an open standard that has buy-in from a bunch of different vendors and different products out there, it allows you to easily switch between them and choose the right tool for the job."Cody Peterson16:45
Who should watch
You are choosing between local, warehouse, and distributed data systems and want one Python-facing interface across them.
Your data science team writes pandas code locally, then hands it to engineers to make it run at scale.
You are evaluating open standards such as Apache Arrow and Ibis, or deciding whether text-to-SQL and RAG are useful in a real product.
