Pandera began as Niels Bantilan's way to describe and validate data-frame schemas, then merged into Union as a foundation for tracking data quality over time.
2
Infrastructure can collect workflow metadata that may eventually predict resource needs, failures, memory spikes, and GPU underuse before a workload runs.
3
ML and LLM systems still need models, data, infrastructure, lineage, reproducibility, and resource isolation, even when teams build on newer model platforms.
Summary
Niels Bantilan explains how Pandera grew from a personal solution to data-frame type errors into an open-source data validation project that later merged into Union. He argues that orchestration systems should collect structured metadata about workflows, inputs, resources, failures, and utilization. That information could support simple rules first, then predictive systems that warn about likely failures or poor hardware use before a job runs. The conversation also compares traditional ML infrastructure with the newer LLM stack. Bantilan sees LLMs as another kind of ML model, with additional retrieval and application components, while many teams still train models on tabular, image, video, audio, and text data. He is skeptical of speculative automation that encourages people to outsource judgment to generated code. For mature teams, practical needs such as iteration speed, lineage, reproducibility, resource isolation, and the ability to move from a laptop to different compute systems matter more than grand visions of AGI.
Pandera started as a way to make data-frame assumptions visible in code
Niels Bantilan built Pandera while working heavily with pandas data frames and running into incorrect types, date formatting problems, negative values, and unexpected nulls. Long transformation chains made it hard to know what a data frame contained. Pandera lets a function declare the schema of its output, including column names, data types, constraints, and ranges. Niels says users still need to inspect data before writing a schema, but the schema then becomes something they can read directly in the codebase.
Pandera's users wanted control in their own cloud rather than a hosted data-reporting service
Niels considered turning Pandera into a company, but conversations with users changed his view of the product. Users told him they did not want him to touch their data. They wanted Pandera running in their own cloud, producing error messages and reports they controlled. A hosted interface for visualizing reports did not interest him enough to build. He became more interested in tracking how a data set changes over time and what kinds of errors appear, which led to merging Pandera into Union.
Data quality and workflow history could become signals for infrastructure decisions
Demetrios Brinkmann asks whether the history collected by Pandera and an orchestrator could support ML inside the infrastructure layer. Niels says the orchestrator should collect structured or semi-structured information about workflow behavior. Examples include a workflow that often fails for certain inputs or data sizes, or one that produces unusual memory spikes. Simple cases could be surfaced with deterministic rules. More complex error summaries and patterns could later use ML, but collecting the underlying data comes first.
Predicting workload resource needs requires connecting specifications with outcomes
Niels describes a possible system that maps workflow code and resource specifications to success or failure on particular hardware. Data set size and model size may provide much of the predictive signal, with other features adjusting the result. He compares this with an algorithmic approach used at Meta's FB Learner, where resource settings are increased or reduced after observing failures or successes. Niels does not propose reinforcement learning as the immediate solution. He frames the problem more simply as classification or empirical experimentation.
Generated infrastructure code can encourage people to give up their own judgment
The discussion turns to generative DevOps and systems that turn natural-language requests into Terraform. Niels sees the application as plausible, but Demetrios is more skeptical about whether generated infrastructure creates more problems than it removes. Niels says the risk is complacency. People may begin accepting generated code without reviewing it, especially when the output is accurate often enough. He also worries about using one LLM to write code and another to review it, creating an open-ended chain that still needs a person to judge the result.
LLMs add components to ML systems, but they do not remove the lower infrastructure layer
Niels describes OpenAI's developer platform as a new compute platform built around tokens, where an LLM can produce instructions or function calls. He still expects teams to build bespoke systems for tabular data, computer vision, video, audio, and simpler text tasks such as sentiment prediction. He treats LLMs as ML models and says ML systems combine model architecture, data set, and infrastructure. LLM applications add pieces such as retrieval, vectors, and similarity metrics, but they still depend on established software and infrastructure underneath.
Mature organizations adopt orchestration after feeling the cost of fragmented tools
Niels says teams often begin with a laptop, a small data set, scikit-learn or XGBoost, and a simple Flask or FastAPI service. As their systems grow, they add tools for experiment tracking, data transformation, observability, data quality, and resource management. An orchestration layer helps connect these purpose-built tools. Teams that have already experienced problems with lineage and reproducibility are more likely to see the value of Flyte. LLM Ops will face similar production requirements, including reproducibility and resource isolation.
Flyte is being shaped around iteration speed, portability, and separate artifact lineage
Niels says Union is working on making Flyte and the Flytekit SDK useful across the path from a local laptop to systems such as Spark, Databricks, Ray, or a Flyte cluster. He does not want users forced into one approved platform. Flyte should provide escape hatches for existing tools and organizational constraints. A related problem is tracking artifacts across separate workflows. One workflow may produce a data set that another consumes to train a model, and Niels wants that artifact's lineage to remain visible without tightly coupling the workflows.
"I prefer to work in this less speculative space, and that kind of means hey what does a customer need today for in a week."Niels Bantilan34:23
Who should watch
You are deciding whether an orchestration layer can connect the separate tools in an ML platform without forcing every team onto one stack.
Your team has accumulated workflow metadata, failures, resource limits, or GPU utilization data and wants to understand what it could support beyond dashboards.
You are building LLM applications and want a grounded discussion of which traditional ML and infrastructure concerns remain underneath the LLM layer.
