Inference is becoming a new kind of data transformation because companies need to turn unstructured text, images, and other data into usable structure at scale.
2
LLM applications are easy to demonstrate but difficult to run reliably, because model calls add non-determinism, variable latency, token costs, context limits, and weak failure recovery.
3
Data engineers and ML engineers can move into AI infrastructure by applying their experience with reliability, SLAs, lineage, testing, and production systems to LLM workloads.
Summary
Kostas Pardalis and Yoni Michael argue that LLMs are changing the work of data and ML engineers. Existing platforms were designed around structured, tabular data and workloads such as BI, while modern AI applications process transcripts, documents, images, and other unstructured inputs. The speakers describe inference as a new form of transform that can turn unstructured data into structured outputs. The hard part is production. Teams must manage context windows, chunking, token costs, rate limits, model latency, GPU dependencies, retries, evaluation, and lineage across multi-stage pipelines. They argue that chat demos and interactive agents are only an early step. Production systems need deterministic behavior around non-deterministic models, with traceability from final outputs back to the data and model calls that produced them. Data engineers and ML engineers already have relevant skills in reliability and operations, but they need to adopt LLMs instead of rejecting them. The discussion also stresses that AI projects need measurable business outcomes and should avoid black-box tooling when systems must be maintained at scale.
Existing data platforms were built for structured analytics rather than AI workloads
Kostas Pardalis says many dominant data tools were created 12 or 13 years earlier for cloud migration, big data, BI, and analytics. Since then, ML, embedded analytics, product data, and AI have created different workloads. Yoni Michael adds that AI introduces unstructured text, images, video, and other modalities that existing platforms do not treat as first-class inputs. Hardware has also changed. Large single-node instances and DuckDB can handle workloads that once required distributed Spark clusters. The result is a mismatch between current workloads and the assumptions built into older data systems.
AI made data more directly connected to products and business outcomes
Pardalis describes a shift from software that automated human activities toward products that use the data created by those activities. BI began with financial reporting and expanded into marketing and sales reporting. ML added recommenders and other systems where data directly influences revenue or user behavior. LLMs widen access to unstructured data such as customer-support calls and transcripts. The speakers argue that AI accelerated this shift because more people can now do things with company data that previously required highly mature teams at companies such as Lyft, Uber, or Apple.
The gap between an AI demo and a production system is still large
The speakers compare application development with data and ML delivery. A front-end and back-end engineer can combine their work into a functioning application, while a data scientist often hands a notebook to a data engineer who rewrites it for production. AI makes experimentation easier, but it does not solve deployment. Companies commonly start with chat interfaces, LangChain, or LlamaIndex, then discover that scaling the system requires handling non-deterministic model outputs. The production challenge is to build stable, predictable pipelines around those models.
Inference is a new transform for turning unstructured data into structured data
Michael says teams are collecting large volumes of transcripts and want to perform semantic analytics without a human reviewing every item. Pardalis describes inference as a powerful transform that can convert unstructured data into structured results. A call center, for example, might process thousands of conversations and extract information for later analysis. Batch inference becomes important when the goal is to process large data sets rather than respond immediately to a user. This requires data teams to apply familiar pipeline ideas to model calls while accounting for model-specific behavior.
LLM production pipelines need controls that older platforms do not provide
Running AI at scale introduces context windows, tokenization, partitioning, chunking, rate limits, model latency, time to first token, model cascading, and inference cost. A transcript cannot always be split into isolated messages because the surrounding conversation may be required for interpretation. Sending too much context raises token costs. The speakers say teams often build brittle Spark UDFs or Lambda-based integrations because existing platforms lack these capabilities. They argue for engines that provide experimentation, data-quality checks, and production controls for unstructured data directly.
Reliability work is an opportunity for data and ML engineers
Pardalis argues that data engineers and ML engineers already understand the operational problems LLM systems create. Data engineers work with SLAs, data quality, retries, and systems that must deliver outputs on time. ML engineers have experience with models that are not fully deterministic. LLMs push those problems further. The speakers predict growing demand for AI platform and AI reliability roles that can make model-based systems dependable. Their warning to engineers is direct: rejecting LLM work means missing a place where their production experience is especially valuable.
Evaluation must cover the whole pipeline rather than one model call
Pardalis explains that a transcript workflow can involve speech recognition, cleanup, chunking, per-chunk summaries, and a final summary. Evaluating each model call separately can miss an early error that affects every later step. Agents create an even more complicated version of the same problem because they may make many calls and revisit earlier steps. The speakers want evaluation and observability systems that trace the final result through every input, prompt, model call, and intermediate output. They compare this need with distributed-system tracing and argue that ordinary column-level lineage is insufficient.
AI projects should start with business value and a credible path to production
The speakers advise teams to choose use cases using business impact and confidence that the system can actually be implemented. Content moderation is one example because the cost of human review is clear and the business outcome can be measured. Insurance call review is another: a pipeline can compare a representative's statement with the customer's declarations page and escalate possible misrepresentation. They warn against building disconnected demos, downloading CSVs from black-box services, or adding tools without fitting them into existing workflows. Engineers should invest in maintainable infrastructure and feedback loops rather than assuming a better model will solve the whole problem.
"The models are going to keep getting better, keep getting more accurate, probably cheaper and a lot more performant. It's all around the infrastructure you have around it now."Yoni Michael1:27:06
Who should watch
You operate data or ML pipelines and need to understand how LLM workloads change reliability, cost, lineage, and evaluation.
Your team has built an impressive AI demo but is struggling with batch processing, context management, retries, or production SLAs.
You lead AI projects and need a practical way to choose use cases that have measurable business value and fit existing workflows.
