AI agents change data platforms because they generate continuous, unpredictable queries and need fresh context while they make decisions.
2
Data platforms need deterministic guardrails around generated SQL and Spark jobs, including semantic layers, access checks, cost limits, query plans, and human review for risky cases.
3
Open table formats, faster execution engines, feature stores, GPU-native storage, and automated table services are being combined to support higher concurrency and lower-latency workloads.
Summary
OpenXData Conference 2026 covers how open data systems are changing as AI agents become major consumers of data. The keynote argues that lakehouses must evolve from systems of record into context-serving layers that provide fresh, joinable, governed data at low latency. Other sessions address guardrails for generated SQL and Spark jobs, Apache Hudi optimizations at JD.com and Uber, online feature serving at Booking.com, and lakehouse migrations using Hudi and StarRocks. Speakers also discuss faster Spark runtimes, GPU-native columnar storage with Vortex, serverless multi-cloud streaming on Kubernetes, personal lakehouses, and agent orchestration. Across the sessions, the practical concerns are consistent: control query cost, preserve provenance, keep data fresh, reduce operational toil, and make probabilistic AI systems depend on deterministic infrastructure where possible.
AI agents create a different workload for data platforms
The keynote argues that most data systems were built around human users who ask a question, wait for results, interpret them, and decide what to do next. Agents do not have those limits. They can run continuously, generate queries autonomously, act on results without a review step, and create cascades of tool calls or subagents. A single workflow may produce narrow lookups, scans, aggregations, semantic searches, and join-heavy relationship queries. This changes failure modes and cost control. The speaker warns that treating agents as faster humans can lead to warehouse cost explosions, production database load, and decisions made from stale or incomplete data.
Agent context must be fresh, joinable, and cheap to query
Context is defined as the data an agent needs to reason about the world, including facts, history, current state, relationships, and previous outcomes. The speaker describes useful context as fresh, joinable, cheap-to-query runtime data. Stale data can cause an agent to act on the wrong state, slow joins can delay inference, and broken pipelines can make the agent reason from the past. Enterprise context is often fragmented across OLTP systems, warehouses, lakes, APIs, and vector databases. Agents cannot rely on people to explain where data lives, so the platform must connect the data and its meaning before the agent makes a decision.
A lakehouse needs a serving layer for agent workloads
The proposed architecture turns the lakehouse from a system of record into a context-serving layer. It should provide low latency, high concurrency, mixed access patterns, and consistent freshness without forcing every dataset into another silo. The speaker compares this with federated querying, where a request crosses databases and APIs and becomes hard to operate or bound for cost. A context-serving layer would expose one governed endpoint over open formats and multiple data types. The talk also predicts that tables will need to hold documents, images, embeddings, metadata, and indexes, while file formats and compute engines add better support for these access patterns.
Generated SQL needs deterministic checks before execution
The guardrails session identifies several ways generated SQL or Spark code can fail. A query may be inefficient, violate column permissions, infer sensitive information by joining permitted tables, answer a different question than the user asked, or run an unbounded scan. Other risks include hallucinated relationships, Cartesian products, policy bypasses, and Spark anti-patterns such as collecting large data to the driver. The proposed pre-flight path parses the query, resolves schemas and lineage, checks access policy, inspects the execution plan, estimates cost, and applies environment-specific rules. SQLite or sample data can provide a cheaper confidence gate before a full Spark run.
Apache Hudi is being extended for high-concurrency lakehouse workloads
The JD.com session describes custom Apache Hudi features for large real-time workloads. Its LSM-tree design separates incremental data from compacted stable files, while native engine formats and Parquet support reduce conversion and improve pruning. A disruptor ring buffer, balanced partition assignment, managed memory, and binary in-memory sorting support Flink ingestion. The team also uses hybrid storage, placing recent writes on higher-performance HDFS and moving consolidated data to lower-cost shared storage. The presentation reports two to four times better write performance and four to six times faster reads than its baseline format in its benchmark setup.
Feature stores balance real-time latency, scale, and cost
The Booking.com feature platform combines real-time and batch ingestion with online, offline, and vector storage. Kafka events update online features within milliseconds to seconds, while scheduled pipelines produce more static features. Redis on Amazon ElastiCache handles low-latency key-value serving, and Snowflake stores training and historical data. The platform uses dedicated Redis clusters for feature groups, read replicas, multi-region replication, and infrastructure as code. The speaker describes a performance test for a feature group with more than 400 million records that kept latency below 50 milliseconds under high throughput.
The query-layer session argues that an agent can produce different valid SQL queries for the same natural-language question, and existing query logs, cost dashboards, and dbt tests may not reveal the semantic difference. The recommendation is to reduce the model's choice space. A semantic layer can expose metrics, dimensions, and filters while a compiler generates deterministic SQL. A constrained SQL sandbox can handle questions outside the semantic layer, with full warehouse access reserved for rare, human-approved cases. The speaker also recommends storing agent traces as primary data so teams can reconstruct the context and decisions that produced an answer, while separating observed events from the agent's interpretation.
GPU-oriented storage changes the file-format tradeoffs
The Vortex session starts from changing hardware: GPUs, faster network cards, NVMe, and remote object storage create different bottlenecks than older CPU-and-disk systems. Vortex is presented as an extensible columnar format designed for SIMD and SIMT execution, random access, and lightweight cascaded encodings. The project keeps the format open and composable, with implementations for multiple languages and compute backends. The speaker reports that Vortex can outperform Parquet for GPU scans, selective filters, compression, and some analytics workloads, while acknowledging that its main drawbacks are narrower ecosystem support and a Rust API that still changes frequently.
"The more that you can ensure that that data is encapsulated and carried along so that you can make a decision at the end, I think that's just incredibly important."David Ronek3:55:12
Who should watch
You are building a data platform that will support AI agents and need to understand why ordinary warehouse scaling does not solve context, freshness, and query-spike problems.
Your team is exposing SQL, Spark, APIs, or data stores to language models and needs practical guardrails for permissions, cost, performance, provenance, and semantic correctness.
You are evaluating open table formats, feature stores, faster Spark engines, GPU-oriented storage, or automated lakehouse operations at large scale.
