Sammy Sidhu learned to optimize small machine learning models for autonomous vehicles by cleaning training data, searching for hardware-specific architectures, and mapping models across constrained chips.
2
Daft combines a Python data-frame interface with a Rust engine so teams can query and process images, videos, audio, 3D assets, and other complex data without stitching together SQL, file lists, and Spark jobs.
3
Eventual is building storage and model execution around Daft, with features for unstructured data such as governance, schema evolution, time travel, efficient container formats, and compiled model inference.
Summary
Sammy Sidhu describes a career that moved from high-performance computing at Berkeley to autonomous vehicle systems at DeepScale, Tesla, Lyft Level 5, and Toyota. He explains how tiny hardware forced his teams to clean data carefully, distill models, optimize distributed training, and build compilers for automotive chips. Those problems led him to Eventual and Daft, an open-source query engine for complex data. Daft presents a Pythonic distributed data frame while running its engine in Rust. It can read large collections of files, transform images or other multimodal data, run models, and execute the resulting query on a cluster. Sammy argues that existing systems often split this work across a database, object storage, and Spark, which makes optimization and failure recovery difficult. He also describes future storage features and model execution without Python frameworks. The discussion covers retrieval for language models, safety feedback loops, and why open-source models may take years rather than months to catch up.
Small models require clean data and hardware-specific search
Sammy says autonomous vehicle models had to run on hardware with roughly a fraction of the computing power of a Raspberry Pi. His team first improved data quality because inconsistent examples required more model capacity. They also developed a technique described in his paper, "Squeezed Ass," which used a neural network to search for architectures that balanced accuracy and latency on a particular hardware target. Shrinking the model created new system problems: GPU work became faster while CPU data ingestion stayed expensive, so distributed training became bottlenecked in unexpected places.
Automotive deployment required compilers and awkward hardware compromises
The vehicle chips came from companies such as Renesas and NXP, and their accelerators handled different operations. Sammy's team built a compiler that divided a model across those hardware components. Models had to use layers that mapped cleanly to several parts of the chip. One platform had a compiler that only ran on Windows, which Sammy calls the worst month of his life. Safety requirements added failovers and redundancies because a chip failure could not simply stop the driving system.
Complex data pipelines break when metadata and assets live apart
After working at DeepScale, Tesla, Lyft Level 5, and Toyota, Sammy saw that teams needed database-like tools for complex data. A typical system stored metadata in a database and left images, videos, or 3D scans in S3. An analyst wrote SQL to select files, another process used Spark to process them, and the results were written back for further analysis. This split prevented end-to-end query optimization. Spark also lacked native abstractions for these assets, so a worker could load too many images into memory, fail, and trigger repeated cluster rebalancing.
Daft gives multimodal processing a lazy Python query interface
Daft is an open-source query engine for complex data. Sammy describes it as a distributed data frame with a Python interface, while the engine runs in Rust. A user can read millions of files from S3, process them as images, crop or resize them, and pass them to a machine learning model for inference or training. The user builds a lazy query, and Daft creates a plan that can run locally or on a cluster. Sammy says the engine understands the data it processes, which helps avoid loading large files into memory in unsafe ways.
Eventual is extending Daft toward storage for unstructured data
Sammy says Daft's users asked how to store complex data for better retrieval after solving the query-processing problem. Eventual is working on an open-source and managed enterprise product for this purpose. The goal is to provide database-style features for unstructured data, including governance, schema evolution, time travel, permissions, column changes, and retention policies. Eventual is also developing container formats that load images efficiently and can seek to selected parts of an image directly from cloud storage, rather than treating every image as an isolated object.
Daft can support retrieval and generation workflows for language models
Sammy describes language-model use cases that involve filtering datasets, tokenizing content, running open-source models, and performing hybrid search over embeddings and text. He also wants language models to write Daft queries against multimodal data lakes, in the same way that language models can write SQL for tabular databases. This could let a model search large collections of documents, images, or other complex assets and then interpret the returned results. He gives legal documents as an example of a workflow involving image processing, text extraction, language-model calls, and retrieval.
Complex data systems must optimize compute, placement, and recovery
Sammy contrasts tabular analytics with complex data processing. Tabular queries often read much more data than they compute over, while an image or a short work of Shakespeare can require very large amounts of computation for model processing. For complex data, Daft needs native knowledge of data types, memory costs, transfer costs, placement, and scheduling. Sammy says a complex workflow mapped onto a Spark cluster may use only about 20 percent of the hardware. Daft is designed to improve utilization and avoid repeating expensive work when a worker fails.
Compiled model execution could reduce dependence on scarce GPUs
Eventual wants Daft to run Python-authored queries that execute in a cloud cluster and shut down when finished. Sammy says many models currently depend on Python, PyTorch, or TensorFlow, so users write glue code around them. His team is exploring embedding models in Daft, compiling them for CPUs or GPUs, and running them without Python or a model framework at execution time. He points to llama.cpp-style work as an example of packaging a smaller model into a single optimized binary. This could help when teams cannot obtain enough GPUs.
"What we want to do is kind of bring the functionality that you're familiar with, like a database or data warehouse for tabular data, but to the complex domain."Sammy Sidhu14:03
Who should watch
You are building pipelines around images, videos, audio, 3D data, or other files that do not fit comfortably into a tabular warehouse.
Your current workflow passes file lists between SQL, object storage, Spark, and model code, and you want one query plan to cover the process.
You are evaluating how retrieval systems and language models could work directly over large multimodal datasets, especially when GPU capacity is limited.
