Physical AI covers software that applies intelligence to the physical world, including autonomous robotics, spatial computing, and other systems that analyze or act on real-world data.
2
Robotics systems need separate online and offline systems, but their data is difficult to store, align, visualize, query, and prepare for training.
3
Rerun is building an open source visualization layer and a commercial data stack that can work with multimodal, time-changing physical AI data across the full lifecycle.
Summary
Nikolaus West defines physical AI as software that applies intelligence to the physical world. The category includes robotics, spatial computing, and systems that analyze real-world activity. He says modern AI matters because the physical world is messy, ambiguous, and constantly changing, while traditional robotics software often works by tightly constraining the environment. West describes two systems in a physical AI lifecycle. Online systems run on or for the robot, handling perception, planning, decisions, and actions. Offline systems record, inspect, transform, curate, and train on the resulting data. The data is hard to manage because cameras, motion sensors, point clouds, audio, and internal metrics run at different rates and may use different clocks. West argues that existing robotics and ML data tools do not fit this combination. Rerun addresses the problem with visualization, time alignment, a multimodal data model, and query infrastructure. Its commercial product is intended to reduce the number and complexity of custom data pipelines.
Physical AI applies intelligence to software that interacts with the real world
Nikolaus West uses physical AI as a broad category for products that apply intelligence to the physical world. It includes autonomous or semi-autonomous robotics, spatial computing, security applications, sports analytics, and systems that analyze or act on real-world activity. He distinguishes this from simply giving robotics a new name. Demetrios Brinkmann summarizes it as AI that is out in tangible physical space, and West agrees. He says most economic activity happens in the physical world, while software has historically managed appointments, schedules, or invoices rather than doing the physical work itself.
AI makes less constrained physical products possible
West says the physical world has an unending number of messy situations and things that can change. Traditional software can handle a narrow setting, such as a coffee-making robot or one repeatable manufacturing cell, because engineers can write rules for each expected case. Those systems become harder to extend when the environment is ambiguous. Modern ML and AI can make products more flexible and allow them to address larger markets. West connects that larger market to hardware economics: scale can support better and cheaper hardware, while larger fleets create more data for improving models.
Physical AI depends on both a hardware flywheel and a data flywheel
West describes two reinforcing loops. More capable and affordable hardware can reach more users and collect more data. Better data can improve the models, which can then support more advanced deployments and produce better data. He also says investment and belief matter because hardware requires substantial upfront work. The popularity of large language models helped create that investment climate. Within robotics, West points to the RT-1, RT-2, and RTX line of papers as examples of methods that appeared to improve when given more data and compute, a property that had long been associated with language models rather than robotics.
Physical AI has online systems and offline systems
West separates a physical AI lifecycle into online and offline systems. Online systems are the components running while a robot operates. They understand the world, plan, make decisions, and act, whether the computation is on the robot or partly accessed through an API. Offline systems run on laptops, workstations, or data centers. They provide fleet observability, algorithm prototyping, analytics, data collection and curation, transformations, training preparation, model training, and deployment. The split gives teams a way to reason about the systems that operate in the world and the systems that improve them.
Time is a system constraint because the world keeps changing
West says the most important difference between physical AI and ordinary software interactions is time. A robot cannot simply take as long as it needs to process an input because the object or situation may change while it is computing. Teams need to track real-world time, compute time, algorithm iterations, and the age of the information behind a decision. A decision made after processing may already relate to an earlier state of the world. This makes time a central concern in system design, rather than a matter of user experience alone.
Robot data is multimodal, asynchronous, and difficult to store together
On a robot, cameras, motion sensors, and other sources produce data at different rates. West gives 30 frames per second for video and 1,000 hertz for motion sensors as examples. A robot can also behave like a distributed system with different clocks. The resulting data looks like logs of multimodal streams, with nested structures, video, audio, 3D information, point clouds, internal metrics, and fast small signals mixed with larger tensors and images. Systems often use specialized file formats that write this data quickly, but moving it into centralized storage can become complicated at larger scale.
Visualization is part of data debugging, not an optional interface
West argues that teams need live visualization to see camera feeds, 3D maps, robot state, and algorithm internals. They also need to scroll through recorded sessions when something goes wrong. Existing offline data tools generally expect tables with columns, while physical AI data is often unaligned and log-structured. Without visualization between pipeline stages, teams may not notice errors such as an orientation being flipped during training. West says this leaves robotics companies with an online stack that understands robot data and an offline stack that supports large-scale learning, while the two stacks do not communicate well.
Rerun aims to unify physical AI logging, visualization, storage, and queries
West describes Rerun as an attempt to build one data stack for online and offline physical AI workflows. Its open source project logs and visualizes multimodal data that changes over time, including tensors, point clouds, 3D scenes, metrics, and video. It has SDKs for Python, Rust, and C++. The project was built from scratch in Rust, with a data model inspired by entity component systems. The commercial product adds storage and indexing, support for varied robotic file formats and tabular data, a physical AI data model, time-aware queries, visualization, and a data catalog. West wants teams to choose which pipeline results to materialize instead of being forced into many rigid steps.
Robotics teams choose different trade-offs between learning and control
West describes several design choices in physical AI. Teams differ in how many sensors they use, how much they trust learned behavior, and whether they want mathematical guarantees. They also differ in how strongly they value modular components compared with end-to-end neural networks. Some use little ML beyond perception, relying on SLAM, classic planners, and tested control methods. West says that approach can work well in structured settings such as warehouses. More end-to-end learning may simplify the online system while making the offline data and training system much more complex. He presents these as engineering trade-offs rather than a single winning architecture.
